How Databricks Delta Cache Behaves on an Autoscaling Cluster

This article is about how Delta cache behaves on an auto-scaling cluster, which removes or adds nodes as needed.

  • When a cluster downscales and terminates nodes:

    A Delta cache behaves in the same way as an RDD cache. Whenever a node goes down, all of the cached data in that particular node is lost. Delta cache data is not moved from the lost node.

  • When a cluster upscales and adds new nodes:

    Whenever a cluster adds a new node, data is not moved between caches. Lost data is re-cached the next time an application accesses the data or tables again.