To save cluster resources, you can terminate a cluster. A terminated cluster cannot run notebooks or jobs, but its configuration is stored so that it can be reused at a later time.
Azure Databricks retains the configuration information for up to 70 interactive clusters terminated in the last 30 days and up to 30 job clusters recently terminated by the job scheduler.
You can manually terminate a cluster or configure the cluster to automatically terminate after a specified period of inactivity.
You can also set auto termination for a cluster. During cluster creation, you can specify an inactivity period in minutes after which you want the cluster to terminate. If the difference between the current time and the last command run on the cluster is more than the inactivity period specified, Azure Databricks automatically terminates that cluster.
A cluster is inactive if all commands on the cluster have finished executing, including Spark jobs, Structured Streaming, and JDBC.
You can opt out of auto termination by clearing the Auto Termination checkbox or by specifying an inactivity period of “0” in the API.
Auto termination is best supported in the latest Spark versions. Older Spark versions have known limitations which may result in inaccurate reporting of cluster activity. For example, clusters running JDBC, R, or streaming commands may report a stale activity time which will lead to premature cluster termination. You are strongly recommended to upgrade to the most recent Spark version to benefit from bug fixes and improvements to the autotermination feature.
Clusters do not report activity resulting from the use of DStreams. This means that an autoterminating cluster may be terminated while it is running DStreams. Turn off autotermination for clusters running DStreams or consider using Structured Streaming.
Databricks records information whenever a cluster is terminated.
This page lists common termination reasons and describes potential steps for remediation.
This termination reason occurs when Databricks fails to acquire virtual machines from Azure. The error code and message from the API are propagated to help you troubleshoot the issue.