Terminate a Cluster

To save cluster resources, you can terminate a cluster. A terminated cluster cannot run notebooks or jobs, but its configuration is stored so that it can be reused at a later time.

Azure Databricks retains the configuration information for up to 70 interactive clusters terminated in the last 30 days and up to 30 job clusters recently terminated by the job scheduler.

To keep an interactive cluster configuration even after it has been terminated for more than 30 days, an administrator can pin a cluster to the cluster list.

You can manually terminate a cluster or configure the cluster to automatically terminate after a specified period of inactivity.

Manual termination

You can manually terminate a cluster from the

  • Cluster list

    ../../_images/terminate-list.png
  • Cluster detail page

    ../../_images/terminate-details.png

Automatic termination

You can also set auto termination for a cluster. During cluster creation, you can specify an inactivity period in minutes after which you want the cluster to terminate. If the difference between the current time and the last command run on the cluster is more than the inactivity period specified, Azure Databricks automatically terminates that cluster.

A cluster is inactive if all commands on the cluster have finished executing, including Spark jobs, Structured Streaming, and JDBC.

../../_images/auto-terminate.png

You can opt out of auto termination by clearing the Auto Termination checkbox or by specifying an inactivity period of “0” in the API.

Note

Auto termination is best supported in the latest Spark versions. Older Spark versions have known limitations which may result in inaccurate reporting of cluster activity. For example, clusters running JDBC, R, or streaming commands may report a stale activity time which will lead to premature cluster termination. You are strongly recommended to upgrade to the most recent Spark version to benefit from bug fixes and improvements to the autotermination feature.

Warning

Clusters do not report activity resulting from the use of DStreams. This means that an autoterminating cluster may be terminated while it is running DStreams. Turn off autotermination for clusters running DStreams or consider using Structured Streaming.

Termination reason

Databricks records information whenever a cluster is terminated.

../../_images/termination-reason.png

This page lists common termination reasons and describes potential steps for remediation.

Cloud provider launch failures

This termination reason occurs when Databricks fails to acquire virtual machines from Azure. The error code and message from the API are propagated to help you troubleshoot the issue.

Communication lost

This means that Databricks was able to launch the cluster, but lost the connection to the instance hosting the Spark driver.

This might be caused by the driver virtual machine going down or a networking issue.