Cluster Log Delivery

When you create a cluster, you can specify a location to deliver Spark driver and worker logs. Logs are delivered every five minutes to your chosen destination. When a cluster is terminated, Azure Databricks guarantees to deliver all logs generated up until the cluster was terminated.

The destination of the logs depends on the cluster ID. If the specified destination is dbfs:/cluster-log-delivery, cluster logs for 0630-191345-leap375 are delivered to dbfs:/cluster-log-delivery/0630-191345-leap375.

To configure the log delivery location:

  1. On the cluster configuration page, click the Advanced Options toggle.

  2. At the bottom of the page, click the Logging tab.

    Cluster_Log_Delivery
  3. Select a destination type.

  4. Enter the cluster log path.

Note

This feature is also available in the REST API. See Clusters API and Cluster log delivery examples.