When you create a cluster, you can specify a location to deliver Spark driver and worker logs. Logs are delivered every five minutes to your chosen destination. When a cluster is terminated, Azure Databricks guarantees to deliver all logs generated up until the cluster was terminated.
The destination of the logs depends on the cluster ID. If the specified destination is
dbfs:/cluster-log-delivery, cluster logs for
0630-191345-leap375 are delivered to
To configure the log delivery location:
On the cluster configuration page, click the Advanced Options toggle.
At the bottom of the page, click the Logging tab.
Select a destination type.
Enter the cluster log path.