Notebook or job execution stops and returns either of the following errors:
Run result unavailable: job failed with error message Context ExecutionContextId(1731742567765160237) is disconnected.
Can’t attach this notebook because the cluster has reached the attached notebook limit. Detach a notebook and retry.
When you attach a notebook to a cluster, Azure Databricks creates an execution context. If there are too many notebooks attached to a cluster or too many jobs are created, at some point the cluster reaches its maximum threshold limit of 145 execution contexts, and Azure Databricks returns an error.
Configure context auto-eviction, which allows Azure Databricks to remove (evict) idle execution contexts. Additionally, from the pipeline and ETL design perspective, you can avoid this issue by using:
- Fewer notebooks to reduce the number of execution contexts that are created.
- A job cluster instead of an interactive cluster. If the use case permits, submit notebooks or jars as jobs.