Cannot Run Notebook Commands After Canceling Streaming Cell

Problem

After you cancel a running streaming cell in a notebook attached to a Databricks Runtime 5.0 cluster, you cannot run any subsequent commands in the notebook. The commands are left in the “waiting to run” state, and you must clear the notebook’s state or detach and reattach the cluster before you can successfully run commands on the notebook.

Note that this issue occurs only when you cancel a single cell; it does not apply when you run all and cancel all cells.

Version

This problem affects Databricks Runtime 5.0 clusters. It also affects Databricks Runtime 4.3 clusters whose Spark Configuration spark.databricks.chauffeur.enableIdleContextTracking has been set to true.

Cause

Databricks Runtime 4.3 introduced an optional idle execution context feature, which is enabled by default in Databricks Runtime 5.0, that allows the execution context to track streaming execution sequences to determine if they are idle. Unfortunately, this introduced an issue that causes the underlying execution context to be left in an invalid state when you cancel a streaming cell. This prevents additional commands from being run until the notebook state is reset. This behavior is specific to interactive notebooks and does not affect jobs.

For more information about idle execution contexts, see Execution contexts.

Solution

Azure Databricks is working to resolve this issue and release a maintenance update for Databricks Runtime 5.0. In the meantime, you can do either of the following:

  • To remediate an affected notebook without restarting the cluster, go to the notebook’s Clear menu and select Clear State:

    ../../_images/clear-state.png
  • If restarting the cluster is acceptable, you can solve the issue by turning off idle context tracking. Set the following Spark Configuration value on the cluster:

    spark.databricks.chauffeur.enableIdleContextTracking false
    

    Then restart the cluster.