Problem: Cluster Cancels Python Command Execution due to Library Conflict


The cluster returns Cancelled in a Python notebook. Notebooks in all other languages execute successfully on the same cluster.


When you install a conflicting version of a library, such as ipython, ipywidgets, numpy, scipy, or pandas to the PYTHONPATH, then the Python REPL can break, causing all commands to return Cancelled after 30 seconds. This also breaks %sh, which is a macro that expands into Python code.



To solve this problem, do the following:

  1. Identify the conflicting library and uninstall it.
  2. Install the correct version of the library in a notebook or with a cluster-scoped init script.

Identify the conflicting library

  1. Uninstall each library one at a time, and check if the Python REPL still breaks.
  2. If the REPL still breaks, reinstall the library you removed and remove the next one.
  3. When you find the library that causes the REPL to break, install the correct version of that library using one of the two methods below.

Install the correct library

Do one of the following.

Option 1: Install in a notebook using pip3

%sh sudo apt-get -y install python3-pip
  pip3 install <library-name>

Option 2: Install using a cluster-scoped init script

Follow the steps below to create a cluster-scoped init script that installs the correct version of the library. Replace <library-name> in the examples with the filename of the library to install.

  1. If the init script does not already exist, create a base directory to store it:

  2. Create the following script:

     sudo apt-get -y install python3-pip
     sudo pip3 install <library-name>
     """, True)
  3. Confirm that the script exists:

  4. Go to the cluster configuration page and click the Advanced Options toggle.

  5. At the bottom of the page, click the Init Scripts tab:

  6. In the Destination drop-down, select DBFS, provide the file path to the script, and click Add.

  7. Restart the cluster.