Databricks Runtime 5.4 with Conda (Beta)

Azure Databricks released this image in June 2019.

Beta

Databricks Runtime with Conda is in Beta. The contents of the supported environments may change in upcoming Beta releases. Changes can include the list of packages or versions of installed packages.

We’re excited to introduce Databricks Runtime 5.4 with Conda, which lets you take advantage of Conda to manage Python libraries and environments. This runtime offers two root Conda environment options at cluster creation:

  • Databricks Standard environment includes updated versions of many popular Python packages. This environment is intended as a drop-in replacement for existing notebooks that run on Databricks Runtime. This is the default Databricks Conda-based runtime environment.
  • Databricks Minimal environment contains a minimum number of packages that are required for PySpark and Databricks Python notebook functionality. This environment is ideal if you want to customize the runtime with various Python packages.

Both include support for Databricks Library utilities.

Note

The Scala, Java, and R libraries in Databricks Runtime 5.4 with Conda are identical to those in Databricks Runtime 5.4. For details, see the Databricks Runtime 5.4 release notes. For information about how to use Databricks Runtime with Conda, see Databricks Runtime with Conda.

System environment

The system environment in Databricks Runtime 5.4 with Conda differs from Databricks Runtime 5.4 as follows:

  • Python: 3.7.x. Only Python 3 is supported.

Libraries

The following is the exported environment.yml file for default root environments on Databricks Runtime 5.4 with Conda.

Databricks Standard

name: databricks-standard
channels:
  - defaults
dependencies:
  - asn1crypto=0.24.0=py37_0
  - backcall=0.1.0=py37_0
  - blas=1.0=openblas
  - boto=2.49.0=py37_0
  - boto3=1.9.111=py_0
  - botocore=1.12.112=py_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.8.24=py37_1
  - cffi=1.11.5=py37he75722e_1
  - chardet=3.0.4=py37_1
  - cryptography=2.3.1=py37hc365091_0
  - cython=0.28.5=py37hf484d3e_0
  - decorator=4.3.0=py37_0
  - docutils=0.14=py37_0
  - idna=2.7=py37_0
  - ipython=6.5.0=py37_0
  - ipython_genutils=0.2.0=py37_0
  - jedi=0.12.1=py37_0
  - jmespath=0.9.4=py_0
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libopenblas=0.3.3=h5a2b251_3
  - libpq=10.5=h1ad7b7a_0
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - ncurses=6.1=he6710b0_1
  - nomkl=3.0=0
  - numpy=1.15.1=py37h99e49ec_0
  - numpy-base=1.15.1=py37h2f8d375_0
  - openssl=1.0.2p=h14c3975_0
  - pandas=0.23.4=py37h04863e7_0
  - parso=0.3.1=py37_0
  - patsy=0.5.0=py37_0
  - pexpect=4.6.0=py37_0
  - pickleshare=0.7.4=py37_0
  - pip=10.0.1=py37_0
  - prompt_toolkit=1.0.15=py37_0
  - psycopg2=2.7.5=py37hb7f436b_0
  - ptyprocess=0.6.0=py37_0
  - pycparser=2.18=py37_1
  - pygments=2.2.0=py37_0
  - pyopenssl=18.0.0=py37_0
  - pysocks=1.6.8=py37_0
  - python=3.7.0=hc3d631a_0
  - python-dateutil=2.7.3=py37_0
  - pytz=2018.5=py37_0
  - readline=7.0=h7b6447c_5
  - requests=2.19.1=py37_0
  - s3transfer=0.2.0=py37_0
  - scikit-learn=0.19.2=py37h22eb022_0
  - scipy=1.1.0=py37he2b7bc3_2
  - setuptools=40.2.0=py37_0
  - simplegeneric=0.8.1=py37_2
  - six=1.11.0=py37_1
  - sqlite=3.24.0=h84994c4_0
  - statsmodels=0.9.0=py37h035aef0_0
  - tk=8.6.8=hbc83047_0
  - traitlets=4.3.2=py37_0
  - urllib3=1.23=py37_0
  - wcwidth=0.1.7=py37_0
  - wheel=0.31.1=py37_0
  - xz=5.2.4=h14c3975_4
  - zlib=1.2.11=h7b6447c_3
  - pip:
    - cycler==0.10.0
    - kiwisolver==1.1.0
    - matplotlib==2.2.3
    - pyarrow==0.12.0
    - pyparsing==2.4.0
    - seaborn==0.9.0
prefix: /databricks/conda/envs/databricks-standard

Databricks Minimal

name: databricks-minimal
channels:
  - defaults
dependencies:
  - backcall=0.1.0=py37_0
  - blas=1.0=openblas
  - ca-certificates=2018.03.07=0
  - certifi=2018.8.24=py37_1
  - decorator=4.3.0=py37_0
  - ipython=6.5.0=py37_0
  - ipython_genutils=0.2.0=py37_0
  - jedi=0.12.1=py37_0
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libopenblas=0.3.3=h5a2b251_3
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - ncurses=6.1=he6710b0_1
  - nomkl=3.0=0
  - numpy=1.15.1=py37h99e49ec_0
  - numpy-base=1.15.1=py37h2f8d375_0
  - openssl=1.0.2p=h14c3975_0
  - pandas=0.23.4=py37h04863e7_0
  - parso=0.3.1=py37_0
  - pexpect=4.6.0=py37_0
  - pickleshare=0.7.4=py37_0
  - pip=10.0.1=py37_0
  - prompt_toolkit=1.0.15=py37_0
  - ptyprocess=0.6.0=py37_0
  - pygments=2.2.0=py37_0
  - python=3.7.0=hc3d631a_0
  - python-dateutil=2.7.3=py37_0
  - pytz=2018.5=py37_0
  - readline=7.0=h7b6447c_5
  - setuptools=40.2.0=py37_0
  - simplegeneric=0.8.1=py37_2
  - six=1.11.0=py37_1
  - sqlite=3.24.0=h84994c4_0
  - tk=8.6.8=hbc83047_0
  - traitlets=4.3.2=py37_0
  - wcwidth=0.1.7=py37_0
  - wheel=0.31.1=py37_0
  - xz=5.2.4=h14c3975_4
  - zlib=1.2.11=h7b6447c_3
  - pip:
    - pyarrow==0.12.0
prefix: /databricks/conda/envs/databricks-minimal