Azure Databricks supports deep learning libraries for building and applying neural networks. This section gives examples to get started with deep learning in Azure Databricks using several popular libraries. We provide installation instructions as well as accompanying example notebooks to get started.
Graphics processing units (GPUs) are increasingly popular because they can accelerate deep learning and other machine learning tasks. For more information about creating GPU-enabled clusters, see GPU-enabled Clusters. Databricks includes pre-installed GPU hardware drivers and NVIDIA libraries such as CUDA.
You can install TensorFlow, MXNet, and Keras as a Databricks library from PyPI. For all other deep learning libraries, we recommend that you use Cluster Node Initialization Scripts to install required libraries on clusters upon creation and we provide init scripts for those libraries.
Many of these deep learning libraries are also available in Databricks Runtime ML (Beta), a machine learning runtime that provides a ready-to-go environment for machine learning and data science. In particular, Databricks Runtime ML includes TensorFlow, Keras, and XGBoost. It also supports distributed TensorFlow training using Horovod. Instead of installing TensorFlow or Keras using the instructions below, you can simply create a cluster using Databricks Runtime ML. See Databricks Runtime ML (Beta).
- Deep Learning Pipelines
- Distributed Deep Learning
- Integrating Deep Learning Libraries with Apache Spark