Databricks Runtime 4.1 ML provides a ready-to-go environment for machine learning and data science. It contains multiple popular libraries, including TensorFlow, Keras, and XGBoost. It also supports distributed TensorFlow training using Horovod.
Databricks Runtime 4.1 ML will be deprecated soon. We recommend you use Databricks Runtime 5.0 ML.
For more information, including instructions for creating a Databricks Runtime ML cluster, see Databricks Runtime for Machine Learning.
Databricks Runtime 5.0 ML is built on top of Databricks Runtime 5.0. For information on what’s new in Databricks Runtime 5.0, see the Databricks Runtime 4.1 release notes. In addition to the new features in Databricks Runtime 4.1, Databricks Runtime 4.1 ML includes the following libraries to support machine learning. Some of these are also included in the base Databricks Runtime 4.1 and are noted as such.
|Distributed Deep Learning||
|Other machine learning libraries|
Maintenance updates made to Databricks Runtime 4.1 ML since its initial release include:
- July 31, 2018
- Added Azure SQL DW connector to ML Runtime 4.1
- Fixed a bug that could cause incorrect query results when the name of a partition column used in a predicate differs from the case of that column in the schema of the table.
- Fixed a bug affecting Spark SQL execution engine.
- Fixed a bug affecting code generation.
- Fixed a bug (
java.lang.NoClassDefFoundError) affecting Databricks Delta.
- Improved error handling in Databricks Delta.
- Fixed a bug that caused incorrect data skipping statistics to be collected for string columns 32 characters or greater.