This section provides a reference for Apache Spark SQL and Delta Lake, a set of example use cases, and information about compatibility with Apache Hive.

For further information on Spark SQL, see the Spark SQL, DataFrames, and Datasets Guide.

For further information on Delta Lake, see the Delta Lake.


If you write a SQL query, either in a SQL notebook or in %sql magic command in a notebook with a different primary language, you cannot use $ in an identifier because it is interpreted as a parameter. To escape a $ in SQL command cells, use $\. For example, to define the identifier $foo, write it as $\foo.

Compatibility with Apache Hive