SQL Guide

This guide provides a reference for Spark SQL and Delta Lake, a set of example use cases, and information about compatibility with Apache Hive.

For further information on Spark SQL, see the Apache Spark Spark SQL, DataFrames, and Datasets Guide.

For further information on Delta Lake, see the Delta Lake Guide.


If you write a SQL query, either in a SQL notebook or in %sql magic command in a notebook with a different primary language, you cannot use $ in an identifier because it is interpreted as a parameter. To escape a $ in SQL command cells, use $\. For example, to define the identifier $foo, write it as $\foo.

For further information on notebook widgets and parameters, see the Widgets Guide.

Compatibility with Other Systems