The transaction log for a Databricks Delta table contains versioning information that supports Databricks Delta evolution. Databricks Delta tracks minimum reader and writer version separately.
Databricks Delta guarantees backward compatibility. A newer version of Databricks Runtime will always be able to read data that was written by an older version of Databricks Runtime.
Databricks Delta will occasionally break forward compatibility. Older versions of Databricks Runtime may not be able to read and write data that was written by a newer version of Databricks Runtime. If you try to read and write to a table with a version of Databricks Runtime that is too old, you’ll get an error telling you that you need to upgrade.
For existing tables, you’ll be warned about the availability of new version and given the option to upgrade the table. Sometimes you may be required to upgrade before you are able to use new features (for example, append-only tables).
Tables created with Databricks Runtime 4.1 have a new writer version and thus cannot be written to by versions of Databricks Runtime lower than 4.1. To take advantage of Databricks Runtime 4.1 improvements, you must upgrade existing tables. To upgrade a table, first upgrade all jobs that are writing to the table. Then run:
%scala com.databricks.delta.Delta.upgradeTableProtocol("</path/to/table>" or "<tableName>")
Databricks Runtime 4.1 did not change the reader version, so versions of the Databricks Runtime lower than 4.1 can read tables written by Databricks Runtime 4.1.