Secret Scopes

Managing secrets begins with creating a secret scope. A secret scope is identified by its name, unique within a workspace. The names are considered non-sensitive and are readable by all users in the workspace. A workspace is limited to a maximum of 100 scopes.

Overview

There are two types of secret scope: Azure Key Vault-backed and Databricks-backed.

Azure Key Vault-backed scopes

To reference secrets stored in an Azure Key Vault, you can create a secret scope backed by Azure Key Vault. You can then leverage all of the secrets in the corresponding Key Vault instance from that secret scope. Because the Azure Key Vault-backed secret scope is a read-only interface to the Key Vault, the PutSecret and DeleteSecret Secrets API operations are not allowed. To manage secrets in Azure Key Vault, you must use the Azure SetSecret REST API or Azure portal UI.

Note

  • Azure Key Vault-backed secret scopes are in Public Preview.
  • Creating an Azure Key Vault-backed secret scope is supported only in the Azure Databricks UI. You cannot create a scope using the Secrets CLI or API.

Databricks-backed scopes

A Databricks-backed scope is stored in (backed by) an Azure Databricks database. You create a Databricks-backed secret scope using the Databricks CLI (version 0.7.1 and above). Alternatively, you can use the Secrets API.

Scope permissions

Scopes are created with permissions controlled by ACLs. By default, scopes are created with MANAGE permission for the user who created the scope (the “creator”), which lets the creator read secrets in the scope, write secrets to the scope, and change ACLs for the scope. If your account has the Azure Databricks Premium Plan, you can assign granular permissions at any time after you create the scope. For details, see Secret Access Control.

You can also override the default and explicitly grant MANAGE permission to all users when you create the scope. In fact, you must do this if your account does not have the Azure Databricks Premium Plan. See the instructions in this topic for details.

Best practices

As a team lead, you might want to create different scopes for Azure SQL Data Warehouse and Azure Blob Storage credentials and then provide different subgroups in your team access to those scopes. You should consider how to achieve this using the different scope types:

  • If you use a Databricks-backed scope and add the secrets in those two scopes, they will be different secrets (Azure SQL Data Warehouse in scope 1, and Azure Blob Storage in scope 2).
  • If you use an Azure Key Vault-backed scope with each scope referencing a different Azure Key Vault and add your secrets to those two Azure Key Vaults, they will be different sets of secrets (Azure SQL Data Warehouse ones in scope 1, and Azure Blob Storage in scope 2). These will work like Databricks-backed scopes.
  • If you use two Azure Key Vault-backed scopes with both scopes referencing the same Azure Key Vault and add your secrets to that Azure Key Vault, all Azure SQL Data Warehouse and Azure Blob Storage secrets will be available. Since ACLs are at the scope level, all members across the two subgroups will see all secrets. This arrangement does not satisfy your use case of restricting access to a set of secrets to each group.

Create an Azure Key Vault-backed secret scope

Note

This feature is in Public Preview.

  1. Verify that you have Owner permission on the Azure Key Vault instance that you want to use to back the secret scope.

    If you do not have a Key Vault instance, follow the instructions in Quickstart: Create a Key Vault using the Azure portal.

  2. Go to https://<your_azure_databricks_url>#secrets/createScope (for example, https://westus.azuredatabricks.net#secrets/createScope).

    ../../_images/azure-kv-scope.png
  3. Enter the name of the secret scope.

  4. Use the Manage Principal drop-down to specify whether All Users have MANAGE permission for this secret scope or only the Creator of the secret scope (that is to say, you).

    MANAGE permission allows users to read and write to this secret scope, and, in the case of accounts on the Azure Databricks Premium Plan, to change permissions for the scope.

    Your account must have the Azure Databricks Premium Plan for you to be able to select Creator. This is the recommended approach: grant MANAGE permission to the Creator when you create the secret scope, and then assign more granular access permissions after you have tested the scope. For an example workflow, see Secret Workflow Example.

    If your account has the Standard Plan, you must set the MANAGE permission to the “All Users” group. If you select Creator here, you will see an error message when you try to save the scope.

    For more information about the MANAGE permission, see Secret Access Control.

  5. Enter the DNS Name (for example, https://databrickskv.vault.azure.net/) and Resource ID, for example:

    /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/databricks-rg/providers/Microsoft.KeyVault/vaults/databricksKV
    

    These properties are available from the Properties tab of an Azure Key Vault in your Azure portal.

    ../../_images/azure-kv.png
  6. Click the Create button.

  7. Use the Databricks CLI databricks secrets list-scopes command to verify that the scope was created successfully.

For an example of using the secrets when accessing Azure Blob Storage, see Mount an Azure Blob Storage container.

Create a Databricks-backed secret scope

To create a scope using the Databricks CLI:

databricks secrets create-scope --scope <scope-name>

By default, scopes are created with MANAGE permission for the user who created the scope. If your account does not have the Azure Databricks Premium Plan, you must override that default and explicitly grant the MANAGE permission to “users” (all users) when you create the scope:

databricks secrets create-scope --scope <scope-name> --initial-manage-principal users

If your account has the Azure Databricks Premium Plan, you can change permissions at any time after you create the scope. For details, see Secret Access Control.

Once you have created a Databricks-backed secret scope, you can add secrets.

List secret scopes

To list the existing scopes in a workspace:

databricks secrets list-scopes

Delete a secret scope

Deleting a secret scope deletes all secrets and ACLs applied to the scope. To delete a scope:

databricks secrets delete-scope --scope <scope-name>