Deploy Azure Databricks in your Azure Virtual Network (VNet Injection)

Important

If you deployed an Azure Databricks workspace in your own Azure Virtual Network during the VNet injection preview, you must upgrade your preview workspace to the GA version before January 31, 2020. See Upgrade your VNet Injection Preview Workspace to GA.

The default deployment of Azure Databricks is a fully managed service on Azure: all data plane resources, including a virtual network (VNet) that all clusters will be associated with, are deployed to a locked resource group. If you require network customization, however, you can deploy Azure Databricks data plane resources in your own virtual network (sometimes called VNet injection), enabling you to:

Deploying Azure Databricks data plane resources to your own virtual network also lets you take advantage of flexible CIDR ranges (anywhere between /16-/24 for the virtual network and up to /26 for the subnets).

Important

You cannot replace the virtual network for an existing workspace. If your current workspace cannot accommodate the required number of active cluster nodes, we recommend that you create another workspace in a larger virtual network. Follow these detailed migration steps to copy resources (notebooks, cluster configurations, jobs) from the old to new workspace.

Virtual network requirements

The virtual network that you deploy your Azure Databricks workspace to must meet the following requirements:

  • Location: The virtual network must reside in the same location as the Azure Databricks workspace.

  • Subscription: The virtual network must be in the same subscription as the Azure Databricks workspace.

  • Subnets: The virtual network must include two subnets dedicated to Azure Databricks: a private subnet and public subnet. The public subnet allows communication with the Azure Databricks control plane. The private subnet allows only cluster-internal communication.

    Azure Databricks will create these for you when you Create the Azure Databricks workspace in the Azure portal and will delegate your public and private subnet to the Microsoft.Databricks/workspaces service during workspace deployment, which allows Azure Databricks to create Network security group rules and update them when necessary. This also true if you use the All in one ARM template or the Virtual network ARM template. If you use a custom ARM template or the Azure Databricks workspace ARM template, it is up to you to ensure that the subnets have network security groups attached and are properly delegated. For delegation instructions, see Upgrade your VNet Injection Preview Workspace to GA or Add or remove a subnet delegation.

  • Address space: A CIDR block between /16 - /24 for the virtual network and a CIDR block up to /26 for the private and public subnets.

You can use the Azure Databricks workspace deployment interface in the Azure portal to automatically configure an existing virtual network with the required subnets, or you can use Azure-Databricks-supplied ARM templates to configure your virtual network and deploy your workspace.

Create the Azure Databricks workspace in the Azure portal

This section describes how to create an Azure Databricks workspace in the Azure portal and deploy it in your own existing virtual network. Azure Databricks updates the virtual network with two new subnets and network security groups using CIDR ranges provided by you, whitelists inbound and outbound subnet traffic, and deploys the workspace to the updated virtual network.

Note

If you want more control over the configuration of the virtual network—for example, you may want to use existing subnets, use existing network security groups, or create your own security rules—you can use Azure-Databricks-supplied ARM templates instead of the portal UI. See Advanced configuration using ARM templates.

Prerequisites

You must configure a virtual network to which you will deploy the Azure Databricks workspace:

  • You can use an existing virtual network or create a new one, but the virtual network must be in the same region and same subscription as the Azure Databricks workspace that you plan to create.

  • A CIDR range between /16 - /24 is required for the virtual network.

    Warning

    A workspace with a smaller virtual network can run out of IP addresses (network space) more quickly than a workspace with a larger virtual network. For example, a workspace with a /24 virtual network and /26 subnets can have a maximum of 64 nodes active at a time, whereas a workspace with a /20 virtual network and /22 subnets can house a maximum of 1024 nodes.

    Your subnets will be created automatically when you configure your workspace, and you will have the opportunity to provide the CIDR range for the subnets during configuration.

Configure the virtual network

  1. In the Azure portal, select + Create a resource > Analytics > Azure Databricks or search for Azure Databricks and click Create or + Add to launch the Azure Databricks Service dialog.

  2. Follow the configuration steps described in Step 2: Create an Azure Databricks workspace in the Getting Started Guide, and select the Deploy Azure Databricks workspace in your Virtual Network option.

    ../../../_images/vnet-injection-yes.png
  3. Select the virtual network you want to use.

    ../../../_images/vnet-injection-workspace.png
  4. Name your public and private subnets and provide CIDR ranges in a block up to /26:

    • A public subnet will be created with associated Network security group rules that allow communication with the Azure Databricks control plane.
    • A private subnet will be created with associated network security group rules that allow cluster-internal communication.
    • Azure Databricks will have delegated permissions to update both subnets via the Microsoft.Databricks/workspaces service. These permissions apply only to network security group rules that are required by Azure Databricks, not to other network security group rules that you add or to the default network security group rules included with all network security groups.
  5. Click Create to deploy the Azure Databricks workspace to the virtual network.

Note

When you create a new virtual network (VNet) workspace in Azure Databricks, an Azure Databricks Managed Resource Group is created by default, as part of the normal operation of the Azure Databricks system. This resource group is not modifiable, and it is not used to create virtual machines. Only the customer-managed group is used to create virtual machines.

Advanced configuration using ARM templates

If you want more control over the configuration of the virtual network—for example, you want to use existing subnets, use an existing network security group, or add your own security rules—you can use the following ARM templates instead of the portal-UI-based automatic virtual network configuration and workspace deployment.

Important

If you deployed an Azure Databricks workspace in your own Azure Virtual Network during the VNet injection preview, you must upgrade your preview workspace to the GA version before January 31, 2020. The primary upgrade task is to delegate your public and private subnets to the Microsoft.Databricks/workspaces service, which allows Azure Databricks to create Network security group rules and update them when necessary. See Upgrade your VNet Injection Preview Workspace to GA.

If you used ARM templates to deploy a Azure Databricks workspace to your own virtual network during the preview, and you want to continue to use ARM templates to create virtual networks and deploy workspaces, you should use the upgraded ARM templates listed here. Follow the links below to get the latest version of each template.

If you are using a custom ARM template or the Workspace Template for Databricks VNet Injection to deploy a workspace to an existing virtual network, you must create public and private subnets, attach a network security group to each subnet, and delegate the subnets to the Microsoft.Databricks/workspaces service before deploying the workspace.

All in one

To create a virtual network and Azure Databricks workspace all in one, use the All-in-one Template for Databricks VNet Injected Workspaces.

Virtual network

To create a virtual network with the proper public and private subnets, use the Virtual Network Template for Databricks VNet Injection.

Azure Databricks workspace

To deploy an Azure Databricks workspace to an existing virtual network, use the Workspace Template for Databricks VNet Injection.

Important

Your virtual network’s public and private subnets must have network security groups attached and must be delegated to the Microsoft.Databricks/workspaces service before you use this ARM template to deploy a workspace. You can use the Virtual Network Template for Databricks VNet Injection to create a virtual network with properly delegated subnets. If you are using an existing virtual network and you have not delegated the public and private subnets, see Add or remove a subnet delegation or Upgrade your VNet Injection Preview Workspace to GA.

Network security group rules

The following table displays the current network security group rules used by Azure Databricks. In order to ensure that your Azure Databricks service runs smoothly, Azure Databricks can change these rules at any time. This topic and table will be updated whenever such a modification occurs.

Direction Protocol Source Source Port Destination Dest Port Used
Inbound Any VirtualNetwork Any VirtualNetwork Any Default
Inbound TCP ControlPlane IP Any VirtualNetwork 22 Public IP
Inbound TCP ControlPlane IP Any VirtualNetwork 5557 Public IP
Outbound TCP VirtualNetwork Any Webapp IP 443 Default
Outbound TCP VirtualNetwork Any SQL 3306 Default
Outbound TCP VirtualNetwork Any Storage 443 Default
Outbound Any VirtualNetwork Any VirtualNetwork Any Default

Troubleshooting

Workspace creation errors

Subnet <subnet ID> requires any of the following delegation(s) [Microsoft.Databricks/workspaces] to reference service association link
Possible cause: you are creating a workspace in a VNet whose private and public subnets have not been delegated to the Microsoft.Databricks/workspaces service. Each subnet must have a network security group attached and must be properly delegated. See Virtual network requirements for more information.

Cluster creation errors

Instances Unreachable: Resources were not reachable via SSH.
Possible cause: traffic from control plane to workers is blocked. If you are deploying to an existing virtual network connected to your on-premises network, review your setup using the information supplied in Connect your Azure Databricks Workspace to your On-Premises Network.
Unexpected Launch Failure: An unexpected error was encountered while setting up the cluster. Please retry and contact Azure Databricks if the problem persists. Internal error message: Timeout while placing node.
Possible cause: traffic from workers to Azure Storage endpoints is blocked. If you are using custom DNS servers, also check the status of the DNS servers in your virtual network.
Cloud Provider Launch Failure: A cloud provider error was encountered while setting up the cluster. See the Azure Databricks guide for more information. Azure error code: AuthorizationFailed/InvalidResourceReference.
Possible cause: the virtual network or subnets do not exist any more. Make sure the virtual network and subnets exist.
Cluster terminated. Reason: Spark Startup Failure: Spark was not able to start in time. This issue can be caused by a malfunctioning Hive metastore, invalid Spark configurations, or malfunctioning init scripts. Please refer to the Spark driver logs to troubleshoot this issue, and contact Databricks if the problem persists. Internal error message: Spark failed to start: Driver failed to start in time.
Possible cause: Container cannot talk to hosting instance or DBFS storage account. Fix by adding a custom route to the subnets for the DBFS storage account with the next hop being Internet.