An Azure Databricks Notebook or Job API returns the following error:
Unexpected failure while creating the cluster for the job. Cause REQUEST_LIMIT_EXCEEDED: Your request was rejected due to API rate limit. Please retry your request later, or choose a larger node type instead.
The error indicates the Cluster Manager Service core instance request limit was exceeded.
A Cluster Manager core instance can support a maximum of 1000 requests.
Contact Azure Databricks Support to increase the limit set in the core instance.
Azure Databricks can increase the job limit
maxBurstyUpsizePerOrg up to 2000, and
upsizeTokenRefillRatePerMin up to 120. Current running jobs are affected when the limit is increased.
Increasing these values can stop the throttling issue, but can also cause high CPU utilization.
The best solution for this issue is to replace the Cluster Manager core instance with a larger instance that can support maximum data transmission rates.
Azure Databricks Support can change the current Cluster Manager instance type to a larger one.