Databricks adds rate limits and access controls to curb serverless spend

Why cost control moves up a layer

According to Databricks, its serverless compute usage has grown more than 3x in the past year, and thousands of customers now run mission critical notebooks, jobs and pipelines on it. As adoption climbs, platform teams keep asking how to gain better visibility into serverless costs and stay within budget.

On classic compute, the company explains, cost control lived in the infrastructure: node counts, instance types, autoscaling ranges and cluster policies. Serverless removes those knobs on purpose, so control attaches to people and workloads instead. Databricks describes three recurring concerns from platform teams:

• Runaway spend: a single unbounded notebook run can spend thousands in a day, because there is no cluster size or autoscaling ceiling.
• The wrong hands: serverless is available to everyone by default, with no entitlement to gate who can run it.
• No accountability: teams will not adopt serverless without the tag enforcement they had on classic compute.

Databricks says customers running serverless SQL and Model Serving already attribute spend, govern access and monitor consumption. The company is now bringing that functionality to serverless compute for notebooks, jobs and pipelines.

Attribution and monitoring available today

Attribution starts with tags. With Serverless Compute Tags, a team tags a serverless compute object once, and the workloads running on it are attributed to the right team, project or cost center. Every DBU that compute runs carries the tag through to billing. This feature is currently in Beta.

Every serverless DBU also lands in system.billing.usage, broken out by user, product and tag, according to Databricks. That makes a "cost by team" view a single query, and the account-console cost dashboards render it without SQL. The same table can show who is running serverless today and how much they spend, which Databricks suggests checking before changing anyone's access.

For monitoring, account-console budgets send email alerts when serverless spend crosses a threshold, and cost management dashboards track consumption by product, workspace and tag. Monitoring is generally available on AWS, Azure and Google Cloud. Databricks acknowledges its limit: monitoring is reactive and reports an expensive day the next morning rather than preventing it.

Access controls and rate limits

Serverless Compute Access Controls is now generally available. It treats a serverless compute object as the permission boundary. Every workspace gets two by default: Default Serverless Interactive Compute for notebooks and Databricks Connect, and Default Serverless Automated Compute for jobs and Spark Declarative Pipelines. Permissions are the familiar Can Use and Can Manage.

The model is safe by default, Databricks says. Every user starts with Can Use, so enabling the feature changes nothing until an admin acts. An admin can open the Compute page, go to the Serverless tab, open a default compute object, remove "All Users" and add a specific group. Users outside that group see Serverless grayed out in the notebook compute selector, and running workloads are not interrupted.

Access control decides whether a workload runs, not how much it spends. Rate limits address that for automated workloads. They are per-compute limits expressed as t-shirt sizes (S, M, L, XL) that bound how much or how fast a job or pipeline can spend. Admins can create custom compute objects and share them per team, giving each team its own limit.

Databricks notes a difference from DBSQL sizes: for notebooks and jobs, the size is a maximum autoscaling ceiling, not a fixed cluster size. A job that only needs Medium is allocated Medium even if XL is chosen. A new serverless_compute_id in system.billing.usage attributes spend per compute object. Rate limits are in Beta and are enabled through Databricks preview settings.

What it means for platform teams

Shared budgets for interactive workloads are not yet available. Databricks argues a per-session cap alone cannot protect a team's budget, because many "Medium" sessions still add up. Its stated direction is a shared limit for the group, with a per-session maximum underneath it.

For teams adopting serverless now, Databricks recommends a sequence: turn on tag enforcement, set up cost dashboards and a budget alert, audit who runs serverless today, decide the access posture for interactive and automated work, and then enable Serverless Compute Access Controls. Documentation for AWS, Azure and Google Cloud covers each feature.