Databricks Cut Some AI Coding Costs by 90% With a Gateway

From removing barriers to a cost spiral

In a blog conversation titled "The AI Bill is coming", Catherine Brown interviews Patrick, who runs Databricks' AI products and has championed internal AI adoption. He says engineering output is the main bottleneck in the business, and that the return on AI tools changed when the first really good coding models arrived about a year ago. According to Databricks, the company is now close to doubling output with the same team, measured through pull requests, features shipped and lines of code.

The early rollout aimed to get people over the hump: the most expensive model and as many tokens as anyone wanted. Then usage took off. Because these tools are priced on consumption rather than per seat, Patrick says, one user can in principle spend an unbounded amount. Costs rose exponentially while revenue did not, and at one point threatened to reverse the efficiency gains. Databricks says its token spend per head is now roughly flat, even as total usage and value keep growing.

Why hard caps were dropped

A hard spend cap was the obvious first move, but Patrick calls it a bad experience. Developers lose their flow mid-task, and the people hitting the cap are often the best users. Some switch to personal accounts, which removes visibility entirely.

Databricks moved to friction that ramps up, handled by Unity Gateway: first a warning, then an approval step, then a downshift to a cheaper model so work can continue. Cutting someone off is a last resort. Developers also receive a regular breakdown of their own spending with saving suggestions, which Patrick says is worth about 10% on its own.

The four levers and what they delivered

The company names four levers: shift defaults to more efficient models, route each task to the cheapest model that can handle it, give developers visibility into spend, and cut token overhead. Patrick says moving traffic to newer, cheaper models has been the most impactful step, worth 50% or more by itself. Because all traffic passes through Unity Gateway, the switch happens in one place and developers change nothing. He adds that one recent cheap model roughly matches what a frontier model could do about nine months ago at around 1% of the cost per task, though he calls that preliminary.

Databricks also built its own benchmark from real pull requests in its repository, graded by running tests rather than using an LLM as judge. Findings included:

• An open source model, GLM 5.2, tied Opus 4.8 on quality at about a third less per task.
• A model that was 1.7x cheaper per token cost more per task because it used more tokens.
• The same model in a different harness showed more than a 2x cost difference, and tuning harness and caching settings saved close to 50%.

Routing and the Astra rollout

Patrick argues that budgets cannot decide which model should handle a request, and that is a routing job. According to Databricks, its Smart Router cuts average task cost by more than 30% at roughly the quality of the most expensive model, and removes the need to ask users when to pick pricier models.

The Astra rollout to around 3,500 engineers illustrates the trade-off. It was clearly better on hard work such as system design, but spend rose about 60% without obvious gains on everyday tasks. Databricks gave Astra its own budget in Unity Gateway and told engineers to use it for hard problems.

What it means for engineering organizations

The company's takeaway is that every engineering organization needs a control layer. An AI gateway sits between developers and models to handle access, routing, budgets and tracing, making spend observable, steerable and predictable. Databricks also stresses that token price alone is a poor predictor of what a model costs, so teams need to test models on their own code. The source text is cut off before the interview ends, and the figures are Databricks' own claims from its internal use.