AI21 Cuts GitLab Duo Code Review Cost by 98% by Tuning Its Harness

A four-stage pipeline with a costly review step

Duo Code Review is the Code Review Flow of GitLab Duo Agent Platform. It reviews merge requests and posts comments. At the time of testing it ran Claude Sonnet 5 by default, and GitLab published the flow, built on LangGraph, in its source-available ai-assist repository. AI21 says that, together with a public benchmark, is why it picked this system.

The flow has four model stages, each with its own prompt and at most two tools. Exploration lists the repository tree around the changed files. File fetching reads related files such as tests, callers and custom instructions. Analysis turns the fetched material into notes. Review reads the diff and most changed files in full, then posts comments.

AI21 notes that its generic optimizations, built for coding agents like Claude Code that resend one growing conversation, changed zero requests here. Each stage of Duo Code Review starts from a fresh prompt, so there is no history to trim. In AI21's words, each harness requires its own careful analysis.

The baseline cost was $0.589 per pull request. The review stage accounts for 65% of that, and its output alone is a third of the total: Sonnet 5 writes about 18,600 tokens per review, nearly all reasoning. Exploration and fetching make up 27%, and analysis 9%.

Compaction and model switching, step by step

The first change was compacting the review context with rules rather than a model, so nothing is summarized. File windowing keeps each changed hunk plus 25 lines either side of both the old and new line ranges. Diff compaction rewrites the per-line diff as a standard unified diff with the same hunks and line numbers. The median review prompt fell from 59K to 32K tokens, and spending on review context fell by 62%.

The bill fell far less, though. With compaction on, Sonnet 5 wrote about 23,800 output tokens per review instead of 18,600. AI21 says it has not isolated why. On its own, compaction cut cost from $0.589 to $0.475, a 19.4% saving, at 47.2% F1 and 1.16x the latency.

Swapping models stage by stage was next. Claude Haiku 4.5 picked the same files as Sonnet 5 on exploration and fetching, but quality dipped on analysis and dropped sharply on the review. Kept on the two mechanical stages, Haiku cut the bill 15.4% at unchanged F1, and 34.6% with compaction.

AI21 then moved all four stages to DeepSeek V4 Flash, an open-weight model it serves with vLLM on H100s. At Together AI's list price it costs 14x less than Sonnet 5 per input token and 36x less per output token. Cost fell to $0.086, an 85.3% saving, at 48.4% F1 and 0.37x the latency. About one review call in seven on DeepSeek ended without posting, but GitLab's own correction step retried each one, and the retries are included in the costs.

Why both changes were needed

The saving stalled at 85% because of the deployment window. AI21's DeepSeek deployment accepts prompts up to 256K tokens, and longer ones stay on Sonnet 5. Without compaction, some review prompts reach 480K, and 3 of the 50 pull requests crossed the limit, producing 11 fallback calls. Those calls carried 77% of the remaining $0.086 per PR.

Putting compaction back in made the prompts fit. The largest review prompt dropped from 480K to 109K tokens, and nothing fell back. Cost reached $0.0135 per PR, 97.7% below baseline, at 47.2% F1 and 0.35x the latency, against 46.4% F1 for the baseline. A second judge scored within half a point.

Compaction's own token saving at that point is about $0.006 per PR. According to AI21, what it buys is fit: in front of Claude it is a cost optimization, while in front of a 256K deployment it keeps every review on the open-weight model. AI21 says a 1M deployment would have removed the fallback too, but it would not have shrunk what every review reads, and it would rely on the model at lengths where its quality is still debated.

Some ideas did not help. Raising reasoning effort to "max" on 30 trial reviews lifted recall from 38.9% to 52.8%, while F1 moved from 35.0% to 35.4% at 3.5x the latency. Prompt caching found little to reuse, because stages share no prefix with each other.

What it means, and the limits

AI21 draws a simple lesson: the model sets the price, and the harness decides whether you get to use it. For teams running agent pipelines, the result suggests looking at each stage's model and context separately instead of applying one generic optimization.

The caveats are significant. The test used public pull requests, with no GitLab production traffic or customer data. Quality is F1 on one public benchmark, scored by two judges. Cost is token spend at published list prices, including for the self-served model, so it does not reflect AI21's own infrastructure costs. Configurations were chosen and tuned on the same 50 pull requests they were scored on, and AI21 says how they hold on a new codebase is untested.