AI21 Labs: Public coding benchmarks miss how developers really use agents

What seven weeks of developer traffic showed

AI21 Labs developed an internal coding task classifier and ran seven weeks of real traffic from its own developers through it. According to the company, the result covered 15 categories, 84 subcategories and 4,211 turns, and it contradicted the company's assumptions about agentic coding.

The central finding is that developers use agents as thought partners more than as code generators. AI21 reports that only about 11% of real developer turns ask the agent to write code. Most of the time, developers ask clarifying questions, paste context, do analysis or request reviews. The single most common message is a clarifying question.

The interactions are also conversational. Instead of handing over a 1,000-word GitHub issue, developers type short messages of around 80 characters in the middle of a session, paste console logs and iterate on the agent's output. AI21 adds that over 60% of its internal sessions contain more than one distinct task.

Where public benchmarks fall short

AI21 ran several public benchmarks, including SWE-bench, HumanEval and Spider, through the same classifier. It says almost all of them focus on writing code from scratch or fixing a bug so that a test passes. Confirmations, redirections, git commands and pasted context make up about 40% of real developer turns, yet appear in almost no public coding benchmarks, according to the company.

Difficulty is another gap. AI21 argues that benchmarks over-index on complex, multi-file architectural issues, while real developers are more likely to ask a practical question such as "Why do I not see my changes in git status?" The company says this can make smaller or cheaper models look artificially incapable of handling everyday traffic.

Context differs too. A typical benchmark task is a well-written, 1,000-character GitHub issue on a clean code checkout. Real messages arrive on top of a long conversation history, tool outputs and half-finished file changes.

How AI21 built its own benchmark

AI21 describes a tension between three properties: tasks drawn from real work, rich context, and the ability to be graded automatically. Real sessions satisfy the first two but are hard to grade without human review, while public benchmarks are easy to grade but lack real-world context. The company set out to combine automated grading with the messiness of actual traffic, guided by four principles:

• Let real traffic set the quotas. If 40% of real traffic is clarifying questions and codebase analysis, then 40% of the benchmark tests that.
• Mix public tasks with internal code. AI21 mined its own repositories for tasks that were already solved and merged, and verified they could be graded by its own test suites.
• Write like a developer. The prompts were rewritten to resemble short, often ambiguous developer messages, while the code environment and automated tests stayed identical.
• Start in the middle. Internal replay infrastructure restores a real developer session at any point, with accumulated context and modified files, so the developer's next move can be simulated.

The result is a 613-task benchmark that the company says mirrors reality.

Testing the benchmark on a routing gateway

To check whether the benchmark captures the natural distribution of real tasks better than public ones, AI21 turned to the frontier-powered router in its intelligent Gateway. The Gateway sits between its developers and foundation models and has been trained for agentic coding traffic. For each message, the router scores task difficulty and decides whether the request needs an expensive premium model or whether a cheaper, faster one can handle it. It can be tuned to send an expected percentage of traffic to the premium model, trading off cost and quality.

The source text breaks off before the results of this test are reported. For teams that build agents, routers or harnesses, AI21's argument is that evaluations should reflect real usage, since a benchmark that misjudges task difficulty can distort decisions about which models to use.