Pydantic benchmark: code mode roughly halves cost on long agent tasks

A benchmark built from boring agents

Code mode is the idea that an agent should not call tools one by one. The model writes a small program that calls them instead, and only the result returns to the model. Pydantic notes that Cloudflare and Anthropic each have a version, and that OpenCode 2 now sends MCP tools through Code Mode by default. Pi ships a codemode tool.

The author of the Pydantic article says most numbers he had seen came from agents with hundreds of tools or huge payloads, while many real agents are far more modest: a handful of CRUD tools and a job to do. So the benchmark uses five small Pydantic AI apps: a shopping cart, a calendar, a chatroom, a kanban board and an inventory system. Each has only single-item tools (list, get, create, update, delete), with paginated lists and no bulk endpoints.

Work came in three tiers. The 50 everyday tests, ten per app, needed about 13 tool calls per run without code mode. The 12 long-horizon jobs covered 60 to 500 records and needed about 68 calls. The scaling tasks applied one rule at 5, 20, 50, 100 and 200 items, and at 200 items runs without code mode averaged 168 tool calls. In total Pydantic graded 5,276 runs across eight models, with code mode off and on, on an internal benchmark.

What the numbers show

On long tasks, across all eight models, code mode lowered cost by 58% and wall time by 51%, and raised success by 16.0 points, according to Pydantic. Without code mode 67 of 288 runs failed. With it, 21 did.

The gap widened with task size. For gpt-6.1-sol, every 100 extra items cost 30 to 79 seconds and $0.04 to $0.07 per run without code mode. With it, the added time was 4 to 9 seconds on four of the five tasks, and the added cost was under a cent on three. At 200 items, code mode was 47% to 75% cheaper.

Correctness followed the same pattern. From 5 items to 200, success without code mode fell from 92-100% to 62-79%. With code mode, success at 200 items stayed at 92-100%.

In Pydantic's words, "models are much better at writing a loop than at being one, and the gap grows with the length of the task."

Not about round trips, and a tax on small jobs

The usual explanation is that each tool call forces another model turn that resends the growing context. Pydantic's data points elsewhere. Models already batch their calls, and one response carried 200 tool calls. At 200 items, averaged over the eight models, model requests fell by between 1% and 22% per task. deepseek-v4.1-flash fell much further. According to the article, what code mode cuts is what goes into each request.

Small jobs do not benefit. At 5 items, code mode cost more on three of the five tasks, from +21% to +38%. On the 50 everyday tasks it was 16% cheaper overall, but success was flat at +1.6 points, which Pydantic describes as noise.

A cart example and the sandbox behind it

Pydantic walks through one everyday task: remove everything in a cart that costs more than $50 apiece. The cart has 122 lines, the list tool returns at most 50 per page, and the remove tool takes one id. 22 lines need to go. Without code mode, that means three pages of listing and 22 removals, each chosen by the model. With code mode, the model writes a loop that pages through the list, filters by price and removes the matching items concurrently. It never reads the 100 cheaper lines, and Python does the price comparison.

Since the code is written by a model, the runtime matters. In Pydantic AI Harness, the program runs in Monty, Pydantic's Python interpreter written in Rust. It starts with no access to the host's files, environment or clock, and has no network. Access is granted per request, and the benchmark used the built-in OSAccess handler, which provides an in-memory filesystem and environment plus the host clock. By default each run_code snippet gets 30 seconds of execution time and a 256 MiB heap. Pydantic says a new sandbox takes under a millisecond from a running pool. With Pydantic Logfire tracing, every tool call the code makes appears as its own span.

Pydantic contrasts this with Prime Intellect's Prime Agent, whose only model tool is a persistent IPython kernel. That is a full Python process, which the author considers right for a coding assistant on your own machine but not for a web server driven by user prompts.

What it means for agent builders

The results suggest that the value of code mode depends on task length. For agents that routinely loop over dozens or hundreds of records, Pydantic's figures show large savings in cost and time and fewer failures. For agents that mostly handle short requests, the gains were small, and tiny jobs can cost more. The benchmark is Pydantic's own and internal, and it tests its own Harness, so the figures are the company's claims rather than independent results.