Arize benchmark: high prompt cache reuse does not guarantee lower agent cost
A prompt caching benchmark from Arize AI finds that a high cache-read rate does not always translate into lower cost. Claude Opus 5.5 read 89.8% of its prompt tokens from cache and still had the highest estimated cost, because output tokens drove most of the spend.
- DeepSeek had the highest cache-read rate (93.6%) and the lowest Phoenix-estimated cost ($1.17 for 100 runs), according to Arize.
- Claude read 89.8% of prompt tokens from cache but had the highest estimated cost, $14.63, because output tokens drove most of the spend.
- Cache reuse rose with conversation length for all four models; GPT went from 34.2% at 5 turns to 87.0% at 20 turns.
- Costs are Phoenix estimates, not reconciled against provider bills, and model and provider path changed together.
How Arize ran the test
Prompt caching lets an agent reuse previously processed prompt tokens instead of processing the same input again. For multi-turn agents that keep carrying forward instructions, application context or conversation history, Arize says this can reduce the cost of repeated input.
To test it, the company built a Wonder Toys shopping assistant and a dataset of 20 multi-turn conversations, ranging from 5 to 20 turns. Each conversation was run five times per model/provider path, giving 100 conversation traces per model. Arize used Harbor, a framework for evaluating agents in sandboxed environments, to run the tasks, and the Phoenix Harbor plugin recorded the jobs as versioned datasets and experiments and sent traces to Phoenix.
The product catalog, task instructions, conversation dataset and evaluation criteria stayed constant. Only the model and provider changed. DeepSeek and GLM ran through OpenRouter, GPT through OpenAI directly, and Claude through Anthropic directly. Because model and provider changed together, Arize says the results compare the observed behavior of each path rather than isolating the caching efficiency of a model.
Cost in the benchmark means Phoenix estimates based on token prices from each provider's pricing page. Arize did not reconcile them against actual provider bills.
Results: cache reuse, cost and latency
DeepSeek had the highest cache-read rate at 93.6%, followed by Claude at 89.8%, GLM at 84.3% and GPT at 77.7%. DeepSeek also had the lowest estimated total cost across 100 runs, at $1.17, and the shortest average trace latency, at 34.7 seconds. Arize cautions that the results do not establish that caching alone caused either advantage.
- DeepSeek V4 Pro 0813: 93.6% cache read, $1.17, 34.7s
- GLM 5.3 Prime: 84.3% cache read, $4.49, 91.4s
- GPT-6.1 Sol: 77.7% cache read, $2.74, 55.6s
- Claude Opus 5.5: 89.8% cache read, $14.63, 81.2s
Cache reuse rose with conversation length for all four models, which Arize expected, since later turns carry more catalog, instruction and history content that can be reused. GPT showed the biggest swing, from 34.2% at 5 turns to 87.0% at 20 turns. DeepSeek started at 85.7% and reached 95.6% at 20 turns. GLM went from 70.0% to 88.3%, and Claude from 84.1% to 91.7%.
Arize notes that teams whose agents mostly hold short conversations may find the 5-turn results more relevant than the aggregate ranking, while those with long threads should look at later-turn behavior.
Why output volume matters for cost
The cache-read percentage shows how much prompt input was reused, but not the whole cost picture. Claude's estimated total of $14.63 was the highest in the test, and Arize says output accounted for most of that spend.
GPT offers another example. Its cache-read rate was lower than GLM's, yet its estimated total cost was also lower, at $2.74 versus $4.49. According to Arize, GPT generated 157,314 completion tokens across the runs, compared with 725,047 for GLM. Output volume and token pricing therefore still matter.
In Phoenix, each conversation appears as a trace with one LLM span per model call. Arize points to one Claude call in which 60% of total tokens were classified as cache reads, yet output accounted for 81% of its estimated cost. Span-level views like this let teams check whether caching is working while debugging, rather than inferring it later from aggregate spend.
What it means for teams
Arize advises measuring caching on the conversations an agent actually runs: keep the task, dataset and repeated runs fixed with Harbor, compare cache-read rate, estimated cost and latency in Phoenix experiment charts, and inspect individual LLM spans. It also recommends comparing cache reuse with output volume and task quality before changing models or provider paths.
The company stresses that the results are specific to this workload and these model/provider paths. Its conclusion is that once cache reuse, cost, latency and output volume are visible together, teams can judge which configuration is efficient for their own agent workloads.
