Everpure benchmark finds frontier models overused for coding tasks

A benchmark built around company code

Everpure says public model leaderboards do not show how well a coding agent will perform on a particular company’s codebase. Its approach is to build a benchmark from code history, using real issues, fixes and tests created by engineers.

The company created an internal version for its private codebase and evaluates new models as they ship. It has now released a public version based on the open-source Zulip team chat application. The repository includes the tasks, harness configurations, grading and raw results, allowing organizations to copy the setup and point it at their own code.

In this context, a harness is the software that drives a model by reading files, running commands and deciding what to try next. Everpure tested 21 model and harness combinations on 107 shared tasks. Each combination made one attempt per task, with no retries or human guidance.

Large cost differences between models

Claude Opus 5 running in the Claude Code harness resolved 94.4% of the tasks, according to Everpure, at a cost of $3.68 per task. GPT-5.6-terra running in Codex resolved 85.0% at $0.54 per task. Gemma-4-31B, an open-weight model hosted on the company’s own hardware and run through the open-source pi harness, resolved 56.1% at $0.044 per task.

Everpure describes that as an 84X price difference between the most expensive and least expensive examples, alongside a 38-point difference in resolve rate. Compared with Claude Opus 5 in Claude Code, the company calculates that GPT-5.6-terra costs 85% less per task with a resolve rate 9.3 percentage points lower. Gemma-4-31B costs 99% less, with a resolve rate 38.3 points lower.

The reported costs for self-hosted models use the public per-token rate a hosting provider would charge to serve the model. They represent the cost of renting the model, rather than the cost of owning GPUs, supplying power or staffing a cluster.

Routing routine work away from frontier models

Everpure says it moved power users from using Claude exclusively to an on-premises GPU cluster. Its system routes routine work to hosted models and sends more demanding tasks to cloud frontier models, which the company defines as the largest and most expensive class of model rented per use. It also selects the cheapest frontier model that fits a task when the cluster is full.

The company says this shift recouped its GPU investment in less than a quarter, while cautioning that the result reflects its own mix of work and may differ for other organizations. Its benchmark is intended to identify which tasks cheaper models can handle and which still require a frontier model.

Everpure says the Zulip sample is small, so differences of one or two tasks can be noise. Its ranking treats combinations within 2 percentage points of one another as practically equal, with the cheaper option taking precedence. For users, the source’s central recommendation is to measure performance on their own task history before setting a routing policy, rather than sending every coding request to the model at the top of a public leaderboard.