Artificial intelligence news.
Straight from the original announcements.

OpenRouter outlines a practical method for benchmarking LLMs

November 12, 2026By the AI News Bites Editorial TeamSource: OpenRouter
OpenRouter outlines a practical method for benchmarking LLMs
Image: OpenRouter

OpenRouter has published a guide to benchmarking large language models with application-specific tasks, code-based grading and repeatable runs. Its example study found GPT-6.1 Sol passed 180/180 trials, while DeepSeek recorded 157/180 passes at the lowest response cost.

Key points
  • OpenRouter recommends application-specific tasks with script-checkable answers.
  • Graders should test correct, incorrect and malformed model responses.
  • GPT-6.1 Sol passed 180/180 trials on OpenRouter’s 60 synthetic tasks.
  • DeepSeek had the lowest response cost and passed 157/180 trials.
  • The public ranking showed weak agreement with the study, with a 0.109 Spearman correlation.

Build tests around the application

OpenRouter recommends choosing models using evidence from the inputs and output formats used by an application, rather than relying only on public rankings. Each benchmark task should define an input, an expected answer and an output contract that a script can check.

The guide describes tasks covering invoice arithmetic, dates and schedules, email extraction, Python functions, output constraints and refund decisions. For the study described by OpenRouter, the team wrote 40 baseline tasks and 20 hard tasks before the final run. The examples include requirements such as returning a number with exactly two decimal places, producing a JSON object with specified keys and values, or returning only APPROVE or DENY.

OpenRouter also advises computing references independently and keeping prompt-tuning cases separate from final evaluation. In its refund examples, the benchmark tests policy boundaries, including exactly $200 and 30 days, as well as values just above each threshold.

Code-based grading needs its own checks

The guide recommends exact matching for fixed outputs and schema or value checks for JSON. Graders should normalize only permitted differences. Accepting prose around a number, OpenRouter says, can conceal a failure when the application requires only a number.

Graders should be tested against known-correct, wrong and malformed responses. OpenRouter reports that a development checker initially rejected valid Python because it omitted ord and type. The team corrected the issue before the final study and specified CPython 3.12.10/WASI with resource limits. The full-study runner isolates generated code and stops when sandbox setup is missing, while the teaching example executes no generated code.

These safeguards matter because a faulty checker can turn infrastructure or grading errors into apparent model failures. The guide says its offline fixtures cover boundary decisions, invalid formats and truncated answers without calling a model.

The example supports controlled comparisons

OpenRouter provides a companion ZIP with a standard-library teaching runner, a JSON example and comparison tools. Users are told to compare the archive’s SHA-256 checksum with a38ee17a2eb7090f9f4849a605cc8214691c889ca0ae40c121259603592178e8 before extracting it. The example requires Python 3.12 or later and its offline checks passed on Windows x64 and Linux. macOS remains untested, while the bundled full-study sandbox and native Ori setup support Windows x64 only.

The three-task example checks an invoice total, UTC conversion and a refund decision. Its default check performs zero inference calls. Live inference requires an explicit live option and an allowance, helping keep validation separate from spending.

In the benchmark cited by OpenRouter, GPT-6.1 Sol passed 180/180 trials across three repeats on 60 synthetic tasks. DeepSeek had the lowest response cost and passed 157/180 trials. Agreement with the public ranking was weak, with a Spearman correlation of 0.109. For users, the message is to compare identical tasks using graded accuracy, billed cost and request time, while checking run counts, missing answers and verified bills before trusting a score.

Source
OpenRouter: How to Benchmark LLMs

Related news

JetBrains asks 12 AI models for 1,200 programming hot takes
ResearchNovember 15, 2026, 18:02

JetBrains asks 12 AI models for 1,200 programming hot takes

Twelve AI models overwhelmingly attacked Python, Rust and TypeScript in a JetBrains experiment that collected 1,200 opinions about programming languages. The models also…

OpenRouter finds no clear quality gap between fp8, fp4 and fp16
ResearchNovember 11, 2026

OpenRouter finds no clear quality gap between fp8, fp4 and fp16

OpenRouter found no detected accuracy difference between fp8, fp4 and fp16 in a test of Qwen3.8 27B, although the company cautions that the result covers one model and…

Claude Opus 5.5 Leads OpenRouter’s Scanned Document OCR Test
ResearchNovember 10, 2026

Claude Opus 5.5 Leads OpenRouter’s Scanned Document OCR Test

Claude Opus 5.5 was the most accurate model in OpenRouter’s test of OCR for scanned documents, correctly identifying 262 of 264 key values. Mistral OCR delivered the…