Artificial intelligence news.
Straight from the original announcements.

GitHub Launches ReviewBench, an Open Benchmark for AI Code Review Agents

October 5, 2026, 17:59By the AI News Bites Editorial TeamSource: GitHub
GitHub Launches ReviewBench, an Open Benchmark for AI Code Review Agents
Image: GitHub

GitHub has released ReviewBench, an open offline benchmark for comparing AI code review agents. According to GitHub, it is modeled on more than 100 million real pull requests and is available through a research preview.

Key points
  • GitHub released ReviewBench, an open offline benchmark for AI code review agents, as a research preview.
  • It contains 219 pull requests from 187 public repositories across 19 languages, modeled on 103.9 million GitHub pull requests.
  • The golden set combines human reviewers, follow-up commits, static analysis and frontier LLMs, graded by Claude Sonnet 5 under a published rubric.
  • Senior engineers' labels agreed with ReviewBench 96.6% of the time.
  • Results can be sliced by severity and category, and the Fβ weighting can be adjusted to re-rank the leaderboard.

What ReviewBench is built on

GitHub says the quality of existing AI reviewers is hard to measure, and that teams need to know a reviewer's strengths before relying on it. Some reviewers surface more issues, some produce less noise, and some are better at critical problems while others also flag smaller improvements. According to GitHub, existing benchmarks often trade off label quality, coverage and how well they represent real-world code review.

To address that gap, GitHub analyzed 103.9 million pull requests to characterize real review workloads. ReviewBench contains 219 pull requests from 187 public open source licensed repositories, spanning 19 languages. Its language and repository-size distributions closely match GitHub overall, and the complete dataset is publicly available.

GitHub made one deliberate adjustment: pull request size is weighted toward the reviewable middle and tail. This reduces the overrepresentation of tiny, single-file changes and keeps more substantive, multi-file pull requests where review quality matters most.

How the golden set and scoring work

GitHub argues that no single reviewer, human or model, can find everything worth flagging in a pull request. The golden set is therefore built in three stages:

  • Gather candidates. Findings come from human reviewers, issues inferred from author follow-up commits, deterministic analysis tools and multiple frontier LLMs across model families.
  • Deduplicate. Overlapping findings that point to the same underlying issue are merged semantically, so agreement between sources does not inflate the set.
  • Validate. A finding counts as a true positive only if it is true, relevant and non-trivial, regardless of its source. GitHub uses Claude Sonnet 5 as the LLM grader and publishes both the rubric and the judge.

Every finding is labeled for severity (critical, medium, low) and category, such as correctness, security, reliability, maintainability and testing.

The benchmark reports metrics in two families. Grounded precision, recall and F1 use only the existing gold-set labels, giving a strict like-for-like comparison. Augmented precision, recall and F1 also assess findings that match nothing in the golden set, with the judge deciding whether they are true or false positives. This lets a reviewer earn credit for valid issues no source in the golden set found. Because augmented recall expands its denominator based on what each agent discovers, GitHub uses grounded recall as the headline cross-system comparison and augmented metrics as a per-system diagnostic.

Configurable rankings and validation

GitHub says there is no single best review experience. Some developers want only critical issues, others value lower-severity findings, and some prioritize broad coverage or minimal noise. Results can be sliced by severity and category, and users can adjust β in the Fβ score to weight recall or precision more heavily. The leaderboard is re-ranked as these preferences change.

Before release, senior engineers who had not helped build the dataset re-labeled every ground-truth finding from scratch. Their true/false-positive judgments agreed with ReviewBench 96.6% of the time, according to GitHub. The company versions the dataset, judge and matcher used in each evaluation, and publishes its validation methodology, agreement measurements and known threats to validity.

What it means for users and teams

GitHub says ReviewBench has made its own offline evaluation of Copilot code review more effective at anticipating the direction of production experiments, which gives it greater confidence that measured improvements reflect meaningful gains for users. The company also says improvements and regressions tend to show up online as well.

The research preview is available on the ReviewBench website, where users can explore the benchmark, compare code review agents and bring their own agent to evaluate and iterate. For teams choosing or building review tools, the public dataset and published rubric are meant to make results inspectable and reproducible.

Related news

Google DeepMind introduces Gemini 4 Hydrogen for faster agentic workTop story
ModelsNovember 12, 2026, 19:00

Google DeepMind introduces Gemini 4 Hydrogen for faster agentic work

Google DeepMind has introduced Gemini 4 Hydrogen, a model designed for rapid execution across everyday coding, agentic workflows and consumer tasks. The company says…

OpenRouter measures the token cost of MCP, CLI tools and skills
For developersNovember 11, 2026

OpenRouter measures the token cost of MCP, CLI tools and skills

OpenRouter’s comparison of MCP, command-line tools and skills found that the way an AI agent receives a capability can dramatically change its context usage and cost. In…

GPT-6 Astra led coding agents, but finished only 26.4% of tasks
ResearchNovember 10, 2026, 17:08

GPT-6 Astra led coding agents, but finished only 26.4% of tasks

OpenAI’s GPT-6 Astra passed an average of 69.9% of hidden acceptance criteria in a benchmark of coding agents, yet fully finished only 26.4% of tasks. The Co-Design…