Ai2 study finds LLM judges of research novelty are unreliable

Why novelty judges matter

The study, titled Old Ideas, Novel Problems, looks at what Ai2 calls novelty judges: LLMs that assess whether a research idea is new relative to prior work, either on its own or in comparison with another idea. Ai2 says it is publishing the paper alongside a benchmark, data and code for testing such judges, plus an automated pipeline that researchers can use to build new evaluation sets.

The work follows Ai2's own efforts to build AI systems for scientific discovery. The institute points to DataVoyager, which helps researchers explore and analyze structured data, AutoDiscovery, which generates and tests hypotheses, and Theorizer, which synthesizes evidence across large bodies of literature into candidate scientific theories. Generating ideas is only part of the task, Ai2 argues, because the ideas must also contribute something genuinely new.

There is no standard way to build a novelty judge. According to Ai2, teams choose a model, prompt, evaluation format, reasoning budget and sometimes a literature-retrieval system, and those choices can materially affect the verdict. The institute also notes that judges are often validated only on human-written papers rather than the AI-generated ideas they are meant to evaluate.

How the benchmark was built

Ai2 built an automated pipeline around ICLR 2026 peer reviews. Instead of treating paper acceptance as a proxy for novelty, the team mined the reviews for passages where reviewers explicitly discussed a contribution's originality. A paper can be accepted without being particularly novel if it is strong elsewhere, and a rejected paper can still be original.

• The high-novelty pool has 154 human-authored ideas from accepted papers in the top-rated 10% of their ICLR research area, where reviewers positively identified novelty and none disputed it.
• The lower-novelty pool has 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none positively identified novelty.

In a check of the review-mining process, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases, and none of the disagreements would have changed how a reviewer was counted.

These pools form the Human-Only setting. A second setting, Human+Generated, pairs the same 154 validated human ideas with 154 ideas from a simple LLM generator that received only a research area and an instruction to propose a novel idea. Ai2 says the generated ideas are assumed to be less novel, and describes these as weak labels. An expert who reviewed 30 of the pairs preferred the human idea in all 30 and judged none of the generated ideas novel.

Small changes, large swings

Ai2 tested six judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. It varied one factor at a time, including the prompt, reasoning budget, access to related papers, idea formatting, the number of judgments combined and the generator model, using both pointwise ratings and pairwise comparisons.

Prompting caused the largest shifts. GPT-5.4 saw Human+Generated pairs with two prompts. One said reviewers had found exactly one idea in each pair to be novel and asked which; the other simply asked which idea was novel. With the simpler prompt, accuracy fell from 93.5% to 40.9%, which is below chance, and the model changed its verdict on more than half of the pairs.

On the Human-Only dataset the same change moved GPT-5.4's accuracy by only about seven percentage points, though it moved Claude Opus 4.6 by 21 points. Ai2 concludes that a judge can look stable on human-authored research while behaving very differently on AI-generated ideas.

No prompt was universally better. Dropping an instruction that warned against treating trivial or incremental contributions as novel helped some judges and hurt others. When Ai2 compared ideas from two AI generators (Claude Opus 4.5 against Claude Sonnet 4.5, and GPT-5.4 against GPT-5.1), roughly a quarter to a third of verdicts changed on average across the six judges under the tested prompt changes.

More search and reasoning did not fix it

Ai2 also tested whether heavier machinery helps. It gave GPT-5.6 Sol an extra-high reasoning budget and live, iterative search restricted to arXiv, and compared it with judges that each saw five pre-retrieved abstracts. On Human+Generated, the agentic judge scored 0.78 macro-F1. That is 11 points behind Claude Opus 4.6 at 0.89 and in roughly the same range as simpler Claude Opus 4.5 and GPT-5.4 setups, while costing roughly 10× more per decision at current API prices.

Ai2 says retrieval and additional reasoning do not consistently improve judgments, and that two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.

What it means

Ai2 says its results suggest "novelty evaluation itself is far from reliable." For teams that use an LLM to rank or filter AI-generated research ideas, the findings imply that the choice of judge prompt can change the outcome, including comparisons between ideation systems. Ai2 notes that many of its core findings do not depend on the labels, since two reasonable setups giving opposite answers for the same pair of ideas already show instability.

The released benchmark, data, code and pipeline are meant to let others study the problem. Ai2 describes them as tools for studying it, while showing how much remains unresolved.