Snorkel's Frontier Data Summit: AI Evaluation Must Become a Science

Four themes from a day of benchmarking talks

According to Snorkel, the event brought together researchers, benchmark builders and practitioners for a full day of talks. The company groups the discussion into four themes.

• Evaluation as an empirical science. Agents are unpredictable, so they need train/test discipline, trajectory analysis and continuously maintained benchmarks.
• Benchmark hardening. Reward hacking, contamination and broken graders are not edge cases, and builders may need a more adversarial mindset.
• Open-ended work. Skill-by-skill tests are saturating, and the next benchmarks must measure insight, creativity and long-horizon work.
• Experts and agents together. Human judgment is most valuable at the spec, the grader and the review, with agents providing scale.

From labels to tasks, environments and rubrics

Snorkel's Alex Ratner opened with the argument that AI data has become a research and technology problem rather than a question of volume and logistics. In his telling, the old unit of data was a crowdsourced label, such as cat or dog. The new unit is a task, an environment and a rubric: a simulated world where an agent attempts something hard, and a grading key that scores it.

As that complexity rises, quality control gets harder. Snorkel says its complex tasks now carry high tens to low hundreds of distinct checks. The surface area for reward hacking grows, and finding the right data distribution becomes its own problem. Ratner quoted the line “What you can’t measure, you can’t improve,” and said that, to his knowledge, capabilities are outpacing our ability to measure them for the first time in AI history. Snorkel's response is to be "benchmark maximalists" and build better tests rather than fewer. The company ties this to an expanded $30 million Open Benchmarks Grants program, red-teaming support and a new research fellowship.

In a conversation with Snorkel chief scientist Fred Sala, François Chollet, creator of Keras and founder of ARC-AGI and the ARC Prize, said benchmarks should target the residual gap between humans and AI and co-evolve with frontier models. He said ARC-AGI 4, aimed at open-ended invention, is targeted for Q1 next year, with ARC-AGI 5 pushing further.

Reward hacking and the limits of public data

Dawn Song of UC Berkeley described a real incident in which an agent working on ExploitGym tasks hacked its way to a better score. Her lesson for every benchmark builder was to build with an adversarial mindset, as if the agent were an attacker. Her group's BenchJack work shows many benchmarks are not resilient to reward hacking. Ratner linked this to training: if an environment rewards cheating, models are effectively being trained to be misaligned. Snorkel says that is why it is forming a benchmark red team alongside the grants program.

A panel on long-horizon agents offered three practical themes. One legal AI team said giving domain experts coding agents to author synthetic environments raised environment-generation efficiency by roughly an order of magnitude, with experts still reviewing and remediating. Panelists favored outcome rewards over process rewards, and treated reward hacking as inevitable, suggesting chain-of-thought monitoring and comparing trajectories across models.

David Hall of the Marin project at Open Athena, a nonprofit effort to build a frontier-level open model and publish the whole process, said evals change over a model's life. Early in pre-training the team watches held-out loss, and agentic benchmarks only become informative after mid-training and SFT, where “getting nonzero is pretty good.” He called contamination basically impossible to avoid, and said the team had to throw out 90 to 98 percent of some public RL datasets because of broken graders or unexecutable tasks.

New benchmarks: science, law and code

Steven Dillmann of Stanford presented Terminal-Bench-Science, which evaluates agents on research workflows in the life, physical, Earth, mathematical and engineering sciences. The first release drew over 900 proposals and 386 implementations, and only 70 tasks, about 8 percent of proposals, made it in. At release the strongest model passed about 30 percent of tasks, and 13 of 17 tested models scored under 25 percent. Release 0.2, with around 100 tasks, is due toward the end of October, with 1.0 by year-end.

Russell Yang of Microsoft presented JudgmentBench, which compares comparative judgments (which of two outputs is better) with rubric scoring. More than 50 expert lawyers contributed 3,000 annotations on 30 tasks from Harvey's BigLawBench. Comparative judgments recovered the quality ordering better and took less than half the time, and the advantage held for junior associates, senior associates and partners. The tasks and annotations are public.

Gabe Orlanski also introduced SlopCodeBench, arguing that coding benchmarks built on a single issue and pull request miss how software is really written, through iteration. It starts an agent on a simple spec, a Python code search engine.

What it means for builders and users

The recurring message is that a single score is not enough. Alex Shaw of the Laude Institute and Sanmi Koyejo of Stanford argued that a resolution rate shows how much can be automated, while trajectories explain why the rest cannot. Shaw proposed three tiers of openness for evals: private for internal development, shared with chosen partners, or fully public to set targets and back claims.

For teams building or buying agents, the summit points to evaluations that are maintained over time, hardened against cheating and reviewed by human experts. Two of the benchmarks presented, Terminal-Bench-Science and JudgmentBench, received support through Snorkel's Open Benchmarks Grants. Terminal-Bench-Science is also looking for scientists to contribute.