Snorkel AI Raises Open Benchmarks Grants Tenfold to a $30M Commitment

What the expanded program includes

According to Snorkel, the expansion has three parts. The first is more funding for the next wave of open benchmarks and measurement methods, covering safety and alignment, cybersecurity, physical AI, long-running agent work, open-ended outputs, dynamic settings and human uplift.

The second is the Open Benchmarks Red Team, a new program meant to help benchmark builders continuously find and fix weaknesses. Snorkel lists reward-hacking exploits, contamination risks, broken verifiers, and distributional and diversity gaps among the targets. The third is the Snorkel Research Fellowship, which supports independent evaluation research with access to Snorkel's research team, domain experts, compute and engineering support.

The grants program began with an initial $3M commitment. Snorkel says that through it and its wider research collaborations it has partnered with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents' Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench and others. The company says it helped design evaluation methodologies, develop task construction pipelines and scale quality control. It also states that benchmarks funded by the grants have appeared on the latest model cards from every major frontier lab.

Why Snorkel says benchmarks are falling behind

Snorkel argues that benchmarks guide and drive AI development, since setting a metric and then "hillclimbing" against it is central to how AI systems are evaluated and built. When benchmark development lags behind model development, the company says, their value for guiding evaluation and alignment drops too.

It also points to "benchmaxxing". When a set of benchmarks becomes too simple, static and correlated, models can overfit to them, much like a student studying directly for the questions expected on an unchanged test. Snorkel says this reduces the benchmarks' value and distorts incentives in model development. Its answer is more benchmarks that are robust, diverse, continuously updated and independently developed.

The company frames this as a matter of scale: massive investments keep accelerating model development, and it calls for equally ambitious investment in open measurement.

Three axes and the problem of trust

Snorkel describes three axes along which benchmark coverage must grow:

• Environment and input complexity: real operating environments involve specialized knowledge, messy context, multimodal inputs, complex toolsets and coordination with people and other agents.
• Autonomy horizon: benchmarks should test how long an agent can operate before reliability breaks down, including whether it keeps context, recovers from mistakes and adapts to changing goals.
• Output complexity: as agents produce more complex work, evaluation of both final outputs and training reward signals must become more nuanced.

The company also stresses measurement robustness. Contamination, ambiguous tasks, exploitable verifiers and insufficiently diverse or dynamic data can weaken the link between scores and the capabilities they are meant to measure. Snorkel says open benchmarks let the field inspect methods, reproduce findings and strengthen tasks and grading, and that keeping them trustworthy needs ongoing red-teaming and maintenance from the community.

It further says AI research must become a more empirical science, with benchmarks as one of the key types of controlled experiments. As examples of emerging capabilities, it cites OpenAI's 10,000-agent mathematics efforts and the covert coordination METR documented in the Hugging Face incident.

Focus areas, red team and fellowship

Snorkel is extending its open invitation for proposals, especially where stakes are highest and measurement is hardest. Listed areas include safety and alignment, cybersecurity, physical AI, scientific agents, long-running work, open-ended outputs, dynamic settings and human uplift, which the company defines as measuring how AI assistance changes human performance compared with working without it.

The Red Team will use Snorkel's data development and evaluation platform, combining domain expert review with specialized quality control and error analysis systems. Snorkel says findings will go to the benchmark team first, and that it will work with teams to address concrete issues.

Fellows in the research program will set their own research questions and lead work from design through public release, working with a Snorkel research co-advisor and receiving funded access to domain experts, compute and research and engineering support.

What it means for the field

For independent benchmark builders, including academics, domain experts and open-source teams, the expansion means a larger pool of funding and, according to Snorkel, practical help with quality control and adversarial testing. For the wider field, the company's position is that evaluation needs to keep pace with frontier models, and that a broader ecosystem of contributors is needed to build and maintain the measurement tools. Snorkel invites anyone with a capability worth measuring, or a better way to measure it, to propose a project.