MLCommons to Build MLPerf Agents Benchmark With $3.5 Million NSF-Backed Award

What MLPerf Agents will measure

According to MLCommons, MLPerf Agents will track efficiency from a delegated goal all the way to a verified outcome. The program is supported by the U.S. National Science Foundation Directorate for Technology, Innovation and Partnerships (NSF TIP).

The nonprofit consortium, which is made up of companies, academics and nonprofits, says existing benchmarks for the time, cost and energy of inference remain important but need to evolve. Its argument is that an agent differs fundamentally from a single model answering a single prompt. An agent works across many turns, calls tools, retrieves information, carries growing context, recovers from failures and checks its own work.

MLCommons notes that its MLPerf Inference working group is already developing agentic inference benchmarks that reflect the demands agents place on model serving: multiple turns, tool calls, growing context and cache reuse. MLPerf Agents is meant to go further. It expands measurement to the complete workflow, including the agent harness, model selection, agentic inference, retrieval, tool execution, orchestration, state management, failure recovery and validation. According to MLCommons, this wider scope makes it possible to study how both harnesses and models affect end-to-end efficiency, not only how fast a model serves tokens.

Four planned benefits

MLCommons lists four things MLPerf Agents is intended to deliver:

• Efficiency tied to success. Cost, energy, latency and throughput are measured for successfully completed tasks. Success is verified with techniques influenced by the MLCommons AI Risk and Responsibility working groups.
• Whole-stack scope. Resources are counted across the entire agent stack, showing where inefficiencies arise.
• Comparable results across architectures. Common workloads, tracing, measurement rules and validation let buyers compare complete solutions.
• Trustworthy verification. Hidden validation datasets and MLPerf's review and audit process reduce benchmark gaming.

Initial workloads are planned to cover software engineering, research and data analysis, and customer service.

Who is involved

MLPerf Agents will build on eight years of MLPerf benchmarks, including MLPerf Inference and MLPerf Power. It will follow the same open, collaborative process, with workloads, rules and reference implementations developed by the MLCommons community and results subject to peer review.

David Kanter, MLCommons founder and head of MLPerf, and Professor Vijay Janapa Reddi, co-founder and VP of MLCommons, will lead the work with the community. Support comes from Qualcomm, NVIDIA, Intel, AT&T, Workato, Gimlet Labs and others, according to the announcement.

The AI Efficiency Challenge is run by STRIDE Ventures, a national platform for breakthrough innovation launched by the U.S. National Science Foundation and Start2 Group. Start2 Group administers the challenge under an Other Transaction Agreement with NSF.

What it means for buyers and developers

MLCommons says the goal is to give buyers evidence for choosing efficient agentic systems, and to help developers find the improvements that cut cost and energy as agentic AI scales. Reddi, MLCommons board member and vice president of research, said: "As agentic AI scales, we are just beginning to understand how to optimize across the harness, model calls, tool invocation, retries and more to deliver the best cost and energy efficiency."

For organizations weighing agent products, the planned approach would allow comparison of complete solutions on common terms rather than by model speed alone. The source does not give a release date or a timeline for first results beyond the two-year award period.