METR Outlines How Assessors Could Test for AI-Driven Acceleration in AI R&D

Four questions an assessment would answer

According to METR, an assessment of this kind would address four areas. The first is capabilities progress: how fast internal capabilities are improving and how much that progress has accelerated recently. The second is acceleration from AI: if there is an acceleration, how much was driven by AI automation versus other factors.

The third area is bottlenecks to future growth, meaning whether constraints could slow growth or limit acceleration in the near term. The fourth is limitations: whether the methods used would reliably identify a much more dramatic acceleration if it had occurred.

METR says the work would involve analyzing the capabilities of unreleased models, understanding the growth of key AI R&D inputs (labor, compute, data, and algorithms), and talking openly with relevant researchers about how AI is affecting their work.

How progress and AI's role would be measured

METR argues that AI capabilities cannot be understood through any single approach, so the assessor should combine several methods. These include benchmarks and aggregate measures such as ECI, intermediate indicators such as pretraining loss, and researchers' own perceptions gathered through interviews or surveys. Other proxies could be added where they matter.

To judge whether AI is behind any acceleration, the assessor would study the inputs most likely to be affected by automation. One question is how much of the capabilities acceleration comes from algorithmic progress or data, versus inputs such as scaling compute that are less affected by AI automation. A second is how much of the acceleration in algorithmic progress or data is attributable to AI assistance or automation, versus inputs like experiment compute and labor.

METR acknowledges that determining a "No AI" counterfactual is challenging. It expects that a well-calibrated estimate will require several lines of evidence, including spend on AI agents (ideally by team and over time, in tokens and inference cost), researchers' qualitative accounts of how progress is made with models, and labor productivity estimates such as lines of code per researcher-day or self-reported uplift.

Data and access the assessor would need

The assessor would require measurements of inputs and outputs across the model development pipeline. METR says data should cover the maximum possible timeframe, carry precise dates so changes can be compared with a historical baseline, and be broken down by team or domain where relevant.

The necessary access listed in the article includes:

• Capability measures across internal models, including benchmarks and key intermediate measures
• Algorithmic progress estimates in pretraining, posttraining and other key domains
• Measurements of key R&D inputs, such as training compute for each model, experimental compute and labor over time
• Interviews with researchers about progress and AI's impact on their work

For benchmarks, METR asks for scores on all measured benchmarks across all frontier models, including internal-only ones, along with example questions, scoring details and sample model answers. On algorithmic progress, it expects developers already keep some estimates, possibly similar to a published method that maps research advances to a reduction in the compute needed to reach certain performance levels. The assessor does not need details of the specific research advances, only their estimated effects over time and between models.

On security, METR says data could be analyzed on developer hardware, possibly on site at developer premises, and that only minimal access to developer infrastructure is needed.

What it means

The proposal is a methodological outline rather than a report of findings. It indicates what a credible outside check on claims about AI-accelerated AI research would require from developers: detailed, dated internal data and open conversations with staff. It also builds in a test of its own reliability, asking whether the methods would have caught a much greater acceleration, and how internal evidence compares with public evidence. That is aimed at helping the public judge how confident it can be that a dramatic acceleration would be noticed and communicated.