METR Says Anthropic May Have Crossed an AI R&D Warning Threshold

METR challenges Anthropic’s threshold assessment

In a review published on November 5, METR examined the “Risks from automated R&D” section of Anthropic’s public August 2026 Risk Report. The review formed part of a high-access assessment of AI R&D acceleration, for which Anthropic shared a private version of its report and other non-public data.

METR broadly agrees with Anthropic’s conclusion that internally deployed models posed a low risk of directly automating research and development in key domains or causing acute catastrophic risks from accelerated capability progress by July 15. But it argues that Anthropic’s claim that its early warning thresholds had not been crossed depends on judgment calls that the public report does not communicate clearly.

Researchers at both METR and Anthropic, according to METR, lean toward the threshold not having been crossed. METR nevertheless considers it plausible that it had been crossed and says the public report understates the level of acceleration taking place.

Anthropic’s two automated R&D thresholds

METR says Anthropic’s Risk Management Policy, or RSP v3.4, sets two capability conditions for “Automated R&D in key domains.” The first is a Full Substitution Threshold. This would be triggered if Anthropic developed a model able to “fully substitute for our entire set of Research Scientists and Research Engineers, at competitive costs (i.e., within a factor of 5).”

Anthropic’s report argues that this threshold had not been crossed because models were not substituting for all of its research staff. The company says researchers still complete work at Anthropic even though it has strong incentives to replace as much research work with models as possible. The report states: “we believe it would be clear to us if models were capable of substituting, even at high cost, for the technical work of our entire research staff.”

The second condition is a 2x Acceleration Threshold, defined as “dramatic acceleration” in AI progress that is likely connected to automating AI R&D. Anthropic reported meaningful acceleration beginning in early-to-mid 2025, but said it was less than a factor of 2. It also said it was fairly confident that factors other than AI drove the acceleration in 2025, while AI models helped faster trends continue through the report’s coverage date.

Why METR sees uncertainty around AI-driven acceleration

METR finds Anthropic’s argument against crossing the Full Substitution Threshold broadly compelling. It describes that threshold as a high bar and agrees Anthropic would probably have noticed greater progress in automating its research if models as of July 15 were capable of meeting it.

At the same time, METR has less confidence than in the past because high-confidence, unsaturated benchmarks showing important capability limits are lacking. METR says this absence raises concern that alignment, tooling, oversight or elicitation problems could be holding back automation even if models had obtained, or were close to obtaining, the necessary capabilities.

METR considers Anthropic’s case that the 2x Acceleration Threshold had not been crossed to be overstated. Anthropic describes that threshold as a potential early warning rather than evidence that catastrophic risks have already materialized. Its report rated the overall risk as “low,” while acknowledging that task-based evaluations had “saturated” and that there were early signs of possible acceleration.

For users and the wider AI market, the review signals a disagreement about how to interpret progress in AI research automation. METR does not say the models had already created the catastrophic conditions in Anthropic’s threat model. Instead, it argues that the evidence may have been sufficient to make crossing an early warning threshold plausible, even while the direct risk remained low.