AI-generated imageFor eleven weeks, from August 12 to October 29, researchers from the evaluation nonprofit METR examined how fast Anthropic is improving its AI models and how much of that pace comes from the AI itself. Their report, published November 5, concludes that Anthropic is likely improving its models on well-specified tasks at around twice the rate it managed between March 2024 and October 2025. METR adds a second judgment and labels it a guess: perhaps 80 percent of the current acceleration is due to AI.
The report is just as direct about what readers do not get. Under the terms of the engagement, METR writes, Anthropic could redact any non-public information in the post. METR was free to state its top-level conclusions about risk, but Anthropic “redacted a majority of the evidence we found most informative” for them. The company also canceled a component that would have examined how it allocates computing power; METR relays Anthropic’s explanation that it wanted to keep the work focused on risk assessment for now. The redactions themselves, Anthropic said, were meant to remove confidential material that could speed up other AI developers.
What METR could see was wide. It describes the access as unprecedented: internal benchmark results, estimates of algorithmic progress, headcount, training compute, staff surveys, code metrics and about fifteen interviews, covering models that have never been released. It could not query those unreleased models directly.
The result, then, is a report in two layers: conclusions anyone can read, resting partly on evidence that the evaluator judged most telling and that the public cannot inspect. METR’s choice to say plainly that most of that evidence was removed is what makes the second layer visible at all.
The numbers come hedged. “Twice the rate” describes how quickly measured capability is rising on well-specified tasks, mainly on AECI, Anthropic’s version of the Epoch Capabilities Index. METR puts the slope at 1.9 times its earlier pace, with a 95 percent confidence interval of 1.5 to 2.8, and says robustness checks range from 1.1 to 2.9 depending on method. It measures neither researchers’ productivity nor how many jobs the models could do. METR is confident about one narrower point—that the speedup did not come from faster growth in training compute—and much less sure about the next year, writing that its views on key questions went through multiple reversals.
The doubling will sound familiar to readers of this publication. Three days ago we reported that Anthropic’s amended IPO filing says future models may “imminently meet” the line its Responsible Scaling Policy draws for AI-driven acceleration, a line defined in part as progress at twice a baseline rate. METR’s estimate sits near that figure, but it is not a ruling on the line. METR says it paid no special attention to the policy’s thresholds, its index is not the policy’s yardstick, and the policy leaves the designation to the company. METR also notes that Anthropic’s Haiku 5.5 system card put acceleration at 1.7 times, and attributes its own higher figure mainly to including newer internal-only models.
Anthropic has framed outside evaluation with an analogy of its own. On October 15, announcing that six METR staff had been given company hardware, office space and access to internal tools, it wrote that bank supervision “operates on similar logic,” while granting that the comparison is not exact. Under that program, the company said, it cannot veto publication and may require redactions only for information in predefined categories. METR says the engagement it has just reported began in mid-August and is distinct from the one announced in October. For this report, the redaction right it describes is broader: any non-public information.
The analogy is useful for what it shows about who draws the line. In U.S. bank supervision, the Federal Reserve’s rules classify reports of examination as confidential supervisory information, a category the regulator defines in its own regulation. The boundary is set by the supervisor, in advance. For this report, the boundary was set by the company being examined.
That matters because few others can check the question METR was asked. Only a frontier lab holds the internal data that would show whether AI is speeding up its own research; that is why outsiders are sent in. When the evidence those outsiders found most telling stays unpublished, the public is left largely with METR’s judgment and METR’s standing—along with the fact, which METR disclosed, that one team member had held job discussions with Anthropic and equity in a competitor. METR also reports that most of its analysis relied on Claude, Anthropic’s own model, and that this could have colored how its staff read the data. From the published version, no one outside can tell whether it did.
None of this makes the conclusions wrong. METR reported its reversals and limits with unusual candor, and Anthropic gave an outside group access that METR calls unprecedented. But the case for the headline judgment now rests in large part on material only two parties have seen. Two things would change that: a record of what was redacted and on what grounds—by category and amount, if not by content—and redaction rules for future reports that are fixed before the evaluator begins and published alongside the findings. Until then, an outside measurement of whether AI is accelerating its own development comes with its most informative evidence held by the company it measured.
Sources
Links in the text lead to the original material the journalist used.
- report metr.org
- amended IPO filing sec.gov
- wrote anthropic.com
- confidential supervisory information law.cornell.edu
Revision history
No revisions.
Discussion
Questions, counter-arguments and sourced corrections are welcome. Comment policy
Loading comments…