Can public trust AI safety when redactions hide the most informative evidence?
The evaluation group says redactions force readers to trust their judgment more than they consider ideal for transparent AI safety assessment.
Key points
- METR submitted a roughly 100-page full assessment report to Anthropic.
- Anthropic redacted a majority of the evidence METR found most informative.
- The public version forces reliance on METR's judgment rather than raw data.
Techmeme reported that the Machine Evaluation Team (METR) stated Anthropic redacted significant portions of a safety assessment.reported The evaluation group submitted a comprehensive document to the AI developer, but the public release lacks much of the underlying data.An AI safety group says the company hid key details from their public report.METR reports that Anthropic's redactions of high-signal evidence degrade the verifiability of their safety assessment.
Evidence: reported
METR: Anthropic redacted much of the "most informative" evidence from METR's 100-page report, so readers "must rely on our judgment more than we consider ideal" (@metr_evals)aifleet-techmeme
The announcement
METR disclosed that Anthropic removed a majority of the evidence deemed most informative by the evaluators. This redaction applies to the public version of the report. The team noted that this limits what readers can independently verify.
The context
The original submission to Anthropic was approximately 100 pages long.reported METR provided this full report as part of their safety assessment process. The public release differs significantly from this internal document due to the extensive redactions applied by Anthropic before publication.
Evidence: reported
METR: Anthropic redacted much of the "most informative" evidence from METR's 100-page report, so readers "must rely on our judgment more than we consider ideal" (@metr_evals)aifleet-techmeme
Reading between the lines
METR explicitly stated that readers must now rely on the team's judgment more than they consider ideal. This suggests a gap between the internal assessment and the public record. The group implies that transparency is reduced when key evidence is withheld from the final published document.
The business case
For AI platform architects, vendor choice often depends on verifiable safety claims. When a vendor like Anthropic redacts the most informative evidence from a third-party assessment, it complicates due diligence. Architects cannot independently validate the specific risks or mitigations cited in the public report. This forces a reliance on the evaluator's summary rather than raw data, which may affect trust in the vendor's transparency practices. The cost here is not financial but informational; less data means higher uncertainty in risk modeling.
Why it matters
- Architects cannot independently verify the specific safety findings due to redacted evidence.
- Trust in the assessment relies on METR's judgment rather than accessible data points.
- Vendor transparency is questioned when key informative evidence is removed from public reports.
- Due diligence processes are hampered by the lack of access to the original 100-page findings.
Topics: AI safety, METR, Anthropic, transparency, vendor assessment
Updated
Then / now / next
METR submitted a full 100-page assessment report to Anthropic.
The public version has had most informative evidence redacted by Anthropic.
It is unclear if unredacted versions will be shared with other stakeholders.
Story timeline
- Then METR submitted a full 100-page assessment report to Anthropic.
- 05 Nov 2026 METR: Anthropic redacted much of the "most informative" evidence from METR's 100-page report, so readers "must rely on our judgment more than we consider ideal" (@metr_evals)
- Now The public version has had most informative evidence redacted by Anthropic.
- Next It is unclear if unredacted versions will be shared with other stakeholders.
The strongest doubt
Counterpoint from the sources
The source material presents METR's perspective without including any counter-argument or justification from Anthropic regarding the redactions.
For readers
Researchers
- Can examine how redactions affect the reproducibility of safety evaluations.
- May need to request direct access to unredacted data for thorough analysis.
Business
- Must weigh vendor transparency as a factor in procurement decisions.
- Should assess the risk of relying on summarized judgments over raw data.
Policy
- Can observe how industry standards for safety reporting are applied in practice.
- May consider regulations requiring full disclosure of safety assessment evidence.
Ask the article
Local keyword search of this article; no generated answers.Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Terms in this story
METR
Machine Evaluation Team, a group that assesses AI safety.
Redaction
The removal of sensitive or specific information from a document.
Assessment
A formal evaluation of safety risks or performance.
Questions readers ask
Why did Anthropic redact the METR report?
The source material does not state Anthropic's reason for the redactions, only that they occurred.
How long was the original report?
METR submitted a report that was approximately 100 pages in length.
What is the impact on readers?
Readers must rely on METR's judgment more than the team considers ideal for transparency.
Connected stories
Reading receipt
- METR submitted a roughly 100-page full assessment report to Anthropic.
- Anthropic redacted a majority of the evidence METR found most informative.
- The public version forces reliance on METR's judgment rather than raw data.
Still open: What specific evidence was redacted from the report?
Still open: Will Anthropic provide unredacted access to other parties?