Start free trial

TRUE, BUTCherry-Picked Slice

Only problems OpenAI's model failed

The '59.4% of SWE-bench is broken' stat comes from an audit that only examined the problems OpenAI's own model kept failing.

“OpenAI retired SWE-bench Verified after its audit found 59.4% of tasks had flawed test cases, and every frontier model trained on the solutions”

Eval roundups and tech aggregators compressing OpenAI's February 2026 retirement post from spring onward (byteiota and others)

Recorded 14 August 2026. Our ruling means: literally true, materially misleading.

Read the story →

Receipts pack

The evidence file for this verdict

3 receipts: 1 support the claim, 1 contradict it, 1 add context. 3 figures on the record, 1 traced to a quoted receipt.

The receipts

Each source, the line we quote from it, its date and which way it cuts. Free to read and to cite.

  1. 1Supports the claimweb.archive.org ↗

    First, 59.4% of audited problems contain flawed test cases that reject correct solutions.

    Look up an archived copy ↗

  2. 2Contradicts the claimweb.archive.org ↗

    We conducted an audit of 138 SWE-bench Verified problems that OpenAI o3 did not consistently solve over 64 independent runs.

    Look up an archived copy ↗

  3. 3Contextweb.archive.org ↗

    In 59.4% of the 138 tasks, the test design or the problem statement itself was flawed.

    Look up an archived copy ↗

Members' extras

What each receipt proves, and what would change our mind.

  • What each receipt above proves, in our words
  • The figures table, each number traced to the receipt that states it
  • The trick behind the claim, explained, with the guide to spotting it
  • What would change our verdict
  • Our call on the record, with its due date

The verdict, the opening and every receipt above stay free. Membership adds our analysis to the evidence file for every checked story.

See Membership →Already a member? Sign in →

First membership: 30 days free, then A$89 a year. One introductory trial per customer. Card required; renews annually until cancelled. Cancel before the trial ends to avoid the first charge.