The trick: Lab Not Field
As eight states sue OpenAI over ChatGPT's alleged harms, its own headline safety score is 91% on suicide conversations.
OpenAI says that score came from a hard test set whose error rates are not representative of average production traffic.
OpenAI says automated evaluations score its new GPT-5 model 91% compliant with desired behaviors in self-harm and suicide conversations, up from 77%.
Before you read on. Your call?
TRUE, BUT
91%
Members · 30 days free
See what the evidence actually shows.
Members read the full check on every story: what the evidence shows, why it matters to you and the one line to take with you. Every past edition, re-verified, and the Receipts Pack come with it. A$89 a year, about A$0.24 a day.
- At sign-up: 30 days free. Payment card required. One introductory trial per customer.
- Your trial ends 30 days after you start. Unless you cancel before then in Account → Manage subscription, we charge A$89 for the first year. It renews automatically at A$89/yr until cancelled.
- You can ask for a full refund within 14 days after any annual payment, renewals included, with no reason needed, by emailing hello@bskiller.com. This voluntary refund does not limit your rights under the Australian Consumer Law.
By starting your trial, you agree to the Terms.
BS Killer is published by Inferno Tech Pty Ltd, ABN 27 647 413 474.
There’s more to this story.
Membership opens the full investigation, the strongest counterargument and what to do with what you’ve learned.
30 days free, then A$89 a year. Payment card required. Cancel before the trial ends to avoid the first charge.
Review my 30-day free trial →- At sign-up: 30 days free. Payment card required. One introductory trial per customer.
- Your trial ends 30 days after you start. Unless you cancel before then in Account → Manage subscription, we charge A$89 for the first year. It renews automatically at A$89/yr until cancelled.
- You can ask for a full refund within 14 days after any annual payment, renewals included, with no reason needed, by emailing hello@bskiller.com. This voluntary refund does not limit your rights under the Australian Consumer Law.
By starting your trial, you agree to the Terms.
BS Killer is published by Inferno Tech Pty Ltd, ABN 27 647 413 474.
Get tomorrow’s check free by email.
Couldn't check your access. That's on us.
The trick has a name
We call it Lab Not Field: it works in the test conditions, not the deployed ones. You'll see it again. Learn to spot it →
Receipts
- Supports iowaattorneygeneral.gov:
OpenAI is hurting kids and they are lying to Iowans
- Context iowaattorneygeneral.gov:
Arkansas, Arizona, Idaho and Montana have also sued OpenAI in their own state courts.
- Supports web.archive.org:
around 0.15% of users active in a given week have conversations that include explicit indicators of potential suicidal planning or intent
- Context web.archive.org:
On a model evaluation consisting of more than 1,000 challenging self harm and suicide conversations
- Supports web.archive.org:
our new automated evaluations score the new GPT
- Supports web.archive.org:
model at 91% compliant with our desired behaviors, compared to 77% for the previous
- Refutes web.archive.org:
error rates are not representative of average production traffic
- Context web.archive.org:
examples are adversarially selected for high
- Context engadget.com:
eight states have filed lawsuits against OpenAI alleging the company has violated state-level consumer protection laws and endangered children
- Context engadget.com:
Engadget has asked OpenAI to comment on the state lawsuits
Open the Receipts Pack → What each source proves, every figure traced, and what would change our verdict.
