Learn · a trick with a name
Lab Not Field
it works in the test conditions, not the deployed ones.
Controlled tests remove the mess that real work is made of. Results rarely survive the trip.
How to spot it
- Benchmarks or demos, no deployment data
- Clean inputs, curated prompts, perfect conditions
- No numbers from paying customers in production
The one question to ask“Has this worked outside the test, for someone like me?”
At work
Pilot on your own messy data before you buy. A week of your tickets beats a month of their benchmarks.
Caught in the wild
Every time we've caught it so far. Try calling each one before you open it.
- TRUE, BUTThe doubling is real on one ruler: Anthropic's internal benchmark index. METR says that ruler covers well-specified tasks, and Anthropic redacted most of the rest.
- TRUE, BUTEpoch AI says an open-weight model broke six Firefox releases for about $1,000 each. That is the cost per version after a build that, per AlphaSignal, ran past $100,000 in API fees.
- TRUE, BUTThe Home Office piloted AI summaries of asylum interviews and reported 23 minutes saved per case with 9% inaccurate. After rollout, 45 of the 203 summaries experts checked failed. They checked about 0.3% of them.
- TRUE, BUTForced to pick, Grok chooses Republicans, says the headline. Turn off search and give it the names, and Grok declined 441 races. The lean sat in the forced prompt and the search.
- TRUE, BUTTexas AI emits 8.5 times more CO2 than Sweden. That is a model of one example request, and the grid gap behind it is closer to 42 times.
- TRUE, BUTMicrosoft's AI really did more than double doctors' accuracy in Science. The doctors were generalists working alone, the patients were simulated, and every author works for Microsoft.
- TRUE, BUTThe AI cancer 'treatment' is one plant compound that worked alongside an existing drug on cancer cell clumps in a dish. No mice, no patients yet.
- TRUE, BUTAn Anthropic co-founder told the IMF the singularity has begun. His evidence is test scores and anecdotes, and his own speech calls them first signs.
- TRUE, BUTAn AI assistant chasing a £350 train refund sent 187 messages, 69 of them to officials and the press. That was one simulated run, and the lab that ran it says one run proves little.
- TRUE, BUTMeta's 816-billion-parameter model runs on two Mac Studios. Each one needs 512GB of memory, a model MacRumors says Apple has not released yet.
- TRUE, BUTAs eight states sue OpenAI over ChatGPT's alleged harms, its own headline safety score is 91% on suicide conversations. OpenAI says that score came from a hard test set whose error rates are not representative of average production traffic.
- TRUE, BUTTen minutes of AI erodes your ability to persist, Berkeley's headline says. The study measured skipped problems on a paid online task, three test problems each, and its largest trial found the skip gap was not significant.
