Learn · a trick with a name
Lab Not Field
it works in the test conditions, not the deployed ones.
Controlled tests remove the mess that real work is made of. Results rarely survive the trip.
How to spot it
- Benchmarks or demos, no deployment data
- Clean inputs, curated prompts, perfect conditions
- No numbers from paying customers in production
The one question to ask“Has this worked outside the test, for someone like me?”
At work
Pilot on your own messy data before you buy. A week of your tickets beats a month of their benchmarks.
Caught in the wild
Every time we've caught it so far. Try calling each one before you open it.
- TRUE, BUTTen minutes of AI erodes your ability to persist, Berkeley's headline says. The study measured skipped problems on a paid online task, three test problems each, and its largest trial found the skip gap was not significant.
- TRUE, BUTGitHub's famous number says Copilot users finished a task 55% faster. A Harvard paper tracking 718 firms finds AI agents raised lines of code 30%, while finished work showed no statistically significant change.
- TRUE, BUTKhan Academy sells Khanmigo as an always-available tutor. A two-year trial found students rarely asked it anything.
- TRUE, BUTThe AI worm is real. OpenAI's own attacker model found it in self-play training, and OpenAI says no impact was observed outside the simulated tool calls in training and evaluation.
- TRUE, BUTThe pitch is that frontier models can generate genuine research ideas. A new blind benchmark handed seven of them a paper's reference list, scrubbed of anything they could have memorized, and asked for the paper's core idea. They got it 3 to 15 percent of the time.
- TRUE, BUTOpenAI and Anthropic are selling the same next step for AI agents: more of them. Claude Code now forks subagents by default, and Sol Ultra fans a problem across up to 64. Google Research ran the controlled test, and the answer is a split: more agents help work that breaks into independent pieces and hurt work that runs as one dependent chain, by up to 70%. Which one your task is decides whether the swarm is an upgrade or a tax.
- TRUE, BUTAnthropic studied whether you read permission prompts. You don't. So today it stopped showing them.
- TRUE, BUTThe 96% accurate deepfake detector is 96% accurate on the deepfakes it was shown. On new ones it is closer to a coin flip.
- TRUE, BUTNVIDIA built a model that talks four times faster. The work arrives 30 percent sooner.
