Learn · a trick with a name
Lab Not Field
it works in the test conditions, not the deployed ones.
Controlled tests remove the mess that real work is made of. Results rarely survive the trip.
How to spot it
- Benchmarks or demos, no deployment data
- Clean inputs, curated prompts, perfect conditions
- No numbers from paying customers in production
The one question to ask“Has this worked outside the test, for someone like me?”
At work
Pilot on your own messy data before you buy. A week of your tickets beats a month of their benchmarks.
Caught in the wild
Every time we've caught it so far. Try calling each one before you open it.
- TRUE, BUTThe pitch is that frontier models can generate genuine research ideas. A new blind benchmark handed seven of them a paper's reference list, scrubbed of anything they could have memorized, and asked for the paper's core idea. They got it 3 to 15 percent of the time.
- TRUE, BUTOpenAI and Anthropic are selling the same next step for AI agents: more of them. Claude Code now forks subagents by default, and Sol Ultra fans a problem across up to 64. Google Research ran the controlled test, and the answer is a split: more agents help work that breaks into independent pieces and hurt work that runs as one dependent chain, by up to 70%. Which one your task is decides whether the swarm is an upgrade or a tax.
- TRUE, BUTAnthropic studied whether you read permission prompts. You don't. So today it stopped showing them.
- TRUE, BUTThe 96% accurate deepfake detector is 96% accurate on the deepfakes it was shown. On new ones it is closer to a coin flip.
- TRUE, BUTNVIDIA built a model that talks four times faster. The work arrives 30 percent sooner.