The trick: Lab Not Field
The 96% accurate deepfake detector is 96% accurate on the deepfakes it was shown.
On new ones it is closer to a coin flip.
deepfake detectors hit around 96% accuracy, so synthetic video can be caught reliably. Intel's FakeCatcher, marketed at a 96% accuracy rate, is the number everyone quotes.
Before you read on. Your call?
TRUE, BUT
78% in the wild
that figure is a controlled-lab result. The DeepFake-Eval-2024 study found leading commercial detectors around 78% on in-the-wild deepfakes, NIST evaluations put accuracy below 40% when the generator differs from the training software, and the Vector Institute named the pattern the Generalization Illusion, where benchmark scores stay high while real-world detection quietly declines. Real capability, badly oversold.
There’s more to this story.
Membership opens the full investigation, the strongest counterargument and what to do with what you’ve learned.
30 days free, then A$89 a year. Payment card required. Cancel before the trial ends to avoid the first charge.
Review my 30-day free trial →- At sign-up: 30 days free. Payment card required. One introductory trial per customer.
- Your trial ends 30 days after you start. Unless you cancel before then in Account → Manage subscription, we charge A$89 for the first year. It renews automatically at A$89/yr until cancelled.
- You can ask for a full refund within 14 days after any annual payment, renewals included, with no reason needed, by emailing hello@bskiller.com. This voluntary refund does not limit your rights under the Australian Consumer Law.
By starting your trial, you agree to the Terms.
BS Killer is published by Inferno Tech Pty Ltd, ABN 27 647 413 474.
Get tomorrow’s check free by email.
Couldn't check your access. That's on us.
The trick has a name
We call it Lab Not Field: it works in the test conditions, not the deployed ones. You'll see it again. Learn to spot it →
Receipts
- Supports newsroom.intel.com:
the company has developed FakeCatcher, a technology that can detect fake videos with a 96% accuracy rate
- Refutes helpnetsecurity.com:
The Vector Institute team calls the result the Generalization Illusion. Benchmark scores stay high. Real-world detection performance quietly declines.
- Context realitydefender.com:
the DeepFake-Eval-2024 study found that leading commercial detectors achieved approximately 78% accuracy on in-the-wild deepfakes, a significant drop from published benchmark numbers
Open the Receipts Pack → What each source proves, every figure traced, and what would change our verdict.
