The trick: Lab Not Field
Microsoft's AI really did more than double doctors' accuracy in Science.
The doctors were generalists working alone, the patients were simulated, and every author works for Microsoft.
Microsoft's MAI-DxO is more than twice as accurate as physicians, per a Science paper.
Before you read on. Your call?
TRUE, BUT
56
It scored 83.2% on NEJM case conferences against 36.1% for generalists using references, on 56 held-out cases in a simulator. The authors, all Microsoft employees, say it shows no superiority over specialists, teams or real practice.
There’s more to this story.
Membership opens the full investigation, the strongest counterargument and what to do with what you’ve learned.
30 days free, then A$89 a year. Payment card required. Cancel before the trial ends to avoid the first charge.
Review my 30-day free trial →- At sign-up: 30 days free. Payment card required. One introductory trial per customer.
- Your trial ends 30 days after you start. Unless you cancel before then in Account → Manage subscription, we charge A$89 for the first year. It renews automatically at A$89/yr until cancelled.
- You can ask for a full refund within 14 days after any annual payment, renewals included, with no reason needed, by emailing hello@bskiller.com. This voluntary refund does not limit your rights under the Australian Consumer Law.
By starting your trial, you agree to the Terms.
BS Killer is published by Inferno Tech Pty Ltd, ABN 27 647 413 474.
Get tomorrow’s check free by email.
Couldn't check your access. That's on us.
The trick has a name
We call it Lab Not Field: it works in the test conditions, not the deployed ones. You'll see it again. Learn to spot it →
Receipts
- Supports microsoft.ai:
MAI-DxO is more than twice as accurate as physicians using textbooks and the internet
- Supports microsoft.ai:
Physicians achieved 22% without external reference resources and 36% with them, making MAI-DxO about 3.7 and 2.3 times more accurate, respectively.
- Context microsoft.ai:
Benchmarks matter, but the true test is performance in real clinical environments.
- Context microsoft.ai:
In a prospective study, patients will interact with the system directly, with performance measured against physicians’ independent assessments.
- Context microsoft.ai:
Through a new partnership with Beth Israel Deaconess Medical Center in Boston, we will evaluate clinical reasoning capabilities across different hospital settings.
- Supports api.crossref.org:
outperforming generalist physicians (36.1 and 46.8%, respectively)
- Context api.crossref.org:
Although these findings were obtained in a controlled benchmark setting
- Refutes unite.ai:
The authors state that the comparisons do not show superiority over specialists, multidisciplinary teams, or real-world clinical practice
- Refutes unite.ai:
A further stated limitation is the comparison against individual physicians rather than multispecialty collaboration
- Context unite.ai:
conducted on 56 cases held out from engineering work
- Context unite.ai:
averaging 22.3% accuracy closed-book at $3,049 per case and 36.1% open-book at $4,725
- Context unite.ai:
achieved 83.2% accuracy at an average cost of $5,431 per case
- Supports unite.ai:
Cost-constrained configurations reached 80.9% accuracy at $2,241 and 79.6% at $1,303
- Context unite.ai:
the authors state that all of them are Microsoft employees holding equity in the company
Open the Receipts Pack → What each source proves, every figure traced, and what would change our verdict.
