The trick: Lab Not Field
Khan Academy sells Khanmigo as an always-available tutor.
A two-year trial found students rarely asked it anything.
Khan Academy markets Khanmigo as an always-available tutor that never gives you the answer.
Before you read on. Your call?
TRUE, BUT
17%
A two-year randomized trial in 18 Tennessee middle schools, by Oreopoulos and Low, found assignment raised maths by 1.3 national percentile ranks per term, about 0.06 to 0.08 standard deviations a year.
The twist
The authors say these gains resemble Khan Academy practice without AI. 96 percent tried Khanmigo, but the median student messaged it in only 17 percent of sessions with a mistake.
There’s more to this story.
Membership opens the full investigation, the strongest counterargument and what to do with what you’ve learned.
Start your 30-day free trial →First membership: 30 days free, then A$89 a year. One introductory trial per customer. Card required; renews annually until cancelled. Cancel before the trial ends to avoid the first charge. Already a member? Sign in
Couldn't check your access. That's on us.
The trick has a name
We call it Lab Not Field: it works in the test conditions, not the deployed ones. You'll see it again. Learn to spot it →
Receipts
- Context edworkingpapers.com:
Assignment raises math achievement by 1.3 national percentile ranks per term, or about 0.06 to 0.08 standard deviations over a school year
- Refutes edworkingpapers.com:
These gains resemble those from Khan Academy practice without AI assistance.
- Context edworkingpapers.com:
96 percent of students tried Khanmigo at least once
- Refutes edworkingpapers.com:
and in only 17 percent of the exercise sessions in which they made a mistake.
- Refutes edworkingpapers.com:
Messages that students did send were mostly bare answers or clicks on suggested prompts.
- Context edworkingpapers.com:
The binding constraint appears to be engagement
- Context edworkingpapers.com:
a two-year cluster randomized trial in 18 Tennessee middle schools
- Supports edworkingpapers.com:
the implied effect of a full year of active participation reaches 0.14 standard deviations.
- Context edworkingpapers.com:
Generative AI has been promoted as the technology that could transform education by providing every student a personal tutor.
- Supports khanmigo.ai:
Khanmigo is your always-available tutor on just about every topic.
- Supports khanmigo.ai:
Unlike ChatGPT, Khanmigo never gives you the answer.
- Context news.ycombinator.com:
AI tutoring with Khanmigo in a two-year school experiment
Open the Receipts Pack → What each source proves, every figure traced, and what would change our verdict.