The trick: Lab Not Field
GitHub's famous number says Copilot users finished a task 55% faster.
A Harvard paper tracking 718 firms finds AI agents raised lines of code 30%, while finished work showed no statistically significant change.
GitHub's 2022 post says developers using Copilot finished a task 55% faster.
Before you read on. Your call?
TRUE, BUT
30%
That task was writing one HTTP server in JavaScript. A Harvard working paper using 300 million work events across 718 firms finds AI agents brought a 30 percent increase in lines of code, but the resolution rate for issues and epics did not change in a statistically significant way. Review time balloons 49 percent.
There’s more to this story.
Membership opens the full investigation, the strongest counterargument and what to do with what you’ve learned.
30 days free, then A$89 a year. Payment card required. Cancel before the trial ends to avoid the first charge.
Review my 30-day free trial →- At sign-up: 30 days free. Payment card required. One introductory trial per customer.
- Your trial ends 30 days after you start. Unless you cancel before then in Account → Manage subscription, we charge A$89 for the first year. It renews automatically at A$89/yr until cancelled.
- You can ask for a full refund within 14 days after any annual payment, renewals included, with no reason needed, by emailing hello@bskiller.com. This voluntary refund does not limit your rights under the Australian Consumer Law.
By starting your trial, you agree to the Terms.
BS Killer is published by Inferno Tech Pty Ltd, ABN 27 647 413 474.
Get tomorrow’s check free by email.
Couldn't check your access. That's on us.
The trick has a name
We call it Lab Not Field: it works in the test conditions, not the deployed ones. You'll see it again. Learn to spot it →
Receipts
- Supports github.blog:
55% faster than the developers who didn’t use GitHub Copilot
- Context github.blog:
timed how long it took them to write an HTTP server in JavaScript
- Context arxiv.org:
completed the task 55.8% faster than the control group
- Context fion.ac:
a novel proprietary dataset from an engineering analytics platform, covering 300 million work events
- Context fion.ac:
across 718 firms. We use a staggered difference-in-differences design
- Refutes fion.ac:
However, productivity gains do not fully pass through to changes in software output or employment.
- Refutes fion.ac:
For AI agents, this incomplete pass-through reflects a bottleneck from code review
- Context arstechnica.com:
Harvard University researchers Fiona Chen and James Stratton made use of aggregated analytics data from
- Context arstechnica.com:
leads to a 30 percent increase in total lines of code generated
- Refutes arstechnica.com:
did not change in a statistically significant way after AI tools were introduced
- Context arstechnica.com:
the resolution rate for Issues and Epics (i.e. wholesale software features) tracked by tools like Jira
- Refutes arstechnica.com:
balloons 49 percent on average after AI agents are introduced
- Context arstechnica.com:
a 14 percent increase in the share of workers performing code reviews
- Refutes arstechnica.com:
absorbed by downstream constraints in the production process
- Context arstechnica.com:
significant updates and upgrades to their output even since this study
Open the Receipts Pack → What each source proves, every figure traced, and what would change our verdict.
