Start free trial

TRUE, BUTLab Not Field

Finished work: no significant change

A Harvard paper tracking 718 firms finds AI agents raised lines of code 30%, while finished work showed no statistically significant change.

“Developers who used GitHub Copilot completed the task 55% faster than the developers who didn't use GitHub Copilot.”

GitHub, research blog post (7 Sep 2022); tested at firm level in a Harvard working paper (current version 4 Aug 2026)

15 receipts1 support5 refute9 context

Issue 4313 October 20263 min
A developer at a desk beside a tall pile of paper pages, with a narrow doorway behind where a single reviewer sits.
AI illustration

Members · 30 days free

See what the evidence actually shows.

Members read the full check on every story: what the evidence shows, why it matters to you and the one line to take with you. Every past edition and the Receipts Pack come with it. A$89 a year, about A$0.24 a day.

What members getAlready a member? Sign in →
  1. At sign-up: 30 days free. Payment card required. One introductory trial per customer.
  2. Your trial ends 30 days after you start. Unless you cancel before then in Account → Manage subscription, we charge A$89 for the first year. It renews automatically at A$89/yr until cancelled.
  3. You can ask for a full refund within 14 days after any annual payment, renewals included, with no reason needed, by emailing hello@bskiller.com. This voluntary refund does not limit your rights under the Australian Consumer Law.

By starting your trial, you agree to the Terms.

Inferno Tech Pty Ltd t/a BS Killer, ABN 27 647 413 474

There’s more to this story.

Membership opens the full investigation, the strongest counterargument and what to do with what you’ve learned.

30 days free, then A$89 a year. Payment card required. Cancel before the trial ends to avoid the first charge.

Start free trial →
  1. At sign-up: 30 days free. Payment card required. One introductory trial per customer.
  2. Your trial ends 30 days after you start. Unless you cancel before then in Account → Manage subscription, we charge A$89 for the first year. It renews automatically at A$89/yr until cancelled.
  3. You can ask for a full refund within 14 days after any annual payment, renewals included, with no reason needed, by emailing hello@bskiller.com. This voluntary refund does not limit your rights under the Australian Consumer Law.

By starting your trial, you agree to the Terms.

Inferno Tech Pty Ltd t/a BS Killer, ABN 27 647 413 474

Already a member? Sign in

The trick has a name

We call it Lab Not Field: it works in the test conditions, not the deployed ones. You'll see it again. Learn to spot it →

Say this in tomorrow's meeting“GitHub's 55% faster figure timed one task. A Harvard study of 718 firms found AI agents added 30% more code but no statistically significant rise in work shipped.”

Receipts

  1. Supports github.blog: 55% faster than the developers who didn’t use GitHub Copilot
  2. Context github.blog: timed how long it took them to write an HTTP server in JavaScript
  3. Context arxiv.org: completed the task 55.8% faster than the control group
  4. Context web.archive.org: a novel proprietary dataset from an engineering analytics platform, covering 300 million work events
  5. Context web.archive.org: across 718 firms. We use a staggered difference-in-differences design
  6. Refutes web.archive.org: However, productivity gains do not fully pass through to changes in software output or employment.
  7. Refutes web.archive.org: For AI agents, this incomplete pass-through reflects a bottleneck from code review
  8. Context arstechnica.com: Harvard University researchers Fiona Chen and James Stratton made use of aggregated analytics data from
  9. Context arstechnica.com: leads to a 30 percent increase in total lines of code generated
  10. Refutes arstechnica.com: did not change in a statistically significant way after AI tools were introduced
  11. Context arstechnica.com: the resolution rate for Issues and Epics (i.e. wholesale software features) tracked by tools like Jira
  12. Refutes arstechnica.com: balloons 49 percent on average after AI agents are introduced
  13. Context arstechnica.com: a 14 percent increase in the share of workers performing code reviews
  14. Refutes arstechnica.com: absorbed by downstream constraints in the production process
  15. Context arstechnica.com: significant updates and upgrades to their output even since this study

Open the Receipts Pack → What each source proves, every figure traced, and what would change our verdict.

Checked by AI at the BS Killer desk · ruled 13 Oct 2026 · How we check →

Get tomorrow's check in your inbox.

One AI claim a night, checked against independent sources, receipts attached. Free.

Free email updates. Unsubscribe any time.

This story is a stable, citable object. If you can falsify a verdict, tell us. Corrections are loud here.