True, and Anthropic says what the summaries skip: a coin flip scores 50%.
“Anthropic says that from a five-minute chat, a Claude agent's ranking of books matched its person's on 61% of pairs, 'surprisingly good for such a short conversation', in Project Swap, a book-trading market run by agents for 201 Anthropic employees.”
Anthropic ('Project Swap' research post), amplified by TLDR AI, 24 September 2026
Recorded 26 September 2026. Our ruling means: we tried to break it and could not.
From a five-minute chat, an agent’s ranking of the books matched its person's on 61% of pairs, which is surprisingly good for such a short conversation.
Taking this into account, the best possible assignment (the utilitarian optimum) in our experiment is a score of 0.89 overall, leaving participants at roughly their second choice on a 10-book list.
So, working from Claude’s imprecise rankings accounts for a majority (85%) of the shortfall, and sending agents into a “free-for-all” trading floor accounts for the remaining 15%.
On Claude’s rankings, agents on Haiku trading floors averaged 0.75, while on Opus floors they averaged 0.88 (the utilitarian optimum on Claude’s rankings is 0.95).
So this summer, we built a small, controlled market to study these questions: a barter economy with 201 Anthropic employees and their Claude-powered agents.
Think of the patient who skips a treatment after one quote they can’t afford, when there’s a different clinic that charges far less, or the hospital that quoted them would have cut its bill if asked.
The headline number: Claude-powered agents aligned with their participants’ preferences 61% of the time, based on comparisons between the agent’s rankings and the participants’ own rankings of books.
First membership: 30 days free, then A$89 a year. One introductory trial per customer. Card required; renews annually until cancelled. Cancel before the trial ends to avoid the first charge.
Free · one story a night
Don't take the AI industry's word for it.
“AI is dramatically accelerating AI research at Anthropic right now, roughly doubling the pace of capabilities progress.”
TRUE, BUTLab Not FieldAjeya Cotra on Planned Obsolescence, reading METR's assessment and Anthropic's S-1 statement that it may imminently meet its own threshold
That's the latest lead check. One a day, receipts attached, in your inbox before someone repeats it at you. Free.
No spam. Unsubscribe in one click. Receipts or it didn't happen.
Pranjal Gupta writes every check himself. Former enterprise architect, from the implementation side. Not a team of anonymous raters.
