Then the graders checked whether they knew Rails existed.
“The Agents on Rails benchmark, published on the official Rails blog, finds frontier agents solve most atomic Rails tasks (up to 92%) while mostly hand-rolling code, with Rails API recall running from 8% to 35%, and price stops predicting score: Luna's full run cost 91 cents while Opus charged 132x the price for nineteen points.”
Agents on Rails team (Kryukov and Petrov, published on the official Rails blog), 13 August 2026
Recorded 14 August 2026. Our ruling means: we tried to break it and could not.
Rails API recall runs from 8% (DeepSeek) to 35% (Fable). Those same two models are 1.4× apart on score and four times apart on recall: knowing Rails separates the field far more sharply than solving the tasks does.
Luna ran all 63 of its runs for 91 cents and beat two models that cost more. Sol costs 36× Luna and buys eleven points. Opus costs 132× and buys nineteen.
First membership: 30 days free, then A$89 a year. One introductory trial per customer. Card required; renews annually until cancelled. Cancel before the trial ends to avoid the first charge.
Free · one story a night
Don't take the AI industry's word for it.
“AI is dramatically accelerating AI research at Anthropic right now, roughly doubling the pace of capabilities progress.”
TRUE, BUTLab Not FieldAjeya Cotra on Planned Obsolescence, reading METR's assessment and Anthropic's S-1 statement that it may imminently meet its own threshold
That's the latest lead check. One a day, receipts attached, in your inbox before someone repeats it at you. Free.
No spam. Unsubscribe in one click. Receipts or it didn't happen.
Pranjal Gupta writes every check himself. Former enterprise architect, from the implementation side. Not a team of anonymous raters.
