Learn · a trick with a name
Self-Marked
graded by the party that benefits from the grade.
A score sounds like a measurement. It is only a measurement if someone without a stake took it.
How to spot it
- The number comes from the company's own blog, eval or harness
- No independent lab has published the same test
- The method section is thinner than the headline
The one question to ask“Who ran the test, and do they win if it looks good?”
At work
Before you quote a benchmark in a deck, find the same number from someone who doesn't sell the model. If you can't, call it a vendor claim.
Caught in the wild
Every time we've caught it so far. Try calling each one before you open it.
- TRUE, BUTKimi K3.1 Pro's weights really are downloadable. But one benchmark sits close to Opus 5.5 while another shows under half its score, and the licence carries a revenue clause.
- TRUE, BUTOpenAI says its AI proved a pillar of complexity theory false, checked by machine on 17 October. The claim is the full hypothesis. The checking so far is OpenAI's own.
- TRUE, BUTThe 42.9% is 36 people. All 84 in the study were screened at Biograph, and every author works for Biograph.
- CONTESTEDOpenAI called its Hugging Face breach an unprecedented cyber incident, then a rare outlier. Its own report, as quoted, shows a monitoring alert on 27 June and a halt on 19 July.
- TRUE, BUTTypeSafe says a third of the Fortune 500 use Jev. Its lead investor, announcing the same round, says 25%.
- TRUE, BUTGoogle calls EmbeddingGemma 2 best-in-class for its size. The 9.92-point code gain, from 68.76 to 78.68, is the lab's own measurement.
- TRUE, BUTQ2D-Web is a public leaderboard of 69,721 queries from Perplexity's own traffic. The paper says the benchmark stays private.
- TRUE, BUTOpenAI dropped hundreds of AI math papers at once. Its own README says about 42% of the headline results are machine checked.
- BSNvidia says its new agent safety platform could have prevented the Hugging Face hack. Per CNBC, that line came from an unnamed Nvidia representative on a press call. No test against the actual hack is in the record we checked.
- TRUE, BUTOpenAI built a mental health test, had its own model grade it, and its own model came top. Answers written by licensed clinicians scored 38.5%. GPT-6 Astra scored 57.3%.
- TRUE, BUTApple blamed "the rapid expansion of AI data centers" for the price hikes on your next Mac or iPad. Apple's own decision to cram more RAM into its devices for its own AI features is a second AI-driven cost its statement never mentions.
- TRUE, BUTOpenAI's president says we're in the AGI era. The benchmark's own inventor scored the same model 37 points lower.
