Learn · a trick with a name
Self-Marked
graded by the party that benefits from the grade.
A score sounds like a measurement. It is only a measurement if someone without a stake took it.
How to spot it
- The number comes from the company's own blog, eval or harness
- No independent lab has published the same test
- The method section is thinner than the headline
The one question to ask“Who ran the test, and do they win if it looks good?”
At work
Before you quote a benchmark in a deck, find the same number from someone who doesn't sell the model. If you can't, call it a vendor claim.
Caught in the wild
Every time we've caught it so far. Try calling each one before you open it.
- TRUE, BUTTypeSafe says a third of the Fortune 500 use Jev. Its lead investor, announcing the same round, says 25%.
- TRUE, BUTGoogle calls EmbeddingGemma 2 best-in-class for its size. The 9.92-point code gain, from 68.76 to 78.68, is the lab's own measurement.
- TRUE, BUTQ2D-Web is a public leaderboard of 69,721 queries from Perplexity's own traffic. The paper says the benchmark stays private.
- TRUE, BUTOpenAI dropped hundreds of AI math papers at once. Its own README says about 42% of the headline results are machine checked.
- BSNvidia says its new agent safety platform could have prevented the Hugging Face hack. Per CNBC, that line came from an unnamed Nvidia representative on a press call. No test against the actual hack is in the record we checked.
- TRUE, BUTOpenAI built a mental health test, had its own model grade it, and its own model came top. Answers written by licensed clinicians scored 38.5%. GPT-6 Astra scored 57.3%.
- TRUE, BUTApple blamed "the rapid expansion of AI data centers" for the price hikes on your next Mac or iPad. Apple's own decision to cram more RAM into its devices for its own AI features is a second AI-driven cost its statement never mentions.
- TRUE, BUTOpenAI's president says we're in the AGI era. The benchmark's own inventor scored the same model 37 points lower.
- TRUE, BUTOpenAI just crossed a cybersecurity line no model has crossed before. Its own timeline shows it saw this coming a month early.
- TRUE, BUTTechCrunch called it a peek at self-improving AI. Anthropic's own paper calls its own benchmarks only proxies, and admits the automated system tried to game them 39 times.
- TRUE, BUTGoogle is moving its 90-person AI risk team out of DeepMind and into the division that handles lobbying. Leadership says nothing changes. The team that decides how close Gemini gets to bioweapon risk now reports through the same building as the lobbyists.
- TRUE, BUTOpenAI called its own security incident unprecedented. The company it hacked says the flaws were ordinary.
