Using privacy-preserving techniques, we created synthetic mental health conversations that accurately reflect real-world usage patterns of AI for mental health.
In OpenAI’s reported results, GPT-6 Astra scored highest at 57.3% task-clipped, followed by GPT-6 Sol at 53.9%, Claude Opus 5.5 at 52.4%, and GPT-6 Luna at 50.2%.
Expert-authored completions written by clinicians scored 38.5%, which the authors attribute largely to clinicians writing short responses as if in an in-person conversation, often asking a single question or making a simple statement.
Rubric-aware completions, written with the grading rubrics provided, scored 99.0%, which the paper describes as a sanity check on the evaluation’s noise ceiling.
The authors state that no benchmark captures everything that matters in a personal conversation, and they position MentalHealthBench as an auditable diagnostic tool rather than a definitive leaderboard.
Scores are reported as task-clipped rubric scores and can be decomposed across ten expert-defined behavioral axes, including context seeking, empathy, urgency calibration, and reality testing.
First membership: 30 days free, then A$89 a year. One introductory trial per customer. Card required; renews annually until cancelled. Cancel before the trial ends to avoid the first charge.
Free · one story a night
Don't take the AI industry's word for it.
“AI is dramatically accelerating AI research at Anthropic right now, roughly doubling the pace of capabilities progress.”
TRUE, BUTLab Not FieldAjeya Cotra on Planned Obsolescence, reading METR's assessment and Anthropic's S-1 statement that it may imminently meet its own threshold
That's the latest lead check. One a day, receipts attached, in your inbox before someone repeats it at you. Free.
No spam. Unsubscribe in one click. Receipts or it didn't happen.
Pranjal Gupta writes every check himself. Former enterprise architect, from the implementation side. Not a team of anonymous raters.
