How HealthBench works
OpenAI · 5,000 conversations · updated August 17, 2026
HealthBench is OpenAI's May 2025 rubric benchmark for health conversations. It holds 5,000 multi-turn conversations, each graded against criteria physicians wrote for that specific conversation. There are 48,562 criteria in total, from 262 physicians across 60 countries, 26 specialties, and 49 languages. Scores are rubric credit, shown as 0 to 100 on some sites and 0 to 1 on others.
| publisher | OpenAI |
|---|---|
| released | 2025-05 |
| size | 5,000 conversations |
| scale | 0-100 rubric-point percentage (some sites display 0-1), higher better; length-adjusted and unadjusted variants |
Rubrics instead of answer keys
Health questions rarely have one correct answer, so HealthBench does not store one. Each conversation carries a list of criteria: things a good response should include, ask about, or avoid. The median conversation has 11 criteria, ranging from 2 to 48. Weights run from -10 to +10, and negative criteria subtract points when a response does something harmful.
An example's score is earned points divided by the maximum positive points for that example, so an example can fall below zero. The overall score is the mean across examples, clipped to 0 to 1. Criteria fall into seven themes, among them accuracy, completeness, context awareness, and instruction following.
How the grader was chosen
The grader is a model, chosen by meta-evaluation: candidates were scored against physician grades on the same responses. GPT-4.1 won with a macro-F1 of 0.709, ahead of o4-mini at 0.692, o3 at 0.681, GPT-4.1-mini at 0.661, and GPT-4.1-nano at 0.580. In five of the seven themes it agreed with physician consensus more often than the average individual physician did.
Rerunning the same model is stable: the standard deviation across 16 repeats is about 0.002. Run noise is not what separates rows on a leaderboard, but grader configuration is.
Reading a score
Physician baselines give the scale meaning. Physicians writing unaided scored 0.13. With model-written references from September 2024 they scored 0.31. With April 2025 references they reached 0.48, no longer improving on those references, which scored 0.49 alone.
Cross-source comparison is the main trap. One vendor reports raw scores, another leads with length-adjusted numbers at maximum reasoning effort, and at least one ran its competitors itself. Length adjustment subtracts about 2.99 points per 500 characters past 2,000 here. Two rows are comparable only when the protocol behind them matches.
Limits
OpenAI has said the parent set is approaching a noise ceiling for frontier models and points to HealthBench Professional for continued measurement, so movement at the top increasingly reflects grader behavior more than capability. The rubrics encode the judgment of the physicians who wrote them, one defensible standard among others. Scoring rewards responses that cover listed criteria, which favors length; the length adjustment corrects for that rather than removing it. And most published rows come from vendors running their own configurations, so protocol mismatch causes more confusion here than measurement error does.
Current standings are indexed at clinicalbenchmarks.ai, with dedicated boards for HealthBench Hard and HealthBench Professional.
The grading machinery behind pages like this one is covered in the rubric grading and model graders guides.