How Rubric Grading Works in Health AI Evals
updated August 17, 2026
Rubric grading scores a model's answer against a written checklist instead of against one correct answer. Each item on the checklist is a criterion that a physician wrote for that specific case, and each criterion carries a weight. A grader decides whether the response meets each criterion, the weights of the met criteria are added up, and that total is divided by the points the case made available. HealthBench, published by OpenAI in May 2025, works this way across 5,000 conversations and 48,562 physician-written criteria. This guide covers where criteria come from, how weights work, what negative criteria do, and how the final number is computed.
Criteria are written per case
An exam-style benchmark has one answer key. A rubric benchmark has a separate checklist for every case, because a health conversation has no single right answer. In HealthBench the median conversation carries 11 criteria, and the count ranges from 2 to 48. The criteria cover accuracy, completeness, context awareness, communication, and instruction following, so a response can lose points for being correct but incomplete, or for answering a question the user did not ask.
Who writes the criteria matters as much as what they say. HealthBench's 48,562 criteria came from 262 physicians in 60 countries, across 26 specialties and 49 languages. HealthBench Professional, released in April 2026, added more review: 525 tasks that physicians selected out of 15,079 real clinician conversations, criteria from 190 physicians in 50 countries, and a three-stage adjudication in which at least three physicians signed off on each criterion.
Weights and negative criteria
Criteria are not worth the same. HealthBench weights run from -10 to +10, and the physicians who wrote the case set them. Naming the condition can be worth several times what a clear closing sentence is worth.
Criteria with weights below zero are negative criteria. They describe what the response should not do, and they subtract points when the grader finds them present. This is how a rubric benchmark scores harm instead of only rewarding good content. Positive criteria carry the other half of the problem, which is what the response failed to say: the First, Do NOHARM v1 study found that errors of omission caused more than 80 percent of its severe errors.
The scoring formula
For a single example, HealthBench divides the points the response earned by the maximum positive points that example offered. Because negative criteria subtract, one example can score below zero. The overall benchmark score is the mean across examples, clipped to the range 0 to 1, which is why a published score never goes negative even though individual cases can.
Two adjustments sit on top of that formula. Length adjustment penalizes verbosity at about 2.99 points per 500 characters beyond 2,000 on HealthBench and about 1.47 on HealthBench Professional; adjusted and unadjusted numbers are separate measurements and should not be compared with each other. Run variability is small at the aggregate level: repeating the full HealthBench run 16 times moved the overall score by a standard deviation of about 0.002. A small gap between two rows is rarely sampling noise, but it is often a difference in configuration, which is the subject of reading a leaderboard.
What a rubric score is worth
A rubric score is not the percentage of the answer that was correct. It is the share of available rubric points the response collected, and rubrics are written to include points most responses will miss. The physician baseline makes this concrete: on HealthBench Professional, physician-written responses score 0.437 against the same rubrics. On the parent HealthBench, physicians writing without model help scored 0.13, and physicians writing with April 2025 model responses in front of them scored 0.48.
Scores also do not transfer between benchmarks. Health Optimization Bench reports rubric credit from 0 to 100, grades each task blind with a panel of three independent model families, and publishes 95 percent bootstrap confidence intervals, so its numbers and HealthBench numbers describe different tasks, rubrics, and graders. Current standings live at clinicalbenchmarks.ai, with dedicated boards at healthbenchprofessional.com, healthbenchhard.ai, and healthoptimizationbench.com.
can a rubric benchmark score be negative?
On a single example, yes. HealthBench divides the points a response earned by the maximum positive points that example offered, and negative criteria subtract points for content that should not be there, so one case can finish below zero. The published overall score is the mean across all examples clipped to the range 0 to 1, so the headline number itself never goes negative.
what does a rubric score of 0.5 mean in a health AI benchmark?
It means the response collected about half the positive rubric points available, not that half the answer was wrong. Rubrics are written to include points most responses miss, so the useful reference is the human baseline: physician-written responses score 0.437 against the HealthBench Professional rubrics, and on the parent HealthBench physicians writing unaided scored 0.13. Scores are only comparable within one benchmark, one grader, and one scoring variant.
Keep going: model graders in health ai evaluations, how to read an ai health leaderboard. All the mechanics pages are on the front page.