Running your own rubric health evaluation

Check what is actually released

Health benchmarks differ in how much of themselves they publish. HealthBench was released in May 2025 with a paper, arXiv 2505.08775. HealthBench Hard is a subset of the same corpus graded on the same physician-written rubrics, so whatever you build for one applies to the other. Health Optimization Bench describes an 89-task v1 release set covering incretin therapeutics evidence.

Others are not runnable in the same sense. PhysicianBench and EHR-Complex report their scores in papers with no standalone public leaderboard. The current frontier table for MedXpertQA MM is vendor-reported rather than independently run. OpenAI's dynamic mental health evaluations are internal, cover OpenAI models only, and are not independently runnable at all.

Before you plan around any of this, confirm the license, the file format and the harness with the publisher directly. This guide tracks what benchmarks measure rather than the state of their repositories, and release terms change.

Match the default grader configuration

The grader is part of the benchmark definition, not an implementation detail. HealthBench's default grader is GPT-4.1, selected by meta-evaluation against physician grades, where it reached macro-F1 0.709 ahead of o4-mini at 0.692 and o3 at 0.681. HealthBench Professional uses a different default: GPT-5.4 at low reasoning effort, with a length adjustment applied.

A newer or cheaper grader will give you a different number, possibly a better-looking one, and it will not be comparable to any published score on that benchmark. If you substitute a grader, treat the result as a new benchmark and label it that way. Record the grader model, its reasoning effort, the grader prompt version, and whether length adjustment was switched on.

Scoring rules that move your number

HealthBench scores each example as earned rubric points divided by the maximum positive points available for that example. Because negative criteria subtract, an individual example can score below zero. The overall score is the mean across examples, clipped to the 0 to 1 range. Rubric size varies: a median of 11 criteria per conversation, ranging from 2 to 48, with weights from -10 to +10.

Length adjustment is the other rule that moves things. The published adjustment penalizes about 2.99 points per 500 characters beyond 2,000 on HealthBench, and 1.47 on HealthBench Professional. Adjusted and unadjusted scores are both legitimate and they are not interchangeable. OpenAI leads with length-adjusted numbers at maximum reasoning effort and reports its production settings separately, so state which variant you ran before anyone asks.

Why your numbers will differ, and what to report

Sampling noise does not explain most gaps on rubric sets. HealthBench's run-to-run standard deviation is about 0.002 over 16 repeats, so a difference of a point or more came from configuration rather than luck. The usual sources are the reasoning effort you ran at, the system prompt, how refusals are handled, and the API version you called.

Vendor-reported numbers use the vendor's own configuration and often differ by points from independent runs of the same benchmark. That is not evidence of anyone cheating. It is what happens when two setups are both defensible and different, and it is why mixing graders or configurations on one axis breaks comparability.

Report the grader and its settings, the subset and version you ran, how many repeats you averaged, and an uncertainty estimate. Health Optimization Bench publishes 95 percent bootstrap confidence intervals with every score, which is a pattern worth copying, and it grades blind across model families so that an authoring family never grades its own task. For an interpretable anchor, physician-written responses score 0.437 on the HealthBench Professional rubrics. For current standings, use clinicalbenchmarks.ai rather than a table in an older paper, and if you are running an agentic set, read consistency and Pass^k before you pick a trial count.

which grader model does HealthBench use

HealthBench's default grader is GPT-4.1, chosen by a meta-evaluation against physician grades where it reached macro-F1 0.709, ahead of o4-mini at 0.692 and o3 at 0.681. HealthBench Professional uses a different default: GPT-5.4 at low reasoning effort, with a length adjustment. A score produced with a substituted grader is not comparable to a published one, so record the grader model and its settings next to any number you report.

why does my HealthBench score not match the published number

Run-to-run noise is small on HealthBench, about 0.002 standard deviation over 16 repeats, so a meaningful gap almost always comes from configuration rather than sampling. Common causes are a different grader model or reasoning effort, length-adjusted versus unadjusted scoring, a different system prompt, and differences in how refusals are handled. Vendor-reported numbers in particular use the vendor's own configuration and often differ by points from independent runs of the same set.