How to Read an AI Health Leaderboard

Read the configuration before the rank

The same model produces different scores on the same benchmark depending on how it was run. On HealthBench, adjusted and unadjusted scores are separate columns: length adjustment removes about 2.99 points per 500 characters beyond 2,000 characters, and about 1.47 on HealthBench Professional. A verbose model loses more under one column than the other, so a rank can move without the model changing at all.

On agentic boards, the row is often not a bare model. HealthAgentBench evaluates agent harnesses such as Claude Code, Codex, and Copilot end to end rather than models on their own. CHI-Bench published 45 harness configurations in its August 12, 2026 update and reported that harness choice matters as much as model choice. If a board has a configuration column, read it before you read the rank.

Vendor-reported, independent, or official

There are three provenances. Vendor-reported means the model's maker ran the evaluation under its own protocol; the current MedXpertQA multimodal frontier table is vendor-reported from Meta's Muse Spark launch evaluation and mirrored display-only elsewhere. Independent means a third party ran every model itself, which is what Vals AI does for MedCode and MedScribe. Official leaderboard means the benchmark's maintainers run it, as Stanford CRFM does for MedHELM. Boards marked mixed contain rows of more than one kind.

Vendor numbers are not wrong, but they use the vendor's configuration and often differ by points from independent runs of the same benchmark. Some vendor evaluations are not cross-vendor at all: OpenAI's dynamic mental health evaluations cover OpenAI models only, are not independently runnable, and come with OpenAI's note that the error rates are not representative of average production traffic. Boards that accept community submissions, such as CHI-Bench, mix author-run and submitted rows for the same reason.

Field size and dates

Some metrics are relative to the field. MedHELM ranks by mean win rate, computed against the set of models evaluated, so a model's score moves when the cohort changes even if its behavior does not. That board is at version 5.0.0, last updated May 14, 2026, on a roughly quarterly cadence.

Dates decide what is missing. MedHELM has no Claude 5 family or GPT-5.6 rows. WHBench froze its model set in March 2026. HealthAdminBench has no GPT-5.6 or Claude 5 rows and states no refresh cadence. MAST is marked preview, was last updated August 15, 2026, and publishes component breakdowns only for First, Do NOHARM v2. An absent model is an unrun model, not a weak one.

Ceilings and repeat runs

Near the top of a saturated benchmark, differences stop meaning much. Exam-style medical question sets passed 95 percent by 2025 and were retired by most trackers. MedScribe's top scores cluster near 90. OpenAI has said the parent HealthBench is approaching a noise ceiling for frontier models and points to HealthBench Professional for continued measurement. Difficulty selection is the usual answer: HealthBench Hard took the 1,000 conversations where frontier models scored worst, and at its release in May 2025 the top score on it was o3's 0.320, against 0.60 for the same model on the full set.

Single-run scores also hide reliability. Pass^k metrics require k consecutive successes on the same task, and the drop is steep: EHR-Complex reports consistency below 50 percent at Pass^4 for nearly every model, and CHI-Bench's launch reported no agent above 20 percent across three identical runs. A pass@1 headline and a Pass^3 number answer different questions, and for deployment the second one usually matters more. For the mechanics behind rubric numbers, see rubric grading and model graders; current standings across boards are at clinicalbenchmarks.ai.

what does vendor-reported mean on an AI benchmark leaderboard?

It means the model's own maker ran the benchmark and published the result under its own configuration, rather than a third party running every model the same way. The vendor chooses the reasoning effort, the grader, and the scoring variant, so vendor numbers often differ by points from independent runs of the same benchmark. Treat them as evidence about the model under that configuration, not as a like-for-like comparison against a row someone else produced.

why do the same model's health benchmark scores differ between sites?

Usually configuration rather than error. The grader model and its effort setting, the reasoning effort of the model being tested, whether the score is length-adjusted, the agent harness in agentic evaluations, and the version of the task set all change the number. Field size matters too: a metric like MedHELM's mean win rate is computed relative to the cohort evaluated, so adding or removing models shifts scores without any model behaving differently.