Benchmark saturation and what replaces it

How a benchmark saturates

The first kind of saturation is visible on the page. When the top scores sit near the maximum, the headroom left is smaller than the differences you are trying to detect, and small changes in the test set move the ranking more than ability does. MedScribe is close to this: its top scores cluster near 90 on a 0 to 100 scale, and the board does not always display decimals below first place.

The second kind is harder to see, because the scores still look unremarkable. Rubric benchmarks are graded by a model, and that grader disagrees with physicians some of the time. HealthBench selected GPT-4.1 as its default grader through a meta-evaluation against physician grades, where it reached macro-F1 0.709, ahead of o4-mini at 0.692, o3 at 0.681, GPT-4.1-mini at 0.661 and GPT-4.1-nano at 0.580. It exceeds the average physician's agreement in 5 of 7 themes. Agreement that good is still not exact agreement, and once the gap between two models is smaller than the grader's own error, their order carries no information about the models.

Exam-style sets saturated first

MedQA and the MultiMedQA family are multiple-choice exam questions. Frontier models passed 95 percent on them by 2025, and the trackers that carried them archived them. The Open Medical-LLM Leaderboard on Hugging Face was built on those sets and has taken no frontier submissions in 2026.

Part of that saturation was a measurement artifact. A multiple-choice item supplies the answer set, which removes the failure mode clinical work is most exposed to: leaving something out. First, Do NOHARM found potential for severe harm in up to 24.6 percent of directly applied consultation recommendations in its v1 study, with errors of omission behind more than 80 percent of the severe cases. An exam item cannot score an omission when the options are printed on the page.

Rubric sets last longer, then reach the ceiling too

HealthBench replaced the exam format with 5,000 multi-turn conversations graded against 48,562 criteria written by 262 physicians. Criteria carry weights from -10 to +10, and negative criteria subtract points for harmful content, so a response can lose points it never had a chance to earn. Partial credit of this kind spreads scores out and delays saturation.

It delayed saturation rather than preventing it. The HealthBench Consensus subset became near-saturated, and frontier runs stopped reporting it separately. OpenAI has since said the parent set is approaching a noise ceiling for frontier models, and points to HealthBench Professional for continued measurement.

Hard subsets and successor benchmarks

The cheap response is to keep the corpus and select the hard part of it. HealthBench Hard was built by having five frontier models of the time score every conversation in the parent set, removing the roughly 1.5 percent where no model earned any positive credit, and keeping the 1,000 lowest-average conversations. The rubrics did not change. At the May 2025 release, the best score on Hard was o3 at 0.320 against 0.60 for the same model on the full set, and three of the older models in that comparison scored 0.00.

A hard subset inherits its difficulty from the models that built it, so it ages. HealthBench Professional is the rebuild: 525 tasks selected by physicians out of 15,079 real clinician conversations, difficulty enriched about 3.5 times, roughly a third of them adversarial, with 190 physicians across 50 countries and a three-stage rubric adjudication that puts at least three physicians on each criterion. Physician-written responses score 0.437 on those rubrics, which tells you how much headroom the set still has.

The other response is to change the axis rather than the difficulty. Agentic boards ask whether a system can complete a workflow instead of whether it can answer a question. Safety boards score harm rather than credit. Health Optimization Bench targets freshness, with tasks written against primary sources and graded blind by model families that did not author them. Composites such as MAST bundle several of these into one number, and that board is currently marked as a preview.

The per-benchmark pages for HealthBench Hard and HealthBench Professional cover construction in more detail, and the dedicated boards at healthbenchhard.ai and healthbenchprofessional.com carry current standings. If you want to run one of these sets yourself, see running your own eval.

is MedQA still a useful benchmark

MedQA and the exam-style medical question sets around it saturated above 95 percent by 2025 and were archived by most trackers, including the Open Medical-LLM Leaderboard on Hugging Face, which has taken no frontier submissions in 2026. The format also hides the failure that matters most in clinical work, because a multiple-choice item cannot score an omission when the options are supplied. Current measurement has moved to rubric-graded conversations and to agentic and safety benchmarks.

what is a benchmark noise ceiling

A noise ceiling is the point where differences between models are smaller than the error in the measurement, so the ranking stops carrying information even though the scores are well below the maximum. On rubric benchmarks the main source of that error is the grader. HealthBench's default grader, GPT-4.1, reached macro-F1 0.709 against physician grades in the selection meta-evaluation and beat the average physician's agreement in 5 of 7 themes, which is close but not exact. OpenAI has said the parent HealthBench is approaching a noise ceiling for frontier models.