How HealthAgentBench works
Microsoft Research · 54 agentic tasks · updated August 17, 2026
HealthAgentBench measures whether an agent can finish a healthcare task, not whether a model can answer a question. Microsoft Research released it in July 2026, documented in arXiv 2606.31179. The environments are terminal based and built from real clinical artifacts, and the rows on the board are agent harnesses such as Claude Code, Codex, and Copilot rather than bare models. That choice reflects how this work actually runs: a model plus tools plus scaffolding.
| publisher | Microsoft Research |
|---|---|
| released | 2026-07 |
| size | 54 tasks across 7 environments; 162 trials (3 attempts per task) |
| scale | mean task success rate, 0-100%, higher better; cost per task also reported |
Seven environments, 54 tasks
The benchmark defines 7 environments with 54 tasks spread across them. An environment is a working setup assembled from real clinical artifacts, and the agent operates inside it through a terminal, reading files, running commands, and changing state until the task is done or it stops. Nothing is multiple choice. Either the end state matches what the task asked for or it does not.
Three attempts per task
Every task is attempted three times, which gives 162 trials. The reported metric is mean task success rate on a 0 to 100 percent scale. The repetition is the part worth noticing: a single attempt cannot separate a capable agent from a lucky one, and agent runs vary enough for that to matter.
The board also reports cost per task, so you can read success against spend. Two harnesses at the same success rate are not equivalent if one spends several times more to get there.
What a harness row means
When the row is a harness, the score covers everything inside it: the model, the prompts, the tool definitions, and the retry behavior. Change the model underneath and the number changes. Change the harness version and it changes again. A HealthAgentBench result is a statement about a configuration on a date, not a property of a model you can carry to another board. Microsoft Research runs the evaluation and includes its own harness among the rows, which is worth knowing when you weigh the source. For an agentic board built on a different task shape, compare /benchmarks/chi-bench.
Limits
54 tasks is a small set, and three attempts gives a coarse read on variance rather than a tight one. Harnesses ship changes often, so a row ages faster than a model benchmark row does. Scoring is per task, so an agent that gets most of the way through earns the same zero as one that never started. Terminal based environments suit some agents better than others and are not how most clinical staff use software, so success here is evidence about tool use rather than about fit at the bedside. Cost per task also moves with vendor pricing, independent of capability.
For who currently sits where on HealthAgentBench, see the results index at clinicalbenchmarks.ai.
The grading machinery behind pages like this one is covered in the rubric grading and model graders guides.