How EHR-Complex works
academic team · 3,915-task test set · updated August 17, 2026
EHR-Complex tests whether an agent can get a correct answer out of a hospital database by writing and running code. An academic team published it in June 2026 (arXiv 2606.23301). The tasks sit on MIMIC-IV: 31 tables, more than 500 million records, and about 365,000 patients. The suite holds roughly 52,000 tasks, with a 3,915-task test set behind the reported results.
| publisher | Academic team (Qiao et al., Ant Group-affiliated; arXiv 2606.23301) |
|---|---|
| released | 2026-06 |
| size | ~52,000 tasks (3,915-task test set) over 365K patients, 31 tables, 500M+ records |
| scale | exact-match accuracy, 0-1, higher better |
What the tasks ask for
Tasks span six clinical intents and are posed at two levels. Patient-level questions concern a single record. Population-level questions require aggregation across the database.
Many answers depend on temporal evidence paths, meaning the answer follows from the order and timing of events rather than from one stored value. That is what makes the tasks hard. The agent has to work out which tables hold the evidence, join them correctly, and respect the sequence in which things happened.
How agents answer and how answers are scored
The agent writes SQL and Python, executes it against the database, and returns an answer. Scoring is exact match on a scale of 0 to 1. There is no rubric and no judge model, so scores do not drift with a grader's behavior.
Exact match is strict in both directions. A nearly correct answer earns nothing. A correct answer reached by a badly reasoned query earns full credit. The metric checks the result, not the route.
Repeated runs and Pass^k
The paper reports repeated runs of the same tasks. Consistency drops below 50 percent at Pass^4 for nearly every model, where Pass^4 means the same task solved correctly in four consecutive attempts.
That pattern shows up across agentic health benchmarks. CHI-Bench reported at launch that no agent stayed above 20 percent across three identical runs. A single-run accuracy figure will overstate how a system behaves in repeated use.
Limits
Results come from the paper rather than a maintained board, so there is no refresh cadence and no rows for models released afterward. The published rows also mix two setups, a headline 12-model evaluation and human-validated configurations, labeled in the config column, so comparisons hold only within a configuration. And everything sits on one research database. Fluency with the MIMIC-IV schema is not the same as fluency with the schema of a live hospital system carrying local customization.
Current EHR-Complex figures sit at clinicalbenchmarks.ai, alongside the other agentic health boards.
The grading machinery behind pages like this one is covered in the rubric grading and model graders guides.