How CHI-Bench works
actAVA · 75 operations workflows · updated August 17, 2026
CHI-Bench covers the administrative side of US healthcare: prior authorization, utilization management, and care management. actAVA released it on May 20, 2026 and last updated it on August 12, 2026 across 45 harness configurations. The tasks are long, 60 to 80 steps each, and the agent works through application surfaces rather than answering in prose. The launch report led with reliability rather than capability, which tells you what the benchmark was built to expose.
| publisher | actAVA.ai |
|---|---|
| released | 2026-05 |
| size | 75 workflows (25 per domain), 21 healthcare applications, 200+ MCP tools |
| scale | pass@1 with binary 0/1 reward, higher better |
How a workflow is set up
There are 75 workflows, 25 in each of the three domains. The agent works across 21 healthcare applications through more than 200 MCP tools, and each workflow runs 60 to 80 steps across 4 to 6 stages.
Length is the design choice that matters. A long horizon task forces the agent to carry state: what it found in stage one determines what counts as correct in stage four, and an early mistake propagates instead of being scored in isolation.
Two graders on one task
Grading has a deterministic half and a judged half. Unit tests check the mechanical outcome, such as whether the right record changed and whether the required action was taken. An LLM judge checks what a unit test cannot express: whether a conclusion is grounded in evidence the agent actually retrieved, whether consent was handled, and whether the stages stay consistent with one another.
The metric is pass@1 with a binary 0 or 1 reward. A workflow either passes in full or scores nothing. There is no partial credit for getting 50 of 70 steps right.
Reliability is the finding
At launch, actAVA ran identical workflows three times and reported that no agent stayed above 20 percent across all three runs. Read that as a variance result: agents that can do the work sometimes cannot do it dependably. The other launch finding was that harness choice mattered as much as model choice, which is why the August 2026 update spans 45 harness configurations. The board also accepts community submissions, so rows mix results actAVA ran with results other people sent in. Check a row's provenance before comparing it to another, and see /benchmarks/healthagentbench for a different task shape.
Limits
Binary pass@1 is honest about end to end success and blind to everything else: a workflow that fails on the last step looks identical to one that fails on the first. The LLM judge puts a model inside the scoring loop for the grounding and consent checks, so those judgments carry model error of their own. Mixed provenance means two rows may not have been produced under the same conditions. The workflows also encode US payer and provider administrative rules, so a score does not transfer to other health systems.
Current CHI-Bench standings are kept on the results index at clinicalbenchmarks.ai.
The grading machinery behind pages like this one is covered in the rubric grading and model graders guides.