The short answer
MedAgentBench, from a Stanford team including James Zou, Andrew Ng and Jonathan Chen, is a virtual electronic health record that agents use through FHIR. It has 100 patient profiles with over 700,000 data elements and 300 tasks from 10 categories written by physicians. Half the tasks are queries, answered with FHIR reads; half are actions, which need FHIR writes.1
The best model in the paper, Claude 3.5 Sonnet v2, scored 69.67% overall: 85.33% on queries but 54.00% on actions.1 That gap, about 31 points, is the most useful number for anyone building agents on clinical systems. Reading went well. Writing failed nearly half the time, and writes are the part that changes the record.
This post is about system behaviour and safeguards, not clinical judgement.
Deedglass
The server said “saved”. Deedglass looked.
Deedglass reads it back
The setup
Agents got nine FHIR-style functions, for searching conditions, creating vital-sign records, creating medication requests and so on, and could send a GET, send a POST, or finish, for up to 8 rounds per task.1 The environment follows the FHIR standard and uses standard EHR APIs, so the authors note it could be migrated to live systems.1
| Task type | Count | FHIR interaction | Graded by |
|---|---|---|---|
| Query | 150 | GET | Answer vs reference answer |
| Action | 150 | POST, often after GETs | Rule-based checks on the POST payload |
Invalid actions, or running out of rounds, count as failures, and results are pass@1: one try per task.1
Reads vs writes
| Model result | Overall | Query | Action |
|---|---|---|---|
| Claude 3.5 Sonnet v2 (best) | 69.67% | 85.33% | 54.00% |
Most models in the study did better on queries than actions; the paper names Gemini 1.5 Pro and Qwen2.5 as exceptions.1 Writing is harder for several reasons the task design makes visible: the agent must first read the right context, then build a correctly structured resource with the right references, codes and values, in one go, within the round limit.
How agents failed
The paper’s error analysis points to mundane causes more than clinical ones:1
- Invalid actions. Requests that didn’t follow the required format; one model, Gemini 2.0 Flash, produced invalid actions in 54% of cases.
- Format errors. Answering in a sentence where a bare value was expected.
- Malformed GET requests. Searches the server couldn’t use.
These are the same failure shapes general tool-use benchmarks find. They matter more here because the tool writes to a patient record.
They’re also the most preventable. A malformed request or a resource that doesn’t match its profile can be caught before it leaves the agent, by validating it against the FHIR definitions the server expects. That turns a silent failure, or worse, a wrong write that happens to be accepted, into an error the agent can see and fix. Validation doesn’t make a write correct, but it removes a whole class of failures that the benchmark found common, and it costs little compared with cleaning up a bad record afterwards.
What the benchmark checks, and doesn’t
MedAgentBench checks action tasks by inspecting the POST the agent sent.1 That tells you whether the agent asked for the right write. It doesn’t tell you what the server then stored, and those can differ. FHIR has several ways for a correct-looking request to store something else: a conditional create that matches an existing resource stores nothing new3, a server that processes a transaction as a batch can apply part of it4, and a temporary id a client sends on a POST is replaced by the server’s own.3 Our own FHIR benchmark found 11 of 17 synthetic writes where the server’s stored result differed from what the agent meant, and a check that trusts the status code caught none of them.44
The authors list their own limits: the benchmark doesn’t capture full real-world complexity such as coordination between teams, its patient profiles come from Stanford Hospital records, and it doesn’t test reliability across repeated action runs.1 That last point connects to τ-bench’s pass^k: an agent that gets a write right once may not get it right every time.2
What the gap means for safeguards
A system where reads succeed 85% of the time and writes 54% of the time should treat the two differently:
| Reads | Writes | |
|---|---|---|
| Access | Scoped, but broadly allowed | Narrow scopes per task |
| Before | No approval needed for routine reads | Rules on content; a person for anything risky |
| After | Show the source with the answer | Read back the stored version and compare |
| Record | Who read what, for whom | Request, decision, stored version, check result |
The “after” row for writes is where FHIR helps. A create returns a Location with the new version, and vread reads that exact version back, so a client can compare what was stored with what it sent.3 Our FHIR versioning post covers the mechanics, and SMART v2 scopes shows how to grant create without update or delete.
A checklist for FHIR agents
- Separate read and write tools, and grant write scopes only where a task needs them.
- Validate every write against the FHIR profile before sending it; invalid actions were a leading failure.1
- Require a person’s approval for writes to clinical resources until you’ve measured your agent’s write accuracy on your own tasks.
- Read back the stored version after each write and compare the fields sent.
- Run each write task several times in testing; one success isn’t reliability.2
- Record every request, decision and check in an audit trail; see what to put around agents writing to patient records.
For the integrity side in depth, see FHIR writes that said success, and the FHIR glossary entry for basics. To see writes checked against what a real database committed, in your browser, try the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).
Written with AI assistance and checked against our claims register.