FHIR writes that said success: 11 cases where the server stored something else

The short answer

A FHIR write can return 200 or 201 and still store something other than what your agent meant. In 17 synthetic scenarios on one real server, Medplum 5.2.6, 11 did exactly that.¹ A check that trusts the status code caught none of them.¹ To catch more, read the write back at the version the server reported, send If-Match on updates, and count versions in the server’s history.

That is the whole lesson. The rest of this post is the detail, starting with the cases our own gate missed, because those are the ones worth planning for.

Every scenario uses synthetic data, and none of it judges clinical content. The benchmark only asks one question: did the server store what the agent says it stored? Each scenario rests on standard FHIR R4 behaviour, RFC 6902 or Medplum’s own documentation, with the sources cited.² What the server stored is read from Medplum’s own version tables, never from its answers or from the tool being tested.²

The seven we missed

We build Deedglass, a gate that sits between an agent and the systems it writes to, and we wrote this benchmark. That is a conflict of interest, so the misses come first. Deedglass missed 7 of the 11 scenarios where the agent’s report and the server disagreed.¹

The table follows the scenario descriptions in the committed results.¹

Why the misses happen tells you what to fix. Deedglass reads each write back at the version the server reported and compares the fields the agent sent, one by one.⁷ That check is good at one question: is the resource I wrote the way I wrote it? Several of these failures pass it.

• The id, the patch and the missing patient. The server stored exactly what it was sent, at an id it chose, or with an array the patch was entitled to replace, or with a reference nobody checked. A read-back of that version agrees with the write. The problem is the agent’s belief about the write, not the write.
• The lost update. The agent’s version is correct. What’s wrong is the version before it, written by someone else, which the agent overwrote without reading. A read-back of your own version can’t see a version you never knew about.
• The retry. Each create, checked alone, matches. Deedglass did report two creates, which is what the server stored. It just didn’t flag that the agent only meant one.¹
• The conditional update and the transaction. Deedglass’s FHIR connector takes one resource on one type and id per call. These two went through its general REST connector instead, which records the call and the server’s answer but reads nothing back.²

Two of these depend on project settings. Medplum processes a transaction Bundle as a batch unless the project turns on its transaction-bundles feature, with no error or warning.⁵ In the run, a Bundle with one invalid entry was answered 200 and two of its three Observations were stored.⁵ A reference to a missing resource is refused only when the project has reference checks on write turned on.⁹

The four we caught

Deedglass caught 4 of the 11: three after the write and one before it.¹

• A conditional create that stored nothing. Under FHIR R4, a conditional create that matches one existing resource is answered 200 and creates nothing.⁴ The agent reported a new Observation. The read-back saw the existing one and said so.
• A read-only note dropped from a 201. Medplum access policies can mark fields read-only.⁶ On Medplum 5.2.6, a create that set one was answered 201 and stored without it. That drop is what we saw on that version; Medplum’s documentation defines read-only fields but doesn’t describe the write.² The field-by-field comparison named the missing note.
• A replaced author. Medplum records the writing client as meta.author.⁷ The agent set a clinician as author and the server stored its own client instead. The read-back flagged the difference.
• A delete, refused before it ran. The healthcare-fhir policy pack refuses deletes of clinical resources, so this one never reached the server.¹ Its read-back alone would not have helped: a delete’s check only confirms the resource now reads as gone, which was true, while every earlier version stayed readable in history.²

Deedglass raised no false alarms on the 6 controls, where the agent’s report was right.¹ Overall, it reported what the server stored in 10 of 17 scenarios; the agent’s own report was right in 6 of 17.¹

A checklist for FHIR writes

None of this needs our software. If an agent, or any client, writes to a FHIR server, these checks cover the scenarios above.

1. Read back at the version the server reported. FHIR’s vread reads a given version.⁴ Compare the fields you sent, after dropping what the server adds on its own.
2. Treat 200 on a conditional create as “nothing new stored”. It is how the standard says “a match already existed”.⁴
3. Send If-Match with the version you read on every update. If the version has moved, the server answers 412 and updates nothing.⁴ Without it, the last writer wins silently.
4. Use conditional create for anything you might retry. Plain create stores a new resource every time it runs. A timeout followed by a retry is a duplicate waiting to happen.²
5. Append to arrays with /-, never add on the array itself. In JSON Patch, add on an existing member replaces its value.⁸
6. Check every entry of a Bundle response, and know your server’s transaction mode. A 200 for the Bundle says nothing about each entry.⁵
7. Turn on reference checks on write, where your server has them.⁹
8. Count versions, not answers. Overwrites, duplicates and partial Bundles show up in the server’s history, not in the resource you wrote. _history reads a resource’s versions.⁴

Items 3 to 7 stop the failure. Items 1, 2 and 8 notice it after the fact. You want both kinds.

Run it yourself

The benchmark is in the Deedglass repository. make fhir-bench reruns it in Docker, with Medplum’s database in memory, and CI replays the committed run without Docker.² We ran the replay against the v0.9.0 tag today:

Code What the screen or file shows, copied as is.

$ go test ./internal/corpus/fhirbench -run TestGoldenReplay -v -count=1
=== RUN   TestGoldenReplay
--- PASS: TestGoldenReplay (0.02s)
PASS
ok  	github.com/deedglass/deedglass/internal/corpus/fhirbench	0.437s

$ jq -c '.totals[] | {subject, caught, missed, false_alarms, true_effect_reported, checkable_record}' corpus/fhir/results/latest.json
{"subject":"none","caught":0,"missed":11,"false_alarms":0,"true_effect_reported":6,"checkable_record":0}
{"subject":"http-status","caught":0,"missed":11,"false_alarms":0,"true_effect_reported":6,"checkable_record":0}
{"subject":"deedglass","caught":4,"missed":7,"false_alarms":0,"true_effect_reported":10,"checkable_record":17}

The last column is the third score. Deedglass left a record a third party can check offline with deedglass verify for all 17 scenarios.¹ The benchmark also takes any checker as a command beside the baseline, so you can score your own the same way.² The plain-language version, with every scenario, is on the FHIR benchmark page, and the Postgres equivalent is here.

What this doesn’t show

• One server, one version, one set of settings. Medplum 5.2.6 only. Another server, or the same server configured differently, behaves differently. Two pairs of scenarios exist to show exactly that.²
• Seventeen small scenarios. One or two resources each. Real systems add client-library retries, subscriptions and several agents writing at once.
• The agent’s report is fixed text. It scores tools, not models.
• No clinical judgement. A write that stored exactly what was sent can still be the wrong thing to write. Rules and a person’s approval decide that before it runs; a read-back reports what happened after.
• We wrote it. The method was written down before the results, the misses are in it on purpose, and anyone can rerun it or send a scenario Deedglass gets wrong.²

If you want the idea behind the read-back in one page, see the read-back check. If your agent is about to write to a patient record, start with what to put around it first. The same three evidence levels run on Postgres in the playground, and npx deedglass scan --postgres-dsn-env DATABASE_URL grades a database your agent can reach (the grade, explained).

Written with AI assistance and checked against our claims register.