The short answer
Picture a vendor offering a free trial of an AI tool on your real patient charts. A safer first step is a pilot on made-up patients, where a wrong write harms no one and every mistake can be counted. Measure what the tool did, not patient outcomes, and go live only when the exit checks pass.
A synthetic-data pilot tests an AI tool’s writes, approvals and records on made-up patients before it goes near a real chart. Generate synthetic FHIR records, point the tool at a sandbox holding only those, and measure operational things: how often what the tool said it wrote differs from what was stored, how many writes waited for a person, how long approvals took, and staff hours. Don’t measure patient outcomes; a synthetic pilot can’t show them, and claiming it does is misleading. Move to real data only when your exit checks pass and IT has reviewed the setup.
Deedglass
How Deedglass keeps an AI agent in check
-
1Before it runs
Your rules decide
- Read a recordAllowed
- Delete a tableNever
- Change a medicationNeeds a person
-
2If it’s risky
A named person approves
NSDr. N. ShahApproved in SlackDeclineApprove -
3After it runs
Checks what really changed
MatchesRecord sealedCheck it offline
Why start synthetic
Three reasons, all practical.
No patient data at risk. A pilot on synthetic records needs no data-sharing agreement, no consent question and no breach exposure. That removes weeks of approvals from the start.
You can break things on purpose. Synthetic data lets you send malformed writes, stale updates and references to missing records and see what happens. In our October 2026 FHIR study, 29 of 72 deliberately invalid writes got a success reply across four server setups.4 You want to find that in a sandbox, not in a ward.
IT can review the real setup. The architecture, logs and approval flow are the same ones that will later carry real data. Only the patients are fake.
Where the data comes from
The most widely used open generator is Synthea, which describes itself as a synthetic patient population simulator that outputs “synthetic, realistic (but not real), patient data”. It exports HL7 FHIR R4 (and STU3 and DSTU2), with C-CDA, CSV and Bulk FHIR ndjson available, under the Apache 2.0 licence.1 Load its R4 output into a FHIR sandbox that holds nothing else.
Two cautions. Synthetic patients are statistically plausible, not your population, so don’t read anything clinical into results. And check that no real identifiers leak in through test fixtures, copied templates or a “quick” import from production.
A four-week plan
| Week | Do | Output |
|---|---|---|
| 0 | Agree scope: which writes, which resources, who approves | One-page scope |
| 1 | Load synthetic records; connect the tool to the sandbox only | Working sandbox |
| 2 | Run normal tasks; read back every write | Mismatch log |
| 3 | Run hard cases: invalid data, conflicts, high-risk orders | Refusal and hold log |
| 4 | IT review; compare results with exit criteria | Go, fix, or stop |
Week 0 matters most. Write down which resource types the tool may create or update, which writes need a person, and who those people are. Without that, you can’t tell a correct hold from a bug.
In weeks 2 and 3, read back every write and compare it with what the tool reported. A FHIR 201 means the server accepted the request, not that it stored what was meant; the write integrity study shows how often those differ by server.
What to measure, and what not to
| Measure | Why | Don’t substitute |
|---|---|---|
| Writes where reported and stored differ | The core integrity signal | “Tool reported success” |
| Writes held for a person, and why | Shows the rules work | Total writes alone |
| Approval time, median and longest | Shows the cost to staff | Anecdotes |
| Refusals by reason | Shows what the server or rules stopped | Error counts without reasons |
| Staff hours spent on review | The honest cost | Vendor estimates |
| Records that verify after the fact | Shows the audit trail holds | “We have logs” |
We follow one rule in our own pilots: reports count data mismatches and staff hours, never patient outcomes.2 A synthetic pilot has no patients to have outcomes. Even a later real-data pilot is too small and too short to say anything clinical, so don’t let a slide imply otherwise.
The IT review pack
Hospital IT will ask the same questions every time, so answer them in writing before they ask. A useful pack covers architecture and data flow, where data lives, the threat model in summary, answers to the usual security questions, the clinical-safety boundary, operations, and how the pilot is measured.2 That’s the outline of the pack our own pilot produces, and it works as a checklist whatever tool you pilot.
Be plain about gaps. For example, we state that we don’t yet have a SOC 2 report or an ISO 27001 audit, single sign-on, or a place on ABDM’s list of approved integrators.2 A reviewer who finds a gap you didn’t mention stops trusting the rest of the pack.
Exit criteria
Agree these in week 0 and don’t move them in week 4:
- Every write the tool made was read back, and every mismatch has an explanation.
- Every high-risk write was held for a named person, and none ran without approval.
- Approval time is something the ward can live with.
- Refusals happened for the right reasons and were visible to the people who needed to see them.
- Records of what happened can be checked by someone who didn’t run the pilot.
- IT has signed off the review pack, including the gaps.
If any fails, fix and rerun that part on synthetic data. Don’t fix it on real patients.
Where synthetic data stops helping
Synthetic data can’t show how real staff behave under time pressure, how messy your real records are, or how your production HIS is configured. It can show whether the tool’s writes, holds and records behave as designed. Treat the synthetic pilot as the gate before a small, supervised real-data pilot, not as a replacement for one.
Questions people ask
Is synthetic data safe to share with a vendor?
Data generated by a tool like Synthea isn’t about real people.1 The risk is real data slipping in through fixtures or imports, so check the sandbox holds only generated records.
How long should a synthetic pilot take?
The plan above takes four weeks, with week 0 for scope. The shortest useful version is an afternoon: one run of normal tasks and one of hard cases, with every write read back.
Should we measure accuracy of the AI’s output?
Measure whether what it wrote matches what was stored, and whether high-risk writes were held. Judging clinical correctness is a different study, and not one a synthetic pilot can do.
Try it. npx deedglass health pilot runs a 15-minute version on synthetic FHIR R4 data on your own machine, with no Docker, server or account.2 Its four scenes include a 201 whose stored value differs and a high-risk order held for a clinician2, the FHIR store stays in memory with synthetic records only3, and --print-review-pack prints the IT pack as Markdown.3 More on the hospital page and in the record spec. To try approvals and read-backs on a real database in your browser, open the playground; to grade what an agent can reach on your own machine, run npx deedglass scan (the grade, explained). To see the same kind of actions in a simple app instead of a terminal, npx deedglass (0.15.0 and later) opens Deedglass in your browser on your own computer5, with what agents did, what waits for a person and the locked records.5
Written with AI assistance and checked against our claims register.