How often do LLMs hallucinate in clinical notes? Two studies, two very different numbers

The short answer

Published error rates for LLM-generated clinical notes vary by more than an order of magnitude, because studies count different things. A 2025 study in npj Digital Medicine, by a company that builds such tools, found that 1.47% of 12,999 clinician-annotated sentences contained a hallucination and that 3.45% of transcript sentences were omitted.¹ A 2024 study in the Journal of Medical Internet Research found an average of 23.6 errors per simulated case, mostly omissions, and concluded the notes didn’t meet the standard for clinical use.²

Neither number is “the” hallucination rate. Each measures a specific pipeline, model, prompt and denominator. What they agree on is more useful: omissions are more common than inventions, errors vary between runs of the same case, and every draft needs a person to check it before it enters the record.

This post is about measurement and records, not clinical judgement.

Study 1: per-sentence rates

Asgari and colleagues built a framework for finding and grading errors in generated notes: a taxonomy, an iterative experiment setup, a safety framework for grading harm, and a review tool. All authors worked for Tortus AI, which makes clinical documentation software, as the paper’s conflict-of-interest statement says.¹

Their numbers, across 18 experimental configurations:¹

Hallucinations broke down as 43% fabrication, 30% negation (stating the opposite), 17% contextual and 10% causality errors.¹ By refining prompts and workflows, the authors report reducing major errors below previously reported rates for human note-taking.¹

Study 2: errors per case

Kernberg, Gold and Mohan at Oregon Health and Science University took simulated outpatient encounters, generated three SOAP notes for each from the transcript, and compared each note with the known facts of the case.²

Accuracy fell as transcripts got longer and as cases had more facts to capture.²

Why the numbers differ

Put the two side by side and the gap looks like a contradiction. It isn’t.

A 1.47% per-sentence rate, with about 29 sentences per note in that study (12,999 across 450), works out to roughly four hallucinated sentences in every ten notes. Twenty-three errors per case, most of them missing facts, describes a different system measured a different way. Both are informative; neither transfers to your pipeline without measuring it.

The second study’s replicate finding deserves attention: only about half the data elements were right in all three versions of the same note.² That’s the same pattern τ-bench measures for customer-service agents with its pass^k metric: success once doesn’t mean success every time.³

Omissions are the bigger category

In both studies, leaving things out was more common than making things up: 3.45% of transcript sentences omitted against 1.47% of note sentences hallucinated in the first, and 86% of errors as omissions in the second.¹²

Omissions are also harder to catch in review. A reviewer reading a fluent note can spot a statement that seems wrong; noticing what isn’t there requires comparing against the source. Any review step should make the source easy to check, not just the draft. In practice that means showing the transcript beside the draft, highlighting which parts of the source the draft used, and making it quick to jump from a sentence in the note to the passage it came from. A review screen that shows only the polished note invites approval of exactly the errors these studies found most often.

Questions to ask of any rate

When a vendor, paper or internal pilot reports an error rate for generated notes:

1. Per what? Sentence, note, case or fact. Convert before comparing.
2. Hallucinations only, or omissions too? A low hallucination rate says nothing about missing information.
3. Graded by whom, against what? Clinicians against transcripts, or a model against a summary?
4. How severe? What share would change care if uncorrected? Study 1 graded 44% of hallucinations as major.¹
5. Same pipeline as yours? Model, prompt, workflow and specialty all change the number.
6. Who ran the study? Vendor studies are valuable and should say so, as this one did.
7. How variable? Run the same case several times.

What a documentation system should record

Because errors happen at some rate in every system, the records around a generated note matter as much as the model. For each note, a system should be able to show:

• The source: which transcript or recording the draft came from, by reference.
• The draft: exactly what the model produced, before any edit.
• The edits: what the clinician changed, so error rates can be measured in your own setting.
• The approver and time: who accepted it into the record, and when.
• What was written: the version id the record system stored, read back to confirm it matches what was approved.

Drafts that are approved without edits, every time, in seconds, are worth looking at. Either the system is excellent or review has become a click. Records can tell you which.

Our posts on AI agents writing to patient records and HIPAA audit controls go into the record design, and FHIR AuditEvent vs Provenance shows how to attach “drafted by an AI, approved by this clinician” to the data itself. The human-in-the-loop approval entry covers the approval step. To see checks and approvals on a real database in your browser, try the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.