How accurate are LLMs at medical coding? Three studies, and what to build around them

The short answer

General-purpose LLMs are inconsistent medical coders. In a 2026 study of CPT codes from 191 operative notes, ChatGPT-3.5 and 4 matched the human-assigned codes exactly in 22% and 14% of cases.¹ In a 2025 study on 300 hospital discharge summaries, ICD-9 code prediction was correct 14.6% to 45.3% of the time across five models, far below their accuracy at naming the primary diagnosis.³ A 2024 preprint found that adding retrieval over a million past visits raised exact-match rates sharply, and physician reviewers often preferred GPT-4’s codes where they differed from the provider’s.²

The common thread: coding accuracy depends heavily on the setup, plain prompting does poorly, and every study calls for human review. For claims automation, that means a coder checks before submission, and a record shows what the AI proposed and what was changed.

Why coding is hard for a general model

Coding isn’t summarising. A code set has thousands of entries with precise rules: which code is more specific, which modifiers apply, which combinations are allowed. Two codes can describe almost the same thing and pay differently. A model that writes a fluent summary can still pick a code that’s one level too general, miss a modifier, or invent a code that looks plausible.

The studies below show those failure modes in numbers. They also show that the setup matters as much as the model. A model with no access to your organisation’s past coding decisions is guessing at local conventions as well as at the code set; a model given retrieval over thousands of similar, already-coded encounters has something concrete to match against. The preprint below shows the size of that effect: retrieval raised exact matches several-fold.

The three studies

CPT codes in otology (peer-reviewed, 2026). Researchers at the University of Cincinnati prompted ChatGPT-3.5 and 4 for CPT codes from about 191 operative notes by one surgeon and compared them with existing billing data.¹

Agreement with human coders was slight; both models failed to apply modifiers and tended to assign fewer work RVUs than the human coders.¹ The pattern is telling: high accuracy on the common, distinctive procedure, near zero on the less common one.

ICD-9 from discharge summaries (peer-reviewed, 2025). Indiana University and Emory researchers gave zero-shot prompts, through public web interfaces, to five models on 300 MIMIC-IV discharge summaries.³

Coding was the weakest task for every model, none met clinical standards across all tasks, and the authors call for task-specific fine-tuning and human-in-the-loop checking.³

ICD-10-CM in the emergency department, with retrieval (preprint, 2024). A Mount Sinai team compared retrieval-augmented LLMs with provider-assigned codes on 500 emergency visits, using over a million past visits for retrieval. This is a preprint and hasn’t been peer reviewed.² Retrieval raised exact-match rates with provider codes substantially, for example from 0.8% to 17.6% for one small model and from 7.2% to 26.4% for another. In 200 cases where GPT-4 and providers disagreed, a review panel of four physicians and two LLMs preferred GPT-4 for accuracy 447 times against 277 for providers.²

What they agree on

1. Plain prompting is weak at coding. Exact matches from 14% to 45% across the two peer-reviewed studies.¹³
2. Rare and specific cases fail first. Cartilage grafting at 0% to 4.2% sensitivity, against 94% to 96% for cochlear implantation.¹
3. Setup changes everything. Retrieval over past coded visits raised exact matches several-fold.²
4. Errors aren’t random in their direction. In the CPT study, models tended to under-assign work RVUs and missed modifiers.¹ Systematic errors add up across a claims batch.
5. Human review is the conclusion of every study.

Note the third study’s comparison is with provider-assigned codes, which aren’t a perfect gold standard either; that’s why its reviewers sometimes preferred the model. Measuring coding accuracy is itself hard.

How to read a coding accuracy claim

Before trusting a vendor’s or pilot’s number:

• Exact or partial match? “Partial” can mean the right code family and the wrong payment.
• Against what? Professional coders’ final codes, provider-entered codes, or a model’s judgement?
• Which code set and setting? ICD-10-CM in emergency medicine isn’t CPT in surgery.
• Common cases or all cases? Ask for accuracy on the rare categories.
• Modifiers and secondary codes included? They carry much of the money.
• Retrieval or fine-tuning involved? Results with your own historical data will differ.

Controls for AI-assisted coding and claims

Whether a model suggests codes or an agent assembles a whole claim, the same controls apply:

The third row is the one most pilots skip. If you don’t keep what the AI proposed beside what the coder submitted, you can’t tell whether the system is helping, and you can’t show an auditor how a code was chosen.

What this post isn’t

This is about administrative and billing data quality, not clinical decisions. Nothing here judges diagnoses or care. Coding rules and payer policies differ by country, but the review-and-record controls above apply wherever claims are submitted.

For the documentation side, see LLM hallucination rates in clinical notes; for what an approval and record should contain, HIPAA audit controls and AI agents and human-in-the-loop approval. To see approvals and checks on a real database in your browser, try the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.