Automation bias and human-in-the-loop: what health IT research says about approving AI

The short answer

Automation bias is the tendency to over-rely on automation. A 2012 systematic review in JAMIA, drawing on 74 studies, found it is driven by trust and confidence in the system, by experience, and by workload, task complexity and time pressure, and that it is reduced by training, by making users accountable, and by how the advice is presented.¹ A 2017 review found it shows up even when people do one task at a time, especially tasks that are hard to verify, and linked it to cognitive load.²

For AI agents, the lesson is direct. A person approving each action is only a safeguard if checking is easy, prompts are rare enough to read, and the approver is accountable. Otherwise approval becomes a click. OWASP’s agentic list names this risk ASI09, Human-Agent Trust Exploitation, including humans approving actions without independent validation.³

What automation bias is

The 2012 review defines automation bias as over-reliance on automation and sets it beside a related idea, automation-induced complacency: insufficient monitoring of what the automation produces.¹ Its starting observation is worth quoting in spirit: decision support usually improves performance overall, but it can also introduce new errors, and those often go unrecognised.¹

Two kinds of error follow from over-reliance:

• Commission: following the system’s suggestion when it’s wrong.
• Omission: missing a problem because the system didn’t flag it.

Both map onto agent approvals. An approver who accepts a wrong action commits the first; one who assumes “the agent would have asked if something was risky” commits the second.

What makes it worse

The 2012 review grouped what makes automation bias more likely:¹

Every row describes a normal week for someone approving agent actions. They trust the agent because it’s usually right, they’re busy, the actions are technical, and the queue is long.

Hard-to-check tasks are the danger

The 2017 review by Lyell and Coiera tested a common belief from the human-factors literature: that automation bias only happens when people are multitasking. Across 40 included studies, six of them in health care, they found it in single tasks too, typically diagnostic tasks rather than monitoring, and with high verification complexity.² Their conclusion: it appears tied to cognitive load, and mitigation should focus on reducing that load.²

Verification complexity is the useful idea for agent design. Ask how hard it is for the approver to check, independently, that the proposed action is right:

The last two rows are where approval is most likely to become a formality, and they’re exactly the actions teams most want a person to look at. A migration or a bulk update is put behind approval precisely because it’s risky, and then presented in a form that’s hardest to check. The fix isn’t to remove approval from those actions; it’s to change what the approver sees, so the check is something a person can actually do in the time they have.

What reduces it

The 2012 review found mitigation in two places: how the system is introduced, through training and emphasising user accountability, and how it’s designed, through the position of advice on screen, updated confidence levels attached to the output, and providing information rather than a recommendation.¹

That last point deserves attention. A screen that says “Approve?” with the agent’s summary is a recommendation. A screen that shows what will change, with the evidence, and asks the approver to confirm is information. The second invites checking; the first invites agreeing.

Designing agent approvals with this in mind

Translating the research into concrete choices:

1. Make the action easy to verify. Show the exact call, the rows or records it will touch, and a diff, not the agent’s description of its intent.
2. Ask less often. Let rules handle clear cases automatically, allowed or refused, and send only the uncertain or high-impact ones to a person. Fewer prompts means each gets attention.
3. Make the approver accountable. Record who approved what and when, and review approvals periodically. The 2012 review lists accountability as a mitigator.¹
4. Split complex actions. A 40-line migration approved as one block is hard to check; one step at a time, with the effect of each shown, is easier.
5. Use two people for the highest-impact actions. A second approver for bulk changes to patient data or large refunds.
6. Watch approval timing. Approvals given in a second or two, all day, are a sign review has become a habit.

The opposite pattern is “Always Allow”. In the GitHub MCP case, Invariant Labs noted that many users choose “Always Allow” for tool confirmations and stop watching individual actions, which is how a prompt meant as a safeguard stops being one.⁴

What this research can’t tell you

The studies behind these reviews are mostly about clinical decision support: alerts, diagnostic aids, monitoring systems. Agents acting through tools are newer, and the reviews themselves note the literature is fragmented and few studies report effects against a control.² The direction of the findings is consistent, but the size of the effect for agent approvals hasn’t been measured.

So treat these as design principles to test, not numbers to rely on. Measure your own approvals: how many, how fast, how often a later review finds one was wrong.

For the practical side of setting up approvals, see requiring approval for MCP tool calls and the human-in-the-loop approval entry. The closely related problem of too many prompts is in alert fatigue research and agent approvals, and lockdown for hospitals shows a two-person design. To see approvals on a real database in your browser, try the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.