The short answer
Picture three departments each trying a different AI tool, and nobody able to list what any of them can change. That is how an unnoticed write ends up in a chart. Governance can be small: a list of tools, one review group, rules for which actions need a person, monitoring and a regular audit.
A hospital can govern AI tools that act in its systems with five parts: an inventory of every AI tool and what it can write; one review group that approves new tools and changes; write tiers that decide which actions need a person; monitoring built on records of what each tool actually did; and a periodic audit. Those five map onto the four functions of NIST’s AI Risk Management Framework, Govern, Map, Measure and Manage, which is voluntary1, and onto India’s ICMR guidance that AI able to cause serious harm needs human oversight and regular internal and external audits.2
Deedglass
How Deedglass keeps an AI agent in check
-
1Before it runs
Your rules decide
- Read a recordAllowed
- Delete a tableNever
- Change a medicationNeeds a person
-
2If it’s risky
A named person approves
NSDr. N. ShahApproved in SlackDeclineApprove -
3After it runs
Checks what really changed
MatchesRecord sealedCheck it offline
What the frameworks ask
| Source | Status | What it asks, in short |
|---|---|---|
| NIST AI RMF 1.0 (January 2023) | Voluntary | Govern, Map, Measure, Manage; a Generative AI Profile followed in July 20241 |
| ICMR ethical guidelines (2023) | Guidance | Compulsory human oversight and manual override for AI that can cause severe or bodily injury; regular internal and external audits2 |
| India AI Governance Guidelines (November 2025) | Voluntary measures | Monitoring standards, audit trails and human-in-the-loop at critical decision points3; seven principles including accountability and “understandable by design”3 |
| SAHI (February 2026) | National strategy | A governance framework and roadmap for responsible AI in healthcare4 |
The India AI Governance Guidelines also propose graded liability based on function, risk and due diligence, and an AI Governance Group with sector regulators keeping enforcement powers.3 None of these documents tells a hospital how to staff governance. That’s the gap this post fills.
Advisers versus actors
Most hospital AI governance started with tools that advise: a risk score, a summary, a suggested code. A person reads the output and acts. Agents and scribes increasingly act: they draft into the chart, queue orders, submit claims, update records.
That shift changes what governance must watch. For an adviser, the question is “is the output good?”. For an actor, the questions add “what did it change, who allowed it, and does the record match what it said it did?”. People also over-rely on automated suggestions, especially under time pressure and workload5, so “a person reviews it” is a control only if the review is designed to be real.
The five parts
1. Inventory. One list of every AI tool in use, including features switched on inside existing products. For each: what it reads, what it can write, which credentials it uses, who owns it.
2. One review group. Small: a clinical lead, the CMIO or equivalent, IT security, health information management, the privacy officer, and someone from quality. It approves new tools, new write permissions and major version changes. Meeting monthly is usually enough, with a fast path for urgent changes.
3. Write tiers. For each tool, classify its writes: read-only, write with read-back, write with approval, two approvers, never. Our EHR write tiers post gives the method.
4. Monitoring from records. Track, per tool per month: writes made, writes held for approval, mismatches between what it reported and what was stored, refusals, time to approve. Use the system of record, not the tool’s own dashboard.
5. Audit. Once a year, or after a serious incident, review a sample of each tool’s actions end to end. ICMR asks for both internal and external audits for higher-risk AI.2
Who does what
| Task | Clinical lead | IT security | HIM | Privacy officer | Review group |
|---|---|---|---|---|---|
| Add a tool to inventory | Consulted | Responsible | Consulted | Consulted | Approves |
| Set write tiers | Responsible | Consulted | Consulted | Informed | Approves |
| Grant credentials | Informed | Responsible | Informed | Informed | Approves |
| Monthly monitoring | Reviews | Produces | Reviews | Reviews | Discusses |
| Incident response | Assesses impact | Leads | Corrects records | Decides on notice | Learns |
| Yearly audit | Participates | Participates | Participates | Participates | Commissions |
Our incident response runbook has the steps for the incident row.
The inventory template
| Field | Example |
|---|---|
| Tool and version | Ambient scribe, vendor X, 4.2 |
| Owner | Dr A, Medicine |
| Reads | Encounter audio, problem list |
| Writes | DocumentReference drafts; ServiceRequest drafts |
| Write tier | Drafts: read-back; orders: approval |
| Credentials | Client id scribe-prod, scopes listed |
| Data leaves the hospital? | Audio to vendor cloud, region stated in contract |
| Records kept | Draft, edits, signer, read-back |
| Last reviewed | 2026-10 |
| Known issues | Unit errors in 2 of 500 sampled notes |
A spreadsheet is fine. What matters is that it’s complete and has an owner.
Mapping to the NIST AI RMF
| NIST function | Hospital part |
|---|---|
| Govern | Review group, policies, roles |
| Map | Inventory: what each tool touches and why |
| Measure | Monitoring from records; audits |
| Manage | Write tiers, incident response, retiring tools |
The RMF is being revised as part of the White House AI Action Plan1, but these four functions are a stable way to explain your setup to a board or a regulator.
Our own model for AI writes in hospitals follows the same lines: rules before each write, a read-back after it, a named person or two for high-risk writes, a signed record anyone can check offline, and data on the hospital’s own servers.6 The India white paper lays it out in full.
Questions people ask
Do we need a separate AI committee?
Not necessarily. Many hospitals add AI to an existing clinical informatics or digital health committee. What matters is that one group owns the inventory and approves write permissions.
Which framework should an Indian hospital follow?
ICMR’s guidelines are the health-specific reference, and the India AI Governance Guidelines set the national direction.23 NIST’s RMF is a useful, voluntary structure for organising the work.1
How do we govern AI features inside products we already use?
Put them in the inventory like any other tool. A feature that writes to the chart needs the same tiers and records whoever built it.
Hospitals can see how this works in practice on our healthcare page, and how each action can be recorded in the provenance spec. To try approvals and read-backs on a real database in your browser, open the playground; to grade what an agent can reach on your own machine, run npx deedglass scan (the grade, explained).
Written with AI assistance and checked against our claims register.