An AI agent security test suite that runs on every release

The short answer

A useful security test suite for an AI agent’s tool layer doesn’t measure whether the model can be fooled. It assumes it already has been, and tests what happens to the tool calls that come out: does the destructive SQL run, does the stolen key work, does the secret leak into a log. Write each case’s expected outcome before its code exists, run every case against real processes and a real database, and rerun the whole suite on every release.

That is how our own suite works. It assumes the model has already been manipulated and tests what Deedglass does with the calls that follow; it doesn’t measure how often a model falls for an injection.² On release 0.14.1, all 82 cases matched their expected outcome, with 0 differing and 0 waiting.¹

Start from a fooled model

Prompt-injection research asks a model-level question: given this attack text, does the model follow it? That matters, and it changes with every model release. A tool layer can’t control the answer. What it can control is what a manipulated model is able to do.

So every case starts at the moment the model has already decided to do the wrong thing. The input to the test is the tool call, not the prompt:

• “The model sends DROP TABLE orders after reading a poisoned row.”
• “The model calls a tool with a key that was revoked an hour ago.”
• “An upstream MCP server changes a tool’s schema after it was listed.”

This framing has a practical benefit: the tests are deterministic. No model, no sampling, no flaky pass rates. A case either produces the expected outcome or it doesn’t.

Write the expected outcome first

The single most important rule: decide what should happen before you write the code that checks it. Otherwise it’s too easy to run a case, see what happened, and write that down as the expectation.

In our suite, each case’s expected outcome is written in a metadata file before its code exists, and a test pins a digest of the expectations. A case that turns out differently is shown as “differs”, and its expectation is never edited to pass.² Changing an expectation means changing the pinned digest in a commit someone can see.

An expected outcome should be specific enough to fail. “Refused” is too vague. Compare:

The strong versions name the reason, the count and the side effects. A refusal for the wrong reason, or a refusal after the write already happened, then shows up as a difference.

What to attack: a starter list

Our cases are grouped by what the attacker controls. The list below is a starting point for your own suite, whatever tool layer you run; each group is one in our suite’s report.²

Two groups are easy to forget. Upstreams as attackers: a tool server can lie in its descriptions, change a definition after you approved it, or return a 50 MiB result. Our post on tool poisoning covers the first two. Side effects you didn’t send: a trigger or cascade changes more than the statement named. The 1 row vs 301 rows case is the example.

Run it against real systems

Mocks are where security tests go to pass. Our suite runs against a real deedglass serve process and a real Postgres; a full run takes about three and a half minutes on a laptop.² A real database catches things a mock won’t: a statement that a parser reads one way and Postgres reads another, a trigger that fires, a transaction that commits when you thought it rolled back.

Not every check needs the full harness. Another 23 library- and CLI-level attack cases run with go test ./attack on every pull request.² Static analysis runs too: CodeQL for Go, JavaScript and TypeScript, Python and the workflows, on every pull request and weekly.⁴ Fast checks on every change, the full suite on every release.

Check invariants after every case

A case that passes its own assertion can still break something else. After every case, check properties that should always hold:

1. The record chain is intact. No record was lost or changed.
2. The target is unchanged wherever nothing should have run.
3. Every action left a record, including refusals.
4. No secret leaked. Plant a credential with a unique marker and search every log line and response for it.

On the 0.11.0 run, both organisations’ record chains were intact after every case, and the planted credential was seen 0 times across 7,811 server log lines.¹ The invariant checks are cheap to write and catch the regressions nobody wrote a case for.

Our numbers, and what they don’t say

The suite has matched on each release we’ve published a run for: 82 of 82 on 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0 and 0.9.0,¹¹¹¹¹¹ then on 0.10.0 and 0.11.0,¹¹ and on 0.12.0, 0.13.0, 0.14.0 and 0.14.1.¹

A suite you wrote yourself, passing every time, should make you suspicious rather than comfortable. It means the cases test what we built, and an attacker will try what we didn’t think of. So the report lists its own gaps. It doesn’t cover a compromised host, database superuser or signing key; poisoned tool results; denial of service at scale; or effects outside Postgres.² Instruction-bearing tool descriptions are found by a heuristic, and a reworded instruction can pass.

That is also why we publish a separate benchmark where our checks lose. Where our gate loses shows the cases we miss, which a suite like this one, by design, never will.

If you build your own suite, start with five cases from the table above, written expectation-first, and run them in CI on a real database. To see the checks behave on a database without installing anything, try the playground. To grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained). The prompt injection entry explains why the model-level question stays open, and a tool-call policy shows what decides the calls this suite throws at it.

Written with AI assistance and checked against our claims register.