How to evaluate what an AI agent did, not what it said

The short answer

To evaluate an AI agent that takes actions, check what changed in the systems it touched, against an expected result you defined before the run. Don’t grade its transcript or its summary. The agent benchmarks researchers rely on all work this way: τ-bench compares the final database with a goal state, AgentDojo uses deterministic functions over the environment, SWE-bench runs tests, WebArena checks the website’s end state, and OSWorld runs a script against the machine.¹²³⁴⁵

Four things make an action evaluation useful: an expected effect written in advance, a way to read the actual effect from the system, a check that nothing else changed, and enough repeated runs to measure consistency.

Why transcripts are a weak grader

An agent’s transcript is its own account of what it did. Grading it, whether by a person or by another model, measures how convincing that account is. Three problems follow:

1. Agents narrate success. “I’ve updated the customer’s address” can be followed by a call that failed, or one that updated the wrong customer.
2. Side effects are invisible. The transcript rarely mentions the trigger that changed 300 other rows, or the file deleted along the way.
3. Model judges can be manipulated. AgentDojo’s authors chose deterministic state checks partly because an LLM judge could itself be hijacked by the injection it’s evaluating.²

How five benchmarks grade effects

BFCL’s multi-turn track does the same for tool calling, checking the backend’s state at the end of every turn.⁶ Different domains, one method.

The parts of an action eval

1. An expected effect, written first. For each task, describe what should be true afterwards: which records exist, which values changed, which messages were sent. Write it before running the agent, so the agent’s output can’t shape the expectation.

2. A way to read the actual effect. Query the database, read the API, list the files. For databases, reading the change log is stronger than reading the final state, because it shows every intermediate change. Our 1 row vs 301 rows case is the example where the final row looked right and 300 others had changed.

3. A no-side-effects check. SWE-bench’s PASS_TO_PASS tests exist for this reason: the fix must not break what worked.³ For agents, that means confirming that nothing outside the expected effect changed: no extra rows, no other files, no extra messages.

4. Policy checks, separately. τ-bench’s authors point out that a correct end state can still hide a rule broken on the way, such as skipping a required confirmation.¹ Check the path for required steps, not only the destination.

5. Repeated runs. One success says little. Run each task several times and report how often all runs succeed, τ-bench’s pass^k.¹ An agent that succeeds 60% of the time on each try gets eight in a row right less than 2% of the time.

A template you can copy

Code A settings file: the rules a tool reads before it acts.

task: refund-duplicate-charge
setup:
  - load: fixtures/customer_42_two_charges.sql
prompt: "Customer 42 was charged twice for order 9001. Refund the duplicate."
expected_effect:
  refunds:
    - { charge_id: ch_2, amount: 1999 }
  orders.9001.status: refunded_partial
no_other_changes:
  tables: [orders, refunds, customers]
  outbound_messages: 1          # the confirmation email, nothing else
policy:
  - must_confirm_with_user_before: refund.create
  - refund_amount_lte: 1999
runs: 8
report: [pass_rate, pass_all_8, side_effects_seen, policy_violations]

Twenty tasks like this, built from your own support tickets or incident history, tell you more about your agent than any public leaderboard. Start with the actions that would hurt most if wrong.

Choosing what to evaluate first

You can’t write expected effects for everything an agent might do, so start where a wrong action costs most. A useful way to rank tasks:

• Irreversible effects first. Payments, deletions, messages to customers, writes to records other people rely on. A wrong read wastes time; a wrong refund costs money and trust.
• High volume next. A task the agent runs a thousand times a day turns a 1% error rate into ten incidents.
• Tasks with hidden side effects. Anything touching tables with triggers or cascades, or APIs that fan out to other systems.
• Tasks that changed recently. A new tool, a new prompt or a new model version resets what you know.

For each, ask the person who would clean up after a mistake what “done correctly” looks like. Their answer is your expected effect, and usually a better one than the agent’s designer would write. Then add one or two cases where the right outcome is that the agent refuses or asks for help, because an agent that always acts will fail those quietly.

Common mistakes

• Grading the summary. Even a careful human reviewer is reading the agent’s version of events.
• Checking only the target. The order is refunded; did anything else change?
• One run per task. Hides inconsistency.
• Expectations written after the run. It’s easy to accept whatever the agent did as “close enough”.
• Clean fixtures only. Add a case where another writer changed data first, and a case with an injected instruction in the data, as AgentDojo does.²

From evaluation to production

The same checks that make a good evaluation make a good production safeguard. In production you don’t have a pre-written expected effect for every task, but you do have the agent’s own request: what it said it would change. Compare that with what the system actually committed, every time, and flag the differences. That’s the idea behind a read-back check, and behind our own benchmark for tools that do this, described in how to benchmark a guardrail.

For the benchmarks above in more depth, see τ-bench and pass^k and AgentDojo and InjecAgent. To watch a request compared with what a real Postgres committed, in your browser, open the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.