The short answer
AppWorld, from Trivedi and colleagues (ACL 2024), is a simulated world of 9 everyday apps, such as Gmail, Venmo, Amazon, Spotify and a file system, exposed through 457 APIs and populated with about 100 fictitious users, plus 750 agent tasks.1 It grades with state-based unit tests on the database diff: the changes after a task must be the expected ones plus a small allowed set, so any unexpected change counts as collateral damage and the task fails.1 In the paper, GPT-4o solved about 49% of normal tasks and 30% of challenge tasks.1 For anyone deploying agents that write to systems, the diff-based grading is the idea worth copying.
Deedglass
What the AI said, and what really changed
The AI said
1 row
“1 row updated”
What really changed
301 rows
From the database’s own log
The world
AppWorld’s engine is about 60,000 lines of code: real implementations of apps with databases, not mock responses.1 Fictitious users have contacts, messages, payment histories, playlists and files, so tasks happen in a populated world with plenty of things to get wrong.
Agents act by writing code that calls the apps’ APIs, which makes it a test of interactive coding agents as much as of tool calling.1
The tasks
| Element | Value |
|---|---|
| Tasks | 750, from 250 scenarios of 3 tasks each1 |
| Splits | Train, dev, normal test, challenge test1 |
| Challenge set | Needs an API from an app not seen in training1 |
| Metrics | Task goal completion (TGC); scenario goal completion (SGC)1 |
Scenario goal completion is the stricter metric: an agent gets credit for a scenario only if it passes every task variant in it.1 That measures consistency, not luck.
Grading by database diff
This is what sets AppWorld apart. After the agent runs, the evaluator compares the database before and after:
Code What the screen or file shows, copied as is.
before ──▶ agent runs ──▶ after
diff = after − before
pass if: diff ⊇ expected changes
and diff ⊆ expected changes ∪ small allowed set
Expected changes must appear; anything else must not.1 Because the check is on final state, different valid paths to the same result all pass.1
| Agent behaviour | Process-based grader | AppWorld |
|---|---|---|
| Completes task the expected way | Pass | Pass |
| Completes task by a different valid path | Might fail | Pass |
| Completes task and deletes a wish list on the way | Might pass | Fail: collateral damage |
| Completes task and starts an unrequested return | Might pass | Fail: collateral damage |
The examples in the last two rows are the paper’s own kind of collateral damage.1
Results and failure modes
| Model (paper) | Normal test (TGC) | Challenge test (TGC) |
|---|---|---|
| GPT-4o with ReAct | ~49% | ~30% |
Source:1. The paper reports other models solving at least 16% fewer. The failure modes it lists are instructive:1
- Not looking things up. Hallucinating information instead of checking the environment.
- API misuse. Inventing fields or using the wrong API, such as sent payment requests instead of received ones.
- Partial instruction following. Approving all requests when told to approve only roommates’ requests.
- Commonsense slips. Confusing the date a song was added with its release date.
- Repetition. Forgetting prior actions and redoing work.
The partial-instruction row is the dangerous one in production: the agent did something, confidently, to more records than it should have.
It’s also the failure a diff catches and a goal check misses. “Approve my roommates’ payment requests” has a goal that’s satisfied whether the agent approves two requests or twenty: every roommate request ends up approved. Only a comparison of the expected changes with the actual changes shows that eighteen extra requests were approved for people who aren’t roommates. Translate that to a hospital, a bank or a store, and the extra eighteen are the incident.
Why collateral damage is the right test
Most real agent incidents aren’t failures to complete a task. They’re tasks completed with something extra: the cleanup that removed too much, the update that touched every row, the refund issued twice. A grader that only checks “was the goal reached?” can’t see any of that.
Other strong benchmarks share AppWorld’s state-based approach. τ-bench compares the final database state in a simulated customer-service domain2, and AgentDojo grades with deterministic functions over environment state.3 GAIA, by contrast, grades only the final answer and doesn’t evaluate the path.4 AppWorld makes the “nothing else changed” condition explicit, and that’s the condition production teams care about most.
Applying the idea to your agent
- Snapshot before, diff after. For every agent task in testing, capture the state of the systems it can touch and diff after.
- Write the expected diff. For each task, list the changes it should make. Everything else is suspect.
- Allow a small, explicit set. Timestamps, logs and caches change; list them so they don’t drown the signal.
- Grade consistency. Run variants of the same task, as scenario goal completion does. An agent that passes one variant and fails another isn’t reliable.
- Do it in production too. A read-back after each write, comparing what changed with what the agent said it changed, is the same diff at run time. Our 1 row vs 301 rows post shows it on Postgres.
Questions people ask
What is AppWorld?
A benchmark world of 9 simulated apps with 457 APIs and 750 tasks for interactive coding agents.1
How does AppWorld grade agents?
With state-based unit tests on the database diff; unexpected changes count as collateral damage and fail the task.1
How well did models do?
GPT-4o solved about 49% of normal and 30% of challenge tasks in the paper.1
To see a diff-based check catch a write that changed more than it said, try the playground. The provenance spec records the claim and the observed diff side by side, and our healthcare page shows where that matters most. To grade what an agent can reach on your own machine, run npx deedglass scan (the grade, explained).
Written with AI assistance and checked against our claims register.