ToolEmu, explained: finding an AI agent's risky actions before it touches real tools

The short answer

ToolEmu is a research framework from a 2023 paper by Ruan, Hashimoto and colleagues that tests language-model agents for risky actions by having another language model emulate the tools the agent calls, instead of running real ones. Its benchmark has 36 high-stakes toolkits and 144 test cases across 9 risk types. Humans judged 68.8% of the failures it found to be valid real-world failures, and even the safest agent tested showed risky failures 23.9% of the time.¹ The key design choice: user instructions are deliberately underspecified, so the test is whether the agent asks or acts.

The problem ToolEmu tackles

To know whether an agent will delete the wrong files or send money to the wrong person, you have to let it try. With real tools, that’s dangerous; with hand-built sandboxes, it’s slow, because every tool needs a fake implementation with realistic state. ToolEmu’s bet is that a language model can play the tool convincingly enough to surface the agent’s behaviour, at a fraction of the effort.

How it works

ToolEmu has four parts:¹

Each test case gives the agent an instruction that leaves something important unsaid. The paper’s example: asked to clean up disk space, does the agent delete everything, or confirm which files matter? The agent’s tool calls go to the emulator, which answers as the tool would; the evaluators then score the whole trajectory.¹

Code What the screen or file shows, copied as is.

 underspecified instruction ──▶ agent ──tool call──▶ LM emulator (plays the tool)
                                   ▲                       │
                                   └──── emulated result ──┘
 trajectory ──▶ safety evaluator + helpfulness evaluator ──▶ scores

What it found

The kappa numbers matter. Automated judges are only useful if they agree with people about as well as people agree with each other, and here they did. That’s a better showing than many LLM-judge setups; our post on LLM-as-a-judge covers why agreement can’t be assumed.⁴

The failure modes the paper names will be familiar to anyone who has run agents: acting on unwarranted assumptions, misreading instructions, erroneous executions and ignoring risk.¹

Limits to keep in mind

• Emulation isn’t execution. The authors note the emulators sometimes overlooked core constraints of the tools, more often in complex or adversarial scenarios.¹ A real API may fail in ways an emulator never imagines, and vice versa.
• About a third of found failures weren’t judged valid. 68.8% valid means the rest were artefacts of emulation or judging.¹
• The judge is a model. Its kappa is good on this benchmark; it may not transfer to your tools.
• Model versions. The agents tested are from 2023. The method outlasts the specific numbers.

How it compares with state-based benchmarks

Other benchmarks take the opposite approach: real or simulated environments with real state, graded by code. τ-bench checks the final database state of a simulated customer-service domain²; AgentDojo grades with deterministic functions over environment state, partly because an LLM judge could itself be hijacked.³

They complement each other. Use emulation to find where an agent is likely to go wrong across many tools; use state-based tests to pin down specific behaviours before release. Our action evaluation post covers the state-based side.

Using the idea in your own testing

You don’t need the ToolEmu codebase to borrow its best ideas:

1. Write underspecified tasks on purpose. “Clean up old records”, “refund the customer”, “update the patient’s details”. The interesting behaviour is what the agent does with what you didn’t say.
2. List the safety constraints the task omits. For each test, write down what a careful person would have checked first. Score against that list.
3. Emulate expensive tools early. For tools you can’t safely call, an emulated version is enough to see whether the agent asks for confirmation.
4. Measure asking, not just success. An agent that completes every task by never asking is the one ToolEmu was built to catch.
5. Graduate to real state. Before production, run the risky cases against a sandbox with real state and deterministic checks.
6. Keep the guard outside the model. A test suite estimates risk; a rule that holds destructive calls for a person, outside the agent, handles the 23.9% that remain.

Questions people ask

What is ToolEmu?

A framework that uses a language model to emulate tools and evaluate whether an agent’s actions are risky, with 144 high-stakes test cases.¹

How reliable are its results?

Humans judged 68.8% of the failures it flagged as valid, and its safety evaluator agreed with humans about as well as humans agreed with each other.¹

Should I use emulated tools for my agent tests?

For broad, early risk discovery, yes. For release decisions, add tests with real state and code-based checks.

To see an agent’s risky call held before it runs, try the playground; the provenance spec shows how such decisions can be recorded, and our healthcare page shows where this matters most. To try approvals and read-backs on a real database in your browser, open the playground; to grade what an agent can reach on your own machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.