The short answer
τ-bench (tau-bench) is a benchmark for agents that talk to a user and call tools under domain rules. A language model plays the user; the agent has API tools and a policy document; and the run is scored by comparing the database state at the end of the conversation with an annotated goal state.1 It introduced pass^k: the chance that all k independent tries at a task succeed, averaged across tasks, as opposed to pass@k, the chance that at least one does.1
The headline from the 2024 paper: state-of-the-art function-calling agents such as gpt-4o succeeded on under 50% of tasks, and pass^8 was under 25% in the retail domain.1 Those numbers are about models of that time; the method is what lasts.
Deedglass
What the AI said, and what really changed
The AI said
1 row
“1 row updated”
What really changed
301 rows
From the database’s own log
How a task works
The paper has two domains: τ-retail, with 115 tasks such as cancelling or modifying pending orders, returns and exchanges, and address changes; and τ-airline, with 50 tasks around booking, changing and cancelling reservations and refunds.1
Each task gives the simulated user an instruction, such as “exchange the shoes you ordered for a larger size, and don’t change anything else”. The user is played by a language model, gpt-4-0613 at temperature 1.0.1 The agent sees only the conversation, its tools (read and write functions over a database) and the domain policy: what it may do, what it must confirm, what it must refuse.
Because the user is sampled at temperature 1.0, the same task plays out differently each time. That randomness is deliberate. It’s what makes measuring consistency possible.
Scoring by database state
τ-bench doesn’t ask a judge whether the conversation went well. It checks two things and multiplies them:1
- Action: does the database at the end match the goal state for that task?
- Output: did the agent’s replies contain the information the user needed, such as a specific value?
This is the most important design choice in the benchmark. An agent that says “I’ve exchanged your shoes” but changed the wrong order fails. An agent that changed the right order but also cancelled another one fails. What counts is the effect, compared with a known-correct target.
The same principle sits behind our own benchmarks: score against what the system actually committed, never against what the agent or the tool reports. Our post on benchmarking a guardrail applies it to gateways instead of agents.
pass^k vs pass@k
Code benchmarks made pass@k popular: run k attempts and count the task solved if any one works. That suits a developer who can try several suggestions and keep the good one.
An agent acting for a customer doesn’t get to keep the good one. Every conversation is live. So τ-bench defines pass^k: the probability that all k independent trials of a task succeed, averaged across tasks. With n trials of a task and c successes, it’s estimated as C(c, k) / C(n, k).1
| pass@k | pass^k | |
|---|---|---|
| Question | Can it ever do this? | Does it do this every time? |
| As k grows | Goes up | Goes down |
| Suits | Picking the best of several drafts | An agent acting unattended |
Worked numbers
If an agent succeeds on a task independently with probability p on each try, pass^k for that task is p to the power k. Two agents:
| Per-try success | pass^1 | pass^2 | pass^4 | pass^8 |
|---|---|---|---|---|
| 90% | 0.90 | 0.81 | 0.66 | 0.43 |
| 60% | 0.60 | 0.36 | 0.13 | 0.017 |
An agent that looks decent at 60% on a single try handles the same request correctly eight times in a row less than 2% of the time. Even at 90%, eight in a row happens less than half the time. That is why the paper’s pass^8 numbers were so much lower than its single-try numbers.
The estimator for real data works on counts. Suppose a task was run 8 times and succeeded 6. Then pass^1 is 6/8 = 0.75, pass^2 is C(6,2)/C(8,2) = 15/28 ≈ 0.54, and pass^4 is C(6,4)/C(8,4) = 15/70 ≈ 0.21.1 Average those across tasks to get the benchmark score.
What it doesn’t measure
The authors list the limits themselves, and they matter if you’re reading a leaderboard:1
- A reward of 1 is necessary, not sufficient. An agent can reach the right database state while breaking policy, for example by not asking the user to confirm. The benchmark wouldn’t notice.
- The simulated user is imperfect. Instructions can contain typos or ambiguity, and the user model can reason badly or drift from its instruction.
- Prompts were tuned with a specific agent, which may bias results.
Add one more from the deployment side: τ-bench measures agents in a sandbox with a clean database and two domains. Your agent’s tools, data and users are different. The benchmark tells you how to measure; it can’t tell you your numbers.
Lessons for your own agents
- Score effects, not transcripts. For every agent task you care about, define the expected end state and compare. A database diff is a better grader than any summary.
- Run each task many times. One success tells you little. Report pass^k for the k that matches your usage.
- Check policy separately. A correct end state reached the wrong way, without a required confirmation, is still a failure in production.
- Expect inconsistency, and design for it. If pass^k is low, put a check after every write and a person in front of the risky ones, rather than hoping the next try goes better.
That last lesson is the practical one. If an agent will sometimes get it wrong, you want to know when it does, every time, from the database rather than from its report. A read-back check does that for one write; our 1 row vs 301 rows case shows a report and a database disagreeing. To see effect-based checks on a real Postgres in your browser, open the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).
Written with AI assistance and checked against our claims register.