WebArena and OSWorld: how computer-use agents are tested, and what the scores mean

The short answer

WebArena and OSWorld are benchmarks for agents that operate software the way a person does. WebArena gives agents fully working websites in four domains, e-commerce, forums, collaborative software development and content management, and checks whether long tasks were completed correctly.¹ OSWorld runs agents on real Ubuntu, Windows and macOS machines, with 369 tasks across web and desktop apps, file handling and multi-app workflows; each task has a setup state and its own script to check the result.²

When they were published, the gap was wide: the best GPT-4-based agent completed 14.41% of WebArena tasks against 78.24% for humans, and the best model completed 12.24% of OSWorld tasks against over 72.36% for humans.¹² Models have improved since. The method, checking the end state rather than the agent’s account, is what to take from them.

WebArena: real websites

WebArena’s authors built self-hosted, fully functional websites rather than using live sites, so tasks are repeatable and nobody’s real account is touched.¹ To picture the kind of task (our illustration): find a product, compare options and place an order on the shopping site, or change a setting on a project in the code-hosting site.

Two design choices stand out:

• Long-horizon tasks. Many steps, several pages, information carried from one page to another.¹
• Functional correctness. The check is whether the site’s state ends up right: the order exists with the right items, the setting has the right value.¹

OSWorld: real desktops

OSWorld goes a level lower, to the operating system: agents work on real machines running Ubuntu, Windows or macOS.² Tasks cover web and desktop applications, file input and output, and workflows that span several apps, such as taking data from a spreadsheet into a document.

Each task comes with two pieces of code: a setup configuration that puts the machine in a known starting state, and an execution-based evaluation script that checks the result.² The authors found agents struggled mainly with GUI grounding, finding the right element on the screen, and with operational knowledge of how applications work.²

Why they check the end state

Both benchmarks share a principle with τ-bench and SWE-bench: grade by what actually changed. τ-bench compares the database at the end of a conversation with a goal state; SWE-bench runs tests against the patched code.³⁴ WebArena checks the website’s state and OSWorld runs a script against the machine.¹²

The alternative, asking a model or a person to read the agent’s transcript and decide, rewards agents that describe their work convincingly. A computer-use agent can say “I’ve saved the file” when it saved it in the wrong folder. The end-state check doesn’t care what it said.

The gap they found

The launch numbers, side by side:

These numbers are from 2023 and 2024 and have moved a lot since; check current leaderboards for today’s figures. Two points still hold. Humans don’t get every task right either, because tasks are genuinely hard or ambiguous; the human score is the ceiling to compare against. And the failure modes the OSWorld authors named, grounding and operational knowledge, are the kind that produce confident wrong actions: clicking the wrong button, editing the wrong file.

Reading a computer-use leaderboard

Newer results on these benchmarks are reported constantly, so a few questions help:

1. Which version of the benchmark? Task sets get fixed and extended; scores across versions aren’t comparable.
2. What could the agent see? Screenshots only, or extra structure from the page or the OS? More structure usually means higher scores and a less realistic test.
3. How many steps and how much time was the agent allowed? Generous limits raise scores and costs.
4. Was there any human help or task-specific tuning?
5. What’s the human baseline on the same version? A score means more next to it.

A big jump in a leaderboard number is often a change in one of these, not only a better model.

What these benchmarks don’t cover

• Safety of the path. A run that deletes a folder on the way to the right end state can still pass, if the checker only looks at the target.
• Real accounts and data. Everything is sandboxed by design. Your environment isn’t.
• Adversarial pages. A web page that tries to instruct the agent is a separate research area; see AgentDojo and InjecAgent.
• Consistency. A single success rate doesn’t say whether the agent succeeds every time; τ-bench’s pass^k does.³

Lessons for running a computer-use agent

1. Give it a disposable machine. Both benchmarks run in resettable environments for good reason. A virtual machine or container you can throw away turns the worst case into a rebuild.
2. Check the end state yourself. After each task, verify the result with a script or a read, not with the agent’s summary.
3. Limit what the machine can reach. No saved passwords, no production credentials, no access to drives with data you can’t replace. Our post on an agent that deleted files shows how quickly a cleanup becomes a deletion.
4. Keep a person on irreversible steps: purchases, sends, deletions outside a scratch folder.
5. Record the actions, with screenshots if you can, so you can see the path, not only the result.

The blast radius entry explains why lesson 3 matters most. For the other benchmarks in the table, see τ-bench, SWE-bench and the function calling leaderboard. To see effect-based checks on a real database in your browser, try the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.