The short answer
GAIA, from Mialon, Fourrier, Swift, Wolf, LeCun and Scialom (2023), is a benchmark of 466 real-world questions for general AI assistants that need reasoning, handling of several media types, web browsing and tool use.1 Each question has exactly one short answer (a string, number or list), graded by quasi exact match. In the original paper, humans scored 92% and GPT-4 with plugins 15%.1 GAIA measures whether an assistant can find the right answer. It doesn’t look at how it got there: the authors say it doesn’t evaluate the trace leading to the answer.1 For agents that change things in the world, that’s the part that matters most.
Deedglass
What the AI said, and what really changed
The AI said
1 row
“1 row updated”
What really changed
301 rows
From the database’s own log
How GAIA is built
Questions are split into three levels by how many steps and tools they need:1
| Level | Questions | Typical demand |
|---|---|---|
| 1 | 146 | No tools or one tool, at most about 5 steps |
| 2 | 245 | Roughly 5 to 10 steps, combining tools |
| 3 | 75 | Arbitrarily long action sequences |
The design principles: real-world questions rather than closed synthetic environments, conceptually simple tasks whose reasoning is easy to check, answers that aren’t sitting in plain text on the web, and factoid answers for simple, zero-shot use.1 Answers to 300 questions are kept private to power a leaderboard; the rest are public.1
How it grades
Every answer is a short string, a number or a comma-separated list, scored by quasi exact match against one ground truth.1 That makes grading cheap, objective and hard to argue with. No LLM judge, no partial credit, no rubric.
It also makes GAIA a pure outcome measure. A correct answer reached by a sensible path and a correct answer reached by luck, scraping something it shouldn’t, or trying fifty things score the same.
The original results
| Respondent | Overall | Level 1 | Level 2 | Level 3 |
|---|---|---|---|---|
| Humans | 92% | |||
| GPT-4 with plugins | 15% | ~30% | ~10% | 0% |
Source:1. The authors call the plugin results an “oracle” estimate, because plugins were chosen by hand and the runs can’t be reproduced exactly.1 Scores on the public leaderboard have risen a lot since 2023; always check which split, which date and which system a quoted number refers to.
What a GAIA score doesn’t tell you
GAIA was designed to measure assistant capability, and it does that well. It wasn’t designed to measure four things buyers of agents often care about:
- The path. The trace isn’t evaluated.1 An agent that reached the answer by an unsafe action scores the same as one that didn’t.
- Side effects. GAIA questions ask for information; they don’t require changing anything. So there’s no notion of collateral damage.
- Consistency. One run per question says little about whether the agent gets it right every time.
- Robustness to hostile input. Web pages the agent reads aren’t adversarial by design.
The authors list their own limits: no trace evaluation, the cost of writing unambiguous questions, English-only coverage, and decay over time as data contamination grows or web sources disappear.1
GAIA next to action benchmarks
Other benchmarks grade what an agent did, not just what it answered:
| Benchmark | Grades | Side effects checked? |
|---|---|---|
| GAIA | Final answer, exact match1 | No |
| τ-bench | Final database state of a simulated domain2 | Yes, via state |
| WebArena | Functional correctness of tasks on working websites3 | Not a stated focus |
| AppWorld | Database diff, with unexpected changes counted as collateral damage4 | Yes, explicitly |
AppWorld’s approach is the clearest contrast: a task passes only if the database changes are the expected ones plus a small allowed set, so an agent that completes the task but also deletes something it shouldn’t fails.4 Our AppWorld post goes deeper.
Think of the difference with a concrete case. Asked “how many orders did customer 42 place last year?”, an agent can get the right number by running a read query, or by running a query that also locks a table for a minute, or by exporting the whole customer table to a scratch file it then reads. GAIA would score all three the same. An action benchmark, or a production gate, would not.
If you deploy an agent that writes to systems, a GAIA score is a reasonable signal of research and tool-use skill. It’s not evidence the agent will act safely.
Reading a GAIA claim
When a launch says “X% on GAIA”, ask:
- Which split? Validation (public answers) or test (private)? Public answers can leak into training.
- Which level? An average hides that level 3 is much harder.
- Which date? Web sources change; old questions can become easier or impossible.
- How many runs? One lucky run vs a mean.
- What tools? Browsing, code execution and file reading change results a lot.
- What did it cost? Some systems try many times or run for a long time per question; a score without cost and time per task hides that.
Questions people ask
What is the GAIA benchmark?
A set of 466 real-world questions for general AI assistants, in three difficulty levels, each with one short correct answer.1
How is GAIA scored?
By quasi exact match between the assistant’s final answer and the ground truth.1
Does a high GAIA score mean an agent is safe to deploy?
No. GAIA doesn’t evaluate the steps taken or their side effects.1 Use action-level tests and controls for that.
To see what action-level checks look like on a real database, try the playground. The provenance spec shows how actions, not answers, can be recorded, and our healthcare page shows why that matters. To try approvals and read-backs on a real database in your browser, open the playground; to grade what an agent can reach on your own machine, run npx deedglass scan (the grade, explained).
Written with AI assistance and checked against our claims register.