Text-to-SQL benchmarks: what BIRD and Spider 2.0 say about letting agents write SQL

The short answer

Text-to-SQL benchmarks measure whether a model can turn a question in plain language into SQL that returns the right answer. BIRD has 12,751 question-and-SQL pairs over 95 real databases, 33.4 GB in total, with messy values; at publication, the best model reached 40.08% execution accuracy against 92.96% for people.¹ Spider 2.0 has 632 enterprise workflow tasks, often on databases with over 1,000 columns in BigQuery or Snowflake; a code agent built on o1-preview solved 21.3%, compared with 91.2% on the original Spider and 73.0% on BIRD.²

The pattern is clear: the closer the task is to a real company database, the lower the score. For agents that query production data, that means two controls matter more than the model: a role that can only read, and a check on what the query returned.

BIRD: real data, dirty values

BIRD, published at NeurIPS 2023, set out to test models on large databases with real contents rather than tidy toy schemas. Its 95 databases span 37 professional domains, and the benchmark focuses on three things toy benchmarks miss: dirty values, external knowledge needed to connect a question to the data, and the efficiency of the SQL.¹

Dirty values are a good example of why this matters. A question about “orders from New York” may need to match NY, New York, new york and NYC in the same column. A model that writes WHERE city = 'New York' produces valid SQL and a wrong answer.

Spider 2.0: enterprise workflows

Spider 2.0, an oral paper at ICLR 2025, moves further toward real work. Its tasks are workflows, not single queries: they often need several queries, in different SQL dialects, sometimes over 100 lines long, and they involve reading metadata, dialect documentation and project code.² Databases come from real applications and often have more than a thousand columns.²

Its headline is the comparison:

The authors conclude that models need substantial improvement before they’re adequate for real-world enterprise use.² Scores have risen since, but the gap between benchmarks remains the lesson.

What execution accuracy measures

BIRD scores by execution accuracy: run the model’s SQL and the reference SQL, and compare the results.¹ That’s the right idea, the same one agent benchmarks use when they compare end states, and it has limits worth knowing:

• Same result, different meaning. Two queries can return the same rows on today’s data and differ tomorrow. A query that forgets a filter can match by luck.
• Read-only by design. Benchmarks run SELECTs. They don’t test what happens when a model writes UPDATE, DELETE or DDL.
• No cost check by default. A correct query that scans a huge table at peak time can still be a problem. BIRD’s efficiency analysis is one of the few attempts to measure this.¹

Why scores fall on real schemas

Reading the two papers together, the drop comes from things every production database has:

1. Size. Hundreds or thousands of columns, many with unhelpful names. The model must find the right ones.
2. Messy values. Inconsistent spellings, nulls meaning different things, codes instead of labels.
3. Domain knowledge. “Active customer” means something specific in your business that isn’t in the schema.
4. Dialects. BigQuery, Snowflake and Postgres differ in functions, quoting and date handling.
5. Multi-step work. Real questions need intermediate tables and several queries, so errors compound.

Your database almost certainly has all five.

What this means for “chat with your data” tools

Many products now let anyone in a company ask questions of the warehouse in plain language. The benchmarks suggest how to set expectations. On tidy, well-documented tables with clear names, accuracy can be high. On the long tail of a real warehouse, with legacy tables, overlapping definitions and undocumented codes, a large share of answers can be wrong while looking perfectly plausible.

That doesn’t make such tools useless. It makes them drafting tools: they produce a query and an answer that a person who knows the data should check before the number goes into a decision. The most useful feature such a tool can have is showing its SQL, because that’s what a knowledgeable reader can verify in seconds. A tool that hides the query asks you to trust a benchmark score you can’t see.

Giving an agent SQL safely

If an agent can be wrong about a quarter or more of real queries, two kinds of wrong matter differently. A wrong SELECT gives a wrong answer, which is bad. A wrong UPDATE changes data, which is worse. So:

Make the connection read-only at the database. A role with SELECT grants only, and read-only transactions on top. In PostgreSQL a read-only transaction disallows INSERT, UPDATE, DELETE, MERGE, DDL and TRUNCATE, though the docs call it a high-level notion of read-only that doesn’t prevent all writes to disk.³ The grants are the real boundary. Our safe Postgres access guide and the read-only role comparison show the setup.

Limit cost. A statement timeout and a row limit on results keep a bad query from becoming an outage.

Check results, not just syntax. Ask the agent to show the query with its answer; sample-check important answers against a known figure; flag answers that changed sharply from last time.

Don’t parse SQL with patterns. If you add a check on the agent’s SQL, use a real parser. Our regex test shows five table-dropping statements walking past a pattern.

Choose the server carefully. Postgres MCP servers differ in their read-only modes; our comparison lists the options.

A checklist

1. Read-only role for question-answering agents, grants per table.
2. Statement timeout and row limits.
3. Show the SQL with every answer.
4. A small private benchmark: twenty real questions with known answers from your own data.
5. Document business terms the agent can read, so “active customer” means one thing.
6. Separate credentials for any agent that must write, with approval for writes.

The read-only database access entry explains why the role matters more than the prompt. To see read and write requests handled on a real Postgres in your browser, open the playground; to grade what an agent can reach in your own database, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.