What SWE-bench measures, and what a high score doesn't tell you

The short answer

SWE-bench measures whether a coding agent can resolve real GitHub issues. Each task is an issue from one of 12 popular Python repositories, with the repository as it was before the fix. The agent edits the code; its patch is applied; and the issue counts as resolved only if all the tests tied to the fix pass, together with the tests that already passed before.¹ When it was published, the best model resolved 1.96% of the 2,294 tasks.¹

It doesn’t measure whether the agent did anything else along the way: commands it ran, files it touched outside the fix, or whether a person would accept the code. A higher score means more issues closed against those tests, nothing more.

How a task is built

The authors collected issues and the pull requests that fixed them from 12 Python repositories.¹ For each, a task gives the agent:

• the repository at the commit before the fix,
• the issue text.

The pull request’s own code change is the reference solution, hidden from the agent. The pull request’s test changes become the grading tests.

That design has one great strength: the tasks are real. They have the messiness of actual bug reports, actual codebases and actual tests, not puzzles written for the benchmark.

How a patch is scored

Two kinds of test decide the result:¹

The agent’s patch is applied with the benchmark’s test patch in place, the tests run, and the task is resolved only if every test in both sets passes.¹ There’s no partial credit, and no human or model judge.

Like τ-bench for customer-service agents, which compares the database at the end of a conversation with a goal state, SWE-bench grades by effect rather than by what the agent says it did.⁴ That’s why both are trusted more than benchmarks judged by another model.

Tests as a grader: strengths and gaps

Grading by tests is what makes SWE-bench credible: no model’s opinion is involved, and anyone can rerun it. But tests encode one particular fix. A patch that solves the issue a different way, perhaps better, can fail a test that checks the reference fix’s exact behaviour. And a patch can pass the tests while doing something a maintainer would reject, such as special-casing the test input. The PASS_TO_PASS set catches some collateral damage, but only where the repository already had tests. Untested code can break without any effect on the score. These aren’t flaws peculiar to SWE-bench; they’re the limits of any test suite used as a judge.

What Verified changed

Real issues bring real problems for grading: an issue may not say enough to pin down the fix, and fixed tests may not accept every valid fix. SWE-bench Verified is the response most leaderboards now use: “a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality”.²

Most current leaderboard numbers are on Verified. Two things follow. Scores on Verified and on the full set aren’t comparable. And a 500-task set is small enough that a few tasks move the percentage noticeably.

The memorization question

The repositories and their fixes are public, and have been for years, so a model may have seen them in training. A 2025 study tested this directly. Given only the issue text, models named the buggy file path with up to 76% accuracy on SWE-bench, but up to 53% on repositories outside it. They also found higher verbatim similarity on SWE-bench Verified than on other coding benchmarks. The authors conclude that gains may be partially driven by memorization rather than problem-solving.³

That doesn’t make scores meaningless. It means a score on public, older issues is an upper bound on what to expect on your private codebase, which no model has seen.

What a score leaves out

The grading looks only at the final patch against the tests. Everything else an agent does on the way is invisible to the score:

The first row is the one that matters for safety. SWE-bench runs agents in disposable environments, which is exactly right for a benchmark. In your environment, the same agent’s commands run against your files, your credentials and sometimes your databases. A model that resolves many issues is a capable model; it isn’t evidence that it’s careful.

Using SWE-bench numbers sensibly

1. Compare like with like: same variant (Verified or full), same harness, similar dates.
2. Read it as capability, not safety. A strong score says the agent can often fix code. It says nothing about what it does when a command fails or a test is in the way.
3. Discount for memorization when predicting performance on private code.³
4. Run your own small benchmark. Twenty issues from your own history, graded by your own tests, tell you more about your codebase than any leaderboard.
5. Put guardrails around the run, independent of the score: a sandbox, scoped credentials, and a hook that asks before destructive commands. Our Claude Code hook post and the checklist before an agent touches production cover the basics.

For a benchmark that scores reliability across repeated tries rather than single passes, see τ-bench and pass^k. For measuring the guardrails themselves, see how to benchmark a guardrail. To see how a gate treats an agent’s database writes in your browser, try the playground; to grade what a coding agent can reach on your machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.