A 2023 benchmark made models fix real bugs, not toy ones

The conversation

In October 2023, researchers introduced SWE-bench in a paper submitted 10 October 2023, later published at ICLR 2024, an evaluation framework built from real, historical GitHub issues rather than hand-written coding puzzles. Instead of asking a model to write a short function from a prompt, SWE-bench hands a model an entire codebase and a filed issue's description, and asks it to produce a patch that resolves it. The benchmark's own current site, swebench.com, now hosts an expanded family of leaderboards, described here as retrieved on 16 September 2026, separate from the original paper's baseline numbers.

What the documents show

The paper describes a three-stage pipeline for building the benchmark from about 90,000 pull requests across 12 popular Python repositories. First, candidate pull requests are scraped. Second, an attribute filter keeps only merged pull requests that resolve a stated issue and add tests, since a contributed test lets the benchmark check whether a later patch fixes the problem. Third, an execution filter runs each candidate's test content before and after the fix, keeping only instances with a test that changes from failing to passing, discarding any producing installation or runtime errors. That pipeline reduced 90,000 candidates to 2,294 verified task instances. At publication, the best baseline, Claude 2 using retrieval to locate relevant files, resolved 1.96% of issues, rising to 4.8% when given the exact files a fix touched, an 'oracle' setting the authors distinguish from realistic retrieval.

The system boundary

SWE-bench measures whether a proposed patch makes a project's own test suite pass, which the paper frames as a verifiable stand-in for whether an issue is genuinely resolved; it does not evaluate code review, deployment safety, or a maintainer's judgement about design quality. The model under test does not merge its own patch or interact with the repository's maintainers; grading is fully automated against test results, and a human curator's earlier filtering, not the model, decided which historical issues counted as valid tasks.

Where it fails

The paper's results show resolving an issue often requires coordinating changes across multiple files and functions, going 'far beyond traditional code generation' for the 2023-era models tested; even the best baseline solved only 'the simplest issues' by the authors' own description. The authors also report different models did not solve overlapping issue sets, so one overall score can obscure very different strengths. Because the score is a resolve rate against tests the original contributors wrote, a builder should not assume it measures judgement about issues lacking an existing test to check against.

• Was the reported SWE-bench score produced under an oracle file-retrieval setting or a more realistic retrieval setting, and which one does the operator's use case resemble?
• Does the repository being worked on resemble the 12 popular, well-tested Python projects the benchmark was built from?
• Is a later system's SWE-bench number a figure the original 2023 paper reported, or a claim from a separate later submission?

SWE-bench's own contribution was showing how little of a real GitHub issue backlog 2023 models could clear; later leaderboard entries describe a different, ongoing measurement, not a revision of that original finding.