A custom computer interface, not a new model, drove a coding gain

The conversation

On 6 May 2024, John Yang, Carlos E. Jimenez and colleagues at Princeton posted a paper introducing SWE-agent, describing it as 'an autonomous system that uses a language model to interact with a computer to solve software engineering tasks.' The paper's central claim is about interface design rather than a new model: it argues that a custom agent-computer interface (ACI), a purpose-built set of commands for viewing, searching and editing files, changes how well a language model can use a computer, separately from which model sits behind it. The system is evaluated on SWE-bench, already on file, a benchmark several of the same authors previously built, though the two are presented as separate contributions in this paper.

What the documents show

Using GPT-4 Turbo (gpt-4-1106-preview) as the underlying model, the paper reports SWE-agent resolved 12.5% of the 2,294 SWE-bench test issues, which it describes as substantially ahead of a prior non-interactive retrieval-augmented baseline's 3.8%. Swapping in Claude 3 Opus (claude-3-opus-20240229) behind the same interface resolved 10.5% of issues, which the authors present as evidence the interface, not just the model, drives performance. The paper also reports successful runs cost a median $1.21 and 12 steps, against $2.52 and 21 steps for unsuccessful ones, and that 93% of resolved instances finished before exhausting their cost budget.

The system boundary

SWE-agent's interface deliberately replaces raw shell file-editing with a single edit command paired with a file viewer that shows surrounding lines and reports errors immediately; the paper's own ablation found removing this editor in favor of shell-only editing dropped the resolve rate to 10.3%. A linting guardrail blocks an edit from applying if it introduces a major syntax error, addressing what the authors call a common failure mode where a model repeatedly edits the same broken snippet. The system still depends entirely on the underlying model's code judgment; the interface only shapes what feedback the model receives and how it acts on it.

Where it fails

The paper documents an inefficient search behavior: when many results match a query, agents tend to page through every one with a 'next' command rather than narrowing the search, which can exhaust the cost or context budget and drop performance below the shell-only baseline. It also reports performance is sensitive to how many lines the file viewer shows at once, with both very short and very long windows scoring worse than a mid-sized one. The project repository, MIT licensed and later presented at NeurIPS 2024, states it can also be applied to offensive-security and competitive-coding tasks beyond SWE-bench.

• Is a reported resolve rate on SWE-bench or SWE-bench Lite, since the two report different denominators?
• How much of a system's performance comes from its interface design versus the underlying model swapped into it?
• Did the run finish inside its cost or step budget, or was it cut off before reaching a resolution?

SWE-agent's own results treat the interface between a model and a computer as a variable worth measuring on its own terms, distinct from both the benchmark it is scored on and the model doing the reasoning.