An open community project set out to test Devin-style coding agents

The conversation

On 23 July 2024, Xingyao Wang and a large academic and industry team, spanning UIUC, Carnegie Mellon, Yale, UC Berkeley and other institutions, posted a paper introducing a platform then named OpenDevin, footnoted as being 'initially inspired by the AI software engineer Devin,' the closed Cognition Labs demo already on file, while stating OpenDevin 'has quickly evolved to support a much wider range of applications beyond software engineering.' The paper describes an open, community-run project rather than a single vendor's product, reporting more than 1,300 contributions from over 160 contributors by the time of writing.

What the documents show

The paper reports results across benchmarks it built into the platform, including SWE-bench and WebArena, both already on file. On SWE-bench Lite, its CodeActAgent v1.8 using Claude 3.5 Sonnet is reported to reach a 26% resolve rate, called 'competitive' against other specialized open-source coding agents; the same agent using gpt-4o-mini solved 6.3% of problems at under 1% of the cost. On WebArena's 812 tasks, its BrowsingAgent v1.0 using GPT-4o reached a 15.5% success rate, ahead of several other reported systems the paper lists in the same table, including a GPT-4 Turbo-based agent from the WebArena paper itself scored at 3.3%. These are the authors' own benchmark runs, compiled into comparison tables alongside other groups' previously published numbers.

The system boundary

The paper describes OpenDevin's architecture as a state and event stream shared between an agent and a sandboxed Linux environment: a Docker container runs the agent's bash commands, and a Chromium browser built on Playwright handles web actions, keeping code execution isolated from the host machine. Multiple agents can be composed through a delegation mechanism, letting one agent hand a sub-task to another. The platform's default sandbox, at the time of the paper, relied on a library called pexpect to pass messages between the sandbox and the controller, which the authors say does not suit other sandbox types and is 'not stable enough.'

Where it fails

The paper's own limitations section, in its appendix, names unresolved gaps directly: multi-modal support is described as not yet handling images or video in a principled way, workflow construction still requires substantial hand-crafted effort rather than automatic generation, and the sandbox runtime is flagged as its own stability risk. The project has since been renamed OpenHands, still released under the MIT license, a change the arXiv paper's own later revisions record in its title.

• Was a reported OpenDevin or OpenHands benchmark score run on the full task set or a cost-limited subset, such as SWE-bench Lite?
• Which underlying model, and at what cost per run, produced the specific success rate being cited?
• Does the sandbox isolate code execution from the host system in the deployment being evaluated?

OpenDevin's own paper frames the platform as a community research base for comparing agents under shared benchmarks and a shared sandbox, not as a finished replication of Devin's closed demonstration.