Cognition's own benchmark showed most Devin runs failed

The conversation

On 12 March 2024, Cognition Labs published "Introducing Devin," describing it as "the first AI software engineer" and "a tireless, skilled teammate, equally ready to build alongside you or independently complete tasks for you to review." The post presents demonstrations of Devin learning unfamiliar technologies, building and deploying an application, finding and fixing bugs, and training its own model, run inside "a sandboxed compute environment" equipped with standard developer tools. Three days later, Cognition published a separate SWE-bench technical report detailing how Devin was evaluated on SWE-bench, described there as asking "agents to resolve real-world GitHub issues found in open source projects like Django and scikit-learn."

What the documents show

The technical report states Devin was evaluated "unassisted," meaning it was not told which files needed editing, while named comparison models, including a Claude 2 baseline, were "assisted" with that information; the report calls out this distinction rather than presenting the scores as measuring identical conditions. On a randomly selected "25% of the SWE-bench test set (570 out of the 2,294)" issues, the report states Devin resolved 79, a "13.86% success rate," against a previous unassisted state-of-the-art the report gives as "1.96%." The report also discloses that success required every provided test to pass, and that it checked for data contamination in the benchmark while noting the comparison baselines "face similar data contamination issues" rather than claiming Devin's figure was uniquely clean.

The system boundary

Cognition's own report draws the evaluation's boundary carefully: a 45-minute runtime limit, a pre-configured environment, and resetting "all of the test files to the original state, in case the agent modified the tests," are documented controls meant to keep the measured task the one a human reviewer would recognize as the original GitHub issue. The introduction post frames Devin as producing work "for you to review," keeping a human checkpoint as the intended way a result is validated, rather than describing its output as ready to merge without review.

Where it fails

A 13.86 percent resolution rate on a quarter of one benchmark, even substantially ahead of the cited prior baseline, means the reported majority of sampled issues were not resolved under Cognition's own test, and the report's own acknowledgment of data-contamination risk across all compared systems is reason to treat any single-run benchmark score as provisional rather than a stable measure of general coding ability. A demonstration recorded in a controlled, sandboxed environment with defined task boundaries is also not the same claim as unsupervised performance on an arbitrary real-world repository.

• Was the reported success rate measured on the specific kind of repository and issue type this task resembles?
• What review step happens before code the agent wrote or a bug fix it made is merged?
• Could the benchmark tasks used to measure this agent have appeared, in some form, in its training data?

Cognition's own two documents, an introduction naming a "teammate" and a technical report naming a 13.86 percent resolution rate under stated conditions, together describe a demonstrated but partial capability, not a finished, unsupervised software engineer.