A self-hosted web benchmark showed GPT-4 browsing agents mostly failed

The conversation

On 25 July 2023, Shuyan Zhou and colleagues at Carnegie Mellon University posted a paper introducing WebArena, a self-hosted set of fully functional websites, covering e-commerce, a social forum, a collaborative software-development site, and a content-management system, built so that autonomous agents could be tested on realistic browsing tasks without touching the live internet. The paper pairs the environment with 812 test tasks and reports baseline results for its own prompted agents, ahead of later browser-agent products such as OpenAI's Operator, already on file.

What the documents show

The authors tested two agent designs, a 'direct' agent that predicts the next action straight from the page's accessibility tree and a 'reasoning' agent that writes out reasoning before acting, using gpt-3.5-turbo-0613 and gpt-4-0613. The paper's own headline figure is that its best configuration, the reasoning agent on GPT-4, completed 10.59% of the 812 tasks end to end; the GPT-3.5 reasoning agent reached 7.38%, and a GPT-3.5 direct agent reached 7.14%. These are the authors' own scores under their own programmatic checkers for task completion, run once per task rather than averaged over repeated trials, which the paper does not describe as a comprehensive account of every possible prompting strategy.

The system boundary

WebArena's tasks are checked programmatically against a defined goal state, such as a specific database entry or account balance, rather than judged by a person or another model, which the authors present as what keeps the benchmark reproducible. The environment gives agents tools like a map and reference manuals to support human-like task-solving, but the paper is explicit that the 812 tasks were annotated by the authors themselves and one external annotator, with disagreements resolved by a third annotator, rather than sourced from real user logs.

Where it fails

The paper's own results section calls its best success rate 'relatively modest' and reports that reasoning before acting did not meaningfully outperform acting directly. Its per-template analysis found the GPT-4 agent reached full success on only one of 41 templates that had at least one successful run, while GPT-3.5 agents reached full success on none. The conclusion attributes common failures to the agent overestimating a task's infeasibility, over-relying on context already visible rather than exploring, and repeating actions after failing to notice a state had not changed. The project's repository, released under the Apache 2.0 license, ships the exact environment and evaluation scripts used to produce these numbers.

• Was a reported browsing-agent success rate measured on WebArena's own 812 tasks, or a different task set entirely?
• Does the agent's checker verify the actual end state, or only that some plausible-looking action was taken?
• How does the agent behave when it wrongly believes a task cannot be completed, versus when it truly cannot be?

WebArena's own numbers describe a 2023 GPT-4 agent failing roughly nine tasks in ten on a fixed, realistic browsing suite; the benchmark's value is that later systems can be compared against that specific, reproducible bar.