A 2023 benchmark tested chatbots as agents across eight worlds

The conversation

In August 2023, a group of researchers released AgentBench, submitted 7 August 2023, describing it as the first benchmark built to evaluate a language model acting as an agent across a deliberately varied set of interactive environments rather than a single task type. The project's own repository lists the eight environments included: five built specifically for this benchmark, covering an operating-system shell, a database, a knowledge graph, a digital card game, and lateral-thinking puzzles, plus three adapted from earlier research projects, covering household tasks, web shopping, and web browsing.

What the documents show

The paper reports evaluating 29 models across these eight environments, split between API-based commercial systems and open-source models. It reports that gpt-4 achieved the paper's highest overall AgentBench score, noting that gpt-4 solved 78% of the web-browsing environment's tasks as a specific example of 'practical usability in this scenario', with claude-2 and an earlier Claude model following behind it. Among open-source models the paper tested, codellama-34b is reported as the strongest, reaching an overall score the paper states as 0.96. The authors summarise the comparison by stating that 'top commercial LLMs present a strong ability of acting as agents in complex environments', while finding 'a significant disparity in performance' between those systems and most open-source competitors sized 70 billion parameters or smaller at the time of testing.

The system boundary

Each AgentBench environment is a bounded simulation with its own scoring rule, whether that means a passing shell command, a correct database query, or a completed purchase in a mock storefront; none of the eight environments is a live production system, and the paper's overall score is a weighted combination the authors constructed to make otherwise very different task scores comparable. The paper is careful to note that even its strongest model was 'not qualified as a practically usable agent' by the authors' own overall judgement, distinguishing a good benchmark score from a system ready to operate unsupervised.

Where it fails

The paper attributes most agent failures to 'poor long-term reasoning, decision-making, and instruction following', naming these as the main obstacles across the environments it tested rather than any single missing feature. It also reports a counterintuitive finding that training a model on code had 'ambivalent impacts on different agent tasks', complicating the common assumption that code training generally helps agent behaviour. Because the reported scores describe specific 2023-era model versions, a reader should treat any claim about a current model's AgentBench performance as a separate, later measurement rather than a figure from this paper.

• Which of the eight AgentBench environments most resembles the task an operator is actually deploying an agent for?
• Is a cited AgentBench score for a current model drawn from this original paper or from an unrelated later run?
• Does the gap this paper found between commercial and open-source agents still describe the models an operator is comparing today?

AgentBench's own contribution was showing that acting as an agent across varied environments exposed weaknesses that single-task benchmarks did not, at the specific point in 2023 when the paper's 29 models were tested.