Anthropic disclosed a fifteen percent score for computer use

The conversation

On 22 October 2024, Anthropic published "Claude 3.5 Sonnet and computer use," announcing an upgraded Claude 3.5 Sonnet, a new Claude 3.5 Haiku, and a public beta letting Claude operate a computer "by looking at a screen, moving a cursor, clicking buttons, and typing text." The post describes an API through which a developer's application supplies Claude screenshots and executes the mouse and keyboard actions Claude requests in return, letting Claude "translate instructions ... into computer commands." Anthropic frames the release explicitly as a beta: "we're releasing computer use early for feedback from developers, and expect the capability to improve rapidly over time."

What the documents show

Anthropic's own benchmark figures are specific: on OSWorld, described as a benchmark that "evaluates AI models' ability to use computers like people do," Claude 3.5 Sonnet "scored 14.9% in the screenshot-only category," which the post says was "notably better than the next-best AI system's score of 7.8%," and "22.0%" "when afforded more steps to complete the task." Anthropic states these are its own reported figures on that benchmark, not an independently audited measure, and names specific weaknesses: "some actions that people perform effortlessly — scrolling, dragging, zooming — currently present challenges for Claude." Retrieved 16 September 2026, the current computer-use documentation describes a maturer tool, with member actions such as screenshot, left_click, and zoom, run in a loop where Claude "analyzes the tool results to determine if more actions are needed."

The system boundary

Both documents locate responsibility for the environment outside the model. The documentation recommends "using a dedicated virtual machine or container with minimal privileges," "avoiding giving the model access to sensitive data," and "asking a human to confirm decisions that might result in meaningful real-world consequences" — none of which the model itself enforces. It also names a specific attack surface: "Claude will follow commands found in content even when they conflict with your instructions" in some circumstances, describing trained resistance and an added classifier layer as mitigations, not a claim the risk is eliminated.

Where it fails

A screenshot-only success rate in the teens, by Anthropic's own account, means most attempted OSWorld tasks did not succeed at launch, and the documented gap between an easy click and a difficult drag or zoom means failure is not evenly spread across the actions a screen-based task might require. The prompt-injection risk the documentation names is a standing one: content on a screen Claude is asked to operate can itself contain instructions.

• What is the task success rate on the specific kind of screen and workflow being automated, not just a general benchmark score?
• Is the computer-use session running in an isolated environment without access to sensitive accounts or data?
• What real-world-consequential actions require a human confirmation step before they execute?

Anthropic's own account, from the 14.9 percent screenshot-only score at launch to the current documentation's security precautions, describes an early and disclosed-as-imperfect capability rather than a finished automation product.