The ReAct paper measured its gains on named benchmarks, not in general

The conversation

In October 2022, Shunyu Yao and coauthors posted 'ReAct: Synergizing Reasoning and Acting in Language Models' to arXiv, proposing a prompting method that has the model generate 'both reasoning traces and task-specific actions in an interleaved manner,' rather than choosing one or the other. The authors' own project page frames the goal as letting reasoning steps help track and revise a plan while actions retrieve outside information, so each kind of output supports the other within a single prompted transcript.

What the documents show

The paper reports evaluating ReAct on question answering with HotpotQA, fact verification with FEVER, and two interactive decision-making benchmarks, ALFWorld and WebShop. On the question-answering and verification tasks, the authors state ReAct 'overcomes issues of hallucination and error propagation' found in chain-of-thought prompting by letting the model query a simple Wikipedia API rather than reason from memory alone. On the decision-making benchmarks, the paper reports ReAct outperforming imitation- and reinforcement-learning baselines by an absolute success rate of 34 percentage points on ALFWorld and 10 points on WebShop, using only one or two in-context examples rather than task-specific training.

The system boundary

ReAct is a prompting method, not a system with its own enforcement layer: the paper's design routes each 'action' step through an external interface, such as a Wikipedia search API or a simulated household environment, that returns an observation the model then reasons over. The method does not itself validate an action before it runs; whatever environment or API ReAct is wired into is what actually performs the lookup or the simulated step, and the paper evaluates the technique inside benchmark environments built for that purpose rather than against arbitrary real-world tools.

Where it fails

The project page states directly that 'ReAct isn't perfect,' and includes a documented failure case from ALFWorld where the interleaved reasoning does not prevent an unproductive action. The reported gains are measured on specific benchmarks with specific models available in 2022, and the authors do not claim the method transfers unchanged to every task, model or tool; a claim that ReAct 'solves' agentic reasoning generally goes beyond what the paper itself measured.

• Which of the paper's benchmarks, if any, resembles the task a builder is actually deploying against?
• Does the interleaved reasoning trace get logged somewhere a person can review it after a failure?
• What happens when the external tool ReAct calls returns an unexpected or malformed observation?

The paper and its project page together establish a specific method, specific benchmarks and a specific, bounded improvement; they do not establish a general claim about reasoning-and-acting models beyond that evaluated scope.