Hugging Face bet that agents should write code, not JSON, for actions

The conversation

On 1 July 2024, Hugging Face researchers Aymeric Roucher and Sergei Petrov published a blog post reporting that a 'Code Agent' built with the library then called transformers.agents had topped the public leaderboard for GAIA, a benchmark for general AI assistants already on file. Five months later, on 31 December 2024, Hugging Face relaunched the framework as a separate, standalone library named smolagents, stating directly that 'smolagents is the successor to transformers.agents, and will be replacing it as transformers.agents gets deprecated in the future.'

What the documents show

The July 2024 post reports its ReactCodeAgent, running on GPT-4o with no fine-tuning, scored 44.2% on GAIA's validation set, ranking first overall, '4 points above the second,' and 33.3% on the test set, ranking second behind an unnamed submission but ahead of what the post identifies as a Microsoft AutoGen submission. Both figures are Hugging Face's own reported scores on the GAIA leaderboard, not an independent audit. The December 2024 post does not repeat that score; it instead frames the library's core design choice as 'Code Agents,' agents that write actions as executable code rather than a JSON block of tool names and arguments, citing outside research it says shows code actions outperform JSON-style tool calls.

The system boundary

Both the July and December posts describe the library's own definition of agency as a spectrum: a plain function call that a program executes has no agency, while a system where a language model's output can change how many steps run, or which tools get called, sits further along that spectrum. The July post is explicit about scope: 'LangChain and LlamaIndex do not support Code actions out of the box to our knowledge,' framing transformers.agents as the library making that action format central, not as claiming those other frameworks cannot ever call code. smolagents still supports the more conventional 'ToolCallingAgent,' which writes actions as JSON or text, alongside its code-writing 'CodeAgent.'

Where it fails

The July post's own account states plainly, 'Our submission was done with GPT-4o (alas),' and speculates a fine-tuned open model might reduce parsing errors and score higher, an acknowledgment the result depended on a closed, general-purpose model rather than one purpose-trained for the task. For execution safety, the December post notes sandboxed execution, via a service called E2B, is offered as an option rather than the default behavior. The smolagents repository, Apache 2.0 licensed, describes itself only as 'a barebones library for agents that think in code,' not a claim about task accuracy generally.

• Was a cited result produced under the transformers.agents name or the later, renamed smolagents library?
• Did the agent run inside a sandboxed environment, or with direct code execution on the host machine?
• Is the underlying model a general-purpose one like GPT-4o, or one fine-tuned for the specific tool-use task?

Hugging Face's own documents describe a single design bet, code as the action format, carried across a name change; the GAIA leaderboard placement is a dated, self-reported result tied to a specific model and submission, not a permanent property of either library name.