Toolformer's authors listed five limits alongside its reported gains

The conversation

On 9 February 2023, Timo Schick and coauthors at Meta AI posted 'Toolformer: Language Models Can Teach Themselves to Use Tools' to arXiv. The paper proposes training a model 'to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction,' using examples the model generates about itself rather than hand-labeled demonstrations. Meta AI's own publication page confirms the method integrates six tools: a calculator, a question-answering system, two search engines, a machine translation system and a calendar.

What the documents show

The paper reports evaluating the resulting model, built on a 6.7-billion-parameter base, on subsets of the LAMA factual-recall benchmark, on math word-problem sets including ASDiv, SVAMP and MAWPS, on the multilingual MLQA question-answering benchmark, and on temporal question sets called TEMPLAMA and a related dataset. Across these benchmarks, the authors report Toolformer 'substantially improves zero-shot performance' and, on several of them, performs competitively with or ahead of the much larger 175-billion-parameter GPT-3, despite Toolformer's own base model being roughly twenty-five times smaller.

The system boundary

Toolformer's own paper is explicit about what the model does not do: the calculator, search engines, translator and calendar it calls are external systems the model queries through defined APIs, and the model incorporates whatever text those APIs return into its own next-token predictions. The model decides whether and when to insert a call during generation, but the tools themselves execute outside the model, in the same client-tool pattern later platforms would formalize.

Where it fails

The paper's own limitations section, numbered section seven, lists several the authors state directly: Toolformer 'is unable to use tools in a chain,' since each API call is generated independently with no training examples of chained tool use; it cannot use a tool interactively, so a search engine's results cannot be browsed or a query refined; the decision to call an API is 'sensitive to the exact wording' of the input; the method is 'very sample-inefficient,' with the authors noting that processing over a million documents yielded only a few thousand useful calculator examples; and the model does not weigh the computational cost of making a call when deciding whether to make one.

• Would a deployment need chained tool calls that this documented single-call design cannot express?
• How sensitive is a given prompt's tool-triggering behavior to small wording changes?
• What is the computational or dollar cost of unnecessary tool calls the model was not trained to avoid?

The paper is a research method evaluated on specific benchmarks with a specific base model; its authors' own stated limitations, not just its reported gains, are part of what the document establishes.