A Berkeley paper fine-tuned a small model to call real APIs correctly

The conversation

On 24 May 2023, Shishir G. Patil, Tianjun Zhang and Joseph E. Gonzalez of UC Berkeley, with Xin Wang of Microsoft Research, posted a paper introducing Gorilla, a LLaMA-7B model fine-tuned specifically to write correct calls to machine-learning APIs. The paper opens by naming the problem directly: 'their potential to effectively use tools via API calls remains unfulfilled,' pointing to GPT-4's tendency to invent arguments or misuse an API's syntax. Rather than prompt a general model to call tools, as OpenAI's and Anthropic's later function-calling launches (already on file) would let developers do, Gorilla trains a smaller model on the calls themselves.

What the documents show

The paper built APIBench, a dataset of 1,645 real API calls collected from three model hubs: 925 from Hugging Face, 626 from TensorFlow Hub's versioned listings, and 94 from Torch Hub. To score a generated call, the authors match its abstract syntax tree against the reference call rather than checking exact text. On this metric, the paper reports zero-shot Gorilla, without a document retriever, scoring 20.43 percentage points higher than GPT-4 and 10.75 points higher than ChatGPT, and as much as 83% higher than the base LLaMA-7B model it was fine-tuned from. These are the authors' own AST-matching scores on their own dataset, not scores on an external, third-party leaderboard.

The system boundary

Gorilla's stated advance is combining fine-tuning with retrieval: when paired with a document retriever, the paper reports the model can adapt to a changed or updated API document at test time without retraining, which it frames as reducing hallucinated API usage. But retrieval quality bounds this benefit; the authors' own ablation reports that swapping in a weaker retriever, BM25 instead of their GPT-based one, degraded accuracy by over 52 percentage points. The model covers only the three API domains in APIBench; it was not trained or evaluated on general-purpose web or enterprise APIs outside those hubs.

Where it fails

The authors' own limitations section is narrow: it flags that APIBench draws on machine-learning APIs whose underlying models can themselves produce biased predictions if trained on skewed data, and frames the released dataset as a way for others to study that risk, rather than describing Gorilla's own calling accuracy as a limitation in that section. The Berkeley team later folded this line of work into a running leaderboard, described on the project's own site as testing 2,000 question-function-answer pairs across several programming languages, a living benchmark distinct from the fixed APIBench numbers in the original paper.

• Was the calling accuracy measured against the same reference APIs the model was trained on, or unseen ones?
• How much of any reported accuracy gain depends on the retriever quality rather than the fine-tuned model itself?
• Does a newer benchmark score for a similar-sounding system come from the same test harness as the one being compared to?

Gorilla is documented evidence that fine-tuning a small model on real API-call examples can outperform prompting a larger general model on the same narrow task, within the three domains and matching method the paper itself defines.