An NVIDIA paper had GPT-4 write and save its own Minecraft skills

The conversation

On 25 May 2023, a team led by Guanzhi Wang and including researchers from NVIDIA, Caltech, UT Austin, Stanford and Arizona State posted a paper introducing Voyager, described as 'the first LLM-powered embodied lifelong learning agent in Minecraft that continuously explores the world, acquires diverse skills, and makes novel discoveries without human intervention.' Voyager queries GPT-4 through its API; the paper is explicit that no model weights are fine-tuned. The project page frames the work as an agent that writes and saves its own code as reusable skills rather than relying on a fixed action set.

What the documents show

Voyager has three parts by the authors' description: an automatic curriculum proposing the next task from the agent's current state, a skill library storing executable code for behaviors already learned, and an iterative prompting loop feeding environment feedback and execution errors back to GPT-4 so it can revise failing code. The paper's headline comparison, against what it calls prior state-of-the-art Minecraft agents, reports Voyager obtaining 3.3 times more unique items, traveling 2.3 times longer distances, and unlocking tech-tree milestones up to 15.3 times faster. These are the authors' own figures from their own evaluation setup, not a third-party leaderboard score, so they describe relative improvement over the specific baselines the paper selected rather than an absolute, externally comparable score.

The system boundary

Voyager's skill library lets it reuse a solution once learned instead of re-deriving it; the paper reports stored skills also transfer to solving novel tasks in a newly instantiated world. But the system does not learn in the reinforcement-learning sense of updating a policy's parameters: all persistence lives in saved code and text, and the boundary of what it can do is set by what GPT-4 can be prompted to write and by the fixed control primitives (movement, crafting, combat) the authors expose. A self-verification step, itself an LLM call, judges task success; the paper documents cases where that judgment is wrong.

Where it fails

The paper's own limitations section names three problems: cost, since GPT-4 access is reported as 15 times more expensive than GPT-3.5 while Voyager depends on GPT-4's code quality; inaccuracies, where the agent gets stuck and the self-verification module misses a success signal, such as not recognizing a dropped spider string as proof a spider was defeated; and hallucination, where the automatic curriculum proposes items that do not exist in Minecraft, such as a 'copper sword,' or generated code calls a fuel source, cobblestone, that the game does not accept. The authors state plainly that they expect future model improvements, not design changes, to close these gaps.

• Is the improvement measured against a comparable baseline, or only the specific systems this paper chose?
• What happens when self-verification itself misjudges success or failure?
• How much of the capability depends on GPT-4, given the paper's own cost comparison to GPT-3.5?

Voyager is documented evidence that an agent can accumulate a growing, reusable code library inside one game across a long-running session; it is not evidence, on the paper's own terms, that the same approach transfers cheaply to domains without Minecraft's scriptable primitives.