← Roadmap

Voyager: An Open-Ended Embodied Agent with Large Language Models

Wang, Xie, Jiang, Mandlekar, Xiao, Zhu, Fan, Anandkumar · 2023

① Intuition

General-purpose agents struggle in open-ended environments like Minecraft because they face two compounding problems. First, tasks have long horizons: surviving and thriving requires hundreds of interdependent steps — chop wood, craft tools, mine stone, smelt iron — none of which can be solved with a single prompt. Second, every skill is potentially useful again: if an agent learns to mine coal in episode 3, it shouldn’t re-derive that procedure from scratch in episode 47. Prior embodied agents either hard-coded a fixed skill set (brittle) or used raw action traces (non-reusable when the world changes). The result: agents that plateau early and never compound their knowledge.

Voyager’s insight is to treat a growing skill library as the agent’s long-term memory and to write skills as executable code rather than recorded action sequences. Code generalizes: mineWood(count=10) works regardless of the agent’s exact starting position, whereas a trace of button presses replays incorrectly whenever the world layout differs. Voyager builds three mutually reinforcing modules on GPT-4: (1) an automatic curriculum that proposes progressively harder tasks, (2) a skill library that stores verified JavaScript/Mineflayer code snippets keyed by natural-language descriptions and retrieved by embedding similarity, and (3) an iterative prompting loop that writes, executes, receives environment feedback, and self-improves the code until it passes.

Before touching the widget: if you slide the episode counter from 0 to 4, you expect to see 4 skills unlock in sequence. Which skill do you predict unlocks last — `craftFurnace`, `mineCoal`, or `mineStone`? And why does the dependency graph matter for that answer?

Interact with the widget: slide the episode counter from 1 to 8 and watch skills unlock one by one. Then toggle “Show dependencies” to reveal the dependency graph. Notice how smeltIron has three incoming edges — it is gated on three earlier skills all appearing in the library. This is the structure that emerges naturally when skills are stored as composable code with explicit function calls.

Voyager — Skill Library

0

library size = 0

  • ○mineWood(ep 1)
  • ○craftPickaxe(ep 2)
  • ○mineStone(ep 3)
  • ○craftFurnace(ep 4)
  • ○mineCoal(ep 5)
  • ○mineIron(ep 6)
  • ○smeltIron(ep 7)
  • ○craftIronSword(ep 8)

After playing with the widget: at episode 6 the library contains 6 skills. If a new task arrives that requires `smeltIron`, which stored skills would the retrieval system likely surface, and why would using code snippets (rather than action traces) make those retrieved skills directly reusable?

Retrieval check

Why does Voyager store skills as executable code rather than as recorded action traces?
Voyager's automatic curriculum proposes new tasks by conditioning on the agent's current state. What three inputs does the curriculum module use?

Answer all 2 correctly to mark this stage mastered.

② Math

The skill library M stores entries of the form (d_i, c_i, e_i) where d_i is a natural-language description, c_i is the executable code, and e_i = \text{embed}(d_i) is the embedding vector. Retrieval for a query description q selects the top-k entries by cosine similarity:

topk(q, M)=argmax⁡k-largesti  ei⋅embed(q)∥ei∥  ∥embed(q)∥\text{top}_k(q,\, M) = \underset{k\text{-largest}}{\operatorname{argmax}}_{i} \;\frac{e_i \cdot \text{embed}(q)}{\|e_i\|\;\|\text{embed}(q)\|}

The automatic curriculum generates the next task t* by maximizing expected exploration conditioned on full agent state:

t∗=LLMcurriculum ⁣(state,  Cdone,  Cfail)t^* = \text{LLM}_{\text{curriculum}}\!\bigl(\text{state},\; C_{\text{done}},\; C_{\text{fail}}\bigr)

where state encodes current inventory and nearby blocks, C_{\text{done}} is the set of completed tasks, and C_{\text{fail}} is the set of recently failed tasks. This is not an explicit optimization — the LLM acts as a learned heuristic that has absorbed exploration principles during pretraining.

Why store skills as code and retrieve them by embedding similarity rather than by exact name lookup?
Show next line
topk(q,M):semantic match⇒reusable code retrieved\text{top}_k(q,M): \text{semantic match} \Rightarrow \text{reusable code retrieved}
The curriculum LLM conditions on both completed tasks and failed tasks. Why are failed tasks necessary — wouldn't completed tasks alone be sufficient to infer what to try next?
Show next line
t∗=LLMcurr(state,  Cdone,  Cfail)⇒avoid already-failed, push frontiert^* = \text{LLM}_{\text{curr}}(\text{state},\; C_{\text{done}},\; C_{\text{fail}}) \Rightarrow \text{avoid already-failed, push frontier}

Retrieval check

In the skill library formula, cosine similarity is used rather than Euclidean distance. What is the key property of cosine similarity that makes it preferable for embedding-based retrieval?
Voyager's iterative prompting loop writes code, executes it, and feeds back the execution result. What information from the execution environment is included in the feedback prompt?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of Voyager’s main episode loop in the correct order:

  1. Mark task complete or failed; update completed_tasks / failed_tasks; advance to next episode
  2. Retrieve top-k skills from library: context_skills = top_k(embed(t), M)
  3. If task succeeded: add new skill to library — M.add(description, code, embed(description))
  4. Curriculum proposes next task t = LLM_curriculum(state, completed_tasks, failed_tasks)
  5. Iteratively prompt GPT-4 to write/debug code with execution feedback until verified or retry limit hit

Fill in the blank — the line that inserts a newly verified skill into the library:

def add_skill(library, description, code):
    embedding = embed(description)
    library.append(_______)
    return library

Retrieval check

In the Voyager loop, skills are only added to the library after the task succeeds. Why not add the skill optimistically after the first code draft, before execution?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement a minimal skill-library retrieval in NumPy. Given a list of skill dicts (each with "description" and "embedding" as a np.ndarray), a query_embedding (np.ndarray), and an integer k, return the top-k skills sorted by descending cosine similarity to the query. If the library has fewer than k entries, return all of them.

Retrieval check

In the retrieve_top_k implementation, both the query and skill embeddings are L2-normalized before the dot product. What happens if you skip normalization and use raw dot products instead?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Voyager: An Open-Ended Embodied Agent with Large Language Models (Wang et al. 2023).

Open in new tab · Download

Suggested reading order: §1 (Introduction — motivation from hard-coded vs. open-ended agents) → §3 (Method — overview of the three modules) → §3.3 (Skill Library — storage, retrieval, and verification details).