← Roadmap

Toolformer: Language Models Can Teach Themselves to Use Tools

Schick, Dwivedi-Yu, Dessì, Raileanu, Lomeli, Zettlemoyer, Cancedda, Scialom · 2023

① Intuition

By 2023, large language models had shown remarkable capabilities in reasoning and generation — but they had a fundamental blind spot: they were frozen in time, unable to perform reliable arithmetic, and incapable of deterministic lookups. A model trained on data from 2021 has no knowledge of 2022 events. Ask it to compute 17% of 3,847 and it will hallucinate a plausible-sounding but wrong answer. Ask it for today’s exchange rates and it fabricates stale numbers with confident prose.

The naive fix — fine-tune the model on tool-augmented data — requires expensive human annotation of when and how to call each tool. Toolformer sidesteps this entirely using self-supervision: the model itself generates candidate tool-call insertions into plain text, executes the tools to get results, and keeps only those insertions that measurably reduce the loss on the tokens that follow. The surviving insertions become training data. No human ever labels a single API call. The model teaches itself, from its own predictions about what information would be useful.

Before using the widget: you have a sentence with a blank — 'The revenue grew from 12M to __ this year.' If you match the correct tool (Calculator) vs. a mismatched tool (Translate), what do you expect to see in the perplexity reduction? Will both tools help the model fill the blank?

Interact with the widget: select a sentence from the dropdown, then try each tool. Observe that only the matched tool (e.g., Calculator for the revenue sentence) produces a large perplexity reduction shown in green. A mismatched tool shows near-zero reduction in red — Toolformer would discard that candidate during training.

The revenue grew from 12M to __ this year.

The revenue grew from 12M to [Calculator(12 * 1.25) → 15M] 15M this year.

Useful completion — perplexity reduction: −4.2 nats

How Toolformer works: During self-supervised training, the model proposes API calls mid-sentence. Only calls that reduce perplexity significantly are kept, teaching the model when to invoke tools.

After playing with the widget: Toolformer uses loss reduction as its only quality signal for tool calls. What is one type of tool call that might slip through this filter incorrectly — i.e., appear useful by this metric but actually be wrong or harmful?

Retrieval check

Toolformer is trained on a dataset of text manually annotated by humans to indicate where tool calls should be inserted. True or false?
Toolformer's tool-call filter keeps a candidate insertion only when the inserted call reduces loss on what part of the text?

Answer all 2 correctly to mark this stage mastered.

② Math

For each position i in a text x and a candidate tool call c with arguments, the model executes the tool to get result r. Define two loss values over the continuation tokens x_{i+1:n}:

L−(x)=∑j=i+1n−log⁡pθ(xj∣x1:i)L_-(x) = \sum_{j=i+1}^{n} -\log p_\theta(x_j \mid x_{1:i})

This is the loss on the continuation without the tool call — conditioning only on the original prefix.

L+(x; c,r)=∑j=i+1n−log⁡pθ(xj∣x1:i, [c→r], xi+1:j−1)L_+(x;\,c,r) = \sum_{j=i+1}^{n} -\log p_\theta(x_j \mid x_{1:i},\, [c \to r],\, x_{i+1:j-1})

This is the loss on the continuation with the tool call and its result inserted. The insertion [c -> r] is placed at position i, so the model conditions on both the original prefix and the tool’s output.

The filtering criterion keeps the insertion if and only if:

L−(x)  −  L+(x; c,r)  ≥  τL_-(x)\;-\;L_+(x;\,c,r)\;\geq\;\tau

where tau is a threshold hyperparameter (set to 1.0 in the paper). Intuitively: if the tool result reduces the cross-entropy on the continuation by at least tau nats, the call was useful and the example is kept for fine-tuning.

Why not keep every candidate tool call rather than filtering by loss reduction? What goes wrong if tau = 0?
Show next line
τ=0⇒noisy calls kept⇒fine-tuning on spurious insertions\tau = 0 \Rightarrow \text{noisy calls kept} \Rightarrow \text{fine-tuning on spurious insertions}
Why is loss reduction on the continuation a valid proxy for 'the tool call was useful'?
Show next line
r informative⇒pθ(xi+1:n∣[c→r])↑⇒L+↓r \text{ informative} \Rightarrow p_\theta(x_{i+1:n} \mid [c \to r]) \uparrow \Rightarrow L_+ \downarrow

Retrieval check

In the Toolformer loss criterion, L_plus conditions on the tool result r. What does this mean computationally?
What is the role of the few-shot exemplars (5–10 per tool) used in Toolformer's data collection phase?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of the Toolformer self-supervised training pipeline in the correct order:

  1. Keep the insertion if L_minus - L_plus >= tau
  2. Fine-tune the language model on the corpus of kept augmented examples
  3. Execute tool c to obtain result r
  4. Compute L_minus (continuation loss without call) and L_plus (continuation loss with [c→r])
  5. Sample position i and generate candidate tool call c with args using few-shot prompting

Fill in the blank — the filtering condition that decides whether a tool call is retained:

def keep_call(l_minus, l_plus, tau):
    return _______

Retrieval check

In the pseudo code pipeline, step 3 computes both L_minus and L_plus using the same base model weights. When are those weights updated?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the Toolformer filtering step in NumPy. Given arrays l_minus and l_plus of per-candidate continuation losses and a threshold tau, return a boolean mask of the candidates that should be kept for fine-tuning.

Retrieval check

The filter returns (l_minus - l_plus) >= tau. What does it mean when this value is negative for a candidate?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al. 2023).

Open in new tab · Download

Suggested reading order: §2 (Method — the self-supervised data generation and filtering pipeline) → §3 (Experiments — benchmark results across all five tools and comparison to GPT-3) → §5 (Discussion — limitations of the loss-reduction heuristic and failure modes).