① Intuition
By 2023, large language models had shown remarkable capabilities in reasoning and generation — but they had a fundamental blind spot: they were frozen in time, unable to perform reliable arithmetic, and incapable of deterministic lookups. A model trained on data from 2021 has no knowledge of 2022 events. Ask it to compute 17% of 3,847 and it will hallucinate a plausible-sounding but wrong answer. Ask it for today’s exchange rates and it fabricates stale numbers with confident prose.
The naive fix — fine-tune the model on tool-augmented data — requires expensive human annotation of when and how to call each tool. Toolformer sidesteps this entirely using self-supervision: the model itself generates candidate tool-call insertions into plain text, executes the tools to get results, and keeps only those insertions that measurably reduce the loss on the tokens that follow. The surviving insertions become training data. No human ever labels a single API call. The model teaches itself, from its own predictions about what information would be useful.
Before using the widget: you have a sentence with a blank — 'The revenue grew from 12M to __ this year.' If you match the correct tool (Calculator) vs. a mismatched tool (Translate), what do you expect to see in the perplexity reduction? Will both tools help the model fill the blank?
Only the Calculator call produces a result that actually helps predict the continuation. The Translate tool returns a Spanish word, which is irrelevant to a sentence about revenue — it adds tokens but no signal, so the model’s cross-entropy on the following text barely changes. Toolformer’s key insight is using this loss reduction as a filter: keep the tool call if and only if it helps the model predict what comes next.
Interact with the widget: select a sentence from the dropdown, then try each tool. Observe that only the matched tool (e.g., Calculator for the revenue sentence) produces a large perplexity reduction shown in green. A mismatched tool shows near-zero reduction in red — Toolformer would discard that candidate during training.
The revenue grew from 12M to [Calculator(12 * 1.25) → 15M] 15M this year.
Useful completion — perplexity reduction: −4.2 nats
After playing with the widget: Toolformer uses loss reduction as its only quality signal for tool calls. What is one type of tool call that might slip through this filter incorrectly — i.e., appear useful by this metric but actually be wrong or harmful?
A tool call can reduce loss on the training-text continuation without being factually correct. For example, a Wikipedia search might return an article snippet that happens to match the wording of the training text (high surface overlap, low loss) even if the semantic content is misleading in context. The filter is purely statistical — it measures correlation with the training corpus, not ground-truth correctness. This is why Toolformer still occasionally calls tools inappropriately at inference time: the self-supervised signal is a useful but noisy proxy for actual usefulness.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
For each position i in a text x and a candidate tool call c with arguments, the model executes the tool to get result r. Define two loss values over the continuation tokens x_{i+1:n}:
This is the loss on the continuation without the tool call — conditioning only on the original prefix.
This is the loss on the continuation with the tool call and its result inserted. The insertion [c -> r] is placed at position i, so the model conditions on both the original prefix and the tool’s output.
The filtering criterion keeps the insertion if and only if:
where tau is a threshold hyperparameter (set to 1.0 in the paper). Intuitively: if the tool result reduces the cross-entropy on the continuation by at least tau nats, the call was useful and the example is kept for fine-tuning.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of the Toolformer self-supervised training pipeline in the correct order:
- Keep the insertion if L_minus - L_plus >= tau
- Fine-tune the language model on the corpus of kept augmented examples
- Execute tool c to obtain result r
- Compute L_minus (continuation loss without call) and L_plus (continuation loss with [c→r])
- Sample position i and generate candidate tool call c with args using few-shot prompting
Fill in the blank — the filtering condition that decides whether a tool call is retained:
def keep_call(l_minus, l_plus, tau):
return _______Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement the Toolformer filtering step in NumPy. Given arrays l_minus and l_plus of per-candidate continuation losses and a threshold tau, return a boolean mask of the candidates that should be kept for fine-tuning.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §2 (Method — the self-supervised data generation and filtering pipeline) → §3 (Experiments — benchmark results across all five tools and comparison to GPT-3) → §5 (Discussion — limitations of the loss-reduction heuristic and failure modes).