← Roadmap

Language Models are Unsupervised Multitask Learners

Radford, Wu, Child, Luan, Amodei, Sutskever · 2019

① Intuition

Before GPT-2, the standard playbook for NLP was pretrain then fine-tune: train a language model on unlabeled text, then add a task-specific head and supervised training for each downstream task. GPT-2’s central claim is more radical: a large enough language model trained on diverse web text can perform many tasks zero-shot — with no gradient updates — simply by conditioning the model on a natural-language prompt that describes the task.

The key ingredients are a decoder-only Transformer (causal, left-to-right attention), 1.5 billion parameters, and 40 GB of high-quality web text (WebText). At inference time, the task is framed as text: “Translate English to French: cheese =>” or “Q: What is the capital of France? A:”. The model completes the prompt without ever seeing labeled examples.

Temperature sampling is central to GPT-2’s generation quality. The model outputs a logit vector over the vocabulary; dividing by temperature T before softmax sharpens (low T) or flattens (high T) the resulting distribution.

Interact with the widget: set T to 0.3, observe that “the” dominates with very high probability. Then raise T to 1.5 — probability mass spreads across more tokens. This single scalar controls the creativity-coherence tradeoff in every GPT-family model.

the
46.4%
a
23.1%
my
12.7%
any
9.4%
each
5.7%
some
2.8%

Entropy: 2.080 bits

GPT-2 was trained as a plain language model (predict the next token) yet it can do translation, summarization, and QA zero-shot. How does the training objective enable task generalization without any task-specific supervision?

Retrieval check

What does 'zero-shot' mean in the context of GPT-2?
What effect does lowering temperature T toward 0 have on the next-token distribution?

Answer all 2 correctly to mark this stage mastered.

② Math

At each generation step, the model produces a logit vector z over the vocabulary. Temperature T rescales these logits before the softmax:

p(xt∣x<t)=softmax ⁣(zT)t=exp⁡(zt/T)∑jexp⁡(zj/T)p(x_t \mid x_{<t}) = \text{softmax}\!\left(\frac{\mathbf{z}}{T}\right)_t = \frac{\exp(z_t / T)}{\sum_j \exp(z_j / T)}

The entropy of the resulting distribution measures its spread:

H=−∑ipilog⁡piH = -\sum_i p_i \log p_i

High entropy means the model is uncertain and assigns non-trivial probability to many tokens (creative, diverse). Low entropy means the model is confident and concentrates on a single token (predictable, coherent).

As T→0, what happens to the softmax output? Derive the limiting behavior.
Show next line
lim⁡T→0softmax(z/T)=earg⁡max⁡z(one-hot on highest logit)\lim_{T \to 0} \text{softmax}(\mathbf{z}/T) = \mathbf{e}_{\arg\max \mathbf{z}} \quad \text{(one-hot on highest logit)}
As T→∞, what happens to the softmax output? Derive the limiting behavior.
Show next line
lim⁡T→∞softmax(z/T)=(1V,…,1V)(uniform)\lim_{T \to \infty} \text{softmax}(\mathbf{z}/T) = \left(\tfrac{1}{V}, \ldots, \tfrac{1}{V}\right) \quad \text{(uniform)}

Retrieval check

Which quantity measures how spread out the next-token distribution is?

Answer all 1 correctly to mark this stage mastered.

③ Pseudo code

Arrange the GPT-2 generation loop in the correct order:

  1. Apply softmax to get probability distribution: p = softmax(z_T)
  2. Tokenize the input prompt into a sequence of token IDs
  3. Sample next token from p (or take argmax if T≈0), append to sequence
  4. Divide logits by temperature T: z_T = z / T
  5. Forward pass through all Transformer decoder layers to get logit vector z

Fill the blank in the temperature-softmax pseudocode:

def generate_next_token(model, tokens, temperature):
    logits = model.forward(tokens)   # shape: [vocab_size]
    logits_T = ______
    probs = softmax(logits_T)
    return sample(probs)

Retrieval check

In the generation loop, when do we stop producing new tokens?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement temperature-scaled softmax in NumPy. Given a logit array and a temperature scalar, return a probability distribution. The function should handle T=1 (standard softmax), T close to 0 (near-greedy), and verify that output probabilities sum to 1.

Retrieval check

Why do we subtract the max logit before exponentiating in the softmax implementation?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download GPT-2: Unsupervised Multitask Learners.

Open in new tab · Download

Suggested reading order: §2 (Approach — WebText dataset, decoder-only architecture, and how tasks are framed as language modeling) → §3 (Experiments — zero-shot results on language modeling, translation, summarization, and QA benchmarks) → §4 (Generalization vs Memorization — analysis of overlap between WebText and test sets).