① Intuition
Before GPT-2, the standard playbook for NLP was pretrain then fine-tune: train a language model on unlabeled text, then add a task-specific head and supervised training for each downstream task. GPT-2’s central claim is more radical: a large enough language model trained on diverse web text can perform many tasks zero-shot — with no gradient updates — simply by conditioning the model on a natural-language prompt that describes the task.
The key ingredients are a decoder-only Transformer (causal, left-to-right attention), 1.5 billion parameters, and 40 GB of high-quality web text (WebText). At inference time, the task is framed as text: “Translate English to French: cheese =>” or “Q: What is the capital of France? A:”. The model completes the prompt without ever seeing labeled examples.
Temperature sampling is central to GPT-2’s generation quality. The model outputs a logit vector over the vocabulary; dividing by temperature T before softmax sharpens (low T) or flattens (high T) the resulting distribution.
Interact with the widget: set T to 0.3, observe that “the” dominates with very high probability. Then raise T to 1.5 — probability mass spreads across more tokens. This single scalar controls the creativity-coherence tradeoff in every GPT-family model.
Entropy: 2.080 bits
GPT-2 was trained as a plain language model (predict the next token) yet it can do translation, summarization, and QA zero-shot. How does the training objective enable task generalization without any task-specific supervision?
A language model trained on diverse web text implicitly learns to predict text that follows questions, answers, translations, summaries, and code — because the web contains all of these. When you prompt with “Translate English to French: cheese =>”, the model has seen similar patterns and learns that “fromage” is the statistically likely continuation. The task emerges from the data distribution, not from labels.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
At each generation step, the model produces a logit vector z over the vocabulary. Temperature T rescales these logits before the softmax:
The entropy of the resulting distribution measures its spread:
High entropy means the model is uncertain and assigns non-trivial probability to many tokens (creative, diverse). Low entropy means the model is confident and concentrates on a single token (predictable, coherent).
Show next line
Show next line
Retrieval check
Answer all 1 correctly to mark this stage mastered.
③ Pseudo code
Arrange the GPT-2 generation loop in the correct order:
- Apply softmax to get probability distribution: p = softmax(z_T)
- Tokenize the input prompt into a sequence of token IDs
- Sample next token from p (or take argmax if T≈0), append to sequence
- Divide logits by temperature T: z_T = z / T
- Forward pass through all Transformer decoder layers to get logit vector z
Fill the blank in the temperature-softmax pseudocode:
def generate_next_token(model, tokens, temperature):
logits = model.forward(tokens) # shape: [vocab_size]
logits_T = ______
probs = softmax(logits_T)
return sample(probs)Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement temperature-scaled softmax in NumPy. Given a logit array and a temperature scalar, return a probability distribution. The function should handle T=1 (standard softmax), T close to 0 (near-greedy), and verify that output probabilities sum to 1.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §2 (Approach — WebText dataset, decoder-only architecture, and how tasks are framed as language modeling) → §3 (Experiments — zero-shot results on language modeling, translation, summarization, and QA benchmarks) → §4 (Generalization vs Memorization — analysis of overlap between WebText and test sets).