① Intuition
GPT-2 showed that a large decoder-only language model could do useful things zero-shot. GPT-3 pushed that idea to an extreme: 175 billion parameters trained on 300 billion tokens drawn from Common Crawl, WebText2, Books, and Wikipedia. At this scale, something qualitatively new emerges — in-context learning.
In-context learning means the model can “learn” a new task purely from examples placed in its prompt, with zero gradient updates. Show it two English→French pairs and a new English phrase, and it translates. Show it three arithmetic examples and a new equation, and it solves it. The weights never change; the “learning” is entirely a forward pass conditioned on the context.
The critical lesson from GPT-3 is that this emergent ability is scale-dependent. Small models (1B params) show nearly no few-shot gains. Above roughly 6B, few-shot accuracy rises sharply. At 175B, GPT-3 matches or beats fine-tuned smaller models on many NLP benchmarks without a single gradient step on those tasks.
Interact with the widget: start with shots=0 and log₁₀(params)=8. Observe the low baseline accuracy. Now slide log₁₀(params) up past 9.5 — accuracy rises, but slowly. Then set shots to 3 or 5 and repeat: the curve lifts noticeably only at large scale. Few-shot in-context learning unlocks at roughly 6B+ parameters (log₁₀ ≈ 9.8).
Accuracy: 26.0%
x-axis: log₁₀(parameters) [8 → 11.25] | y-axis: accuracy | red dot = current setting
Why does in-context learning get better as model size increases, even though no gradient updates happen? What does the model need to 'have' to learn from a few prompt examples?
A model doing in-context learning must already know the relevant concepts from pretraining. With few examples in the prompt, it must recognize the task, map inputs to outputs according to demonstrated examples, and generalize the pattern. Smaller models lack the capacity to store and compose fine-grained world knowledge, so they cannot extract the task signal from the prompt reliably. Larger models have seen more patterns, stored richer representations, and have enough capacity to perform implicit Bayesian updating over their prior — all within the forward pass.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
GPT-3’s training objective is standard autoregressive language modeling — predict each token given all preceding tokens:
At inference time, few-shot prompting is pure prompt-conditioning. Given k demonstrations and a query, the model scores:
No parameter updates. The prompt acts as a soft specification of the task distribution.
Scaling laws (Kaplan et al. 2020, developed alongside GPT-3) describe how the optimal loss scales with model size N when compute budget is fixed:
where N_c is a critical parameter count and alpha is approximately 0.076 for language modeling. Doubling parameters improves loss, but with diminishing returns. This law motivated training a single 175B model rather than many smaller models.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the steps of GPT-3 few-shot inference in the correct order:
- Tokenize the full prompt into token IDs
- Run a forward pass through all 96 transformer decoder layers
- Collect k demonstration pairs (x_1, y_1), ..., (x_k, y_k) for the task
- Concatenate demonstrations and query into a single prompt string
- Sample or argmax from the output distribution to get the predicted answer
Fill the blank in the few-shot prompt construction pseudocode:
def build_kshot_prompt(examples, query, separator='\n'):
parts = [f"{x} {y}" for x, y in examples]
return ______Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement a function build_kshot_prompt(examples, query, separator) that constructs a k-shot prompt string. Each example is a (input_str, output_str) tuple. The prompt should interleave examples with the separator, then append the query at the end separated by the same separator. Verify that the output contains both examples, the query, and that prompt length grows with k.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §1 (Introduction — motivation for in-context learning and scale) → §2.1 (Approach / In-Context Learning — formal definition of zero/one/few-shot) → §3.1 (Results on NLP benchmarks — few-shot vs. fine-tuned baselines).