← Roadmap

Language Models are Few-Shot Learners

Brown et al. · 2020

① Intuition

GPT-2 showed that a large decoder-only language model could do useful things zero-shot. GPT-3 pushed that idea to an extreme: 175 billion parameters trained on 300 billion tokens drawn from Common Crawl, WebText2, Books, and Wikipedia. At this scale, something qualitatively new emerges — in-context learning.

In-context learning means the model can “learn” a new task purely from examples placed in its prompt, with zero gradient updates. Show it two English→French pairs and a new English phrase, and it translates. Show it three arithmetic examples and a new equation, and it solves it. The weights never change; the “learning” is entirely a forward pass conditioned on the context.

The critical lesson from GPT-3 is that this emergent ability is scale-dependent. Small models (1B params) show nearly no few-shot gains. Above roughly 6B, few-shot accuracy rises sharply. At 175B, GPT-3 matches or beats fine-tuned smaller models on many NLP benchmarks without a single gradient step on those tasks.

Interact with the widget: start with shots=0 and log₁₀(params)=8. Observe the low baseline accuracy. Now slide log₁₀(params) up past 9.5 — accuracy rises, but slowly. Then set shots to 3 or 5 and repeat: the curve lifts noticeably only at large scale. Few-shot in-context learning unlocks at roughly 6B+ parameters (log₁₀ ≈ 9.8).

Accuracy: 26.0%

x-axis: log₁₀(parameters) [8 → 11.25]  |  y-axis: accuracy  |  red dot = current setting

Why does in-context learning get better as model size increases, even though no gradient updates happen? What does the model need to 'have' to learn from a few prompt examples?

Retrieval check

GPT-3 uses 175B parameters trained on 300B tokens. What dataset contributed the largest portion of training tokens?
What distinguishes 'zero-shot', 'one-shot', and 'few-shot' prompting for GPT-3?

Answer all 2 correctly to mark this stage mastered.

② Math

GPT-3’s training objective is standard autoregressive language modeling — predict each token given all preceding tokens:

L=−∑t=1Tlog⁡pθ(xt∣x1,…,xt−1)L = -\sum_{t=1}^{T} \log p_\theta(x_t \mid x_1, \ldots, x_{t-1})

At inference time, few-shot prompting is pure prompt-conditioning. Given k demonstrations and a query, the model scores:

pθ(y∣x1,y1,…,xk,yk,xquery)p_\theta(y \mid x_1, y_1, \ldots, x_k, y_k, x_{\text{query}})

No parameter updates. The prompt acts as a soft specification of the task distribution.

Scaling laws (Kaplan et al. 2020, developed alongside GPT-3) describe how the optimal loss scales with model size N when compute budget is fixed:

L(N)≈(NcN)αL(N) \approx \left(\frac{N_c}{N}\right)^{\alpha}

where N_c is a critical parameter count and alpha is approximately 0.076 for language modeling. Doubling parameters improves loss, but with diminishing returns. This law motivated training a single 175B model rather than many smaller models.

GPT-3 was trained with batch size 3.2M tokens. At 175B parameters, roughly how many FLOPs does a single forward+backward pass consume? (Use the rule of thumb: ~6N FLOPs per token for a transformer, where N = number of parameters.)
Show next line
6×N×Btokens≈6×1.75×1011×3.2×106≈3.4×1021 FLOPs6 \times N \times B_{\text{tokens}} \approx 6 \times 1.75 \times 10^{11} \times 3.2 \times 10^6 \approx 3.4 \times 10^{21} \text{ FLOPs}
Few-shot prompting vs. fine-tuning: how many gradient updates does GPT-3 few-shot inference require for a new task?
Show next line
∇θ=0 (frozen weights)⇒few-shot=forward pass only\nabla_\theta = \mathbf{0} \text{ (frozen weights)} \quad \Rightarrow \quad \text{few-shot} = \text{forward pass only}

Retrieval check

According to the Kaplan scaling law L(N) ≈ (N_c/N)^α, what does doubling the number of parameters N do to the loss?
In GPT-3 few-shot inference, what role does the prompt play mathematically?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the steps of GPT-3 few-shot inference in the correct order:

  1. Tokenize the full prompt into token IDs
  2. Run a forward pass through all 96 transformer decoder layers
  3. Collect k demonstration pairs (x_1, y_1), ..., (x_k, y_k) for the task
  4. Concatenate demonstrations and query into a single prompt string
  5. Sample or argmax from the output distribution to get the predicted answer

Fill the blank in the few-shot prompt construction pseudocode:

def build_kshot_prompt(examples, query, separator='\n'):
    parts = [f"{x} {y}" for x, y in examples]
    return ______

Retrieval check

In the few-shot pseudocode above, why do we append the query after the separator at the end?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement a function build_kshot_prompt(examples, query, separator) that constructs a k-shot prompt string. Each example is a (input_str, output_str) tuple. The prompt should interleave examples with the separator, then append the query at the end separated by the same separator. Verify that the output contains both examples, the query, and that prompt length grows with k.

Retrieval check

In a k-shot prompt, why do we include both the input AND output of each demonstration example (not just the input)?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download GPT-3: Language Models are Few-Shot Learners.

Open in new tab · Download

Suggested reading order: §1 (Introduction — motivation for in-context learning and scale) → §2.1 (Approach / In-Context Learning — formal definition of zero/one/few-shot) → §3.1 (Results on NLP benchmarks — few-shot vs. fine-tuned baselines).