① Intuition
Before Chinchilla, the dominant belief about large language model scaling came from Kaplan et al. (2020): parameters matter far more than training tokens. GPT-3 (175B parameters, 300B tokens) was the embodiment of that belief — an enormous model trained on a comparatively small dataset. The Kaplan scaling laws suggested that for a fixed compute budget you should spend almost all of it on parameters, keeping the token count roughly constant.
Hoffmann et al. (2022) ran a much more careful set of scaling experiments and reached a different conclusion. When you hold compute fixed and vary both the number of parameters N and the number of tokens D simultaneously, you find an interior optimum: the loss is minimized when N and D grow at roughly the same rate — not 10:1 as Kaplan implied, but close to 1:1. The paper named their compute-optimal 70B model “Chinchilla” and showed it outperforms the 280B Gopher model on almost every benchmark — using exactly the same compute budget — simply because Gopher was trained on too few tokens.
Before playing with the widget: if you increase log₁₀(params) while keeping log₁₀(tokens) fixed, what do you expect to happen to the loss? Does it always decrease?
Loss is a function of both N and D. Increasing N while holding D fixed does reduce loss — but with diminishing returns. The term A/N^α in the parametric fit shrinks, but B/D^β stays constant. At some point adding more parameters yields almost no gain because the data term B/D^β dominates. That is the core message: a bigger model needs more tokens to justify its size.
Interact with the widget: drag both sliders and watch how the predicted loss changes. Notice that for a fixed log₁₀(C) (shown in the stats panel), moving tokens toward log₁₀(D) = log₁₀(N) + 1.3 (roughly D = 20·N) lowers the loss more than simply adding parameters. Toggle “Show Chinchilla-optimal” to see the green marker at D* = 20·N.
x-axis: log₁₀(tokens) [9 → 13] | y-axis: loss | red = current
After playing: for a fixed compute budget C = 6ND, why does there exist a best (N, D) pair rather than 'more params always wins'?
Because loss has two data-dependent and parameter-dependent terms that trade off. A/N^alpha rewards more parameters; B/D^beta rewards more data. Given C = 6ND, increasing N forces D = C/(6N) to decrease, eventually making the data term worse faster than the parameter term improves. The loss surface is bowl-shaped along the compute isocurve, so a finite optimum exists.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
The Chinchilla paper fits a parametric model to the observed training loss across hundreds of runs:
Here E is the irreducible entropy of natural language, A/N^alpha captures the contribution from model size, and B/D^beta captures the contribution from dataset size. The fitted constants are approximately A = 406, B = 410, alpha = 0.34, beta = 0.28, E = 1.70.
Given a fixed compute budget C, the total training FLOPs are approximated as:
so D = C / (6N). Substituting into L(N, D) and minimising over N (equivalently over D, since C is fixed) yields:
Both exponents are near 0.5, confirming the near-symmetric scaling. The ratio D*/N* = 20 follows from plugging the fitted A, B, alpha, beta values into the first-order condition.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the four steps of a compute-optimal training recipe in the correct order:
- Train a model with N* parameters on D* tokens, monitoring loss
- Compute optimal N* = (C/6)^(beta/(alpha+beta)) * constant and D* = C / (6 * N*)
- Measure final loss L and compare to the parametric prediction L(N*, D*)
- Choose compute budget C (FLOPs available for training)
Fill in the blank: given optimal parameter count N* and compute budget C, the implied token count D* is:
def chinchilla_tokens(N_star, C):
# N_star: optimal parameter count
# C: total compute budget in FLOPs
D_star = ______
return D_starRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement a Chinchilla loss predictor in NumPy. Given parameter count N and token count D, compute the predicted loss using the parametric fit. Then, given a fixed compute budget C and a range of N values, find the D implied by D = C/(6N) and the corresponding predicted loss, and verify that the minimum is achieved near D* = 20*N.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 (Approach — overview of the three methods used to estimate optimal compute allocation) → §3.1 (Approach 1, 2, 3 — IsoFLOP profiles, parametric modelling, and optimal frontier fitting) → §4 (Results — Chinchilla vs Gopher/GPT-3/Megatron on downstream benchmarks).