← Roadmap

Training Compute-Optimal Large Language Models

Hoffmann et al. · 2022

① Intuition

Before Chinchilla, the dominant belief about large language model scaling came from Kaplan et al. (2020): parameters matter far more than training tokens. GPT-3 (175B parameters, 300B tokens) was the embodiment of that belief — an enormous model trained on a comparatively small dataset. The Kaplan scaling laws suggested that for a fixed compute budget you should spend almost all of it on parameters, keeping the token count roughly constant.

Hoffmann et al. (2022) ran a much more careful set of scaling experiments and reached a different conclusion. When you hold compute fixed and vary both the number of parameters N and the number of tokens D simultaneously, you find an interior optimum: the loss is minimized when N and D grow at roughly the same rate — not 10:1 as Kaplan implied, but close to 1:1. The paper named their compute-optimal 70B model “Chinchilla” and showed it outperforms the 280B Gopher model on almost every benchmark — using exactly the same compute budget — simply because Gopher was trained on too few tokens.

Before playing with the widget: if you increase log₁₀(params) while keeping log₁₀(tokens) fixed, what do you expect to happen to the loss? Does it always decrease?

Interact with the widget: drag both sliders and watch how the predicted loss changes. Notice that for a fixed log₁₀(C) (shown in the stats panel), moving tokens toward log₁₀(D) = log₁₀(N) + 1.3 (roughly D = 20·N) lowers the loss more than simply adding parameters. Toggle “Show Chinchilla-optimal” to see the green marker at D* = 20·N.

L (current): 2.3946
log₁₀(C): 20.78

x-axis: log₁₀(tokens) [9 → 13]  |  y-axis: loss  |  red = current  

After playing: for a fixed compute budget C = 6ND, why does there exist a best (N, D) pair rather than 'more params always wins'?

Retrieval check

What was wrong with the Kaplan et al. (2020) scaling prescription that Chinchilla corrected?
Chinchilla (70B params, 1.4T tokens) vs Gopher (280B params, 300B tokens) — same compute budget. What explains Chinchilla's better performance?

Answer all 2 correctly to mark this stage mastered.

② Math

The Chinchilla paper fits a parametric model to the observed training loss across hundreds of runs:

L(N,D)=E+ANα+BDβL(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

Here E is the irreducible entropy of natural language, A/N^alpha captures the contribution from model size, and B/D^beta captures the contribution from dataset size. The fitted constants are approximately A = 406, B = 410, alpha = 0.34, beta = 0.28, E = 1.70.

Given a fixed compute budget C, the total training FLOPs are approximated as:

C≈6NDC \approx 6 N D

so D = C / (6N). Substituting into L(N, D) and minimising over N (equivalently over D, since C is fixed) yields:

N∗∝Ca,D∗∝Cb,a=βα+β≈0.46,b=αα+β≈0.54N^* \propto C^{a}, \quad D^* \propto C^{b}, \quad a = \frac{\beta}{\alpha + \beta} \approx 0.46, \quad b = \frac{\alpha}{\alpha + \beta} \approx 0.54

Both exponents are near 0.5, confirming the near-symmetric scaling. The ratio D*/N* = 20 follows from plugging the fitted A, B, alpha, beta values into the first-order condition.

Given C = 6ND fixed, substitute D = C/(6N) into L(N,D). Why does an interior minimum in N exist — why doesn't L always decrease as N increases?
Show next line
dLdN=−αANα+1+βB⋅6βCβNβ−1=0  ⇒  N∗∝Cβ/(α+β)\frac{dL}{dN} = -\frac{\alpha A}{N^{\alpha+1}} + \frac{\beta B \cdot 6^\beta}{C^\beta} N^{\beta - 1} = 0 \;\Rightarrow\; N^* \propto C^{\beta/(\alpha+\beta)}
Plugging alpha = 0.34 and beta = 0.28 into the optimality condition gives D*/N* ≈ 20. Which direction does the exponent asymmetry push — should the optimal model have more parameters or more tokens per FLOP?
Show next line
D∗≈20 N∗  ⟸  D∗N∗=(A αB β)1/(α+β)≈20D^* \approx 20\, N^* \;\Longleftarrow\; \frac{D^*}{N^*} = \left(\frac{A\,\alpha}{B\,\beta}\right)^{1/(\alpha+\beta)} \approx 20

Retrieval check

The Chinchilla paper fits L(N,D) = E + A/N^alpha + B/D^beta. What does E represent and why can't it be reduced by adding more parameters or data?
Why does computing D* = C / (6 N*) require knowing the fitted exponents alpha and beta, rather than just the compute budget C?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the four steps of a compute-optimal training recipe in the correct order:

  1. Train a model with N* parameters on D* tokens, monitoring loss
  2. Compute optimal N* = (C/6)^(beta/(alpha+beta)) * constant and D* = C / (6 * N*)
  3. Measure final loss L and compare to the parametric prediction L(N*, D*)
  4. Choose compute budget C (FLOPs available for training)

Fill in the blank: given optimal parameter count N* and compute budget C, the implied token count D* is:

def chinchilla_tokens(N_star, C):
    # N_star: optimal parameter count
    # C: total compute budget in FLOPs
    D_star = ______
    return D_star

Retrieval check

In the pseudocode, why is the factor 6 used in C = 6 * N * D?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement a Chinchilla loss predictor in NumPy. Given parameter count N and token count D, compute the predicted loss using the parametric fit. Then, given a fixed compute budget C and a range of N values, find the D implied by D = C/(6N) and the corresponding predicted loss, and verify that the minimum is achieved near D* = 20*N.

Retrieval check

In the chinchilla_optimal function, why do we search over N using np.logspace rather than np.linspace?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Training Compute-Optimal Large Language Models (Chinchilla).

Open in new tab · Download

Suggested reading order: §3 (Approach — overview of the three methods used to estimate optimal compute allocation) → §3.1 (Approach 1, 2, 3 — IsoFLOP profiles, parametric modelling, and optimal frontier fitting) → §4 (Results — Chinchilla vs Gopher/GPT-3/Megatron on downstream benchmarks).