← Roadmap

Classifier-Free Diffusion Guidance

Ho, Salimans · 2022

① Intuition

DDPM and LDM showed that diffusion models can generate stunning images. But left unconditioned, they sample from the full data distribution — there is no way to say “generate a photo of a red fox in snow.” Conditioning on a text prompt or class label is essential. The obvious approach: train an auxiliary classifier on noisy images at every noise level t, and at inference time use its gradient to steer the denoising trajectory toward the desired class. This is classifier guidance (Dhariwal and Nichol, 2021). It works — but it requires training a second model (the noisy-image classifier), which is expensive and tightly coupled to the specific noise schedule of the diffusion model.

Ho and Salimans (2022) asked: can we guide without any classifier at all? The answer is yes. The key insight: a single model can implicitly encode both a conditional score and an unconditional score. During training, the conditioning signal c (e.g. a class label or text embedding) is randomly replaced with a null token ∅ with probability p_uncond (typically ~10%). The network learns to denoise with the condition when it is present, and without it when it is absent. At inference time, you run the network twice per denoising step — once with c and once with ∅ — then combine the two noise predictions with a guidance weight w. Cranking up w trades diversity for sharpness: the generated distribution shrinks toward the conditional mode.

Before using the widget: if w=0 in the CFG formula, what do you expect the guided distribution to look like compared to the conditional and unconditional distributions?

Interact with the widget: drag the guidance scale slider from 0 to 10. The orange curve is the guided density. Toggle “Show both source distributions” to reveal the conditional (blue, narrower peak) and unconditional (grey, wider and shifted left). Watch how increasing w sharpens the orange curve and pulls its mode toward the conditional mode.

Classifier-Free Guidance

mode x: 2.26sharpness: 1.323

After playing with the widget at high w (e.g. w=10): the mode of the guided distribution moves past the conditional mode and the tails narrow dramatically. What does this tell you about the cost of very high guidance?

Retrieval check

What practical problem does Classifier-Free Guidance solve compared to classifier guidance?
During CFG training, why is the condition c randomly replaced with ∅ rather than simply training two separate models?

Answer all 2 correctly to mark this stage mastered.

② Math

The CFG inference formula combines the conditional and unconditional noise predictions with guidance weight w:

ε~θ(xt,c)=(1+w) εθ(xt,c)−w εθ(xt,∅)\tilde{\varepsilon}_\theta(x_t, c) = (1 + w)\,\varepsilon_\theta(x_t, c) - w\,\varepsilon_\theta(x_t, \varnothing)

In score-function form this is equivalent to:

∇log⁡p~(x∣c)=(1+w) ∇log⁡p(x∣c)−w ∇log⁡p(x)\nabla \log \tilde{p}(x \mid c) = (1+w)\,\nabla \log p(x \mid c) - w\,\nabla \log p(x)

The formula linearly extrapolates from the unconditional score toward the conditional score. When w=0 you recover the pure conditional score. When w=1 you get 2conditional - 1unconditional, pushing further toward the conditional mode. The training loss is the standard DDPM objective applied jointly:

L=Ex0,c,ε,t ⁣[∥ε−εθ(xt,cdrop)∥2]\mathcal{L} = \mathbb{E}_{x_0, c, \varepsilon, t}\!\left[\|\varepsilon - \varepsilon_\theta(x_t, c_{\text{drop}})\|^2\right]

where c_drop equals c with probability (1 - p_uncond) and equals ∅ with probability p_uncond. The network sees both branches during training and learns to handle both. This is the entire training change — no new loss terms, no new model.

At w=0, what does the CFG formula ε̃ = (1+w)ε(x_t,c) - w·ε(x_t,∅) reduce to?
Show next line
ε~θ∣w=0=(1+0) εθ(xt,c)−0⋅εθ(xt,∅)=εθ(xt,c)\tilde{\varepsilon}_\theta\big|_{w=0} = (1+0)\,\varepsilon_\theta(x_t, c) - 0\cdot\varepsilon_\theta(x_t,\varnothing) = \varepsilon_\theta(x_t, c)
Why does increasing w beyond 0 sharpen the guided distribution toward the conditional mode?
Show next line
ε~=εθ(xt,c)+w (εθ(xt,c)−εθ(xt,∅))⏟guidance direction: ∇log⁡p(c∣xt)\tilde{\varepsilon} = \varepsilon_\theta(x_t, c) + w\,\underbrace{(\varepsilon_\theta(x_t, c) - \varepsilon_\theta(x_t, \varnothing))}_{\text{guidance direction: } \nabla\log p(c \mid x_t)}

Retrieval check

In the CFG score form ∇log p̃(x|c) = (1+w)∇log p(x|c) - w∇log p(x), what does the expression simplify to after collecting terms?
What is the role of p_uncond (the null-conditioning dropout probability) during CFG training?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the four steps of a single CFG denoising step in the correct order:

  1. Apply standard DDPM update rule using e_guided to obtain x_{t-1}
  2. Combine with guidance weight: e_guided = (1 + w) * e_cond - w * e_uncond
  3. Compute unconditional noise prediction: e_uncond = eps_theta(x_t, null)
  4. Compute conditional noise prediction: e_cond = eps_theta(x_t, c)

Fill in the blank — the CFG combination step:

def cfg_combine(e_cond, e_uncond, w):
    # e_cond, e_uncond: noise predictions, same shape
    # w: guidance scale (>=0)
    e_guided = ______
    return e_guided

Retrieval check

During CFG inference, the model is called twice per denoising step. What are the two calls and what do they return?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the CFG combination in NumPy. Given arrays e_cond and e_uncond (noise predictions of the same shape), and a guidance scale w, compute e_guided. Then verify that w=0 returns the conditional prediction and that w>0 shifts the result in the direction of e_cond - e_uncond.

Retrieval check

The CFG formula can be rewritten as e_guided = e_cond + w*(e_cond - e_uncond). What does the term (e_cond - e_uncond) represent geometrically?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Classifier-Free Diffusion Guidance (Ho, Salimans 2022).

Open in new tab · Download

Suggested reading order: §1 (Introduction — motivation for avoiding a classifier and the dropout trick) → §2 (Classifier-Free Guidance — the training procedure and inference formula) → §4 (Experiments — FID vs. IS tradeoff curves showing the effect of varying w).