① Intuition
Diffusion models learn to generate data by learning to reverse a gradual noising process. The key insight: if you corrupt an image by adding tiny amounts of Gaussian noise at each of T steps, the result is eventually pure noise. A neural network trained to reverse each small step — to denoise — can then generate novel images by starting from pure noise and denoising T times.
Setup: imagine slowly dissolving a photo into static. At step 0 you have the original. At step T (~1000) you have pure Gaussian noise. The model learns to run this film backward.
Predict: at timestep T (~1000), how much of x_0's information remains in x_T? Type your guess.
~0 — the forward chain is designed so that ᾱ_T → 0, meaning x_T converges to an isotropic Gaussian 𝒩(0, I) regardless of what x_0 was. All original signal is destroyed; only noise remains.
Now explore the widget below. Drag the timestep slider and switch between schedules to see how quickly signal is destroyed.
x_t = √ᾱ_t · x_0 + √(1 − ᾱ_t) · ε (ε ~ 𝒩(0, I))
Predict: does the linear schedule destroy information faster or slower than the cosine schedule at early timesteps?
The linear schedule destroys signal faster in the early steps. The cosine schedule keeps ᾱ_t higher for longer, preserving low-frequency structure through more of the forward process. This means the reverse model has more signal to work with at each step, yielding better sample quality — which is why the cosine schedule was introduced as an improvement.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Forward process — each step adds a small amount of Gaussian noise:
Closed-form marginal — the key algebraic trick: repeated reparameterization collapses T steps into one:
where .
Show next line
What should the network predict? Rather than predicting x_0 directly, Ho et al. found predicting the noise ε gives better results. The loss simplifies to:
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the steps of the DDPM training loop in correct order:
- Sample ε ~ 𝒩(0, I)
- Compute x_t = √ᾱ_t · x_0 + √(1−ᾱ_t) · ε
- Compute L = ‖ε − ε_θ‖²
- Predict ε_θ(x_t, t)
- Sample t ~ Uniform(1, T)
- Backprop and update θ
Complete the pseudocode — what variable does the network predict?
def train_step(x0, model, betas):
t = sample_uniform(1, T)
eps = sample_normal(shape=x0.shape)
x_t = sqrt(alpha_bar[t]) * x0 + sqrt(1 - alpha_bar[t]) * eps
eps_pred = model(x_t, t)
loss = mean((eps - eps_pred) ** 2) # predicting: ___
loss.backward()The blank is:
Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement the DDPM forward process in NumPy: compute alpha bars and sample x_t using the closed form.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §2 Background → §3 Diffusion models → §3.2 Reverse process → §4 Experiments.