← Roadmap

Denoising Diffusion Probabilistic Models

Ho, Jain, Abbeel · 2020

① Intuition

Diffusion models learn to generate data by learning to reverse a gradual noising process. The key insight: if you corrupt an image by adding tiny amounts of Gaussian noise at each of T steps, the result is eventually pure noise. A neural network trained to reverse each small step — to denoise — can then generate novel images by starting from pure noise and denoising T times.

Setup: imagine slowly dissolving a photo into static. At step 0 you have the original. At step T (~1000) you have pure Gaussian noise. The model learns to run this film backward.

Predict: at timestep T (~1000), how much of x_0's information remains in x_T? Type your guess.

Now explore the widget below. Drag the timestep slider and switch between schedules to see how quickly signal is destroyed.

x_t = √ᾱ_t · x_0 + √(1 − ᾱ_t) · ε (ε ~ 𝒩(0, I))

Predict: does the linear schedule destroy information faster or slower than the cosine schedule at early timesteps?

Retrieval check

What is the distribution of x_T in the DDPM forward process?
Why does the cosine schedule produce better samples than the linear schedule?

Answer all 2 correctly to mark this stage mastered.

② Math

Forward process — each step adds a small amount of Gaussian noise:

q(xt∣xt−1)=N ⁣(xt;  1−βt xt−1,  βtI)q(x_t \mid x_{t-1}) = \mathcal{N}\!\left(x_t;\; \sqrt{1-\beta_t}\, x_{t-1},\; \beta_t I\right)

Closed-form marginal — the key algebraic trick: repeated reparameterization collapses T steps into one:

xt=αˉt x0+1−αˉt ε,ε∼N(0,I)x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\, \varepsilon, \quad \varepsilon \sim \mathcal{N}(0, I)

where αˉt=∏s=1t(1−βs)\bar\alpha_t = \prod_{s=1}^{t}(1-\beta_s).

What does q(x_t | x_0) equal after T steps as ᾱ_t → 0?
Show next line
q(xt∣x0)=N ⁣(xt;  αˉt x0,  (1−αˉt)I)→αˉt→0N(0,I)q(x_t \mid x_0) = \mathcal{N}\!\left(x_t;\; \sqrt{\bar\alpha_t}\,x_0,\; (1-\bar\alpha_t)I\right) \xrightarrow{\bar\alpha_t\to 0} \mathcal{N}(0, I)

What should the network predict? Rather than predicting x_0 directly, Ho et al. found predicting the noise ε gives better results. The loss simplifies to:

Lsimple=Et,x0,ε ⁣[∥ε−εθ(xt,t)∥2]L_\text{simple} = \mathbb{E}_{t, x_0, \varepsilon}\!\left[\bigl\|\varepsilon - \varepsilon_\theta(x_t, t)\bigr\|^2\right]
Which target is easier for the neural network: predicting x_0 or predicting ε given x_t?
Show next line
ε^=εθ(xt,t),Lsimple=E ⁣[∥ε−ε^∥2]\hat\varepsilon = \varepsilon_\theta(x_t, t), \quad L_\text{simple} = \mathbb{E}\!\left[\|\varepsilon - \hat\varepsilon\|^2\right]

Retrieval check

In the closed-form q(x_t|x_0), what is ᾱ_t?
L_simple in DDPM is a loss on:

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the steps of the DDPM training loop in correct order:

  1. Sample ε ~ 𝒩(0, I)
  2. Compute x_t = √ᾱ_t · x_0 + √(1−ᾱ_t) · ε
  3. Compute L = ‖ε − ε_θ‖²
  4. Predict ε_θ(x_t, t)
  5. Sample t ~ Uniform(1, T)
  6. Backprop and update θ

Complete the pseudocode — what variable does the network predict?

def train_step(x0, model, betas):
    t = sample_uniform(1, T)
    eps = sample_normal(shape=x0.shape)
    x_t = sqrt(alpha_bar[t]) * x0 + sqrt(1 - alpha_bar[t]) * eps
    eps_pred = model(x_t, t)
    loss = mean((eps - eps_pred) ** 2)   # predicting: ___
    loss.backward()

The blank is:

Retrieval check

In the DDPM training loop, which comes first: sampling t or computing x_t?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the DDPM forward process in NumPy: compute alpha bars and sample x_t using the closed form.

Retrieval check

Why use np.cumprod(1 - betas) to compute alpha bars?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download DDPM.

Open in new tab · Download

Suggested reading order: §2 Background → §3 Diffusion models → §3.2 Reverse process → §4 Experiments.