① Intuition
DDPM (Ho et al., 2020) proved that diffusion models can generate high-quality images — but there is a catch. The forward and reverse processes both operate directly on pixel arrays. At 512x512 resolution, that means working with 512 x 512 = 262,144 individual values per image. Every U-Net forward pass during training and every denoising step at inference time must process all those values. Attention scales quadratically in the number of tokens, so moving from 256x256 to 512x512 multiplies the attention cost by 16x. Pushing to 1024x1024 is almost impractical in pixel space.
Rombach et al. (2022) asked: do we actually need to run diffusion on raw pixels? Most of the redundancy in a natural image lives at the perceptual level — the fine-grained texture details — not at the semantic level. A powerful image encoder (the encoder half of a VAE) can compress a 512x512 image into a 64x64 latent grid — 64 times fewer spatial positions — while preserving everything semantically meaningful. Latent Diffusion Models (LDMs) run the entire diffusion process in this compressed latent space and only decode back to pixels once, at the very end. This is the architecture behind Stable Diffusion.
Before playing with the widget: if the VAE encoder uses a spatial downsampling factor f=8 on a 512x512 image, how many latent tokens will there be versus pixel tokens? And roughly how does compute change?
At f=8 the spatial size shrinks from 512 to 512/8=64 in each dimension, giving 64x64=4,096 latent tokens versus 512x512=262,144 pixel tokens — a 64x reduction in sequence length. Attention cost scales as tokens squared, so the FLOP savings are roughly 64^2 = 4,096x for the attention layers. This is the core efficiency argument of the paper.
Interact with the widget: select different resolutions and adjust the downsampling factor. Watch how token count and the compute ratio change. Notice that f=8 at 512x512 gives exactly the 4,096 latent tokens Stable Diffusion uses, compared to 262,144 pixel tokens.
Latent vs Pixel Diffusion Compute
FLOP proxy = tokens² × d_model (d_model = 320). Attention scales quadratically, so latent-space gives ≈ f⁴ FLOP savings.
After playing with the widget: why does the compute ratio grow so fast as f increases? What does that tell you about the architectural choice of f=8 in Stable Diffusion?
Because attention is quadratic in sequence length, halving the token count quarters the cost — so a factor-f downsampling gives f^4 FLOP savings for attention. At f=8, that is 4,096x cheaper. The paper found that f=4 to f=8 gives the best quality-efficiency tradeoff: below f=4, the savings are modest; above f=8, the latent is so compressed that the VAE loses perceptual quality and the diffusion model cannot recover it.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
LDM training happens in two phases. Phase 1 trains the VAE to learn a perceptually compressed latent representation. Phase 2 trains the diffusion model inside that latent space, with optional text conditioning via cross-attention.
Phase 1 — VAE training:
Here E is the encoder that maps image x to latent distribution q(z|x), D is the decoder that reconstructs x from a sample z, and the KL term regularises the latent space toward a standard Gaussian. A small lambda_KL (around 1e-6) provides just enough regularisation without over-compressing the latent.
Phase 2 — LDM training:
where z_t is the noisy latent at timestep t:
This is exactly the DDPM objective — except x_0 is replaced by z_0 (the VAE-encoded latent) and the U-Net now additionally receives a conditioning signal c (e.g., a text prompt encoded by CLIP). The conditioning is injected through cross-attention layers inside the U-Net:
The latent feature map z attends to the text embedding c: each spatial position in z queries which text tokens are most relevant, then aggregates their value vectors. This is how “a cat sitting on a bench” steers the denoised output toward cats and benches.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of LDM inference (text-to-image generation) in the correct order:
- Decode final clean latent z_0 back to pixel image x_0 via VAE decoder D(z_0)
- Sample pure noise latent z_T ~ N(0, I) at full latent resolution (H/f x W/f x C_z)
- Iteratively denoise: for t = T down to 1, predict noise eps_theta(z_t, t, c) and update z_{t-1}
- Encode text prompt to embedding c via frozen CLIP text encoder
Fill in the blank — the final pixel reconstruction step:
def ldm_decode(z_0, vae_decoder):
# z_0: denoised latent of shape (H/f, W/f, C_z)
# vae_decoder: pre-trained, frozen VAE decoder D
x_0 = ______
return x_0 # pixel image of shape (H, W, 3)Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement classifier-free cross-attention in NumPy. Given a latent feature matrix z of shape (N, d) and a text conditioning matrix c of shape (M, d_c), compute the cross-attention output using learned projections W_Q (d x d), W_K (d_c x d), W_V (d_c x d_v). The output should have shape (N, d_v) and each row of the attention weight matrix should sum to 1.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 (Method — overview of the two-stage pipeline) → §3.2 (Latent Diffusion Models — the core diffusion objective and why pixel vs. latent space matters) → §3.3 (Conditioning Mechanisms — cross-attention architecture for text, layout, and other conditioning signals).