← Roadmap

High-Resolution Image Synthesis with Latent Diffusion Models

Rombach, Blattmann, Lorenz, Esser, Ommer · 2022

① Intuition

DDPM (Ho et al., 2020) proved that diffusion models can generate high-quality images — but there is a catch. The forward and reverse processes both operate directly on pixel arrays. At 512x512 resolution, that means working with 512 x 512 = 262,144 individual values per image. Every U-Net forward pass during training and every denoising step at inference time must process all those values. Attention scales quadratically in the number of tokens, so moving from 256x256 to 512x512 multiplies the attention cost by 16x. Pushing to 1024x1024 is almost impractical in pixel space.

Rombach et al. (2022) asked: do we actually need to run diffusion on raw pixels? Most of the redundancy in a natural image lives at the perceptual level — the fine-grained texture details — not at the semantic level. A powerful image encoder (the encoder half of a VAE) can compress a 512x512 image into a 64x64 latent grid — 64 times fewer spatial positions — while preserving everything semantically meaningful. Latent Diffusion Models (LDMs) run the entire diffusion process in this compressed latent space and only decode back to pixels once, at the very end. This is the architecture behind Stable Diffusion.

Before playing with the widget: if the VAE encoder uses a spatial downsampling factor f=8 on a 512x512 image, how many latent tokens will there be versus pixel tokens? And roughly how does compute change?

Interact with the widget: select different resolutions and adjust the downsampling factor. Watch how token count and the compute ratio change. Notice that f=8 at 512x512 gives exactly the 4,096 latent tokens Stable Diffusion uses, compared to 262,144 pixel tokens.

Latent vs Pixel Diffusion Compute

Pixel tokens (H×W)65,536
Latent tokens (H/f × W/f)1,024
Token savings (f²)64×
Pixel attn FLOP proxy (log₁₀)12.14
Latent attn FLOP proxy (log₁₀)8.53
Compute ratio (pixel / latent): 4096.00×

FLOP proxy = tokens² × d_model (d_model = 320). Attention scales quadratically, so latent-space gives ≈ f⁴ FLOP savings.

After playing with the widget: why does the compute ratio grow so fast as f increases? What does that tell you about the architectural choice of f=8 in Stable Diffusion?

Retrieval check

At 512x512 with downsampling factor f=8, how many latent tokens does LDM diffuse over?
What role does the VAE play in Latent Diffusion Models?

Answer all 2 correctly to mark this stage mastered.

② Math

LDM training happens in two phases. Phase 1 trains the VAE to learn a perceptually compressed latent representation. Phase 2 trains the diffusion model inside that latent space, with optional text conditioning via cross-attention.

Phase 1 — VAE training:

LVAE=∥x−D(E(x))∥2+λKL KL(q(z∣x) ∥ p(z))\mathcal{L}_{\text{VAE}} = \|x - D(E(x))\|^2 + \lambda_{\text{KL}}\, \mathrm{KL}(q(z|x)\,\|\,p(z))

Here E is the encoder that maps image x to latent distribution q(z|x), D is the decoder that reconstructs x from a sample z, and the KL term regularises the latent space toward a standard Gaussian. A small lambda_KL (around 1e-6) provides just enough regularisation without over-compressing the latent.

Phase 2 — LDM training:

LLDM=Ez0,ε,t ⁣[∥ε−εθ(zt,t,c)∥2]\mathcal{L}_{\text{LDM}} = \mathbb{E}_{z_0, \varepsilon, t}\!\left[\|\varepsilon - \varepsilon_\theta(z_t, t, c)\|^2\right]

where z_t is the noisy latent at timestep t:

zt=αˉt z0+1−αˉt ε,ε∼N(0,I)z_t = \sqrt{\bar{\alpha}_t}\, z_0 + \sqrt{1 - \bar{\alpha}_t}\,\varepsilon, \quad \varepsilon \sim \mathcal{N}(0, I)

This is exactly the DDPM objective — except x_0 is replaced by z_0 (the VAE-encoded latent) and the U-Net now additionally receives a conditioning signal c (e.g., a text prompt encoded by CLIP). The conditioning is injected through cross-attention layers inside the U-Net:

Q=WQz,K=WKc,V=WVcQ = W_Q z,\quad K = W_K c,\quad V = W_V cAttention(Q,K,V)=softmax ⁣(QK⊤d)V\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{Q K^\top}{\sqrt{d}}\right) V

The latent feature map z attends to the text embedding c: each spatial position in z queries which text tokens are most relevant, then aggregates their value vectors. This is how “a cat sitting on a bench” steers the denoised output toward cats and benches.

Why does LDM train the VAE first in a separate phase, rather than training VAE and diffusion model jointly end-to-end?
Show next line
Ltotal=LVAE⏟perceptual compression+LLDM⏟semantic generation(trained sequentially, not jointly)\mathcal{L}_{\text{total}} = \underbrace{\mathcal{L}_{\text{VAE}}}_{\text{perceptual compression}} + \underbrace{\mathcal{L}_{\text{LDM}}}_{\text{semantic generation}} \quad \text{(trained sequentially, not jointly)}
In the cross-attention equations Q = W_Q z, K = W_K c, V = W_V c, the queries come from z (the latent image features) and the keys/values come from c (the text embeddings). What does this mean intuitively for how text controls the image?
Show next line
Attn(z,c)=softmax ⁣(WQz (WKc)⊤d)(WVc)  : each pixel position queries all text tokens\text{Attn}(z, c) = \text{softmax}\!\left(\frac{W_Q z\,(W_K c)^\top}{\sqrt{d}}\right)(W_V c) \;\text{: each pixel position queries all text tokens}

Retrieval check

In the LDM loss L = E[||eps - eps_theta(z_t, t, c)||^2], what is z_t and how is it constructed from z_0?
Cross-attention in LDM uses Q from z (latent) and K,V from c (text). Why is this asymmetry — rather than self-attention over [z; c] concatenated — the preferred design?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of LDM inference (text-to-image generation) in the correct order:

  1. Decode final clean latent z_0 back to pixel image x_0 via VAE decoder D(z_0)
  2. Sample pure noise latent z_T ~ N(0, I) at full latent resolution (H/f x W/f x C_z)
  3. Iteratively denoise: for t = T down to 1, predict noise eps_theta(z_t, t, c) and update z_{t-1}
  4. Encode text prompt to embedding c via frozen CLIP text encoder

Fill in the blank — the final pixel reconstruction step:

def ldm_decode(z_0, vae_decoder):
    # z_0: denoised latent of shape (H/f, W/f, C_z)
    # vae_decoder: pre-trained, frozen VAE decoder D
    x_0 = ______
    return x_0  # pixel image of shape (H, W, 3)

Retrieval check

During LDM inference, how many times is the VAE decoder called?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement classifier-free cross-attention in NumPy. Given a latent feature matrix z of shape (N, d) and a text conditioning matrix c of shape (M, d_c), compute the cross-attention output using learned projections W_Q (d x d), W_K (d_c x d), W_V (d_c x d_v). The output should have shape (N, d_v) and each row of the attention weight matrix should sum to 1.

Retrieval check

In the cross-attention implementation, why do we subtract scores.max(axis=1, keepdims=True) before applying np.exp?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download High-Resolution Image Synthesis with Latent Diffusion Models (LDM / Stable Diffusion).

Open in new tab · Download

Suggested reading order: §3 (Method — overview of the two-stage pipeline) → §3.2 (Latent Diffusion Models — the core diffusion objective and why pixel vs. latent space matters) → §3.3 (Conditioning Mechanisms — cross-attention architecture for text, layout, and other conditioning signals).