← Roadmap

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Ioffe, Szegedy · 2015

① Intuition

Training a deep network is hard because the distribution of each layer’s pre-activations keeps shifting as the weights below change — the paper calls this internal covariate shift. As distributions drift into the tails of saturating non-linearities (tanh, sigmoid), gradients collapse and learning stalls. Pre-2015, people carefully initialized weights and used tiny learning rates.

Batch Normalization attacks the shift directly: at every layer it subtracts the minibatch mean and divides by the minibatch standard deviation, re-centering activations to roughly zero-mean unit-variance. Two learnable scalars per channel, γ and β, let the network re-scale if it needs to.

Before touching the widget: as you stack more layers (each multiplying by 1.1 and shifting by 0.2), what happens to the pre-activation distribution at layer 10 without any normalization?

Play with the widget — crank layer depth to 10 with BatchNorm OFF, then toggle BatchNorm ON.

mean: -0.010variance: 1.005

Predict: with BatchNorm ON, roughly what mean and variance do you expect at layer 10? Does batch size matter much?

Retrieval check

Why does drift cause training to stall?
What does BN do to a batch at inference?

Answer all 2 correctly to mark this stage mastered.

② Math

Batch Normalization over a minibatch B of pre-activations z:

μB=1m∑i=1mzi,σB2=1m∑i=1m(zi−μB)2\mu_B = \frac{1}{m}\sum_{i=1}^{m} z_i, \quad \sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m} (z_i - \mu_B)^2z^i=zi−μBσB2+ϵ,BN(zi)=γ z^i+β\hat{z}_i = \frac{z_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \quad \text{BN}(z_i) = \gamma\, \hat{z}_i + \beta

γ, β are learned per-feature parameters. ε is a small constant (e.g. 1e-5).

Why add ε inside the square root?
Show next line
z^i=zi−μBσB2+ϵ\hat{z}_i = \frac{z_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}
If BN always forces mean 0, variance 1 — why do we still need learnable γ and β?
Show next line
BN(z)=γ z^+β can recover any affine shape of z\text{BN}(z) = \gamma\,\hat{z} + \beta \text{ can recover any affine shape of } z

At inference we replace μ_B, σ_B² with running averages μˉ,σˉ2\bar{\mu}, \bar{\sigma}^2 accumulated during training.

Retrieval check

The inverse of the normalization can be recovered by:
A typical value of ε is:

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the BN forward pass steps in order:

  1. Normalize: ẑ = (z - μ_B) / √(σ_B² + ε)
  2. Scale and shift: y = γ·ẑ + β
  3. Compute batch mean μ_B
  4. Compute batch variance σ_B²

Fill the blank in this pseudo code:

def batchnorm(z, gamma, beta, eps=1e-5):
    mean = z.mean(axis=0)
    var  = z.var(axis=0)
    zhat = _______    # ← fill
    return gamma * zhat + beta

Retrieval check

Which axis is the mean/variance reduction taken over in standard 2D BN?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement BatchNorm’s forward pass for a 2D tensor of shape (batch, features).

Retrieval check

After BN with γ=1, β=0, what is the feature-wise variance?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Batch Normalization.

Open in new tab · Download

Suggested reading order: §3 (Normalization via Mini-Batch Statistics) → §3.1 (Training and Inference with Batch-Normalized Networks) → §4 (Experiments).