① Intuition
Training a deep network is hard because the distribution of each layer’s pre-activations keeps shifting as the weights below change — the paper calls this internal covariate shift. As distributions drift into the tails of saturating non-linearities (tanh, sigmoid), gradients collapse and learning stalls. Pre-2015, people carefully initialized weights and used tiny learning rates.
Batch Normalization attacks the shift directly: at every layer it subtracts the minibatch mean and divides by the minibatch standard deviation, re-centering activations to roughly zero-mean unit-variance. Two learnable scalars per channel, γ and β, let the network re-scale if it needs to.
Before touching the widget: as you stack more layers (each multiplying by 1.1 and shifting by 0.2), what happens to the pre-activation distribution at layer 10 without any normalization?
It drifts and spreads. Mean walks off (0.2 accumulates across layers); variance blows up (1.1² per layer, ×10 layers ≈ 6.7× variance). Most activations fall in the saturated regions of a downstream non-linearity, so gradients die.
Play with the widget — crank layer depth to 10 with BatchNorm OFF, then toggle BatchNorm ON.
Predict: with BatchNorm ON, roughly what mean and variance do you expect at layer 10? Does batch size matter much?
Mean ≈ 0, variance ≈ 1 — BN renormalizes at every layer, so depth doesn’t accumulate drift. Batch size matters a little: smaller batches give noisier μ, σ estimates, which is why group/layer norm were later invented for batch=1 inference.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Batch Normalization over a minibatch B of pre-activations z:
γ, β are learned per-feature parameters. ε is a small constant (e.g. 1e-5).
Show next line
Show next line
At inference we replace μ_B, σ_B² with running averages accumulated during training.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the BN forward pass steps in order:
- Normalize: ẑ = (z - μ_B) / √(σ_B² + ε)
- Scale and shift: y = γ·ẑ + β
- Compute batch mean μ_B
- Compute batch variance σ_B²
Fill the blank in this pseudo code:
def batchnorm(z, gamma, beta, eps=1e-5):
mean = z.mean(axis=0)
var = z.var(axis=0)
zhat = _______ # ← fill
return gamma * zhat + betaRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement BatchNorm’s forward pass for a 2D tensor of shape (batch, features).
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 (Normalization via Mini-Batch Statistics) → §3.1 (Training and Inference with Batch-Normalized Networks) → §4 (Experiments).