← Roadmap

Deep Residual Learning for Image Recognition

He, Zhang, Ren, Sun · 2015

① Intuition

Before ResNet, the community assumed that adding more layers to a neural network always helps — more parameters means more capacity. But in 2015 He et al. showed something disturbing: when you stack plain layers past ~20 deep on CIFAR-10 or ImageNet, training accuracy gets worse, not just test accuracy. This is not overfitting — the training error itself rises. The network simply refuses to learn the identity mapping through many layers.

The fix is surprisingly simple: give the network a shortcut. Instead of asking each block to directly learn the desired mapping H(x), let it learn the residual F(x) = H(x) − x, then add back the input: H(x) = F(x) + x. If the optimal mapping is close to identity, the residual is near zero — much easier to push towards zero than to fit an identity through stacked weight layers.

Before touching the widget: as you increase depth from 20 to 50 layers in a plain network (no skip connection), what do you expect happens to training accuracy?

Drag the depth slider and toggle “Use skip connection” to see the gap between plain and residual networks.

train_acc_plain: 0.9180train_acc_residual: 0.8965
depth 2→50
Plain Residual

With skip connections enabled at depth 50, the residual curve stays high while the plain curve drops. Why does the shortcut path make identity easier to learn?

Retrieval check

The degradation problem is caused by:
A residual block learns F(x). What does it output?

Answer all 2 correctly to mark this stage mastered.

② Math

A residual block with input x and learned residual function F:

y=F(x,{Wi})+xy = F(x,\{W_i\}) + x

When the input and output dimensions differ (e.g., when doubling channels or halving spatial size), a linear projection is applied to the shortcut:

y=F(x,{Wi})+Ws xy = F(x,\{W_i\}) + W_s\, x

W_s is a 1×1 convolution that matches dimensions. F typically contains two or three conv-BN-ReLU sub-layers; the final ReLU is applied after the addition.

The gradient through a residual block is:

∂L∂x=∂L∂y(1+∂F∂x)\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y}\left(1 + \frac{\partial F}{\partial x}\right)

The “+1” term from the skip connection means the gradient can never vanish completely — even if ∂F/∂x ≈ 0, the loss gradient flows unchanged back to x.

Why is learning F(x) = H(x) − x easier than learning H(x) directly, when the optimal H is close to identity?
Show next line
F(x)≈0⇒y≈x(identity is free)F(x) \approx 0 \Rightarrow y \approx x \quad (\text{identity is free})
Given the gradient formula ∂L/∂x = (∂L/∂y)(1 + ∂F/∂x), what prevents gradient vanishing even with 100+ layers?
Show next line
∂L∂x=∂L∂y ⁣(1+∂F∂x)\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y}\!\left(1 + \frac{\partial F}{\partial x}\right)

Retrieval check

When are projection shortcuts (W_s x) used instead of identity shortcuts?
Where is the final ReLU applied in a residual block?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the residual block forward pass steps in correct order:

  1. out = out + identity (add residual)
  2. out = conv1(x) → BN → ReLU
  3. out = conv2(out) → BN
  4. out = ReLU(out)
  5. identity = x (save input for shortcut)

Fill the blank in this pseudo code:

def residual_block(x, W1, W2):
    identity = x
    out = relu(bn(conv(x, W1)))
    out = bn(conv(out, W2))
    out = _______    # ← fill
    return relu(out)

Retrieval check

In the pseudo code above, why is the final relu applied AFTER the addition?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement a residual block forward pass in NumPy. Use simple matrix multiplication (no actual convolution) to keep it clear — treat each “conv” as a linear layer applied to a flattened input.

Retrieval check

In the zero-weight test above (W1=W2=0), the block outputs relu(x). What does this tell you about skip connections?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Deep Residual Learning for Image Recognition.

Open in new tab · Download

Suggested reading order: §3.1 (Residual Learning) → §3.2 (Identity Mapping by Shortcuts) → §4 (Experiments — Table 6 plain vs residual, Table 4 ImageNet results).