① Intuition
Before ResNet, the community assumed that adding more layers to a neural network always helps — more parameters means more capacity. But in 2015 He et al. showed something disturbing: when you stack plain layers past ~20 deep on CIFAR-10 or ImageNet, training accuracy gets worse, not just test accuracy. This is not overfitting — the training error itself rises. The network simply refuses to learn the identity mapping through many layers.
The fix is surprisingly simple: give the network a shortcut. Instead of asking each block to directly learn the desired mapping H(x), let it learn the residual F(x) = H(x) − x, then add back the input: H(x) = F(x) + x. If the optimal mapping is close to identity, the residual is near zero — much easier to push towards zero than to fit an identity through stacked weight layers.
Before touching the widget: as you increase depth from 20 to 50 layers in a plain network (no skip connection), what do you expect happens to training accuracy?
Training accuracy degrades with plain networks beyond ~20 layers — the degradation problem. Gradients become harder to propagate through many weight layers, and the network cannot learn even the identity mapping. This is not overfitting; the training error itself worsens.
Drag the depth slider and toggle “Use skip connection” to see the gap between plain and residual networks.
With skip connections enabled at depth 50, the residual curve stays high while the plain curve drops. Why does the shortcut path make identity easier to learn?
With a skip connection, the weight layers only need to learn the residual F(x) = H(x) − x. If the true optimal mapping is near identity, F(x) ≈ 0 — easy. In the plain case, each layer must conspire to produce an exact identity through composition of nonlinear transforms, which is hard. Skip connections also provide a direct gradient highway back to early layers, alleviating vanishing gradients.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
A residual block with input x and learned residual function F:
When the input and output dimensions differ (e.g., when doubling channels or halving spatial size), a linear projection is applied to the shortcut:
W_s is a 1×1 convolution that matches dimensions. F typically contains two or three conv-BN-ReLU sub-layers; the final ReLU is applied after the addition.
The gradient through a residual block is:
The “+1” term from the skip connection means the gradient can never vanish completely — even if ∂F/∂x ≈ 0, the loss gradient flows unchanged back to x.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the residual block forward pass steps in correct order:
- out = out + identity (add residual)
- out = conv1(x) → BN → ReLU
- out = conv2(out) → BN
- out = ReLU(out)
- identity = x (save input for shortcut)
Fill the blank in this pseudo code:
def residual_block(x, W1, W2):
identity = x
out = relu(bn(conv(x, W1)))
out = bn(conv(out, W2))
out = _______ # ← fill
return relu(out)Retrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement a residual block forward pass in NumPy. Use simple matrix multiplication (no actual convolution) to keep it clear — treat each “conv” as a linear layer applied to a flattened input.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3.1 (Residual Learning) → §3.2 (Identity Mapping by Shortcuts) → §4 (Experiments — Table 6 plain vs residual, Table 4 ImageNet results).