① Intuition
Before 2012, training deep neural networks with many layers was nearly impossible. The culprit: saturating activations. Both tanh and sigmoid squash their outputs into a bounded range, and once you get far from zero — say, x = 5 or x = −5 — the derivative collapses toward zero. Multiply those near-zero gradients across 8 layers and the signal vanishes long before it reaches early weights.
Setup: play with the widget below. Set x = 5, then cycle through sigmoid → tanh → ReLU. Watch f′(x).
Before touching the widget: at x = 5, what do you expect f′(x) to be for tanh vs ReLU? Write your guess, then check.
At x = 5, tanh′(5) ≈ 0.0001 — effectively zero. Sigmoid′(5) ≈ 0.0066 — also nearly dead. But ReLU′(5) = 1.0, always. That is the entire story of why deep networks became trainable. AlexNet stacked 8 layers (5 conv + 3 FC), 60 million parameters, trained on two GTX 580 GPUs. With tanh they could not have gotten the gradients through; with ReLU they won ImageNet 2012 by 10.9 percentage points over the next best entry.
Drag the slider to explore how gradient (f′) behaves — notice ReLU keeps f′=1 for x>0, while sigmoid/tanh saturate near ±1.
Predict: if you replace all ReLUs with sigmoid in a 5-layer network and keep everything else the same, what happens to training? Why?
Training stalls or diverges. The sigmoid gradient at |x| > 3 is < 0.045. Chained across 5 layers: 0.045⁵ ≈ 1.8 × 10⁻⁷. Weights in early layers receive a signal 5 million times smaller than the last layer — the classic vanishing gradient problem. ReLU short-circuits this: positive activations pass gradient = 1 unchanged.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Three activation functions:
ReLU (Rectified Linear Unit):
Tanh:
Sigmoid:
Why tanh gradient vanishes — derivation:
The tanh derivative peaks at x = 0 where tanh(0) = 0, giving f′(0) = 1 − 0² = 1. As |x| grows, tanh(x) → ±1, so:
Show next line
Softmax output layer:
AlexNet’s final layer outputs a probability distribution over 1000 ImageNet classes using softmax:
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the steps of a forward pass through one AlexNet convolutional layer in correct order:
- Convolve input with learned filters → feature map z
- Local Response Normalize → output to next layer
- Max-pool: take max in each k×k window → downsampled map
- Apply ReLU activation → a = max(0, z)
Fill the blank in this pseudo code:
def conv_layer_forward(x, W, b):
z = convolve(x, W) + b # linear transform
a = _______ # ← activation
pooled = max_pool(a, kernel=3)
out = local_response_norm(pooled)
return outRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement ReLU and its derivative in NumPy.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 Architecture → §3.1 ReLU Nonlinearity → §4 Reducing Overfitting (Dropout) → §6 Results.