← Roadmap

ImageNet Classification with Deep Convolutional Neural Networks

Krizhevsky, Sutskever, Hinton · 2012

① Intuition

Before 2012, training deep neural networks with many layers was nearly impossible. The culprit: saturating activations. Both tanh and sigmoid squash their outputs into a bounded range, and once you get far from zero — say, x = 5 or x = −5 — the derivative collapses toward zero. Multiply those near-zero gradients across 8 layers and the signal vanishes long before it reaches early weights.

Setup: play with the widget below. Set x = 5, then cycle through sigmoid → tanh → ReLU. Watch f′(x).

Before touching the widget: at x = 5, what do you expect f′(x) to be for tanh vs ReLU? Write your guess, then check.

f(x) = 0.0000f′(x) = 0.0000ReLU

Drag the slider to explore how gradient (f′) behaves — notice ReLU keeps f′=1 for x>0, while sigmoid/tanh saturate near ±1.

Predict: if you replace all ReLUs with sigmoid in a 5-layer network and keep everything else the same, what happens to training? Why?

Retrieval check

What is f′(x) for ReLU when x > 0?
AlexNet beat the 2011 ImageNet winner by roughly:

Answer all 2 correctly to mark this stage mastered.

② Math

Three activation functions:

ReLU (Rectified Linear Unit):

f(x)=max⁡(0, x),f′(x)={1x>00x≤0f(x) = \max(0,\, x), \qquad f'(x) = \begin{cases} 1 & x > 0 \\ 0 & x \le 0 \end{cases}

Tanh:

f(x)=ex−e−xex+e−x,f′(x)=1−tanh⁡2(x)f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}, \qquad f'(x) = 1 - \tanh^2(x)

Sigmoid:

f(x)=11+e−x,f′(x)=f(x)(1−f(x))f(x) = \frac{1}{1 + e^{-x}}, \qquad f'(x) = f(x)\bigl(1 - f(x)\bigr)

Why tanh gradient vanishes — derivation:

The tanh derivative peaks at x = 0 where tanh(0) = 0, giving f′(0) = 1 − 0² = 1. As |x| grows, tanh(x) → ±1, so:

f′(x)=1−tanh⁡2(x)  →  1−1=0as ∣x∣→∞f'(x) = 1 - \tanh^2(x) \;\to\; 1 - 1 = 0 \quad \text{as } |x| \to \infty
At x = 2, tanh(2) ≈ 0.964. What is tanh′(2)?
Show next line
tanh⁡′(2)=1−tanh⁡2(2)≈1−0.9642≈0.071\tanh'(2) = 1 - \tanh^2(2) \approx 1 - 0.964^2 \approx 0.071

Softmax output layer:

AlexNet’s final layer outputs a probability distribution over 1000 ImageNet classes using softmax:

softmax(z)j=ezj∑k=1Kezk\text{softmax}(z)_j = \frac{e^{z_j}}{\sum_{k=1}^{K} e^{z_k}}
Softmax outputs must satisfy what constraint?
Show next line
∑j=1Ksoftmax(z)j=∑jezj∑kezk=1\sum_{j=1}^{K} \text{softmax}(z)_j = \frac{\sum_j e^{z_j}}{\sum_k e^{z_k}} = 1

Retrieval check

tanh′(x) reaches its maximum value at:
In softmax, what ensures the outputs are positive?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the steps of a forward pass through one AlexNet convolutional layer in correct order:

  1. Convolve input with learned filters → feature map z
  2. Local Response Normalize → output to next layer
  3. Max-pool: take max in each k×k window → downsampled map
  4. Apply ReLU activation → a = max(0, z)

Fill the blank in this pseudo code:

def conv_layer_forward(x, W, b):
    z = convolve(x, W) + b          # linear transform
    a = _______                      # ← activation
    pooled = max_pool(a, kernel=3)
    out = local_response_norm(pooled)
    return out

Retrieval check

Max pooling in AlexNet primarily serves to:

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement ReLU and its derivative in NumPy.

Retrieval check

np.maximum(0, x) vs np.max(x) — which implements ReLU?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download ImageNet Classification with Deep CNNs.

Open in new tab · Download

Suggested reading order: §3 Architecture → §3.1 ReLU Nonlinearity → §4 Reducing Overfitting (Dropout) → §6 Results.