← Roadmap

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov · 2014

① Intuition

When you train a large network long enough on limited data, neurons start to collude. Unit A fires whenever unit B fires; together they memorize a specific pattern in the training set — a texture, a noise artifact, an ID of one example. This is co-adaptation: neurons rely on each other rather than each learning something independently useful. The result is a network that looks brilliant on training data and flails on anything new.

Dropout’s insight: randomly silence a different random subset of neurons on every forward pass. Now unit A can never count on B being present. Each unit is forced to be useful on its own. At test time every neuron is active, but we scale outputs down by keep_prob to match expected training activations.

Before the widget: if you set keep_prob = 0.5, how many of the 36 neurons do you expect to survive on a given pass? Type your guess.

Play with the widget. Drag keep_prob, hit Resample several times, and toggle Show expected value.

1
1
0
1
0
0
1
1
0
1
1
1
0
1
0
1
1
1
0
1
1
1
1
1
0
1
1
0
1
1
0
0
0
0
1
0

Surviving: 22 / 36 neurons

Step 0 — each neuron is independently kept with probability keep_prob. At test time, all neurons are active but outputs are scaled by keep_prob.

After playing: at keep_prob = 0.9 vs keep_prob = 0.1, how does the surviving network's capacity change? When would you choose each extreme?

Retrieval check

Why does dropout act as an ensemble method?
At test time with standard (non-inverted) dropout, outputs are scaled by:

Answer all 2 correctly to mark this stage mastered.

② Math

Let pp be the keep probability. During training, each unit hih_i is multiplied by a Bernoulli random variable ri∼Bernoulli(p)r_i \sim \text{Bernoulli}(p):

h~i=ri⋅hi,ri∼Bernoulli(p)\tilde{h}_i = r_i \cdot h_i, \quad r_i \sim \text{Bernoulli}(p)

At test time all units are active. To match the expected activation during training:

hitest=p⋅hino-dropouth_i^{\text{test}} = p \cdot h_i^{\text{no-dropout}}

Inverted dropout — the modern variant — moves the scaling to train time so the test forward pass needs no modification:

h~iinverted=rip⋅hi\tilde{h}_i^{\text{inverted}} = \frac{r_i}{p} \cdot h_i

Now E[h~iinverted]=pphi=hi\mathbb{E}[\tilde{h}_i^{\text{inverted}}] = \frac{p}{p} h_i = h_i, matching the test activation exactly.

Why does dividing by p at train time mean test time needs no scaling?
Show next line
E ⁣[rip]=E[ri]p=pp=1  ⟹  E[h~iinv]=hi\mathbb{E}\!\left[\frac{r_i}{p}\right] = \frac{\mathbb{E}[r_i]}{p} = \frac{p}{p} = 1 \implies \mathbb{E}[\tilde{h}_i^{\text{inv}}] = h_i

Ensemble interpretation: with nn hidden units, there are 2n2^n possible subnetworks (masks). Dropout weight-shares all 2n2^n models — the test-time network with scaled weights approximates averaging their predictions.

If a layer has n = 100 neurons and keep_prob = 0.5, how many distinct subnetworks are implicitly trained?
Show next line
∣M∣=2n=2100≈1.27×1030|\mathcal{M}| = 2^n = 2^{100} \approx 1.27 \times 10^{30}

Retrieval check

In inverted dropout with keep_prob = 0.5, a surviving unit's activation is multiplied by:
The ensemble of 2^n subnetworks all share:

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the steps of inverted dropout (train-time forward pass) in correct order:

  1. Multiply activations element-wise: h̃ = h * mask
  2. Scale surviving units: h̃ = h̃ / keep_prob
  3. Sample Bernoulli mask: mask[i] = Bernoulli(keep_prob) for each unit i
  4. Continue forward pass with h̃ as the layer output

Fill the blank in this pseudo code:

def dropout(h, keep_prob, training):
    if not training:
        return h
    mask = bernoulli(keep_prob, shape=h.shape)
    return (h * mask) / _______    # ← fill

Retrieval check

Why is the mask re-sampled on EVERY forward pass rather than once per epoch?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement inverted dropout in NumPy.

Retrieval check

In inverted dropout, what value do surviving units take when x = 1 and keep_prob = 0.5?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Dropout.

Open in new tab · Download

Suggested reading order: §1 Introduction (co-adaptation motivation) → §3 Model Description (Bernoulli mask + scaling) → §4 Learning Dropout Nets (inverted form and hyperparameters) → §6 Experiments (ImageNet, CIFAR, speech).