① Intuition
When you train a large network long enough on limited data, neurons start to collude. Unit A fires whenever unit B fires; together they memorize a specific pattern in the training set — a texture, a noise artifact, an ID of one example. This is co-adaptation: neurons rely on each other rather than each learning something independently useful. The result is a network that looks brilliant on training data and flails on anything new.
Dropout’s insight: randomly silence a different random subset of neurons on every forward pass. Now unit A can never count on B being present. Each unit is forced to be useful on its own. At test time every neuron is active, but we scale outputs down by keep_prob to match expected training activations.
Before the widget: if you set keep_prob = 0.5, how many of the 36 neurons do you expect to survive on a given pass? Type your guess.
18 neurons on average — each of the 36 is independently kept with probability 0.5, so the expected count is 36 × 0.5 = 18. But the actual count varies around that mean each time you resample. That variance is the point: the network sees a different sparse subnetwork every step.
Play with the widget. Drag keep_prob, hit Resample several times, and toggle Show expected value.
Surviving: 22 / 36 neurons
Step 0 — each neuron is independently kept with probability keep_prob. At test time, all neurons are active but outputs are scaled by keep_prob.
After playing: at keep_prob = 0.9 vs keep_prob = 0.1, how does the surviving network's capacity change? When would you choose each extreme?
At keep_prob = 0.9 the network is almost intact each pass — mild regularization, used for output layers or when data is plentiful. At keep_prob = 0.1 only ~4 of 36 neurons survive — extreme regularization, rarely used because it kills most capacity. The paper typically uses 0.5 for hidden layers and 0.8 for inputs.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Let be the keep probability. During training, each unit is multiplied by a Bernoulli random variable :
At test time all units are active. To match the expected activation during training:
Inverted dropout — the modern variant — moves the scaling to train time so the test forward pass needs no modification:
Now , matching the test activation exactly.
Show next line
Ensemble interpretation: with hidden units, there are possible subnetworks (masks). Dropout weight-shares all models — the test-time network with scaled weights approximates averaging their predictions.
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the steps of inverted dropout (train-time forward pass) in correct order:
- Multiply activations element-wise: h̃ = h * mask
- Scale surviving units: h̃ = h̃ / keep_prob
- Sample Bernoulli mask: mask[i] = Bernoulli(keep_prob) for each unit i
- Continue forward pass with h̃ as the layer output
Fill the blank in this pseudo code:
def dropout(h, keep_prob, training):
if not training:
return h
mask = bernoulli(keep_prob, shape=h.shape)
return (h * mask) / _______ # ← fillRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement inverted dropout in NumPy.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §1 Introduction (co-adaptation motivation) → §3 Model Description (Bernoulli mask + scaling) → §4 Learning Dropout Nets (inverted form and hyperparameters) → §6 Experiments (ImageNet, CIFAR, speech).