① Intuition
SGD uses one global learning rate for every parameter. That works badly when the loss landscape is ill-conditioned: parameters with consistently large gradients overshoot, while parameters with tiny gradients barely move. Adding momentum smooths noisy gradients by accumulating a running average — but it still uses the same step size everywhere.
Adam’s insight: maintain two running statistics per parameter. The first moment m is a momentum estimate of the gradient. The second moment v tracks the average squared gradient — a proxy for curvature. Dividing by √v gives each parameter an automatic per-parameter learning rate: large gradients compress the step, small gradients enlarge it.
Before the widget: if β₂ is very close to 1.0, what do you expect v to do over many steps — stabilise quickly or slowly? Type your guess.
Stabilise slowly. β₂ close to 1.0 means the running average forgets old squared gradients very slowly (each new gradient contributes only (1−β₂) ≈ 0.001 weight). v converges to the true average, but it takes many steps. The advantage is a stable, low-variance estimate; the disadvantage is it adapts sluggishly to sudden gradient changes.
Now play with the widget. Drag β₂ toward 1.0 and watch how v changes across steps. Then set step = 1 and toggle bias correction off.
Grey = g_t · Blue = m_t · Amber = m̂_t (bias-corrected) · Red line = current step
At step 1 with bias correction OFF, the displayed update looks very small. Why? What would it look like if you turned bias correction back ON?
At step 1, m₁ = (1−β₁)·g₁ and v₁ = (1−β₂)·g₁². Both are multiplied by tiny factors (≈0.1 and 0.001), so the raw update m/(√v + ε) can still be reasonable — but it is not the true gradient. With bias correction ON, both m and v are divided by (1−β₁^t) and (1−β₂^t) respectively, which at t=1 are exactly (1−β₁) and (1−β₂). This inflates them back to g₁ and g₁², recovering the actual gradient as the first update.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
At each step t, given gradient , Adam maintains:
Both are initialised at zero. Because they start at zero, early estimates are biased toward zero. Bias-corrected estimates:
The parameter update:
Defaults: β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸, α = 0.001.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of one Adam iteration in the correct order:
- Update first moment: m_t = β1·m_{t-1} + (1−β1)·g_t
- Update second moment: v_t = β2·v_{t-1} + (1−β2)·g_t²
- Bias-correct: m̂ = m_t / (1−β1^t), v̂ = v_t / (1−β2^t)
- Compute gradient g_t = ∇L(θ_{t-1})
- Apply update: θ_t = θ_{t-1} − α·m̂ / (√v̂ + ε)
Fill the blank in this pseudo code:
def adam_step(g, m, v, t, beta1, beta2, alpha, eps):
m = beta1 * m + (1 - beta1) * g
v = beta2 * v + (1 - beta2) * g**2
m_hat = m / _______ # ← fill (bias correction for m)
v_hat = v / (1 - beta2**t)
theta_update = alpha * m_hat / (v_hat**0.5 + eps)
return m, v, theta_updateRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement one Adam step in NumPy. Starting from zero-initialised m and v, after one step with gradient g=1.0 and α=0.001, the parameter update should be approximately −α (i.e. θ ≈ −0.001).
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §2 (Algorithm — the pseudocode box) → §3 (Bias correction derivation) → §6.1 (Experiments on logistic regression and MLPs).