← Roadmap

Adam: A Method for Stochastic Optimization

Kingma, Ba · 2014

① Intuition

SGD uses one global learning rate for every parameter. That works badly when the loss landscape is ill-conditioned: parameters with consistently large gradients overshoot, while parameters with tiny gradients barely move. Adding momentum smooths noisy gradients by accumulating a running average — but it still uses the same step size everywhere.

Adam’s insight: maintain two running statistics per parameter. The first moment m is a momentum estimate of the gradient. The second moment v tracks the average squared gradient — a proxy for curvature. Dividing by √v gives each parameter an automatic per-parameter learning rate: large gradients compress the step, small gradients enlarge it.

Before the widget: if β₂ is very close to 1.0, what do you expect v to do over many steps — stabilise quickly or slowly? Type your guess.

Now play with the widget. Drag β₂ toward 1.0 and watch how v changes across steps. Then set step = 1 and toggle bias correction off.

g_t
-0.8183
m_t
0.2232
v_t
0.0052
m̂_t
0.3427
v̂_t
0.5247
update
0.4732

Grey = g_t · Blue = m_t · Amber = m̂_t (bias-corrected) · Red line = current step

At step 1 with bias correction OFF, the displayed update looks very small. Why? What would it look like if you turned bias correction back ON?

Retrieval check

Why does Adam outperform SGD on ill-conditioned losses?
What is the purpose of the first moment m in Adam?

Answer all 2 correctly to mark this stage mastered.

② Math

At each step t, given gradient gtg_t, Adam maintains:

mt=β1mt−1+(1−β1) gtm_t = \beta_1 m_{t-1} + (1-\beta_1)\,g_tvt=β2vt−1+(1−β2) gt2v_t = \beta_2 v_{t-1} + (1-\beta_2)\,g_t^2

Both are initialised at zero. Because they start at zero, early estimates are biased toward zero. Bias-corrected estimates:

m^t=mt1−β1t,v^t=vt1−β2t\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^t}

The parameter update:

θt←θt−1−α⋅m^tv^t+ε\theta_t \leftarrow \theta_{t-1} - \alpha \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \varepsilon}

Defaults: β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸, α = 0.001.

At t=1, m₁ = (1−β₁)g₁. With β₁=0.9, m₁ = 0.1·g₁. Why does this under-represent g₁?
Show next line
m1=(1−β1) g1=0.1 g1≪g1m_1 = (1-\beta_1)\,g_1 = 0.1\,g_1 \ll g_1
What does dividing m₁ by (1−β₁^1) give?
Show next line
m^1=m11−β11=(1−β1)g11−β1=g1\hat{m}_1 = \frac{m_1}{1-\beta_1^1} = \frac{(1-\beta_1)g_1}{1-\beta_1} = g_1

Retrieval check

As t grows large, the bias correction factor (1−β^t) approaches:
ε in the Adam update θ ← θ − α·m̂/(√v̂ + ε) serves to:

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of one Adam iteration in the correct order:

  1. Update first moment: m_t = β1·m_{t-1} + (1−β1)·g_t
  2. Update second moment: v_t = β2·v_{t-1} + (1−β2)·g_t²
  3. Bias-correct: m̂ = m_t / (1−β1^t), v̂ = v_t / (1−β2^t)
  4. Compute gradient g_t = ∇L(θ_{t-1})
  5. Apply update: θ_t = θ_{t-1} − α·m̂ / (√v̂ + ε)

Fill the blank in this pseudo code:

def adam_step(g, m, v, t, beta1, beta2, alpha, eps):
    m = beta1 * m + (1 - beta1) * g
    v = beta2 * v + (1 - beta2) * g**2
    m_hat = m / _______    # ← fill (bias correction for m)
    v_hat = v / (1 - beta2**t)
    theta_update = alpha * m_hat / (v_hat**0.5 + eps)
    return m, v, theta_update

Retrieval check

If you forget the bias correction step, what goes wrong in the first few iterations?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement one Adam step in NumPy. Starting from zero-initialised m and v, after one step with gradient g=1.0 and α=0.001, the parameter update should be approximately −α (i.e. θ ≈ −0.001).

Retrieval check

Why does one Adam step with g=1 give θ ≈ −α rather than something else?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Adam.

Open in new tab · Download

Suggested reading order: §2 (Algorithm — the pseudocode box) → §3 (Bias correction derivation) → §6.1 (Experiments on logistic regression and MLPs).