① Intuition
Before 2017, sequence models read tokens one-by-one (RNN/LSTM). Every position waited on its predecessor. Training long sequences was slow; very distant dependencies were forgotten.
Setup: think of translating the sentence “The cat sat on the mat.” You want each word’s representation to be able to “look at” every other word directly.
Before touching the widget: if each word could look at every other word directly, what would go wrong as sentences get longer? Type your guess.
Two things: (a) the number of query–key pairs grows as n², and (b) without weighting, “looking at everything” is just averaging — you lose position info. The Transformer solves both: the softmax creates weighted attention (so it’s never uniform averaging), and positional encodings re-inject order.
Now play with the widget below. Change d_k. Toggle the √d_k scaling. Watch the attention row for the selected query token.
| The | cat | sat | on | the | mat | |
|---|---|---|---|---|---|---|
| The | ||||||
| cat | ||||||
| sat | ||||||
| on | ||||||
| the | ||||||
| mat |
Hover a cell to see exact value. Row = query token. Darker cell = more attention mass.
Predict: when you DISABLE √d_k scaling at large d_k (say 128), does the attention row become more peaked or more flat? Why?
More peaked — very sharp, often a single cell near 1.0. Dot products of high-dimensional random vectors scale with √d_k, so without the divisor softmax inputs are huge and softmax saturates. Dividing by √d_k keeps softmax inputs order-O(1), giving smooth gradients.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
Scaled dot-product attention:
Where Q ∈ ℝ^(n×d_k), K ∈ ℝ^(n×d_k), V ∈ ℝ^(n×d_v).
Derivation of scaling: assume q and k have iid components with mean 0, variance 1. Then .
Show next line
To keep softmax inputs order-O(1) regardless of d_k, divide by
Multi-head attention:
Each head has its own projection and captures different dependency patterns (syntax, coreference, etc.).
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the steps of scaled dot-product attention in order:
- Divide every entry by √d_k
- Compute QKᵀ (n × n matrix of dot products)
- Multiply by V to get output (n × d_v)
- Apply softmax row-wise
Fill the blank in this pseudo code:
def attention(Q, K, V, dk):
scores = Q @ K.T
scores = scores / _______ # ← fill
weights = softmax(scores, axis=-1)
return weights @ VRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement scaled dot-product attention in NumPy.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 (Model Architecture) → §3.2 (Scaled Dot-Product Attention) → §3.2.2 (Multi-Head) → §5 (Training).