← Roadmap

Attention Is All You Need

Vaswani et al. · 2017

① Intuition

Before 2017, sequence models read tokens one-by-one (RNN/LSTM). Every position waited on its predecessor. Training long sequences was slow; very distant dependencies were forgotten.

Setup: think of translating the sentence “The cat sat on the mat.” You want each word’s representation to be able to “look at” every other word directly.

Before touching the widget: if each word could look at every other word directly, what would go wrong as sentences get longer? Type your guess.

Now play with the widget below. Change d_k. Toggle the √d_k scaling. Watch the attention row for the selected query token.

Thecatsatonthemat
The
cat
sat
on
the
mat

Hover a cell to see exact value. Row = query token. Darker cell = more attention mass.

Predict: when you DISABLE √d_k scaling at large d_k (say 128), does the attention row become more peaked or more flat? Why?

Retrieval check

Why was self-attention attractive vs RNNs?
What does scaling by √d_k prevent?

Answer all 2 correctly to mark this stage mastered.

② Math

Scaled dot-product attention:

Attn(Q,K,V)=softmax ⁣(QK⊤dk)V\text{Attn}(Q, K, V) = \text{softmax}\!\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V

Where Q ∈ ℝ^(n×d_k), K ∈ ℝ^(n×d_k), V ∈ ℝ^(n×d_v).

Derivation of scaling: assume q and k have iid components with mean 0, variance 1. Then q⋅k=∑i=1dkqikiq \cdot k = \sum_{i=1}^{d_k} q_i k_i.

What is Var(q · k)?
Show next line
Var(q⋅k)=dk  ⟹  sd(q⋅k)=dk\mathrm{Var}(q \cdot k) = d_k \implies \mathrm{sd}(q \cdot k) = \sqrt{d_k}

To keep softmax inputs order-O(1) regardless of d_k, divide by before softmax.

Multi-head attention:

MultiHead(Q,K,V)=Concat(head1,…,headh)WO\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W^Oheadi=Attn(QWiQ,  KWiK,  VWiV)\text{head}_i = \text{Attn}(Q W^Q_i,\; K W^K_i,\; V W^V_i)

Each head has its own projection and captures different dependency patterns (syntax, coreference, etc.).

Retrieval check

For iid zero-mean unit-variance q, k, sd(q·k) = ?
Multi-head attention primarily increases:

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the steps of scaled dot-product attention in order:

  1. Divide every entry by √d_k
  2. Compute QKᵀ (n × n matrix of dot products)
  3. Multiply by V to get output (n × d_v)
  4. Apply softmax row-wise

Fill the blank in this pseudo code:

def attention(Q, K, V, dk):
    scores = Q @ K.T
    scores = scores / _______    # ← fill
    weights = softmax(scores, axis=-1)
    return weights @ V

Retrieval check

What happens if you softmax BEFORE scaling?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement scaled dot-product attention in NumPy.

Retrieval check

Stable softmax subtracts row-max before exp to:

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Attention Is All You Need.

Open in new tab · Download

Suggested reading order: §3 (Model Architecture) → §3.2 (Scaled Dot-Product Attention) → §3.2.2 (Multi-Head) → §5 (Training).