① Intuition
Before attention, seq2seq models compressed an entire source sentence into a single fixed-size vector C — the encoder’s final hidden state. The decoder then generated every target token from that one vector alone. For short sentences this works. For long ones, the model must stuff “The European Parliament has rejected the proposal to postpone the amendment to the Tobacco Products Directive” into a 256-dimensional vector. Early words get crushed out by later ones. Translation quality collapses.
Bahdanau’s fix: instead of a static C, give the decoder a dynamic context vector at each output step i. The decoder computes a soft alignment weight over every encoder hidden state , then takes a weighted sum. The decoder “peeks” at the full source sequence each step, focusing on the parts most relevant to the current output word.
Before the widget: when translating 'Je suis étudiant' → 'I am a student', which source word should the decoder attend to most when producing 'student'? Type your guess.
‘étudiant’ — the direct semantic match. Soft alignment lets the decoder assign near-1 weight to ‘étudiant’ and near-0 to everything else when generating ‘student’. This is the core insight: attention is a learned, differentiable lookup over the source sequence.
Now play with the widget. Change the target step (decoder position). Toggle Monotonic mode. Drag the Sharpness slider to the extremes.
Target step t=0. Cell opacity and value show attention weight α over each source token. Learned mode: additive score seeded by Math.sin.
When you set Sharpness very low (near 0.5), the attention weights become flat. What does that mean for the context vector c_i, and why is it bad for translation?
Flat weights make c_i an unweighted average of all encoder states. Every source word contributes equally regardless of relevance — the context loses all alignment information. The decoder can’t distinguish which source word to focus on, so it tends to produce generic, blurry translations. High sharpness forces the attention to concentrate on one or a few source tokens, creating a crisp “soft pointer” into the source sequence.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
The encoder is a bidirectional RNN. For source token j, forward and backward hidden states are concatenated:
At decoder step i, an alignment score measures how well source position j matches the current decoder state :
This is the additive (Bahdanau) score function — a small one-hidden-layer MLP with learned weight matrices and vector .
Softmax over all source positions gives a probability distribution:
The context vector is the resulting convex combination:
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the six steps of one Bahdanau attention pass in the correct order:
- Run forward RNN over source tokens → hidden states h→_1 … h→_Tx
- Run backward RNN over source tokens → hidden states h←_1 … h←_Tx
- Normalise scores: α_{ij} = softmax(e_{ij}) over j
- For each decoder step i: compute scores e_{ij} = v_a^T tanh(W_a s_{i-1} + U_a h_j)
- Concatenate h→_j and h←_j to form h_j for each source position j
- Compute context c_i = Σ_j α_{ij} h_j, then update decoder state s_i
Fill the blank in this pseudo code:
def bahdanau_context(s_prev, H, W_a, U_a, v_a):
# s_prev: (d_s,) H: (T, d_h) W_a: (d_a, d_s) U_a: (d_a, d_h) v_a: (d_a,)
scores = v_a @ np.tanh(W_a @ s_prev + (U_a @ H.T).T) # (T,)
alpha = _______ # ← fill: normalise scores to a distribution
c = alpha @ H # (d_h,)
return c, alphaRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement the Bahdanau additive attention score and softmax weighting in NumPy. Given decoder state s_prev, encoder states H, and projection matrices W_a, U_a, v_a, return the attention weights alpha (shape (T_enc,)) and context vector c (shape (d_h,)).
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 (Learning to Align and Translate — the full model description) → §A.1.2 (score function formula and motivation) → §4.2 (quantitative results and alignment visualisations).