← Roadmap

Neural Machine Translation by Jointly Learning to Align and Translate

Bahdanau, Cho, Bengio · 2014

① Intuition

Before attention, seq2seq models compressed an entire source sentence into a single fixed-size vector C — the encoder’s final hidden state. The decoder then generated every target token from that one vector alone. For short sentences this works. For long ones, the model must stuff “The European Parliament has rejected the proposal to postpone the amendment to the Tobacco Products Directive” into a 256-dimensional vector. Early words get crushed out by later ones. Translation quality collapses.

Bahdanau’s fix: instead of a static C, give the decoder a dynamic context vector cic_i at each output step i. The decoder computes a soft alignment weight αij\alpha_{ij} over every encoder hidden state hjh_j, then takes a weighted sum. The decoder “peeks” at the full source sequence each step, focusing on the parts most relevant to the current output word.

Before the widget: when translating 'Je suis étudiant' → 'I am a student', which source word should the decoder attend to most when producing 'student'? Type your guess.

Now play with the widget. Change the target step (decoder position). Toggle Monotonic mode. Drag the Sharpness slider to the extremes.

0.06
Je
0.19
suis
0.34
étudiant
0.27
à
0.11
l'université
0.03
.

Target step t=0. Cell opacity and value show attention weight α over each source token. Learned mode: additive score seeded by Math.sin.

When you set Sharpness very low (near 0.5), the attention weights become flat. What does that mean for the context vector c_i, and why is it bad for translation?

Retrieval check

Why does vanilla seq2seq struggle with long sentences?
In Bahdanau attention, the context vector c_i is:

Answer all 2 correctly to mark this stage mastered.

② Math

The encoder is a bidirectional RNN. For source token j, forward and backward hidden states are concatenated:

hj=[h→j ; h←j]h_j = \left[\overrightarrow{h}_j\,;\,\overleftarrow{h}_j\right]

At decoder step i, an alignment score measures how well source position j matches the current decoder state si−1s_{i-1}:

eij=va⊤tanh⁡ ⁣(Wa si−1+Ua hj)e_{ij} = v_a^\top \tanh\!\left(W_a\,s_{i-1} + U_a\,h_j\right)

This is the additive (Bahdanau) score function — a small one-hidden-layer MLP with learned weight matrices Wa,UaW_a, U_a and vector vav_a.

Softmax over all source positions gives a probability distribution:

αij=exp⁡(eij)∑k=1Txexp⁡(eik)\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}

The context vector is the resulting convex combination:

ci=∑j=1Txαij hjc_i = \sum_{j=1}^{T_x} \alpha_{ij}\, h_j
Why use additive scoring e = vᵀ tanh(W·s + U·h) rather than the simpler dot product s·h?
Show next line
eij=va⊤tanh⁡(Wasi−1+Uahj)e_{ij} = v_a^\top \tanh(W_a s_{i-1} + U_a h_j)
Why apply softmax along the source-position axis j (not across decoder steps i)?
Show next line
∑j=1Txαij=1,ci=∑jαijhj is a convex combination\sum_{j=1}^{T_x} \alpha_{ij} = 1,\quad c_i = \sum_j \alpha_{ij} h_j \text{ is a convex combination}

Retrieval check

What is the role of the bidirectional encoder in Bahdanau's model?
After computing α_{ij} via softmax, the context vector c_i equals:

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the six steps of one Bahdanau attention pass in the correct order:

  1. Run forward RNN over source tokens → hidden states h→_1 … h→_Tx
  2. Run backward RNN over source tokens → hidden states h←_1 … h←_Tx
  3. Normalise scores: α_{ij} = softmax(e_{ij}) over j
  4. For each decoder step i: compute scores e_{ij} = v_a^T tanh(W_a s_{i-1} + U_a h_j)
  5. Concatenate h→_j and h←_j to form h_j for each source position j
  6. Compute context c_i = Σ_j α_{ij} h_j, then update decoder state s_i

Fill the blank in this pseudo code:

def bahdanau_context(s_prev, H, W_a, U_a, v_a):
    # s_prev: (d_s,)  H: (T, d_h)  W_a: (d_a, d_s)  U_a: (d_a, d_h)  v_a: (d_a,)
    scores = v_a @ np.tanh(W_a @ s_prev + (U_a @ H.T).T)  # (T,)
    alpha = _______          # ← fill: normalise scores to a distribution
    c = alpha @ H            # (d_h,)
    return c, alpha

Retrieval check

Why must the bidirectional encoder finish processing the entire source before decoding starts?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement the Bahdanau additive attention score and softmax weighting in NumPy. Given decoder state s_prev, encoder states H, and projection matrices W_a, U_a, v_a, return the attention weights alpha (shape (T_enc,)) and context vector c (shape (d_h,)).

Retrieval check

Why subtract np.max(scores) before np.exp in the softmax implementation?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Bahdanau Attention.

Open in new tab · Download

Suggested reading order: §3 (Learning to Align and Translate — the full model description) → §A.1.2 (score function formula and motivation) → §4.2 (quantitative results and alignment visualisations).