① Intuition
Before seq2seq, neural machine translation was largely handled by phrase-based statistical models — pipelines of hand-tuned components that decomposed sentences into fragments, translated each fragment, and stitched them back together. These systems were complex, brittle, and required extensive feature engineering for each language pair.
Sutskever, Vinyals, and Le’s key insight: a single end-to-end trained LSTM can map a variable-length source sequence to a variable-length target sequence by first encoding the entire source into one fixed-size vector v (the final hidden state), then decoding from that vector token by token. No phrase tables, no alignment models — just two RNNs and a training objective. The paper also uncovered a surprising empirical trick: reversing the source sequence boosted BLEU by ~5 points. When the source is reversed, the first source words (which must align to the first target words) end up at the end of the encoding trajectory — just a few LSTM steps from where decoding begins. Shorter “temporal distance” makes the gradient path shorter and alignment easier.
Before playing with the widget: if the source is 'the cat sat' and the target starts with 'le', which source token should ideally be closest in LSTM-time to the start of decoding? Type your prediction.
‘the’ — it is the first source token and must align to the first target token ‘le’. In the normal (non-reversed) order, ‘the’ is processed first and its signal must survive T_src − 1 LSTM steps before decoding starts. Reversing puts ‘the’ last, so only 1 LSTM step separates its encoding from the first decoder step. This is the geometric intuition behind the reversal trick.
Adjust the source length and toggle reversal. Watch how the retention bar for the first source word changes.
Distance from final encoder step to “the” = 5
Retention ∝ 0.6(distance from end), normalized. Original order — first word is farthest from the final encoder step.
With the source reversed (checkbox checked) and length set to 6, where does the first source word appear relative to the final encoder step? How does this affect the gradient path during backprop-through-time?
When reversed, the first source word is at distance 0 from the final encoder step — it is the very last token the encoder sees. During backpropagation-through-time, the gradient from the first decoder output to the encoding of the first source word travels through only a handful of steps instead of T_src − 1 steps. Shorter BPTT paths suffer less from vanishing gradients, making it easier for the model to learn that the first source word should influence the first target word.
Retrieval check
Answer all 2 correctly to mark this stage mastered.
② Math
The seq2seq model defines a conditional probability over the target sequence given the source. The encoder reads source tokens and produces a single summary vector:
The decoder then factorises the joint target probability using the chain rule:
Each factor is computed by a softmax over the vocabulary applied to the decoder hidden state , which evolves as:
Training maximises the log-likelihood over all sentence pairs in the corpus:
At inference time, beam search maintains the top-B partial hypotheses at each step, expanding each by all vocabulary tokens and retaining only the B highest-probability continuations.
Show next line
Show next line
Retrieval check
Answer all 2 correctly to mark this stage mastered.
③ Pseudo code
Arrange the five steps of the seq2seq encoder–decoder algorithm in the correct order:
- Run encoder RNN step-by-step over src tokens → hidden states h_1^enc … h_T^enc
- At each decoder step t: compute h_t^dec = RNN(h_{t-1}^dec, [y_{t-1}; v]); output y_t = argmax softmax(W h_t^dec)
- Initialise decoder hidden state h_0^dec = v; set input y_0 = <BOS>
- Extract the final encoder hidden state v = h_T^enc
- Optionally reverse the source token list: src = src[::-1]
Fill the blank in this pseudo code for the decoder loop:
def greedy_decode(v, W_out, rnn, max_steps):
h = v
y = BOS_ID
outputs = []
for _ in range(max_steps):
h = rnn(h, embed(y))
logits = W_out @ h
y = _______ # ← fill: pick next token
if y == EOS_ID:
break
outputs.append(y)
return outputsRetrieval check
Answer all 1 correctly to mark this stage mastered.
④ Code (Python)
Implement a greedy seq2seq decoder in pure Python / NumPy. You are given a toy single-layer RNN cell , a readout matrix , and an encoder context vector v. Complete the greedy_decode function so it runs for exactly n_steps steps and returns the list of predicted token indices.
Retrieval check
Answer all 1 correctly to mark this stage mastered.
⑤ Original paper
Suggested reading order: §3 (The Model — encoder/decoder architecture and training objective) → §3.3 (Reversing the Source — the empirical trick and its intuition) → §5 (Experiments — BLEU results on WMT English→French showing reversal gains).