← Roadmap

Sequence to Sequence Learning with Neural Networks

Sutskever, Vinyals, Le · 2014

① Intuition

Before seq2seq, neural machine translation was largely handled by phrase-based statistical models — pipelines of hand-tuned components that decomposed sentences into fragments, translated each fragment, and stitched them back together. These systems were complex, brittle, and required extensive feature engineering for each language pair.

Sutskever, Vinyals, and Le’s key insight: a single end-to-end trained LSTM can map a variable-length source sequence to a variable-length target sequence by first encoding the entire source into one fixed-size vector v (the final hidden state), then decoding from that vector token by token. No phrase tables, no alignment models — just two RNNs and a training objective. The paper also uncovered a surprising empirical trick: reversing the source sequence boosted BLEU by ~5 points. When the source is reversed, the first source words (which must align to the first target words) end up at the end of the encoding trajectory — just a few LSTM steps from where decoding begins. Shorter “temporal distance” makes the gradient path shorter and alignment easier.

Before playing with the widget: if the source is 'the cat sat' and the target starts with 'le', which source token should ideally be closest in LSTM-time to the start of decoding? Type your prediction.

Adjust the source length and toggle reversal. Watch how the retention bar for the first source word changes.

the
3.3%
cat
5.4%
sat
9.1%
on
15.1%
the
25.2%
mat
42.0%

Distance from final encoder step to “the” = 5

Retention ∝ 0.6(distance from end), normalized. Original order — first word is farthest from the final encoder step.

With the source reversed (checkbox checked) and length set to 6, where does the first source word appear relative to the final encoder step? How does this affect the gradient path during backprop-through-time?

Retrieval check

What is the 'fixed-vector bottleneck' in the original seq2seq model?
Why does reversing the source sequence reduce the 'minimal time lag' between source and target words?

Answer all 2 correctly to mark this stage mastered.

② Math

The seq2seq model defines a conditional probability over the target sequence given the source. The encoder reads source tokens x1,…,xTsrcx_1, \ldots, x_{T_{\text{src}}} and produces a single summary vector:

v=hTsrcencv = h_{T_{\text{src}}}^{\text{enc}}

The decoder then factorises the joint target probability using the chain rule:

p(y1,…,yTtgt∣x1,…,xTsrc)=∏t=1Ttgtp ⁣(yt∣v, y1,…,yt−1)p(y_1, \ldots, y_{T_{\text{tgt}}} \mid x_1, \ldots, x_{T_{\text{src}}}) = \prod_{t=1}^{T_{\text{tgt}}} p\!\left(y_t \mid v,\, y_1, \ldots, y_{t-1}\right)

Each factor p(yt∣v,y<t)p(y_t \mid v, y_{<t}) is computed by a softmax over the vocabulary applied to the decoder hidden state htdech_t^{\text{dec}}, which evolves as:

htdec=LSTM ⁣(ht−1dec, [yt−1; v])h_t^{\text{dec}} = \text{LSTM}\!\left(h_{t-1}^{\text{dec}},\, [y_{t-1};\, v]\right)

Training maximises the log-likelihood over all sentence pairs (X,Y)(X, Y) in the corpus:

L=∑(X,Y)∑t=1Ttgtlog⁡p ⁣(yt∣v, y<t)\mathcal{L} = \sum_{(X,Y)} \sum_{t=1}^{T_{\text{tgt}}} \log p\!\left(y_t \mid v,\, y_{<t}\right)

At inference time, beam search maintains the top-B partial hypotheses at each step, expanding each by all vocabulary tokens and retaining only the B highest-probability continuations.

Why can the joint probability p(y_1,…,y_T | x_1,…,x_T_src) be written as a product of conditionals?
Show next line
p(y1,…,yT∣v)=∏t=1Tp(yt∣v, y1,…,yt−1)(chain rule)p(y_1,\ldots,y_T \mid v) = \prod_{t=1}^{T} p(y_t \mid v,\, y_1,\ldots,y_{t-1}) \quad \text{(chain rule)}
Beam search with width B=1 is equivalent to which simpler algorithm?
Show next line
B=1:  yt=arg⁡max⁡yp(y∣v, y<t)(greedy decoding)B=1:\; y_t = \arg\max_{y} p(y \mid v,\, y_{<t}) \quad \text{(greedy decoding)}

Retrieval check

In seq2seq training, what quantity is maximised?
What is v in the seq2seq model?

Answer all 2 correctly to mark this stage mastered.

③ Pseudo code

Arrange the five steps of the seq2seq encoder–decoder algorithm in the correct order:

  1. Run encoder RNN step-by-step over src tokens → hidden states h_1^enc … h_T^enc
  2. At each decoder step t: compute h_t^dec = RNN(h_{t-1}^dec, [y_{t-1}; v]); output y_t = argmax softmax(W h_t^dec)
  3. Initialise decoder hidden state h_0^dec = v; set input y_0 = <BOS>
  4. Extract the final encoder hidden state v = h_T^enc
  5. Optionally reverse the source token list: src = src[::-1]

Fill the blank in this pseudo code for the decoder loop:

def greedy_decode(v, W_out, rnn, max_steps):
    h = v
    y = BOS_ID
    outputs = []
    for _ in range(max_steps):
        h = rnn(h, embed(y))
        logits = W_out @ h
        y = _______        # ← fill: pick next token
        if y == EOS_ID:
            break
        outputs.append(y)
    return outputs

Retrieval check

During teacher forcing, what is fed as the decoder input at each step?

Answer all 1 correctly to mark this stage mastered.

④ Code (Python)

Implement a greedy seq2seq decoder in pure Python / NumPy. You are given a toy single-layer RNN cell ht=tanh⁡(Whht−1+Ueet)h_t = \tanh(W_h h_{t-1} + U_e e_t), a readout matrix WoutW_{\text{out}}, and an encoder context vector v. Complete the greedy_decode function so it runs for exactly n_steps steps and returns the list of predicted token indices.

Retrieval check

In the greedy decoder, what does np.argmax(logits) return?

Answer all 1 correctly to mark this stage mastered.

⑤ Original paper

Your browser can't embed PDFs. Download Sequence to Sequence Learning with Neural Networks.

Open in new tab · Download

Suggested reading order: §3 (The Model — encoder/decoder architecture and training objective) → §3.3 (Reversing the Source — the empirical trick and its intuition) → §5 (Experiments — BLEU results on WMT English→French showing reversal gains).