Seq2Seq & Beam Search
Encoder-Decoder Idea
Compress input → fixed vector → generate output
4 input tokens → 1 context vector → 4 output tokens. No matter how long the input, the encoder must pack everything into one vector. The decoder never sees the original tokens — only this summary.
Step 1 of 6: Encoder-Decoder Idea
Compress input → fixed vector → generate output
4 input tokens → 1 context vector → 4 output tokens. No matter how long the input, the encoder must pack everything into one vector. The decoder never sees the original tokens — only this summary.
Step 2 of 6: Encoding
The encoder RNN processes input tokens one at a time
Processing "I": h1 is computed from the zero-initialized state + the input embedding. The heatmap shows the 6D hidden vector — each cell's color and value reveals what the RNN "remembers." Green = positive, purple = negative.
Step 3 of 6: Decoding
The decoder generates output tokens autoregressively
Decoding begins: <SOS> + context vector → s1 → softmax picks "J'" (62% confidence). The curved arrow shows this output feeding back as the next step's input — this is autoregressive decoding.
Step 4 of 6: Greedy vs Beam Search
Exploring multiple hypotheses for better translations
Greedy decoding (beam=1): only the single best token survives at each step. 3 candidates pruned at t=1, 2 at t=2. Fast but myopic — the locally best token may not lead to the globally best sequence.
Step 5 of 6: Beam Scores
Ranking candidate translations by log-probability
No length penalty (α=0): raw log-prob sum determines rank. The amber-bordered tokens (first two columns) are the beam paths from step 4 — each hypothesis extends a surviving beam. Shorter sequences win unfairly here. Try increasing α.
Step 6 of 6: Limitations & Attention
The information bottleneck motivates the attention mechanism
Short inputs: Seq2Seq works well! The fixed-size context vector can hold enough information. The attention advantage is small here.