(start of sequence) is the first input to the decoder, giving it something to condition on before any word has been produced. (end of sequence) is a token the decoder learns to emit when the output is complete, which is how the model decides its own output length.

1Text Classification & Sentiment 2Seq2Seq & Beam Search 3Named Entity Recognition 4Language Modeling 5BERT & Masked Language Modeling

Lensa ML

Seq2Seq & Beam Search

Step 1 of 6: Encoder-Decoder Idea

Compress input → fixed vector → generate output

Input sequence length4

Input length

Output length

Bottleneck

1 vector

4 input tokens → 1 context vector → 4 output tokens. However long the input, the encoder packs it into one vector. The decoder never sees the original tokens, only this summary.

Drag Input sequence length from 2 to 7 and compare the input and output token counts. The 2014 paper by Sutskever, Vinyals and Le used two separate 4-layer LSTMs with 1,000 cells per layer, one to read English and one to write French, so each sentence became 8,000 numbers of state. One finding from that paper: reversing the word order of the source sentence raised BLEU from 25.9 to 30.6, because it placed the first source words next to the first target words. Cho et al. proposed a similar RNN encoder-decoder the same year.

Step 2 of 6: Encoding

The encoder RNN processes input tokens one at a time

Time step0

Time step

1 / 4

Token

‖h‖

1.233

Δ from prev

1.233

Processing "I": h1 is computed from the zero initial state and the input embedding. The heatmap shows the 6D hidden vector, the RNN's running memory. Red = positive, purple = negative.

Drag Time step from 0 to 3 to feed "I love deep learning" into the encoder one word at a time. Each step applies the same update, h_t = tanh(W_h h_{t−1} + W_x x_t + b), with the same weights at every position, so the encoder accepts inputs of any length. Real systems use LSTM or GRU cells instead of this plain tanh RNN, because their gates keep gradients from vanishing over long inputs. Bidirectional encoders run a second RNN from right to left and join the two states, so each position sees words on both sides.

Step 3 of 6: Decoding

The decoder generates output tokens autoregressively

Decoding step0

Step

1 / 6

Input

<SOS>

Output

Confidence

62%

Decoding begins: <SOS> + context vector → s1 → softmax picks "J'" (62% confidence). The curved arrow feeds this output back in as the next step's input.

Drag Decoding step from 0 to 5 to generate the translation one token at a time. The decoder factors the output probability as P(y | x) = Π_t P(y_t | y_<t, c), where each term is conditioned on the previous outputs and the context vector c. During training the decoder is fed the correct previous word rather than its own guess, a method called teacher forcing. It speeds up learning but causes exposure bias: at test time the model has to recover from its own mistakes, which it never practiced during training.

Step 4 of 6: Greedy vs Beam Search

Exploring multiple hypotheses for better translations

Beam width1

Beam width

Expanded

Pruned

Best score

-1.48

Greedy decoding (beam=1): only the single best token survives at each step. 3 candidates pruned at t=1, 2 at t=2. Fast but short-sighted: the best token now may not start the best sentence.

Drag Beam width from 1 to 4 and watch how many branches survive. Exhaustive search is impossible: with a 30,000-word vocabulary, a 10-word output has 30,000¹⁰ candidates. Beam search keeps only the k best partial outputs at each step, at about k times the cost of greedy decoding. Sutskever et al. found that a beam of 2 already gave most of the gain. Beam search suits tasks with one right answer, such as translation and speech recognition. Chatbots and story writing usually sample instead, because beams tend to produce bland, repetitive text.

Step 5 of 6: Beam Scores

Ranking candidate translations by log-probability

Length penalty α0.00

0.00

Best

-1.28

Best hyp

Length range

3–5

No length penalty (α=0): the raw log-prob sum sets the rank. Every extra token adds a negative term, so this score favors short outputs: the 3-token "J' aime <EOS>" beats the full translation. Amber-bordered tokens (first two columns) are step 4's beam paths.

Drag Length penalty α from 0 to 1 and watch the normalized scores change. A sentence's score is the sum of its token log-probabilities, and every token can only lower it. Without a correction, beam search prefers short outputs and can stop too early. Google's 2016 neural translation system divided by a softer penalty, lp(Y) = ((5 + |Y|) / 6)^α, rather than by the raw length. It also added a coverage penalty that favors outputs attending to every source word, which targets a related failure: translations that skip part of the input.

Step 6 of 6: Limitations & Attention

The information bottleneck motivates the attention mechanism

Input sequence length4

Input length

Seq2Seq

38.3

+ Attention

41.7

Gap

+3.4

Short inputs: the fixed-size context vector holds enough information, so plain Seq2Seq does well and attention adds little. (The curves are a schematic of the trend reported by Bahdanau et al., 2015, not scores measured here.)

Drag Input sequence length from 1 to 20 and compare the two curves. The numbers are illustrative, but the shape matches Cho et al. (2014), who found that a basic encoder-decoder's translations got much worse as sentences grew longer. Bahdanau, Cho and Bengio fixed this by letting the decoder compute a fresh weighted average of all encoder states at every output step. The Transformer (2017) kept the encoder-decoder layout and the attention, and replaced the recurrent layers with attention as well.

What is a sequence-to-sequence (seq2seq) model?

A sequence-to-sequence (seq2seq) model is a neural network that maps an input sequence to an output sequence of a possibly different length. An encoder reads the input and compresses it into a context vector, and a decoder generates the output one token at a time from that vector. In the original 2014 form, both halves are recurrent networks.

Seq2seq models made neural machine translation practical and replaced phrase-based systems at Google Translate and elsewhere within a few years. The same encoder-decoder layout handles summarization, speech recognition, question answering, and code generation. Its main weakness, squeezing a whole input into one vector, led directly to attention and then to the Transformer, which keeps the encoder-decoder design in models such as T5 and BART.

Key ideas

An encoder turns the input sequence into a context vector; a decoder turns that vector into the output sequence.
Input and output lengths are independent, which suits translation and summarization.
The decoder is autoregressive: each output token is fed back as the next input.
Teacher forcing trains the decoder on correct previous tokens instead of its own predictions.
Beam search keeps the k most probable partial outputs instead of only the single best one.
One fixed-size context vector limits quality on long inputs, which attention removes.

Common questions

What is the difference between seq2seq and a Transformer?

Seq2seq names the task setup, mapping one sequence to another with an encoder and a decoder. The original seq2seq models built both halves from RNNs. The Transformer is an architecture that fills the same two roles with attention layers, so it is itself a seq2seq model, just not a recurrent one.

What are <SOS> and <EOS> tokens?

<SOS> (start of sequence) is the first input to the decoder, giving it something to condition on before any word has been produced. <EOS> (end of sequence) is a token the decoder learns to emit when the output is complete, which is how the model decides its own output length.

How are seq2seq models evaluated?

Translation is usually scored with BLEU, which counts how many n-grams of the output also appear in reference translations. Summarization often uses ROUGE, which focuses on recall of reference n-grams. Both are rough proxies, so human ratings and learned metrics such as COMET are used alongside them.

Is GPT a seq2seq model?

Not in the strict sense. GPT is decoder-only: the input and output share one sequence, and the model continues it token by token. It can still do seq2seq tasks such as translation by reading the source text as a prompt and generating the target after it.

Related explainers

Recurrent Neural Networks — How RNNs read and understand sequences
LSTMs — Gates and memory cells for long sequences
How Self-Attention Works — Query, key, value — the core of self-attention
Cross-Attention — Attending across sequences and modalities
The Transformer Architecture — A visual guide to 'Attention Is All You Need'
Language Modeling — Predict the next token — the objective that powers GPT, BERT, and modern NLP

References

Last updated October 25, 2026