ENID
Lensa ML

Decoder-Only Transformers

Step 1 of 6: The Architecture

Inside a decoder-only transformer block

Decoder blocks (N)5
Structure of a Decoder-Only TransformerFull model (left) and one decoder block (right)Output token vectors#sarecool.!×5Decoder blockDecoder blockDecoder blockDecoder blockDecoder blockPosemb+++++Input token vectorsLLM#sarecool.One decoder blockLayer NormMasked Self-Attention+Layer NormFFNN+Structure of a decoder-only transformer model
Blocks
5
Components/block
5
Attention
Causal

A small decoder-only model with 5 blocks. Input tokens are converted to vectors and combined with position embeddings (so the model knows word order). Each decoder block applies masked self-attention (tokens only see the past) followed by a feed-forward network, with residual connections and layer norms. The output predicts the next token at each position.

Drag the Decoder blocks (N) slider from 5 to 12. GPT-1 (2018) stacked 12 such blocks. The final layer maps each position's vector to one score per vocabulary token, and many models reuse the input embedding matrix for this output layer (weight tying). That saves vocabulary × width parameters, about 38.6 million in GPT-2 Small (50,257 × 768). Because one stack both reads the prompt and writes the answer, the same weights serve every task, which is a large part of why this design displaced encoder-decoder models for general-purpose language models.

Step 2 of 6: Causal Masking

Tokens can only see the past

Query position3
Causal Mask: tokens see only the pastUpper triangle = masked to −∞ before softmaxQueryKeyThecatsatonthematThecatsatonthemat1.00✕✕✕✕✕0.520.48✕✕✕✕0.320.310.37✕✕✕0.220.260.370.15✕✕0.250.190.140.180.25✕0.200.230.220.100.200.06"on" (pos 3) can attend to:The22%cat26%sat37%onquerythemaskedmatmasked
Query pos
3
Visible
4
Masked
2
Entropy
1.93

Position 3 sees 4 tokens. Its entropy is 1.93 bits, close to the maximum log₂(4) = 2.00: these random, untrained weights spread attention almost evenly, so entropy grows with context here. A trained head can stay sharp however much context it sees. At the last position, the model sees the full sequence, as at inference time. This is why decoder-only training is efficient: one forward pass gives 5 training signals (one per position).

Drag the Query position slider from 0 to 5 and watch the ENTROPY readout. Because no token can see later tokens, a token's representation never changes when text is appended after it. That property is what makes caching possible: the keys and values computed for a prompt stay valid for the rest of the conversation. A bidirectional encoder like BERT lacks it, since adding a word changes every earlier representation. Some models use a prefix-LM mask instead, where prompt tokens see each other fully and only the generated part is causal.

Step 3 of 6: Layer Normalization

Stabilizing activations in deep transformers

Activation scale5
Layer NormalizationStabilize activations between matrix multiplicationsRaw Activations (scale = 5×)4.0-6.01.510.5-2.58.0-4.52.0γ · (x − μ) / √(σ² + ε) + βAfter LayerNorm0.44-1.40-0.021.63-0.761.17-1.120.07Pre-Norm (modern LLMs)LayerNorm→Sublayer→Add
Mean
1.63
Std
5.45
RMS
5.69
Variant
LayerNorm

Standard Layer Normalization computes the mean and variance over the embedding dimension for each token independently. It then normalizes to zero mean and unit variance, and applies learnable scale (γ) and shift (β) parameters. With scale=5×, raw activations range from -6.0 to 10.5, but after LayerNorm they are stable between -1.40 and 1.63. Modern LLMs use pre-norm: normalizing before each sublayer (attention, FFN) rather than after, which makes deep models much more stable to train.

Drag the Activation scale slider from 1 to 20, then switch between LayerNorm and RMSNorm. LayerNorm computes y = γ · (x − μ) / √(σ² + ε) + β for each token; RMSNorm computes y = γ · x / √(mean(x²) + ε). The original Transformer normalized after each residual addition (post-norm), which needs a learning-rate warmup to train reliably. Xiong et al. (2020) showed that normalizing before each sublayer (pre-norm) keeps gradients well-behaved at initialization, so warmup matters much less. GPT-2 adopted pre-norm, and almost every LLM since has followed.

Step 4 of 6: Next-Token Prediction

The universal training objective

Temperature1.0
Next-Token Prediction"The cat sat down" → predict next wordThecatsatdownDecoder-Only TransformerLinear (d → vocab_size)Softmax (T=1.0)Probability distribution over vocabularysat17.3%in14.8%mat ✓13.5%cat13.4%down8.3%bed7.2%Training loss (T=1) = −log(P("mat")) = 2.00Every position predicts its next token → 3 training signals per sequence
Temperature
1.0
Top pred
sat
Loss (T=1)
2.00
Perplexity (T=1)
7.4

At temperature 1.0, the distribution is balanced. The model outputs logits (raw scores), divides by temperature, then applies softmax. Training uses T=1: loss = −log(P(correct)) = 2.00. Perplexity 7.4 means the model is as confused as choosing between ~7 equally likely tokens.

Drag the Temperature slider from 0.1 to 3.0 and watch the bars reshape while LOSS and PERPLEXITY, computed at T = 1 as in training, stay fixed. Perplexity is e^loss, the exponential of the average cross-entropy in nats, so a loss of 2.3 corresponds to a perplexity of about 10. Pretraining uses only this objective on raw text, with no labels, which is why it scales to trillions of tokens. Following instructions and holding a conversation come later, from fine-tuning on examples and from preference training such as RLHF. In a batch, the loss is averaged over every position of every sequence.

Step 5 of 6: Autoregressive Generation

How LLMs generate text token by token

Max tokens4
Autoregressive GenerationGenerate one token at a time, append, repeatCurrent sequence (2 tokens)thepromptcatpromptForward Pass (causal masked)Step 1: predict next tokencat22.9%on18.6%mat12.9%in11.0%a7.9%KV Cache: store K,V from previous tokensEach step recomputes only the new token → O(n) instead of O(n²)
Generated
0/4
Seq length
2
Next
cat
Confidence
23%

Starting with prompt "the cat". Press Play or Step to generate. At each step the model reads the sequence so far, and the last position's output distribution picks the next token. This is autoregressive generation: each new token depends on all previous ones.

Press Play or Step, and drag Max tokens from 1 to 8. Real systems run generation in two phases. Prefill runs the whole prompt in one parallel pass and fills the KV cache; decode then adds one token per pass. The cache grows as 2 (keys and values) × layers × width × tokens. For a 7B model with 32 layers and 4,096-wide attention in 16-bit precision, that is about 0.5 MB per token, or 2 GB for 4,096 tokens. Sampling often trims the tail, keeping the top k tokens or the smallest set whose probabilities reach p (nucleus sampling).

Step 6 of 6: Modern Variants & Scale

RoPE, GQA, SwiGLU, and scaling laws

Method50
Modern LLM Architecture ImprovementsSame decoder-only foundation, refined componentsPositional Encoding EvolutionAbsolutesin/cos or learnedGPT-2, BERT012345RoPERotation in pairsLLaMA, Mistral012345ALiBiLinear attention biasBLOOM, MPT0.0-0.1-0.2-0.3-0.4-0.5
Category
Position
Selected
RoPE
Used by
LLaMA

Positional encoding has changed over time. The original Transformer used fixed sinusoidal encodings. RoPE (Rotary Position Embeddings) applies rotation matrices to Q and K at each layer, so relative position shows up as the angle between vectors. ALiBi subtracts a distance-based linear bias from attention scores, with no learned parameters.

Click Position, Attention, FFN, and Scaling, and drag the slider within each. SwiGLU uses three weight matrices instead of two, so LLaMA shrinks the feed-forward width from 4d to about (2/3) · 4d ≈ 2.7d to keep the parameter count the same. Kaplan et al. (2020) found that loss falls as a power law in parameters, data, and compute. The Chinchilla result (Hoffmann et al., 2022) works out to about 20 training tokens per parameter for a fixed budget, though many current models train far longer because a smaller model is cheaper to run.

What is a decoder-only Transformer?

A decoder-only Transformer is a stack of identical blocks, each with masked self-attention and a feed-forward layer, trained to predict the next token of a text from the tokens before it. The causal mask stops each position from seeing later tokens, so one pass over a training text gives a prediction and a loss at every position. At inference time the model generates text by predicting one token, appending it, and repeating.

This is the architecture behind GPT, Llama, Mistral, Gemma, Claude, and most other large language models. The same network reads the prompt and writes the answer, which makes it a single general interface for chat, coding, translation, summarization, and reasoning tasks. Its design choices, such as pre-norm, RMSNorm, RoPE, grouped-query attention, and SwiGLU, now appear in almost every open model.

Key ideas

  • A decoder-only model is a stack of masked self-attention and feed-forward blocks.
  • The causal mask lets each position see only itself and earlier tokens.
  • Training minimizes next-token cross-entropy at every position at once.
  • Generation is autoregressive: predict a token, append it, repeat.
  • A KV cache stores past keys and values so each new token costs one short pass.
  • Modern LLMs add pre-norm RMSNorm, RoPE, grouped-query attention, and SwiGLU to the original design.

Common questions

What is the difference between decoder-only and encoder-decoder models?

An encoder-decoder model such as T5 reads the input with a bidirectional encoder and generates with a separate decoder that attends to it through cross-attention. A decoder-only model puts the input and output in one sequence and uses a single causal stack for both. The encoder-decoder split can be more efficient for tasks with a long input and short output, like summarization, while decoder-only models are simpler to scale and to use for open-ended tasks.

Why are most large language models decoder-only?

The training objective needs no labels and gives a loss at every token, so any text can be used. One model handles any task phrased as text, and the KV cache makes generation efficient. GPT-3 also showed that large decoder-only models can learn new tasks from examples in the prompt, which made scaling this design the main path.

What is a context window?

The context window is the maximum number of tokens the model can attend to at once, prompt and output together. It is limited by the positional encoding range used in training and by memory, since the KV cache and attention cost grow with length. Modern models range from a few thousand to over a million tokens.

Can a decoder-only model produce embeddings or classify text?

Yes. Because of the causal mask, the last token's hidden state is the only one that has seen the whole input, so it is often used as a summary vector for classification or retrieval. Many current embedding models are fine-tuned decoder-only LLMs. For classification, the model can also be prompted to output a label word.

Related explainers

References

  1. Brown et al. (2020), "Language Models are Few-Shot Learners", NeurIPS
  2. Touvron et al. (2023), "LLaMA: Open and Efficient Foundation Language Models"
  3. Xiong et al. (2020), "On Layer Normalization in the Transformer Architecture", ICML
  4. Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models"
  5. Holtzman et al. (2020), "The Curious Case of Neural Text Degeneration", ICLR

Last updated