Decoder-Only Transformers

With 5 decoder blocks stacked, the model builds increasingly abstract representations. Early blocks capture syntax and local patterns; later blocks capture semantics and long-range dependencies. Each block has the same structure: Layer Norm → Masked Self-Attention → Add → Layer Norm → FFNN → Add. GPT-2 Small uses 12 blocks; LLaMA-7B uses 32.

Position 3 sees 4 tokens. More context = higher entropy (1.98 bits) since attention is spread across more tokens. At the last position, the model sees the full sequence — same as inference time. This is why decoder-only training is so efficient: one forward pass generates 5 training signals (one per position).

Standard Layer Normalization computes the mean and variance over the embedding dimension for each token independently. It then normalizes to zero mean and unit variance, and applies learnable scale (γ) and shift (β) parameters. With scale=5×, raw activations range from -6.0 to 10.5, but after LayerNorm they are stable between -1.40 and 1.63. Modern LLMs use pre-norm: normalizing before each sublayer (attention, FFN) rather than after, which dramatically improves training stability for deep models.

At temperature 1.0, the distribution is balanced. The model outputs logits (raw scores), divides by temperature, then applies softmax. Loss = −log(P(correct)) = NaN. Perplexity NaN means the model is as confused as choosing between ~NaN equally likely tokens.

Starting with prompt "the cat". Press Play or Step to generate. At each step, the entire sequence feeds through the model, and the last token's output distribution determines the next token. This is autoregressive generation: each new token depends on all previous ones.

Positional encoding has evolved significantly. The original Transformer used fixed sinusoidal encodings. RoPE (Rotary Position Embeddings) applies rotation matrices to Q and K at each layer — relative positions are encoded via the angle between vectors, enabling better length generalization. ALiBi simply adds a linear bias to attention scores, requiring no learned parameters.