The Transformer Architecture

Short sequences are fine for RNNs. But the Transformer paper showed that self-attention connects every token directly in O(1) hops — no information bottleneck, and fully parallelizable across GPUs.

The Transformer (Vaswani et al., 2017) is an encoder-decoder architecture built entirely from attention — no recurrence, no convolution. The encoder reads the full input bidirectionally; the decoder generates output tokens autoregressively with causal masking. Click a component to explore.

"The" pays 45% attention to "cat". Each token creates a Query vector ("what am I looking for?") and a Key vector ("what do I contain?"). The dot product Q·K measures relevance, divided by √d_k to prevent extreme values, then softmax turns scores into weights that sum to 1.

When "The" queries, Head 1 focuses on "sat" while Head 2 focuses on "down". Each head projects Q, K, V into a different subspace (d_k=64), learning to detect different relationships — one might capture syntax, another semantics.

Position 0 ("The"): each bar above is computed by plugging pos=0 into sin and cos at different frequencies. Low-frequency dimensions (sin0, cos0) change slowly across positions; high-frequency ones (sin3, cos3) change rapidly. Together they form a unique "address" that's added to the word embedding — without it, "cat sat" and "sat cat" would look identical.

Temperature 1.0 is the balanced setting. The encoder processes the full English sentence, the decoder generates French token by token. At each step, the decoder uses cross-attention to "read" the encoder's output — aligning "chat" with "cat", "assis" with "sat". This is the architecture that achieved state-of-the-art translation in 2017.