How Multi-Head Attention Works
One Head's Limit
A single head captures only one attention pattern per query
Query
cat
Top Key
cat
Entropy
2.54
"cat" spreads attention broadly (entropy 2.54). These are the same weights from self-attention — a single head can represent only one attention pattern per query.
Step 1 of 6: One Head's Limit
A single head captures only one attention pattern per query
Query
cat
Top Key
cat
Entropy
2.54
"cat" spreads attention broadly (entropy 2.54). These are the same weights from self-attention — a single head can represent only one attention pattern per query.
Step 2 of 6: Multiple Heads
Split the model dimension across parallel attention heads
d_model
64
# Heads
4
d_k
16
Params
4x
4 heads each get d_k = 16 dimensions. Each head can learn a different attention pattern in its own subspace.
Step 3 of 6: Different Patterns
Each head learns a distinct type of relationship
Head
H1
Pattern
Positional
Top Key
cat
Entropy
2.00
Head 1 (Positional) — "cat" attends most to nearby words. This captures local context and word order.
Step 4 of 6: Concatenation
Stack all head outputs back together
Per Head
16
Concat
64
Output
64
4 heads of size 16 concatenate to 64 = d_model. The linear projection W_O combines all head outputs back to 64 dimensions.
Step 5 of 6: Output Projection
W_O mixes head outputs into the final representation
Concat
64
W_O Shape
64x64
Output
64
W_O is a learned linear projection that mixes all head outputs. Head 1 (Positional) contributes 35% — its local patterns get blended with other heads' signals.
Step 6 of 6: Multi-Head in Action
Compare attention patterns across heads for a full sentence
Head
H1
Query
cat
Top Key
cat
Weight
0.47
Comparing all heads for "cat": Head 1 captures positional proximity, while other heads capture different relationships. Together they give the model a rich, multi-faceted view of the sentence.