How Cross-Attention Works
Self vs Cross-Attention
Where Q, K, V come from
Q from
Same Seq
K from
Same Seq
V from
Same Seq
In self-attention, Q, K, and V all come from the same sequence. Each token attends to every other token in its own sequence.
Step 1 of 6: Self vs Cross-Attention
Where Q, K, V come from
Q from
Same Seq
K from
Same Seq
V from
Same Seq
In self-attention, Q, K, and V all come from the same sequence. Each token attends to every other token in its own sequence.
Step 2 of 6: Encoder-Decoder Attention
Translation with cross-attention arcs
Source
"The"
Le
82%
chat
5%
assis
7%
"The" strongly aligns with "Le" — cross-attention finds the translation pair.
Step 3 of 6: Q from the Decoder
The decoder asks questions of the encoder
Dec Pos
1
Q vec
[0.3, -0.1, 0.5]
Top Attended
"The"
Generating "Le": the decoder's Q queries the encoder and attends most to "The" — matching the article.
Step 4 of 6: Word Alignment
Attention as a soft alignment matrix
Source
"The"
Target
"Le"
Attention
82%
Strong alignment: "Le" directly translates "The". Cross-attention learns these word correspondences.
Step 5 of 6: Multi-Modal Cross-Attention
Images meet text through attention
Region
Cat
Word
"cat"
Attention
80%
The word "cat" strongly attends to the Cat region — cross-attention grounds language in visual content.
Step 6 of 6: Cross-Attention Through Layers
Attention refines across decoder depth
Layer
1 / 6
Entropy
1.57
Top Conf.
39%
Early layers: attention is diffuse — the decoder broadly gathers information from all encoder positions.