1How Self-Attention Works 2Multi-Head Attention 3Positional Encoding 4Cross-Attention 5The Transformer Architecture 6Decoder-Only Transformers

Lensa ML

How Cross-Attention Works

Step 1 of 6: Self vs Cross-Attention

Where Q, K, V come from

Self

Cross

Q from

Same Seq

K from

Same Seq

V from

Same Seq

In self-attention, Q, K, and V all come from the same sequence. Each token attends to every other token in its own sequence.

Drag the slider from Self to Cross and watch where K and V come from. The formula does not change: softmax(Q · Kᵀ / √d_k) · V. Only the inputs do: Q = X_dec · W_Q, while K = X_enc · W_K and V = X_enc · W_V. Because the two sequences can have different lengths, the score matrix is m × n instead of square, with one row per decoder token and one column per source token. The two sides can even have different widths, as long as W_Q and W_K project both to the same d_k.

Step 2 of 6: Encoder-Decoder Attention

Translation with cross-attention arcs

Source word

"The"

Source

"The"

82%

chat

assis

"The" strongly aligns with "Le", so cross-attention has found the translation pair.

Drag the Source word slider across "The", "cat", and "sat". Each percentage reads down one column of the alignment matrix: how strongly each French word attends to the chosen English word. Rows must sum to 100% but columns need not, so a source word can be ignored entirely or used by several target words. In the original Transformer each of the 6 decoder layers has its own cross-attention block, and all of them read the same final encoder output.

Step 3 of 6: Q from the Decoder

The decoder asks questions of the encoder

Decoder pos

"Le"

Dec Pos

Q vec

[0.3, -0.1, 0.5]

Top Attended

"The"

Generating "Le": the decoder's Q queries the encoder and attends most to "The", the matching article.

Drag the Decoder pos slider to generate "Le", "chat", and "assis" in turn. The decoder's query changes at every step, but the encoder's keys and values do not. Implementations compute K and V for the source sentence once and reuse them for every generated token, so cross-attention adds little cost per step. The query comes from the output of the decoder's masked self-attention, so it already encodes the French words produced so far. That is how "assis" can depend on both "sat" and the earlier "chat".

Step 4 of 6: Word Alignment

Attention as a soft alignment matrix

Source

"The"

Target

"Le"

Source

"The"

Target

"Le"

Attention

82%

Strong alignment: "Le" directly translates "The". Cross-attention learns these word correspondences.

Drag the Source and Target sliders to pick one cell of the 3 × 3 matrix. The matrix is almost diagonal because English and French share word order in this phrase. For "the black cat" and "le chat noir" the weights would cross, since French puts the adjective after the noun. Statistical translation systems learned hard word alignments explicitly, starting with the IBM models of the early 1990s. Attention learns soft alignments as a side effect of training for translation, with no alignment labels.

Step 5 of 6: Multi-Modal Cross-Attention

Images meet text through attention

Region

Cat

Word

"cat"

Region

Cat

Word

"cat"

Attention

80%

The word "cat" strongly attends to the Cat region: cross-attention grounds the word in visual content.

Drag the Region and Word sliders to pair a word with one of the 4 image regions. In text-to-image diffusion models such as Stable Diffusion the roles are the other way round: image features are the queries, and the text encoder's token embeddings supply the keys and values, so each patch of the image being generated looks up the words that describe it. Editing methods such as Prompt-to-Prompt change an image by editing these cross-attention maps. Flamingo added gated cross-attention layers to a frozen language model so it could read images.

Step 6 of 6: Cross-Attention Through Layers

Attention refines across decoder depth

Layer

1 / 6

Entropy

1.57

Top Conf.

39%

Early layers: attention is diffuse, and the decoder gathers information from all encoder positions.

Drag the Layer slider from 1 to 6. The softmax of layer 1's logits gives weights of about 39%, 32%, and 29%, while layer 6 puts about 99% on one token. This steady sharpening is a simplification. In trained translation models the clearest alignments often come from one of the later layers but not always the last, and some layers stay diffuse because they gather broad context. Cross-attention maps are used to inspect translations and, in Whisper, to estimate when each transcribed word was spoken.

What is cross-attention?

Cross-attention is an attention operation in which the queries come from one sequence and the keys and values come from another. Each token of the first sequence, such as a word the decoder is generating, scores every token of the second sequence, such as the encoded source sentence, and takes a weighted average of its values. It is the same scaled dot-product attention as self-attention; only the source of K and V changes.

Cross-attention connects the two halves of encoder-decoder models for translation, summarization, and speech recognition, including T5 and Whisper. It is also how text-to-image diffusion models such as Stable Diffusion let the text prompt steer each part of the image, and how some vision-language models let a language model read image features. Any time one stream of data has to look up information in another, cross-attention is a standard choice.

Key ideas

Cross-attention takes queries from one sequence and keys and values from another.
The math is the same as self-attention: softmax(Q · Kᵀ / √d_k) · V.
The score matrix is m × n, so the two sequences can have different lengths.
Encoder keys and values are computed once and reused at every decoding step.
Cross-attention weights act as a soft alignment between the two inputs.
It links text to images in diffusion models and vision-language models.

Common questions

How is cross-attention different from concatenating the two inputs?

A decoder-only model can join both inputs into one sequence and use self-attention over all of it, as many vision-language models do with image tokens. Cross-attention instead keeps the streams separate, so only one side is updated and the other is read. That saves compute when the source is long and lets the source encoder be frozen or reused.

Why do the keys and values come from the encoder?

Keys decide what can be found and values decide what is returned, so both have to describe the information being looked up. The query describes what the current position needs. Swapping the roles would make the source sentence ask questions of the output instead.

How much compute does cross-attention need?

For m queries and n source tokens it computes m × n scores per head, which is linear in each length. Since the source keys and values are fixed, they are projected once and cached during generation. The cost grows with source length, which matters for long documents and high-resolution images.

What is the Perceiver?

The Perceiver (Jaegle et al., 2021) uses cross-attention from a small, fixed set of learned latent vectors to a very large input, such as all the pixels of an image. The cost is then proportional to the number of latents times the input size, not the input size squared. Self-attention then runs only among the latents.

Related explainers

How Self-Attention Works — Query, key, value — the core of self-attention
The Transformer Architecture — A visual guide to 'Attention Is All You Need'
Seq2Seq & Beam Search — Encode an input sequence and decode an output — the architecture behind translation
Multi-Head Attention — Parallel attention heads learning different patterns
Diffusion Models — Generate images by learning to reverse a gradual noising process
Decoder-Only Transformers — The architecture behind GPT and modern LLMs

References

Last updated October 25, 2026