How Self-Attention Works

"cat" attends most to itself (22%). In self-attention, each word in a sentence computes how relevant every other word is to its own meaning — including itself.

The Query matrix is computed by multiplying the input embeddings by W_Q. Each row becomes a query vector — it asks "what am I looking for?"

Score("cat", "cat") = 0.63 is the highest. The attention score is the dot product Q·K — multiply each dimension pairwise and sum. Higher scores mean the query and key are more aligned.

Raw Q·K scores are divided by √d_k = 1.73 to prevent large values from pushing softmax into extremes. Then softmax converts scores into a probability distribution — "cat" puts 21.5% of its attention on "cat".

In bidirectional self-attention, "sat" attends to all 6 tokens equally. This is used in encoders (like BERT) where the full context is available. Compare with causal masking to see the difference.

This is the full attention pipeline for "cat": embed → project to Q,K,V → compute Q·K scores → scale by 1/√3 → softmax → weighted sum of V vectors. The output [0.14, 0.17, 0.45] is a context-aware representation that blends information from all words, weighted by relevance.