Lensa ML
Lensa ML

How Positional Encoding Works

Step 1 of 6

Position Matters

Why word order changes meaning

The problem: Attention computes Q·KT — the dot product between every pair of tokens. This only measures similarity between words, not where they appear. Rearranging words gives the same dot products — so the model can't tell order apart.
Dogemb=[0.8, 0.2]bitesemb=[0.1, 0.9]manemb=[0.5, 0.6]Attention = Q·KT (no position info)0.680.260.520.260.820.590.520.590.61DogbitesmanDogbitesman

Sentence

Dog bites man

Meaning

Normal

Attn change

Same values!

"Dog bites man" — the normal sentence. The attention matrix shows dot products between all word pairs. Try the other orderings and watch what happens.

Step 1 of 6: Position Matters

Why word order changes meaning

The problem: Attention computes Q·KT — the dot product between every pair of tokens. This only measures similarity between words, not where they appear. Rearranging words gives the same dot products — so the model can't tell order apart.
Dogemb=[0.8, 0.2]bitesemb=[0.1, 0.9]manemb=[0.5, 0.6]Attention = Q·KT (no position info)0.680.260.520.260.820.590.520.590.61DogbitesmanDogbitesman

Sentence

Dog bites man

Meaning

Normal

Attn change

Same values!

"Dog bites man" — the normal sentence. The attention matrix shows dot products between all word pairs. Try the other orderings and watch what happens.

Step 2 of 6: Sinusoidal Encoding

Sin and cos waves encode position

PE(pos,2i) = sin(pos / 100002i/dmodel) = sin(10 / 6.31) = 1.000
Dimension →Position →p10d4dim 4 (sin, i=2)1.00+1−1

Position

10

Dim

4 (sin)

Angle

1.58

PE value

1.000

Low dimensions oscillate rapidly — notice the tight vertical stripes on the left side of the heatmap. These encode fine-grained position differences.

Step 3 of 6: Frequency Bands

Different dimensions, different scales

Positiondim 0

Dimension

0

Frequency

1.0000

Wavelength

6.3

Low dimensions have high frequency — they change rapidly across positions, encoding fine-grained position differences.

Step 4 of 6: Relative Distance

Why sinusoidal encodings work

Position Encoding Vectors (first 16 dims)0123456789101112131415pos 3pos 8dot = 6.14

pos_i

3

pos_j

8

Dot product

6.14

Distance

5

As distance grows, the dot product decreases smoothly. The sinusoidal design ensures similarity depends only on relative distance.

Step 5 of 6: RoPE

Rotary position embedding via 2D rotation

How RoPE modifies Q and K (showing pair 0)
q=[0.60, 0.40]→R(3·θ₀)→q'=[-0.65, -0.31]
k=[0.60, 0.40]→R(7·θ₀)→k'=[0.19, 0.70]
q rotated by 3 × θ₀ = 3.00 rad  |  k rotated by 7 × θ₀ = 7.00 rad  |  q'·k' depends only on |3−7| = 4
pair 0Δ=-4.0pair 1Δ=-0.4pair 2Δ=-0.0pair 3Δ=-0.0q (pos 3)k (pos 7)angle between = f(|3−7|) only

q pos

3

k pos

7

q·k

1.18

|q−k|

4

RoPE rotates q and k by different amounts (Δθ depends on |3−7|=4). The dot product depends only on this relative distance — not on absolute position. Move both sliders by the same offset and watch q·k stay constant!

Step 6 of 6: Sinusoidal vs RoPE

Comparing the two approaches

Similarity to position 2 as position n variesposition nsimilaritym=2SinusoidalRoPE
SinusoidalRoPE
Where appliedAdded to inputRotates Q, K
WhenBefore projectionAfter projection
Position typeAbsoluteRelative
Parameters00
ExtrapolationLimitedBetter
Used byOriginal TransformerLLaMA, Mistral, Gemma

|m−n|

6

Sin dot

3.78

RoPE dot

1.97

At large distances, both show low similarity. RoPE's advantage: since it encodes relative position directly into Q·K, it generalizes better to sequence lengths not seen during training. That's why LLaMA, Mistral, and Gemma all use RoPE.