ENID
Lensa ML

How Positional Encoding Works

Step 1 of 6: Position Matters

Why word order changes meaning

The problem: Attention computes Q·KT: the dot product between every pair of tokens. This only measures similarity between words, not where they appear. Rearranging words gives the same dot products, so the model can't tell the orders apart.
Dogemb=[0.8, 0.2]bitesemb=[0.1, 0.9]manemb=[0.5, 0.6]Attention = Q·KT (no position info)0.680.260.520.260.820.590.520.590.61DogbitesmanDogbitesman

Sentence

Dog bites man

Meaning

Normal

Attn change

Same values!

"Dog bites man": the normal sentence. The attention matrix shows dot products between all word pairs. Try the other orderings.

Click each of the three sentence buttons. The same three words give the same nine dot products in every order. Absolute schemes add a position vector to each word embedding before the first layer. Relative schemes leave the embeddings alone and change the attention scores, either with a learned bias for each distance (T5) or by rotating queries and keys (RoPE). A causal mask also leaks some order information, since each token sees a different number of earlier tokens, and some decoder-only models have been trained with no positional encoding at all.

Step 2 of 6: Sinusoidal Encoding

Sin and cos waves encode position

PE(pos,2i) = sin(pos / 100002i/dmodel) = sin(10 / 6.31) = 1.000
Dimension →Position →p10d4dim 4 (sin, i=2)1.00+1−1

Position

10

Dim

4 sin

Angle

1.58

PE value

1.000

Low dimensions oscillate rapidly, so in the left columns the color flips every few rows. They encode fine-grained position differences.

Drag the Position slider from 0 to 49 and the Dimension slider from 0 to 19. The formula is PE(pos, 2i) = sin(pos / 10000^(2i/d)) and PE(pos, 2i + 1) = cos(pos / 10000^(2i/d)), with d = 20 in this heatmap. Every value stays between −1 and 1, so the encoding never swamps the word embedding at large positions. In the original model the embeddings are also multiplied by √d_model before the encoding is added, which keeps the two on a similar scale. Nothing here is learned, so it costs no parameters.

Step 3 of 6: Frequency Bands

Different dimensions, different scales

Positiondim 0

Dimension

0

Frequency

1.0000

Wavelength

6.3

Low dimensions have high frequency: they change rapidly across positions and encode fine-grained differences.

Click dim 0, dim 2, dim 4, and dim 6 and compare the Wavelength readout. With d = 16, the wavelength for dimension 2i is 2π · 10000^(2i/16): about 6.3 positions for dim 0 and 199 for dim 6. Across the full width the wavelengths form a geometric series from 2π to 10,000 · 2π. This works like a binary counter written with smooth waves: fast dimensions tell neighbors apart, and slow ones tell the start of a long document from the end. The base of 10,000 sets the longest wavelength.

Step 4 of 6: Relative Distance

Why sinusoidal encodings work

Position Encoding Vectors (first 16 dims)0123456789101112131415pos 3pos 8dot = 6.14

pos_i

3

pos_j

8

Dot product

6.14

Distance

5

The dot product depends only on the distance: move both sliders by the same amount and it stays at 6.14. It does not fall steadily, though: the fast sin/cos pairs wrap around, so it dips and rises again as the distance grows.

Drag Position i and Position j from 0 to 19 and watch the Dot product card. Each sin and cos pair shares one frequency ω, so PE(i) · PE(j) = Σ cos(ω_k (i − j)), a function of distance only. At i = j every pair contributes 1, giving the maximum of 8 for d = 16. The offset property comes from the angle-addition formulas: sin(ω(p + k)) and cos(ω(p + k)) are a fixed rotation of sin(ωp) and cos(ωp) that depends only on k. Learned projections partly scramble this structure, which helped motivate RoPE.

Step 5 of 6: RoPE

Rotary position embedding via 2D rotation

How RoPE modifies Q and K (showing pair 0)
q=[0.60, 0.40]→R(3·θ₀)→q'=[-0.65, -0.31]
k=[0.60, 0.40]→R(7·θ₀)→k'=[0.19, 0.70]
q rotated by 3 × θ₀ = 3.00 rad  |  k rotated by 7 × θ₀ = 7.00 rad  |  q'·k' depends only on 3−7 = -4
pair 0Δ=-4.0pair 1Δ=-0.4pair 2Δ=-0.0pair 3Δ=-0.0q (pos 3)k (pos 7)angle between = f(3−7) only

q pos

3

k pos

7

q·k

1.18

Offset m−n

-4

RoPE rotates q and k by different amounts (Δθ depends on 3−7 = -4). The dot product depends only on this relative offset, not on absolute position. Move both sliders by the same amount and q·k stays constant. (Here q and k share one content vector, so the sign of the offset happens not to matter; in general it does.)

Drag the Query position and Key position sliders. RoPE splits each query and key into d/2 pairs and rotates pair i by the angle pos · θ_i, with θ_i = 10000^(−2i/d); here d = 8, so there are 4 pairs. Rotating q by angle a and k by angle b leaves q · k depending only on a − b, so the score sees m − n. It does not extend well past the trained length by default, so methods such as position interpolation and YaRN rescale the angles to stretch the context.

Step 6 of 6: Sinusoidal vs RoPE

Comparing the two approaches

Similarity to position 2 as position n variesposition nsimilaritym=2SinusoidalRoPE
SinusoidalRoPE
Where appliedAdded to inputRotates Q, K
WhenBefore projectionAfter projection
Position typeAbsoluteRelative
Parameters00
ExtrapolationLimitedLimited; extended by PI/YaRN
Used byOriginal TransformerLLaMA, Mistral, Gemma

|m−n|

6

Sin dot

3.78

RoPE dot

1.97

Neither curve falls steadily: at distance 6 the similarity is 3.78 (sinusoidal) and 1.97 (RoPE), because the fast pairs wrap around. RoPE's advantage does not show as a different curve here: it builds relative position directly into Q·K and adds no parameters. LLaMA, Mistral, and Gemma all use it.

Drag Position m (query) and Position n (key) and compare the Sin dot and RoPE dot cards. The curves look alike here because both use the same frequencies. The difference is where position enters. Sinusoidal codes are added once at the bottom, and after the first learned layer they mix with content. RoPE acts inside every attention score in every layer. Another relative method is ALiBi (Press et al., 2021), which subtracts a penalty proportional to distance from each score and handles inputs longer than those seen in training.

What is positional encoding in Transformers?

Positional encoding is the information a Transformer adds so that attention can tell where each token sits in the sequence. Attention compares tokens only by their content, so without it "dog bites man" and "man bites dog" look the same. The original Transformer adds a fixed pattern of sine and cosine waves to each token embedding; many newer models instead rotate the queries and keys by a position-dependent angle (RoPE).

Every Transformer needs some form of position information, whether it reads text, image patches, audio frames, or code. The choice affects how well a model handles long inputs: learned position tables stop at a fixed length, while sinusoidal codes, RoPE, and distance-based biases such as ALiBi can be applied at any position. Extending the context window of a large language model is largely a question of how its positional encoding behaves past the training length.

Key ideas

  • Attention compares content only, so position has to be supplied separately.
  • Sinusoidal encoding uses sin and cos waves whose wavelengths grow geometrically across dimensions.
  • The dot product of two sinusoidal encodings depends only on the distance between positions.
  • RoPE rotates each pair of query and key dimensions by an angle proportional to position.
  • With RoPE, the attention score depends on relative position m − n.
  • Behavior beyond the training length differs by method and limits context extension.

Short takes on this topic

Common questions

What is the difference between absolute and relative positional encoding?

Absolute encoding gives each position its own vector, added to the token embedding, so the model learns what position 7 means. Relative encoding changes how two tokens interact based on the distance between them, so the model learns what "3 tokens earlier" means. Relative methods usually generalize better across positions because the same distance looks the same anywhere in the sequence.

Is positional encoding learned or fixed?

Both are used. The original Transformer and RoPE use fixed formulas with no parameters. BERT and GPT-2 learn a table with one vector per position, which works well but cannot go past the table size, 512 positions for BERT and 1,024 for GPT-2. The original paper found learned and sinusoidal versions performed about the same.

Can a model read sequences longer than it was trained on?

It depends on the encoding. Learned tables have no entry for new positions. Sinusoidal codes and RoPE produce values at any position, but models usually degrade beyond the training length because they never saw those angle combinations. ALiBi and RoPE with interpolation or YaRN extend further, often after a short round of fine-tuning on longer text.

How do vision transformers encode position?

A vision transformer cuts an image into patches, such as a 14 × 14 grid, and the original ViT adds a learned vector for each patch index. Variants use 2D sinusoidal codes or 2D RoPE that encode row and column separately. When the image resolution changes, the learned position vectors are interpolated to the new grid size.

Related explainers

References

  1. Vaswani et al. (2017), "Attention Is All You Need", NeurIPS
  2. Su et al. (2021), "RoFormer: Enhanced Transformer with Rotary Position Embedding"
  3. Shaw, Uszkoreit & Vaswani (2018), "Self-Attention with Relative Position Representations"
  4. Press, Smith & Lewis (2021), "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation"
  5. Chen et al. (2023), "Extending Context Window of Large Language Models via Positional Interpolation"

Last updated