How a transformer knows word order, in plain words

A transformer, the model design behind large language models, reads all the words of a sentence at once. Its attention step, where each word gathers context from the others, compares words only by meaning, so it can't tell "dog bites man" from "man bites dog". Positional encoding fixes this by adding a position tag to each word.

1. 1
   Each word enters as an embedding, a list of numbers that stands for it, and its tag is a list of the same length added to it. In the heatmap each row is the tag for one position, 0 to 49 from the top, and each column is one dimension, a slot in the list. Red means +1, blue means −1 and pale gray means 0.
2. 2
   Each column follows one wave: PE(pos, 2i) = sin(pos / 10000^(2i/d)) for the even columns, with cos in place of sin for column 2i + 1. Here pos is the position, 2i the column number, d = 20 the number of dimensions, and ^ means "to the power of". The curve below the heatmap is column 4, sin(pos / 6.31), which peaks at 1.000 at position 10. Columns 0 and 1 repeat every 6.3 positions, so their colors flip about every three rows. Each pair of columns to the right is about 2.5 times slower.
3. 3
   A row's colors are that position's fingerprint. One fast wave is not enough: in columns 0 and 1, positions 2 and 46 read 0.91 and −0.42 against 0.90 and −0.43. Column 2 tells them apart, 0.71 against −0.51, like an hour hand beside a minute hand. The slowest waves span thousands of positions, so over these 50 rows sine stays near 0 (pale gray) and cosine near 1 (red), which draws the wide stripes on the right. In long documents they separate distant positions.

More bites

In 1 picture

Self-attention in 1 picture: every word looks at every word

2 min read