Tokenization & BPE

No merges yet — pure character-level tokenization. Every letter is its own token. The vocabulary is tiny (just 26 letters), but the sequence is very long.

Subword level: BPE discovers meaningful pieces — 'un' (negation), 'believ' (root), 'able' (suffix). Even rare words like 'xylophonist' can be split into known parts. The sweet spot!

Starting from individual characters (plus end-of-word marker </w>). BPE counts every adjacent pair across the corpus — the most frequent pair will be merged first.

5 merges: vocab = 16, total tokens = 36. The curves show the fundamental BPE tradeoff — more merges = larger vocabulary but shorter sequences. Real models use 30k–100k merges.

Merges are applied in the order they were learned during training — rank 1 was the most frequent pair in the corpus, rank 2 the next, and so on. At inference, BPE scans this fixed list top-to-bottom: #1 (w+e→we), #2 (l+o→lo), #4 (s+t→st), #5 (st+</w>→st</w>). Only rules whose pair exists in the current tokens actually fire. This fixed priority is what makes BPE deterministic and fast.

Token "The" → ID 101 → embedding vector. Each token ID maps to a row in the embedding matrix (typically 768 or 1024 dimensions). This dense vector is what the transformer actually processes — not the text itself.