1Tokenization & BPE 2Bag of Words 3TF-IDF 4Word2Vec & Word Embeddings 5CBOW (Continuous Bag of Words)6Skip-gram 7Embeddings

Lensa ML

How Tokenization Works

Step 1 of 6: Why Tokenize?

Words fail on new words, bytes are too long

word lookup (12-word vocabulary)

[UNK]newestlowestwidest[UNK]

raw bytes: 30 tokens

BPE (26 learned merges + byte fallback)

t<0x68>e</w>newest</w>lowest</w>widest</w><0x6D>odel</w>

Word tokens · unknown

5 · 2

Byte tokens

BPE tokens

A model can only take numbers, so text has to be cut into pieces from a fixed list, each with an ID. Whole words are short sequences, but any word not in the list becomes [UNK]: 2 of your 5 words are lost. Raw bytes never fail, since every text is bytes, but the sequence is 30 long and each byte means little. BPE sits in between: frequent words like "newest" become one token, rarer ones are built from pieces, and anything else falls back to bytes, so nothing is ever unknown. Type your own text.

Type your own text and compare the three encodings. A model's input layer is a lookup table indexed by integers, so text must first become a sequence of IDs from a fixed list. Word-level vocabularies need hundreds of thousands of entries and still miss names, typos, and new words. Character or byte vocabularies cover everything with a few hundred entries, but make sequences several times longer. Subword methods such as BPE, WordPiece, and Unigram are the compromise every modern language model uses.

Step 2 of 6: Vocabulary vs Length

Every merge trades a bigger vocabulary for shorter text

Merges10

held-out text "the wildest oldest lower widen"

t<0x68>e</w>wildest</w>oldest</w>lower</w>widen</w>

Vocabulary size

Corpus tokens

Held-out tokens

Each merge adds one entry to the vocabulary and shortens the text. With 10 merges the vocabulary has 21 entries and the training corpus takes 95 tokens (from 204 as single characters). Shorter sequences are cheaper for a Transformer, whose attention cost grows with the square of the length, but every vocabulary entry costs an embedding row and the rare ones get few training examples. Real tokenizers stop at 30k–200k entries.

Drag the merges and watch both the corpus and the held-out text shrink. Vocabulary size is a hyperparameter: GPT-2 uses about 50,000 entries, Llama 2 32,000, and newer multilingual models 100,000 to 250,000. A larger vocabulary shortens sequences and so cuts attention cost, but each entry needs an embedding row and the rare ones are seen in few training examples. Tokenizers are usually trained on a sample of the model's training data, so they compress that kind of text best.

Step 3 of 6: Byte-Pair Encoding Merges

Train it: merge the most frequent pair, repeat

corpus: word × count, split into current tokens

×5

low</w>

×2

lower</w>

×3

lowest</w>

×4

new</w>

×2

newer</w>

×6

newest</w>

×2

wide</w>

×2

wider</w>

×3

widest</w>

×3

old</w>

×2

older</w>

×2

oldest</w>

most frequent adjacent pairs right now

e + s = 14s + t = 14t + </w> = 14w + e = 13e + w = 12

Merges so far

0 / 26

Last merge

—

Next merge

e+s (14)

Vocabulary

Training starts with every word split into characters plus an end-of-word marker </w>. Count every adjacent pair, weighted by how often the word occurs, and merge the most frequent pair everywhere, adding it to the vocabulary. The winner is e + s with 14 occurrences (the amber boxes). Press merge. The ordered list of merges is the tokenizer.

Step through the merges and watch the amber pair. Byte-pair encoding began as a data-compression algorithm (Gage, 1994) and was adapted for machine translation by Sennrich et al. (2016), whose toy corpus of low, lower, newest, and widest this demo follows. Each step counts adjacent pairs weighted by word frequency and merges the most common one; ties are broken by a fixed rule, here alphabetical, so training is deterministic.

Step 4 of 6: Tokenizing New Text

Replay the merges, in order, on a word never seen

start

wildest</w>

merge #1: e+s

wildest</w>

merge #2: es+t

wildest</w>

merge #3: est+</w>

wildest</w>

merge #11: l+d

wildest</w>

Seen in training?

Merges that applied

4 / 26

Tokens

To encode "wildest", split it into characters and replay the learned merges in the same order they were learned, skipping any that don't occur. Only 4 of the 26 merges apply here, and the result is [w, i, ld, est</w>]. "wildest" never appeared in training, yet it is encoded from pieces the tokenizer learned elsewhere, such as the ending "est</w>". Because the order is fixed, the same text always gives the same tokens.

Pick a word and replay the merges. Encoding never looks at frequencies again: it applies the stored merge list in order, which is why a tokenizer is shipped as two files, the vocabulary and the merges. Different but similar words share pieces, so "wildest" reuses the "est" ending learned from newest, lowest, and widest. Real tokenizers also split text on spaces and punctuation first, and GPT-style ones attach the space to the following word, as in "Ġmodel".

Step 5 of 6: Unknown Characters

Why modern tokenizers never output [UNK]

base alphabet learned from the corpus: d e i l n o r s t w </w>

l[UNK][UNK][UNK]</w>

Characters missing from the alphabet

a z y

[UNK]

Byte tokens

The base alphabet only has the characters seen in training. "lazy" contains a, z, y, which no merge can produce, so each becomes [UNK] and the model loses that information for good. A tokenizer that does this can't even spell new names correctly. Switch on byte fallback.

Pick a word, then switch on byte fallback. Any character missing from the base alphabet would otherwise become [UNK], and the original text could not be recovered. SentencePiece-based tokenizers, used by Llama, keep 256 byte tokens as a fallback for this case, while GPT-2's byte-level BPE starts from bytes so every string is covered by construction. The cost is that text in scripts that are rare in the training data, or with many accented letters, takes more tokens per word.

Step 6 of 6: Tokens to IDs

Row numbers in a table of vectors

"newest oldest wider" → tokens → IDs (row numbers in the vocabulary)

newest</w>24oldest</w>34wider</w>36

Token

newest</w>

Its vector

[-0.5, 0.8, -0.8, -0.1]

Each token's ID is just its row number in the learned vocabulary: "newest</w>" is 24. The model's first layer looks that row up in an embedding table and works only with those numbers from then on. The table starts random and is learned in training; real models use rows of 768 to 16,384 numbers. This is why models stumble at spelling: "newest" here is one token, ID 24, so the model never sees its letters and can't directly count its 2 e's. That's the same reason LLMs miscount the r's in "strawberry".

Click each token to follow it to its row. The ID is the token's position in the vocabulary file, and the embedding matrix has one learned row per ID, often thousands of numbers wide. Because the model only sees IDs, it has no direct view of letters, which explains errors in spelling, rhyming, reversing words, and counting characters. Prices and context limits for language models are counted in these tokens, roughly 0.75 English words per token.

What is tokenization and byte-pair encoding (BPE)?

Tokenization is the step that splits raw text into units, called tokens, and maps each one to an integer ID a model can use. Byte-pair encoding (BPE) is the most common way to build the token list: it starts from single characters and repeatedly merges the most frequent adjacent pair until the vocabulary reaches a chosen size. The result is a vocabulary of subwords, pieces between a letter and a whole word.

Every text model starts with a tokenizer. GPT-2, GPT-4, Llama, and RoBERTa use BPE variants, and BERT uses the closely related WordPiece. The tokenizer decides how long each input is, which sets compute cost and how much text fits in a context window, and it decides which words a model can represent at all. API pricing for language models is also counted in tokens, not words.

Key ideas

A tokenizer turns text into a sequence of integer IDs from a fixed vocabulary.
Character vocabularies are tiny but make long sequences; word vocabularies are short but cannot cover rare words.
BPE builds a subword vocabulary by repeatedly merging the most frequent adjacent pair.
Every merge adds one vocabulary entry and shortens the encoded corpus.
New text is encoded by replaying the learned merges in training order, so the output is deterministic.
Each token ID indexes one row of an embedding matrix, and the model works on those vectors.

Common questions

How many tokens is a word?

It depends on the tokenizer and the language. For English text with GPT-style tokenizers, a common rule of thumb is about 4 characters or 0.75 words per token, so 100 tokens is roughly 75 words. Common words are usually one token, while rare words, names, and numbers split into several.

What is the difference between BPE, WordPiece, and SentencePiece?

BPE merges the pair with the highest raw count. WordPiece, used by BERT, picks the merge that most increases the likelihood of the training data, which favors pairs that are frequent relative to their parts. SentencePiece is a library rather than an algorithm: it treats the space as an ordinary symbol and can train either BPE or a unigram language-model tokenizer directly on raw text.

Why do language models struggle to count letters?

The model sees token IDs, not characters. A word like "strawberry" may arrive as two or three subword tokens, so the individual letters are never directly visible. The model has to have memorized the spelling of each token to answer letter-level questions.

Do tokenizers treat all languages equally?

No. A tokenizer trained mostly on English learns long merges for English words and fewer for other languages, so the same sentence in Thai, Hindi, or Amharic can take several times as many tokens. That makes those languages slower and more expensive to process and leaves less room in the context window.

Related explainers

Embeddings — Turning words and categories into vectors
Language Modeling — Predict the next token — the objective that powers GPT, BERT, and modern NLP
Bag of Words — Turn text into numbers — count word occurrences to create the simplest document representation in NLP
Word2Vec & Word Embeddings — Learn dense vector representations where similar words live near each other in space
The Transformer Architecture — A visual guide to 'Attention Is All You Need'

References

Last updated November 4, 2026