How Embeddings Work
Words as Numbers?
How do you turn 'cat' into something a network can read?
Computers need numbers, not words. A one-hot vector puts a 1 at the word's index and 0 everywhere else. But the vector is huge (vocab size!) and treats all words as equally unrelated.
Step 1 of 6: Words as Numbers?
How do you turn 'cat' into something a network can read?
Computers need numbers, not words. A one-hot vector puts a 1 at the word's index and 0 everywhere else. But the vector is huge (vocab size!) and treats all words as equally unrelated.
Step 2 of 6: The Embedding Layer
Learning a dense vector for each word
The embedding layer is a simple lookup table: each word maps to a short, dense vector of learned numbers. This table is a weight matrix — it's trained with the rest of the network. Click different words to see their embeddings.
Step 3 of 6: Similarity in Embedding Space
Similar words live close together
In embedding space, similar words cluster together. "cat" and "kitten" are nearby; "car" and "truck" form their own cluster. The network learned these relationships from data — no one hand-coded them.
Step 4 of 6: Word2Vec — CBOW
Predict the center word from its context
Word2Vec CBOW predicts the missing center word from its context words. By learning to predict well, the embedding table captures meaning — words used in similar contexts get similar vectors.
Step 5 of 6: Vector Arithmetic
king - man + woman ≈ queen
The famous word analogy test: directions in embedding space encode relationships. The vector from "man" to "king" captures the concept of royalty — apply it to "woman" and you land near "queen."
Step 6 of 6: Embeddings Everywhere
Not just words — items, users, graphs, and more
Embeddings aren't just for words. Products, users, and graph nodes can all be embedded. The core principle is always the same: learn dense vectors from context, so similar things end up close together.