1Tokenization & BPE 2Bag of Words 3TF-IDF 4Word2Vec & Word Embeddings 5CBOW (Continuous Bag of Words)6Skip-gram 7Embeddings

Lensa ML

How Embeddings Work

Step 1 of 6: Words as Numbers?

One-hot IDs say every word is equally unrelated

Click a row to see that word's vector.

Numbers per word

5 (= vocab)

cos(cat, kitten)

0.00

cos(cat, car)

0.00

A one-hot ID gives each word its own slot: "cat" is a 1 in slot 0 and 0 everywhere else. Every pair of different words has similarity exactly 0, so "cat" is as unlike "kitten" as it is unlike "truck". The vector also grows with the vocabulary: 50,000 words means 50,000 numbers, almost all zero. Switch to learned vectors.

Switch between one-hot IDs and learned vectors and compare the two tables. With one-hot vectors every pair of different words has cosine 0 and Euclidean distance √2, so the encoding carries no notion of similarity, and its length equals the vocabulary size. The learned vectors here have 8 numbers each and come from the word2vec model trained in step 3. Real models use a few hundred to a few thousand dimensions.

Step 2 of 6: The Embedding Layer

A learned table, one row per word

Click a word to look up its row.

Vocabulary50k

Dimensions768

Table shape

(50k, 768)

Learned numbers

38.4M

Memory (fp32)

153.6 MB

An embedding layer is just a table E with one row per word. Multiplying a one-hot vector by E picks out one row, so frameworks skip the multiply and index the row directly: E[1] for "kitten". The rows start random and are learned like any other weights. With 50k words and 768 dimensions the table holds 38.4M numbers.

Click words to look up their rows, then change the vocabulary and dimension sliders. An embedding layer is a matrix with one row per token, and multiplying by a one-hot vector selects a row, so frameworks implement it as indexing (nn.Embedding in PyTorch). The rows are ordinary weights updated by backpropagation. A 50k × 768 table already holds 38.4 million numbers, and many language models reuse the same table to turn their final hidden state back into token scores, a trick called weight tying.

Step 3 of 6: Learning from Context

Watch word2vec train on a tiny corpus

corpus: cat purrs sofa · cat chases mouse · cat meows · kitten purrs sofa · kitten chases mouse · kitten meows small · … (39 sentences)

Epoch0

Epoch

Loss

—

cos(cat, kitten)

-0.43

cos(cat, car)

-0.14

Every word starts as 8 random numbers, so the dots are a meaningless scatter (the view zooms to fit each epoch). Training reads each pair of nearby words, like (cat, purrs), and nudges their vectors to score high together. For each real pair it also picks a few random words, like (cat, engine), and pushes those to score low. Press Train.

Press Train or step through the epochs. This is skip-gram with negative sampling (Mikolov et al., 2013): for each word and a neighbour within two positions, training raises the dot product of their vectors, and for a few randomly drawn words it lowers it. The corpus has 39 short sentences, and the 8-dimensional vectors train for 300 epochs right in your browser. The 2-D view is a fixed PCA projection of the final vectors, zoomed to fit each epoch. CBOW is the mirror image: it predicts the middle word from the average of its neighbours.

Step 4 of 6: Similarity Is an Angle

Rank words by cosine similarity

Click any word.

Closest

kitten 0.86

Farthest

queen -0.14

Same group in top 3

2 / 3

Cosine similarity measures the angle between two vectors: 1 means they point the same way, 0 means unrelated, negative means opposite. It ignores length, which mostly tracks how often a word appears rather than what it means. "cat" is closest to "kitten" (0.86). Search engines, recommendation systems and RAG all run exactly this: embed the query, then rank everything by cosine. The ranking uses all 8 numbers; the plot above is only a 2-D shadow of them, so dots that look close aren't always the most similar. A corpus this small is noisy: "man" (0.49) sneaks above "puppy" (0.48). More text fixes that.

Click words to rank the others by cosine similarity. Cosine similarity is the dot product of two vectors divided by their lengths, so it compares directions and ignores magnitude. Vector databases store millions of embeddings and use approximate nearest-neighbour indexes such as HNSW to find the top matches quickly. Neighbours in a 2-D picture can disagree with the real ranking, because the projection throws away most of the dimensions.

Step 5 of 6: Directions Carry Meaning

Vector arithmetic, and when it fails

king − man + woman ≈ queen (0.98)

#1 nearest

queen 0.98

#2 nearest

car 0.38

#3 nearest

truck 0.38

The arrow from "man" to "king" is a direction in the space. Adding the same arrow to "woman" lands next to "queen" with cosine 0.98, computed from the 8-D vectors (the inputs are excluded from the answer; in this 2-D shadow the gap can look bigger). The model learned a consistent offset because the corpus uses these words in parallel ways, for example "he king", "she queen". Real models learn many such directions, though they are noisier than this.

Pick an analogy and compare the two arrows. The result b − a + c is computed in 8 dimensions and compared with every other noun by cosine, excluding the three input words, as analogy benchmarks do. Analogies only hold when the corpus uses the words in parallel ways, and the last button shows one that does not. Directions in embedding spaces can also encode unwanted associations, such as gender stereotypes learned from text, which is why embeddings are audited for bias.

Step 6 of 6: Embeddings Beyond Words

Users and movies in one learned space

Recommend to Ana

John Wick

Predicted rating

4.3★

Error on known ratings

0.40★

The same trick works for anything that has "context". Each user and each movie gets 2 learned numbers, trained so that the average rating plus user · movie (a dot product) matches the 22 known stars. Nobody labelled the movies, yet action films end up on one side and romance on the other, and users sit near what they like. For Ana, the best unseen movie is John Wick at 4.3★. Streaming services, shops and search all run this idea at a much larger scale.

Click each user to see their predicted ratings. This is matrix factorization, the method popularized by the Netflix Prize: each user and each movie gets a learned vector, and a rating is predicted as the overall average plus their dot product. Here the vectors have 2 dimensions, fit to 22 known ratings by stochastic gradient descent with a small L2 penalty. The same idea, learning vectors so that related things score high together, gives sentence embeddings, image embeddings such as CLIP, and graph node embeddings.

What are embeddings in machine learning?

An embedding is a learned vector of numbers that represents a discrete item, such as a word, a product, or a user, so that similar items get similar vectors. Instead of a sparse one-hot code with one slot per item, each item gets a short dense vector, usually with a few hundred dimensions. Distances and directions between these vectors carry meaning that the model learned from data.

Embeddings are the input layer of almost every language model, where each token is looked up as a vector before any other processing. They also power semantic search and retrieval-augmented generation, recommendation systems that match users to products, and image search that compares photos with text. Word2vec (2013) made word embeddings widely used, and the same idea now applies to sentences, images, code, and graph nodes.

Key ideas

An embedding maps each discrete item, such as a word, to a short dense vector of learned numbers.
One-hot vectors are sparse and treat every pair of words as equally unrelated.
An embedding layer is a weight matrix, and looking up a row equals multiplying by a one-hot vector.
Words used in similar contexts get similar vectors, so cosine similarity tracks relatedness.
Directions in the space can encode relationships, as in king − man + woman ≈ queen.
The same idea embeds products, users, graph nodes, sentences, and images.

Common questions

What is the difference between word2vec and transformer embeddings?

Word2vec gives each word one fixed vector, so "bank" has the same embedding in "river bank" and "bank account". A Transformer starts from a fixed token embedding but then mixes in the surrounding words, so its output vectors are contextual. Models such as BERT give each occurrence of a word its own vector.

How many dimensions should an embedding have?

Classic word vectors use 100 to 300 dimensions. Language models use 768 to several thousand, and sentence embedding models commonly output 384 to 3,072. More dimensions can store finer distinctions but cost memory and need more data to train well.

Should embeddings be trained from scratch or pretrained?

With little data, starting from pretrained vectors such as GloVe or a pretrained language model usually works better. The embeddings can then be frozen or fine-tuned with the rest of the network. Training from scratch makes sense for items with no pretrained vectors, such as product IDs in a recommender.

What is an embedding model?

An embedding model is a network trained to turn a whole input, such as a sentence or an image, into one vector for comparison. These vectors are stored in a vector database for semantic search and retrieval-augmented generation. They are usually trained with contrastive learning, which pulls matching pairs together and pushes unrelated pairs apart.

Related explainers

Word2Vec & Word Embeddings — Learn dense vector representations where similar words live near each other in space
CBOW (Continuous Bag of Words) — Predict the center word from its context — the averaging architecture behind Word2Vec
Skip-gram — Predict context words from a center word — the architecture that powers Word2Vec with negative sampling
Norms & Distance Metrics — L1, L2, cosine similarity — measuring closeness in vector spaces
Retrieval-Augmented Generation — Ground language model outputs in retrieved documents to reduce hallucination
Tokenization & BPE — How text becomes numbers — from characters to subword tokens via byte-pair encoding

References

Last updated November 4, 2026