ENID
Lensa ML

Word2Vec & Embeddings

Step 1 of 6: Distributional Hypothesis

A word is known by the company it keeps

Context window size2
"You shall know a word by the company it keeps" — Firth, 1957thekingsatonthegoldenthroneandthequeenwatchedwindow = ±2Context words for "king" (window=2):thesatonWords appearing in similar contexts get similar embeddings.
Center
king
Window
±2
Context
3 words

Window ±2: 'the', 'sat', and 'on' are context words. Across a large corpus 'queen' shows up near the same kinds of words as 'king', so their embeddings end up close.

Drag Context window size from 1 to 4 and watch the context set around "king" grow. The quote at the top states the distributional hypothesis, put forward by Zellig Harris in 1954 and popularized by J. R. Firth in 1957. Word2vec turns it into a training signal. Every (center, context) pair inside the window is one training example, so a ±5 window over a billion-word corpus yields about 10 billion examples. The default window in the original tool is 5, and it samples a smaller window for each word so that close neighbors count more often.

Step 2 of 6: Skip-Gram Architecture

Predict context words from a center word

kingInput (center)thronecrownroyalqueen→ Predict context wordsW_embed (king) → hidden → W_context → P(context|king)
Center
king
Contexts
4
Objective
Skip-Gram

Skip-Gram: given "king", predict its context words [throne, crown, royal, queen]. Each training update pulls the input vector of "king" (its row of W_embed) toward the output vector of "throne" (its row of W_context), so words that share contexts end up with similar W_embed rows. After training, the rows of W_embed are the word vectors.

Click king, queen, dog, and car to see each center word's context targets. The network has no hidden nonlinearity. The hidden layer is just the center word's row of W_embed, and a second matrix scores every vocabulary word as a possible neighbor. Word2vec ships two architectures. Skip-gram, shown here, predicts the context from the center word, and CBOW does the reverse, predicting the center from its averaged context. Both are shallow on purpose: the 2013 paper argued that a simple model trained on far more text beats a deeper one trained on less.

Step 3 of 6: Training Objective

Watch embeddings cluster as training progresses

Training step0
kingqueenmanwomanprinceprincessdogcatpuppykittencartruck2D embedding spaceroyaltygenderanimalvehicle
Step
0
Progress
0%

Before training: words sit at random positions. There is no structure yet, so 'king' is no closer to 'queen' than to 'dog'. (This animation is illustrative: it slides each word in a straight line to a hand-placed final position.)

Drag Training step from 0 to 20 to watch random starting positions settle into clusters. The demo interpolates toward the final layout; real training moves the vectors with stochastic gradient descent, one word pair at a time. A full softmax over a 100,000-word vocabulary is too slow for every update, so the follow-up paper introduced negative sampling. For each true pair, the model also scores a few random "negative" words (5 to 20 on small datasets) and learns to separate them with a sigmoid. Frequent words like "the" are randomly skipped, which speeds training.

Step 4 of 6: Vector Arithmetic

king - man + woman ≈ queen

princeprincessdogcatpuppykittencartruckkingmanwomanqueenking−man+woman
king
(2, 2.5)
− man
(2.2, 0.8)
+ woman
(1.2, 0.6)
≈ Result
queen

king−man+woman: the vector from "man" to "king" (pink dashed) captures a relationship. Adding the same offset to "woman" (teal dashed) lands near "queen". The relationship is stored as a direction in the space.

Click king−man+woman, prince−man+woman, and puppy−dog+cat to compare three analogies. In this 2D demo the result lands exactly on the target. In real 300-dimensional vectors it lands near it, and the answer is the closest word after excluding the three input words. Without that exclusion, the nearest vector to king − man + woman is often king itself. The original analogy test set has about 19,500 questions, and even the best model in the first paper got about a third of them wrong.

Step 5 of 6: Cosine Similarity

Measuring word relatedness by the angle between vectors

Query vector angle (°)30
1woman2prince3manprincesskingqueencartruckpuppycatkittendog1woman1.002prince0.993man0.98query
#1 nearest
woman
Sim
0.998
#2
prince
#3
man

At 30°, the nearest neighbor is "woman" (cosine sim = 0.998). Only direction matters: a word far from the origin scores the same as a near one on the same ray.

Drag Query vector angle (°) from 0 to 360 and watch the top three neighbors change. Word vectors are compared with cosine, cos θ = a · b / (|a| |b|), rather than Euclidean distance, partly because a word2vec vector's length tracks how frequent the word is. In 300 dimensions, unrelated words score near 0 and close synonyms often above 0.6. Antonyms such as hot and cold also score high, because they appear in the same contexts. Distributional similarity measures relatedness, not sameness of meaning.

Step 6 of 6: Beyond Word2Vec

From static vectors to contextual embeddings

Context mixing (α)0.00
"bank" (both senses)Word2Vec2013 · 300dGloVe2014 · 300dELMo2018 · 1024dBERT2018 · 768dGPT-22019 · 1600dStatic embeddings: one vector per word
Context mixing
0.00
Type
Static
Distance
0.00
cos(bank, bank)
1.00

Static embeddings (Word2Vec, GloVe): each word gets one vector regardless of context. 'Bank' (river) and 'bank' (money) share the same point, so their cosine similarity is 1.

Drag Context mixing from 0 to 1 and watch the two "bank" points split apart as their cosine similarity falls from 1.00 to 0.17. The slider blends the fixed "bank" vector with the average of its neighbors, a simplified stand-in for what a contextual model does. The difference is architectural rather than size: ELMo and BERT compute each vector from the whole sentence with a deep network, while word2vec looks a word up in a fixed table. Static vectors still have uses. fastText extends word2vec with character n-grams so it can build vectors for unseen words, and static embeddings remain a cheap option for query expansion and for models on low-power devices.

What is word2vec?

Word2vec is a family of shallow neural network models that learn a dense vector, or embedding, for every word in a vocabulary from raw text. It trains on a simple prediction task: guess a word from its neighbors, or its neighbors from the word. After training, words used in similar contexts end up with similar vectors.

Word2vec was released by Tomas Mikolov and colleagues at Google in 2013, with two architectures, CBOW and skip-gram, and a set of training tricks that made it fast on billions of words. Its vectors became standard inputs for text classifiers, named entity recognizers, and search systems, and the famous king − man + woman ≈ queen result showed that the vectors encode relationships. The same idea, learning embeddings from context, is the first layer of every modern language model.

Key ideas

  • Word2vec learns one dense vector per word from the words around it.
  • Words that appear in similar contexts get vectors that point in similar directions.
  • Skip-gram predicts the context from a word; CBOW predicts a word from its context.
  • Negative sampling replaces the expensive full softmax with a few binary decisions per pair.
  • Vector offsets encode relationships, so king − man + woman lands near queen.
  • Word2vec vectors are static: each word has one vector whatever sentence it appears in.

Common questions

What is the difference between word2vec and GloVe?

Word2vec learns from local windows, updating vectors one (word, context) pair at a time. GloVe, from Stanford in 2014, first counts how often every pair of words co-occurs in the whole corpus and then fits vectors whose dot products match the log of those counts. The resulting vectors behave similarly, and which one wins depends on the task and the training data.

How many dimensions do word2vec vectors have?

Usually 100 to 300. The widely used Google News vectors released with the paper have 300 dimensions for 3 million words and phrases. Larger sizes help only when there is enough training text to fill them.

Is word2vec deep learning?

Not in the usual sense. It is a two-layer linear model: an embedding lookup followed by an output layer, with no hidden nonlinearity. Its contribution was to show that a very simple network trained on huge amounts of text learns useful representations.

Do word embeddings contain bias?

Yes. Embeddings learn whatever associations their training text contains, including stereotypes. Bolukbasi et al. (2016) found that Google News vectors complete "man is to computer programmer as woman is to" with "homemaker", and proposed methods to reduce such gender directions.

Related explainers

  • Embeddings — Turning words and categories into vectors
  • CBOW (Continuous Bag of Words) — Predict the center word from its context — the averaging architecture behind Word2Vec
  • Skip-gram — Predict context words from a center word — the architecture that powers Word2Vec with negative sampling
  • Norms & Distance Metrics — L1, L2, cosine similarity — measuring closeness in vector spaces
  • BERT & Masked Language Modeling — Mask a word, predict it from both sides — how BERT learns deep bidirectional representations
  • Named Entity Recognition — Identify and classify named entities — people, places, organizations — in running text

References

  1. Mikolov et al. (2013), "Efficient Estimation of Word Representations in Vector Space"
  2. Mikolov et al. (2013), "Distributed Representations of Words and Phrases and their Compositionality", NeurIPS
  3. Pennington, Socher & Manning (2014), "GloVe: Global Vectors for Word Representation", EMNLP
  4. Bolukbasi et al. (2016), "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings"
  5. Jurafsky & Martin, "Speech and Language Processing" (3rd ed. draft)

Last updated