Word2Vec & Embeddings
Distributional Hypothesis
A word is known by the company it keeps
Window ±2: 'the', 'sat', 'on' are context words. Since 'queen' also appears near 'throne' and 'sat', it will develop a similar embedding to 'king'.
Step 1 of 6: Distributional Hypothesis
A word is known by the company it keeps
Window ±2: 'the', 'sat', 'on' are context words. Since 'queen' also appears near 'throne' and 'sat', it will develop a similar embedding to 'king'.
Step 2 of 6: Skip-Gram Architecture
Predict context words from a center word
Skip-Gram: given "king", predict its context words [throne, crown, royal, queen]. The model learns that "king" and "throne" often co-occur. The embedding matrix W_embed is the learned representation — that's the word2vec output!
Step 3 of 6: Training Objective
Watch embeddings cluster as training progresses
Before training: words are randomly positioned in the embedding space. No structure — 'king' and 'queen' are just as far apart as 'king' and 'dog'.
Step 4 of 6: Vector Arithmetic
king - man + woman ≈ queen
king−man+woman: the vector from "man" to "king" (pink dashed) captures a relationship. Adding that same offset to "woman" (teal dashed) lands near "queen". This works because Word2Vec encodes semantic relationships as geometric directions!
Step 5 of 6: Cosine Similarity
Measuring word relatedness by the angle between vectors
Rotating the query vector to 30° — nearest neighbor is "woman" (cosine sim = 0.998). Cosine similarity measures the angle between vectors, ignoring magnitude. Words in the same direction as the query are most similar.
Step 6 of 6: Beyond Word2Vec
From static vectors to contextual embeddings
Static embeddings (Word2Vec, GloVe): each word gets ONE vector regardless of context. 'Bank' (river) and 'bank' (money) share the same embedding — a fundamental limitation.