ENID
Lensa ML

Skip-gram

Step 1 of 6: Center → Context

Predicting the neighbors

Center word position2
Skip-gram: predict context from center (window ±2)thecatsatonthematTraining pairs generated:sat→thesat→catsat→onsat→thePosition 2 generates 4 pairs. Full sentence of length 6 with window ±2generates ~18 total training pairs.
Center
sat
Pairs
4
Window
±2

"sat" at position 2 produces 4 training pairs. Each pair is a separate example: given "sat", predict that one context word. Nothing is averaged.

Drag Center word position from 0 to 5 to move the center through "the cat sat on the mat". With a ±2 window this six-word sentence yields 18 training pairs: 2, 3, 4, 4, 3, and 2 from left to right. The objective maximizes the average of log P(context | center) over all pairs. Each word has two vectors, an input vector v used when it is the center and an output vector u used when it is a context word. Then P(o | c) = exp(u_o · v_c) / Σ_w exp(u_w · v_c).

Step 2 of 6: The Softmax Bottleneck

Why full softmax is too expensive

Vocabulary size12
P(context | center) = exp(score) / Σ exp(all scores)Denominator: sum over ALL 12 wordstheacatdogsatranonindowmatbedbigComputation cost:144 ops/pairCost = O(V × d) per training pair — V=12, d=6Real Word2Vec: V = 100,000+, d = 300. That's 60,000,000 ops × millions of pairs!→ This is why negative sampling was invented.
Vocab size
12
Ops/pair
144
Bottleneck
Severe

At 12 words, cost is 144 operations per training pair. At V = 100,000 and a real embedding size of d = 300, that becomes 60,000,000 ops per pair, times billions of pairs. The softmax denominator dominates the cost.

Drag Vocabulary size from 3 to 12 and watch the operation count grow in a straight line. The cost sits in the denominator Σ_w exp(u_w · v_c), which touches every vocabulary word. The 2013 papers offered two fixes. Hierarchical softmax arranges the vocabulary as a binary Huffman tree, so frequent words get short paths and a prediction costs about log₂ V sigmoid decisions instead of V. Negative sampling, in the next step, drops the normalized distribution entirely. Language models still compute a full softmax at their output, but on GPUs, as one matrix multiplication.

Step 3 of 6: Negative Sampling

Binary classification instead of softmax

Negative samples (k)5
Negative Sampling: binary classification instead of softmax✓ Positive pair (real context):sat→catσ=0.542want ≈ 1✗ Negative samples (drawn ∝ frequency^0.75): k = 5sat→inσ=0.521want ≈ 0sat→onσ=0.494want ≈ 0sat→theσ=0.567want ≈ 0sat→ranσ=0.490want ≈ 0sat→dogσ=0.523want ≈ 0Objective: log σ(pos) + Σ log σ(−neg) = -4.281Cost: O(k × d) per pair — only 6 dot products instead of 12 (full softmax)
k
5
σ(pos)
0.542
Objective
-4.28
Speedup
2×

k=5: more negatives steady the gradients, with diminishing returns. The speedup is 2× over full softmax. With V = 12 that is modest; at V = 100K it is 100K versus 6 dot products.

Drag Negative samples (k) from 1 to 6 and watch the objective change. For one true pair the loss is −log σ(u_o · v_c) − Σ_n log σ(−u_n · v_c), summed over the k negatives. The paper suggests k = 5 to 20 for small datasets and 2 to 5 for large ones. Real negatives are drawn from word frequencies raised to the 3/4 power, which picks rare words more often than raw frequency would. The demo samples this way too, so frequent words such as "in", "on", and "the" come up first. Levy and Goldberg later showed that this objective implicitly factorizes a shifted PMI matrix of word co-occurrences.

Step 4 of 6: Subsampling Frequent Words

Discarding 'the' to learn better

Subsampling threshold t = 1.0e-3-3.00
P(discard) = 1 − √(t / f(w))higher frequency → more likely discardedthef=0.282 P(drop)=0.94af=0.200 P(drop)=0.93onf=0.141 P(drop)=0.92inf=0.106 P(drop)=0.90catf=0.059 P(drop)=0.87dogf=0.052 P(drop)=0.86satf=0.042 P(drop)=0.85ranf=0.035 P(drop)=0.83matf=0.028 P(drop)=0.81bigf=0.023 P(drop)=0.79downf=0.019 P(drop)=0.77bedf=0.014 P(drop)=0.73Expected: 10% of occurrences kept · all 12 words stay in vocab
Threshold
1.0e-3
Occurrences kept
10%
Words dropped ≥99%
0

t=1.0e-3: "the" and "a" are usually discarded ("the": 94%), so content words like "cat" and "dog" make up a larger share of the training pairs.

Drag the Subsampling threshold t slider from 10⁻⁵ to 10⁻¹ and watch the occurrences of common words thin out, while every word stays in the vocabulary. Each occurrence of word w is discarded with probability 1 − √(t / f(w)), where f(w) is the word's share of the corpus. The paper recommends t around 10⁻⁵ for billion-word corpora. This toy corpus has only 426 tokens, so word shares are huge and even t = 0.1 discards some "the". Dropping frequent words also widens the effective window, since the surviving words move closer together.

Step 5 of 6: Training Dynamics

Watching embeddings cluster

Training step0
Learning rate0.10
Embedding space (illustrative)theacatdogsatranonindownmatbedbigLoss (lr=0.10)functionanimalsverbsprepositionsother
Step
0
Loss
3.20
LR
0.10

Before training: words are scattered at random, with no structure. Training will pull co-occurring words together and push negative samples apart. (This step is a scripted illustration, not a real training run: the positions, loss curve and learning-rate effects are drawn to show typical behaviour.)

Click Play or drag Training step from 0 to 30, then change Learning rate between 0.01 and 0.50 and replay. The curves are a simulation of the qualitative effect. The original tool starts skip-gram at a learning rate of 0.025 (0.05 for CBOW) and decays it linearly to nearly zero, so early updates move vectors far and late ones fine-tune them. Many CPU threads update the shared vectors at once without locks. That works because each update touches only a few rows of the two matrices.

Step 6 of 6: Embedding Quality

Nearest neighbors in the embedding space

Query word2
Nearest neighbors by cosine similarity (illustrative positions)thea (0.35)catdog (1.00)sat (0.09)ran (-0.04)onindownmatbed (-0.10)big
Query
cat
#1 nearest
dog
Sim
0.998
#2
a

"cat" is nearest to "dog" (sim=1.00): both are animals. "cat" and "dog" fill the same slots ("the ___ sat", "a ___ ran"), so their embeddings cluster.

Drag Query word from 0 to 11 to see each word's nearest neighbors. Embeddings are judged in two ways. Intrinsic tests check the geometry directly, with analogy questions or by correlating cosine scores with human similarity ratings such as WordSim-353 and SimLex-999. Extrinsic tests plug the vectors into a real task, such as named entity recognition or sentiment classification, and measure the gain. The two do not always agree, so the target task is the better guide. Skip-gram's main advantage on both is rare words.

What is the skip-gram model?

Skip-gram is the word2vec architecture that predicts the surrounding words from a single center word. Each (center, context) pair inside a sliding window is one training example, and the model adjusts the center word's vector until it scores its real neighbors highly. In practice it is trained with negative sampling, a cheap stand-in for the full softmax.

Skip-gram with negative sampling (SGNS) is the most widely used word2vec setup and produces strong vectors for rare words. The same recipe works on any sequence of tokens: DeepWalk and node2vec apply it to random walks on graphs, and recommender systems apply it to sequences of products or songs to learn item embeddings. Its training tricks, negative sampling and frequent-word subsampling, show up again in later contrastive learning methods.

Key ideas

  • Skip-gram predicts each context word separately from the center word.
  • Every word has an input vector and an output vector; the input vectors are kept as embeddings.
  • A full softmax costs one dot product per vocabulary word for every training pair.
  • Negative sampling turns each prediction into k + 1 binary logistic decisions.
  • Subsampling discards frequent words with probability 1 − √(t / f(w)).
  • Skip-gram is slower than CBOW but learns better vectors for rare words.

Common questions

Why is it called skip-gram?

The name comes from skip-grams in language modeling: n-grams that allow gaps, so "the mat" counts as a pair in "the cat sat on the mat" even with words between them. Pairing a center word with every word in its window, near or far, is the same idea.

What does SGNS mean?

SGNS stands for skip-gram with negative sampling, the combination that most word2vec implementations use by default when skip-gram is selected. The term is common in research papers that compare word2vec with count-based methods.

Can skip-gram learn vectors for phrases?

Yes. The second 2013 paper first finds word pairs that occur together far more often than chance, such as "New York" or "Air Canada", and merges them into single tokens. Skip-gram then learns one vector per phrase, which helps where the phrase means more than its parts.

Is skip-gram used outside of text?

Yes. Any data that comes as sequences of discrete items works. DeepWalk and node2vec generate random walks on a graph and train skip-gram on them to embed nodes, and music and e-commerce recommenders train it on listening or browsing sessions to embed songs and products.

Related explainers

References

  1. Mikolov et al. (2013), "Distributed Representations of Words and Phrases and their Compositionality", NeurIPS
  2. Mikolov et al. (2013), "Efficient Estimation of Word Representations in Vector Space"
  3. Goldberg & Levy (2014), "word2vec Explained: Deriving Mikolov et al.'s Negative-Sampling Word-Embedding Method"
  4. Rong (2014), "word2vec Parameter Learning Explained"

Last updated