Skip-gram

"sat" at position 2 produces 4 training pairs. Each pair asks: given "sat", can you predict each context word? Unlike CBOW (which averages context), skip-gram treats each (center, context) pair independently.

At 12 words, cost is 144 operations per training pair. Now imagine V = 100,000 — that's 1,200,000 ops per pair, times millions of pairs. The full softmax becomes the dominant cost. This is why the original Word2Vec paper proposed negative sampling.

k=5: the model trains to push σ(positive) toward 1 and σ(negatives) toward 0. Each step requires only 6 dot products instead of 12. The negatives are sampled proportional to word frequency^0.75 — common words are more likely to be negatives.

t=1.0e-3: "the" and "a" are frequently discarded (94%), giving rare content words more relative weight. This dramatically improves embedding quality for words like "cat" and "dog" — they get more meaningful training signal.

Before training: words are randomly scattered. No semantic structure. The skip-gram objective will pull co-occurring words together and push non-co-occurring words apart.

"cat" is nearest to "dog" (sim=1.00) — both are animals! Skip-gram learned that "cat" and "dog" appear in similar contexts ("the ___ sat", "a ___ ran"), so their embeddings cluster together.