1Tokenization & BPE 2Bag of Words 3TF-IDF 4Word2Vec & Word Embeddings 5CBOW (Continuous Bag of Words)6Skip-gram 7Embeddings

Lensa ML

CBOW

Step 1 of 6: Context Window

Selecting the input words

Context window size2

Window

±2

Context

4 words

Target

sat

Window ±2: CBOW averages 4 context embeddings into one vector, then predicts the center word "sat". Averaging is the "bag" in CBOW: context order doesn't matter.

Drag Context window size from 1 to 4 and watch which words around "sat" feed the prediction. The sentence has only six words, so at ±3 and ±4 the window already covers the whole sentence and the context stays at 5 words. The name explains the design: "bag" because context order is ignored, "continuous" because the context is a dense vector rather than a sparse count vector. The original 2013 experiments used four words on each side of the target, and the input layer was shared across all positions.

Step 2 of 6: Averaging Embeddings

From multiple vectors to one

Inspect context word0

Target

sat

Inspecting

the

Emb dim

Context size

The embedding for "the" is a 6-dimensional vector. Every context word counts equally in the average. Slide to inspect each one.

Drag Inspect context word from 0 to 3 to step through the, cat, on, and the, the four words around "sat". The hidden vector is h = (v_the + v_cat + v_on + v_the) / 4, so "the" counts twice because it appears twice. Averaging is CBOW's main design choice. It costs almost nothing and gives the model a fixed-size input whatever the window size, but it erases which word sat where. Some later variants weight context positions or concatenate the vectors instead, trading speed for order information.

Step 3 of 6: Projection Layer

From hidden state to vocabulary scores

Embedding dimension4

Emb dim

Vocab

Params

Target logit

0.95

4 dimensions: the averaged context vector is multiplied by the 12×4 output weight matrix to produce one logit per vocabulary word. The target "sat" has logit 0.95. Softmax turns these scores into probabilities next.

Drag Embedding dimension from 2 to 6 and watch the logits shift. The model has two matrices of size 12 × d: W_input supplies the context vectors, and the rows of W_output score each candidate center word. At d = 6 that is 144 parameters. At real scale the same layout is large: a 100,000-word vocabulary with 300 dimensions needs 30 million weights per matrix. Word2vec keeps only W_input as the word vectors. GloVe, a related method, adds its two matrices together instead.

Step 4 of 6: Softmax Prediction

Turning scores into probabilities

Temperature1.00

Temperature

1.0

P(sat)

0.115

Top-1

ran

Entropy

3.26

Temperature 1.0: close to the plain softmax (T = 1) that CBOW trains with. "sat" has probability 0.115; with untrained weights "ran" leads, and training has to move probability onto "sat".

Drag Temperature from 0.1 to 3.0 and watch the distribution sharpen and flatten. The weights here are untrained, so "ran" outscores "sat" at every temperature; only training moves probability onto the right word. CBOW itself always trains at T = 1. Cost is the bigger issue: a full softmax normalizes over the whole vocabulary, which means 100,000 dot products per training example at real scale. Word2vec avoids it with hierarchical softmax, a binary tree that needs about log₂ V ≈ 17 decisions per word, or with negative sampling.

Step 5 of 6: Training with Cross-Entropy

Learning from mistakes

Training step0

Step

Loss

2.16

P(sat)

0.115

Before training: the weights are random, P("sat") is 0.12 and "ran" scores higher. The cross-entropy loss, −log P("sat"), is 2.16.

Click Play or drag the Training step slider from 0 to 30 to follow the loss and probability curves. Training minimizes cross-entropy, L = −log P(center | context), so the starting P = 0.115 costs about 2.16 and the final P = 0.75 costs about 0.29. The gradient flows back through the average, so every context vector gets the same update divided by the number of context words. The curves come from 30 steps of gradient descent on this one window, while the 2D scatter is only a sketch. Real training sweeps billions of windows with stochastic gradient descent and a learning rate that decays linearly toward zero.

Step 6 of 6: CBOW vs. Skip-gram

Two sides of the same coin

CBOW

Skip-gram

Direction

Context → Center

Center → Context

Speed

Faster (1 update/window)

Slower (2C updates/window)

Rare words

Averaged out

Each gets its own gradient

Frequent words

Slightly more accurate

Slightly less accurate

Best for

Frequent words, syntax

Rare words, semantics

Model

CBOW

Training pairs

1/window

Best for

Frequent

CBOW averages all context embeddings into one vector and predicts the center word, so each window gives one update. That makes it fast and good for frequent words. A rare word's signal is diluted by the neighbors it is averaged with.

Click CBOW and Skip-gram to compare the two architectures. In the original paper CBOW trained faster than skip-gram and scored higher on syntactic analogies, while skip-gram did far better on semantic ones. The speed comes from all context words sharing one prediction. A practical rule follows: CBOW suits very large corpora where training time matters, and skip-gram suits smaller corpora or tasks where rare words matter. Both produce vectors of the same shape, so either can feed the same downstream model.

What is CBOW (continuous bag of words)?

CBOW, or continuous bag of words, is the word2vec architecture that predicts a word from the words around it. It looks up the embedding of each context word, averages them into one vector, and uses that vector to score every vocabulary word as the missing center word. Training adjusts the embeddings until the right word gets a high score.

CBOW is the faster of the two word2vec architectures, which made it a common choice for training word vectors on very large corpora. It is the default mode in tools such as gensim's Word2Vec, and its averaging idea reappears in fastText's text classifier, which averages word and n-gram vectors to label documents. The fill-in-the-middle task it uses is also an early form of the masked word prediction that BERT later scaled up.

Key ideas

CBOW predicts the center word from the average of its context word embeddings.
Averaging makes the context a bag: word order inside the window is ignored.
Scores for every vocabulary word come from one matrix product with the averaged vector.
Training minimizes cross-entropy on the true center word.
Hierarchical softmax or negative sampling avoids normalizing over the full vocabulary.
CBOW trains faster than skip-gram but learns weaker vectors for rare words.

Common questions

Is CBOW supervised or unsupervised?

It is self-supervised. The labels come from the text itself: every position in the corpus provides a context and the word that belongs in the middle, so no human annotation is needed. The prediction task is only a means to an end, since the useful output is the embedding matrix.

How do you train a CBOW model in Python?

In gensim, Word2Vec(sentences, sg=0) trains CBOW, and sg=0 is the default. The main settings are vector_size (dimensions), window (context size on each side), min_count (drop rare words), and negative (number of negative samples). The cbow_mean option chooses between averaging and summing the context vectors.

Can CBOW be used as a language model?

Not directly. It predicts a missing word from words on both sides, so it cannot generate text left to right, and it ignores order inside the window. It is a training trick for embeddings rather than a model of sentence probability.

What is the difference between CBOW and bag-of-words?

Bag-of-words represents a whole document as sparse word counts and involves no learning. CBOW learns a dense vector for each word by predicting it from a small window of neighbors. Both ignore order, but only CBOW places related words near each other.

Related explainers

Word2Vec & Word Embeddings — Learn dense vector representations where similar words live near each other in space
Skip-gram — Predict context words from a center word — the architecture that powers Word2Vec with negative sampling
Softmax, Logits & Temperature — Turning raw scores into probabilities
Embeddings — Turning words and categories into vectors
Bag of Words — Turn text into numbers — count word occurrences to create the simplest document representation in NLP
Loss Functions — How networks measure their mistakes

References

Last updated October 25, 2026