1Tokenization & BPE 2Bag of Words 3TF-IDF 4Word2Vec & Word Embeddings 5CBOW (Continuous Bag of Words)6Skip-gram 7Embeddings

Lensa ML

Bag of Words

Step 1 of 6: Building a Vocabulary

From corpus to word list

Minimum frequency threshold1

Vocab size

Removed

Total tokens

Minimum frequency = 1: every word in the corpus is kept, from common ones like 'the' (5×) to rare ones like 'big' (1×).

Drag Minimum frequency threshold from 1 to 5 and watch the 14-word vocabulary shrink. The five toy documents hold 30 tokens in total. Building the vocabulary is the only fitting a bag-of-words model needs: tokenize, count, filter, and give each surviving word an index. Real pipelines filter from both ends. scikit-learn's CountVectorizer has min_df to drop rare words and max_df to drop words found in too many documents, and many pipelines also lowercase the text and remove a stop-word list. Words outside the vocabulary are ignored at prediction time.

Step 2 of 6: One-Hot Encoding

One word, one dimension

Select word0

Word

Index

Dimensions

Non-zero

"a" is at index 0 → its one-hot vector has a 1 at position 0 and 0s everywhere else. Common words get low indices here. Each word owns exactly one dimension, with no notion of similarity between words.

Drag Select word across the 14 indices to move the single 1 through the vector. Any two different one-hot vectors have a dot product of 0 and a Euclidean distance of √2, which is the exact sense in which one-hot encoding knows nothing about meaning. Models rarely store these vectors. Multiplying a one-hot vector by a weight matrix only selects one row of the matrix, so neural networks replace the multiplication with a table lookup. That lookup table is an embedding layer.

Step 3 of 6: Count Vectors

Summing one-hots into a document vector

Select document0

Document

Doc 0

Tokens

Unique

Sparsity

64%

Doc 0 has 6 tokens mapping to 5 unique words. Words like "the" appear 2×. The count vector records frequency, not position.

Drag Select document from 0 to 4 to see each sentence's count vector. A count vector is the sum of the one-hot vectors of its tokens: Doc 0 has 6 tokens, and its two copies of "the" add up to a 2 in one slot. Summing throws away position, so word order cannot be recovered. The vector still carries plenty of signal. Naive Bayes and logistic regression on count vectors were the standard spam filters and sentiment classifiers for years, and on small datasets they remain hard to beat.

Step 4 of 6: Document-Term Matrix

The full picture — all documents, all terms

Vocabulary size10

Matrix

5×10

Non-zero

Sparsity

56%

Terms

Every document is a row and every term is a column. With 10 terms, the matrix is 56% sparse. Real corpora are 99%+ sparse.

Drag Vocabulary size up and watch columns of mostly zeros join the matrix. Rows are documents and columns are terms, so reading a column instead gives each word a vector of the documents it appears in. Factorizing this matrix with SVD is latent semantic analysis (LSA, 1990), one of the first methods to place related words near each other. Sparse formats such as CSR store only the non-zero cells and their positions. A dense matrix of a million documents by 100,000 terms would need 100 billion cells.

Step 5 of 6: Binary vs. Count vs. Normalized

Three flavors of BoW

Mode

Count

Value range

0-2

Non-zero

Count BoW: raw word counts. 'the' appears 2× in Doc 0, so its cell is 2. Frequency is kept, but longer documents get larger counts, which skews comparisons. (Every toy document here has exactly 6 tokens, so that skew can't appear in this table.)

Click Binary, Count, and TF-Normalized to switch how each cell is filled. The right choice depends on the model. Bernoulli naive Bayes expects binary inputs, multinomial naive Bayes expects counts, and linear models trained with gradient descent usually do better on normalized or log-scaled values, so that no single feature dominates. In short texts such as tweets or search queries words rarely repeat, and binary and count vectors come out almost identical. The normalized version is the TF part of TF-IDF.

Step 6 of 6: Limitations

What BoW loses

Ordering

BoW vector

[1,1,1]

Sparsity

64%

"Dog bites man": an ordinary headline with BoW vector [1,1,1]. Click the other orderings. The meaning changes, but the vector does not.

Click the four orderings of dog, bites, and man and compare the vectors. A partial fix is n-grams: adding word pairs as extra columns gives "dog bites" and "man bites" separate columns, so the two headlines finally differ. The cost is size, since a vocabulary of V words can produce up to V² bigrams. Bag-of-words has other blind spots. "not good" and "good" share the feature "good", and synonyms like "film" and "movie" are unrelated columns. Word embeddings and sequence models such as RNNs and transformers address these problems.

What is the bag-of-words model?

The bag-of-words (BoW) model represents a document as the counts of the words it contains, ignoring their order. A fixed vocabulary assigns each word a position, and each document becomes a vector holding how many times each vocabulary word appears. Two sentences with the same words in a different order get the same vector.

Bag-of-words is the simplest way to turn text into numbers that a machine learning model can use. Spam filters, sentiment classifiers, topic models such as LDA, and early search engines were all built on it, usually paired with naive Bayes or logistic regression. It is still a quick, interpretable baseline, and TF-IDF is a reweighted version of the same vectors.

Key ideas

Bag-of-words represents a document by its word counts and discards word order.
A vocabulary maps each kept word to one fixed index in the vector.
A document's count vector is the sum of the one-hot vectors of its tokens.
Stacking the vectors gives a document-term matrix that is almost entirely zeros.
Cells can be binary, raw counts, or counts divided by document length.
N-grams recover some local word order at the cost of a much larger vocabulary.

Common questions

What is the difference between bag of words and TF-IDF?

Bag-of-words fills each cell with a count, so frequent function words get large values. TF-IDF starts from the same counts and multiplies each by an inverse document frequency weight, which shrinks words that appear in most documents and boosts rare, topical ones. Both produce vectors of the same shape.

Why is it called a "bag" of words?

In mathematics a bag, or multiset, is a collection that records how many times each item occurs but not their order. The idea of describing text by word distributions goes back to Zellig Harris's 1954 paper "Distributional Structure", which uses the phrase "bag of words" when discussing language.

How does bag-of-words handle words it has never seen?

A word missing from the vocabulary has no column, so it is dropped. The hashing trick avoids a fixed vocabulary by hashing each word to one of a set number of columns. That handles any word but lets unrelated words collide in the same column.

Is bag-of-words used outside text?

Yes. The bag-of-visual-words model clusters small image patches into a codebook of "visual words" and describes each image by a histogram of those clusters. It was a standard image classification method before convolutional networks, and the same counting idea appears in audio and genomics.

Related explainers

Tokenization & BPE — How text becomes numbers — from characters to subword tokens via byte-pair encoding
TF-IDF — Score how important a word is to a document in a collection — the backbone of search engines and text retrieval
Naive Bayes Classifier — Probabilistic classification with Bayes' rule
Text Classification & Sentiment — Teach a model to assign labels to text — from spam detection to sentiment analysis
Embeddings — Turning words and categories into vectors
Singular Value Decomposition — Factor any matrix into rotation, scaling, and rotation — the Swiss army knife of linear algebra

References

Last updated October 25, 2026