ENID
Lensa ML

TF-IDF

Step 1 of 6: Term Frequency

Counting words in a document

Corpus of 4 documents — counting "rocket"
D1: Space2 of 9 tokens → TF = 0.222
therocketlaunchedintospacetherocketorbitedearth
D2: Ocean0 of 10 tokens → TF = 0.000
thewhaleswaminthedeepoceanthewhalesurfaced
D3: Space1 of 8 tokens → TF = 0.125
theastronautfloatedinspaceneartherocket
D4: Food0 of 10 tokens → TF = 0.000
thechefcookedthefreshfishinthedeeppan
TF("rocket") per documentD1 Space0.222D2 Ocean0.000D3 Space0.125D4 Food0.000
TF("rocket", doc) = count("rocket") ÷ total tokens in doc
TF (D1)
0.222
TF (D2)
0.000
TF (D3)
0.125
TF (D4)
0.000

"rocket" has non-zero TF in D1, D3 and zero in the others. Terms concentrated in a few documents tell them apart.

Click each word in the vocabulary row, from rocket to the, to see its count and TF in all four documents. Dividing by document length keeps a long document from winning just by having more words: "rocket" appears twice in D1's 9 tokens (TF = 0.222) and once in D3's 8 tokens (TF = 0.125). It is one of several TF variants. Raw counts, a binary present-or-absent flag, and 1 + log(count) are all common. The log version stops a word that appears 50 times from counting 50 times as much as one that appears once.

Step 2 of 6: Inverse Document Frequency

Rare words matter more

In how many documents does "rocket" appear?
D1: Spacecontains
therocketlaunchedintospacetherocketorbitedearth
D2: Oceanabsent
thewhaleswaminthedeepoceanthewhalesurfaced
D3: Spacecontains
theastronautfloatedinspaceneartherocket
D4: Foodabsent
thechefcookedthefreshfishinthedeeppan
IDF = log(N / DF), N = 412340.000.691.39documents containing the term (DF)0.693
IDF("rocket") = log(4 / 2) = 0.693
N (docs)
4
DF
2
IDF
0.693

"rocket" appears in 2 of 4 documents, so IDF = log(4/2) = 0.69. The fewer documents contain a term, the higher its IDF.

Click a rare word such as whale, then a common one such as the, and compare the IDF line. The demo uses the natural log: whale appears in 1 of 4 documents, so IDF = ln 4 ≈ 1.386, and rocket appears in 2 of 4, so IDF = ln 2 ≈ 0.693. Karen Spärck Jones proposed this weighting in 1972, arguing that a term's value falls with the number of documents that use it. scikit-learn smooths it to ln((1 + N) / (1 + df)) + 1, which keeps "the" slightly above zero.

Step 3 of 6: TF-IDF Score

Combining frequency and rarity

TF-IDF("rocket") = TF × IDF for each document
D1: SpaceTF=0.222 × IDF=0.693 = 0.154
therocketlaunchedintospacetherocketorbitedearth
D2: OceanTF=0.000 × IDF=0.693 = 0.000
thewhaleswaminthedeepoceanthewhalesurfaced
D3: SpaceTF=0.125 × IDF=0.693 = 0.087
theastronautfloatedinspaceneartherocket
D4: FoodTF=0.000 × IDF=0.693 = 0.000
thechefcookedthefreshfishinthedeeppan
TF-IDF("rocket") per documentD1 Space0.154D2 Ocean0.000D3 Space0.087D4 Food0.000
TF-IDF = TF × IDF — IDF("rocket") = log(4/2) = 0.693
TF-IDF (D1)
0.154
TF-IDF (D2)
0.000
TF-IDF (D3)
0.087
TF-IDF (D4)
0.000

"rocket" shares one IDF across documents, so its scores differ only by TF. The document that repeats it more gets the higher score.

Click through the terms and watch which document cards light up. The score is a product, so a term needs both parts: whale in D2 has TF = 0.2 and IDF = 1.386, giving 0.277, the largest value in this corpus. A frequent word with zero IDF and a rare word that is absent from a document both score 0 there. Search engines later refined the formula. BM25 keeps IDF but saturates the TF part, so the tenth repeat of a word adds far less to the score than the second does.

Step 4 of 6: Document Vectors

Documents as points in term-space

D1: Space
therocketlaunchedintospacetherocketorbitedearth
TF-IDF vector for D1: Space — one dimension per vocab term:
rocket
0.154
space
0.077
whale
0.000
ocean
0.000
deep
0.000
astronaut
0.000
chef
0.000
fish
0.000
the
0.000
vector = [0.15, 0.08, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00]
Dimensions
9
Non-zero
2
Max value
0.154
Sparsity
78%

D1: Space has a very sparse vector: only 2 non-zero dimensions. Every vocabulary term missing from this document, plus "the", sits at 0.

Click D1 through D4 to see each document as a vector with one dimension per vocabulary term. The demo tracks 9 terms. A real corpus has tens of thousands, and a typical document uses well under 1% of them, so libraries store TF-IDF matrices in sparse formats that keep only the non-zero entries. Each row is usually scaled to unit length (L2 normalization) so long and short documents are comparable. These rows are the classic input to linear classifiers such as logistic regression and to clustering methods such as k-means.

Step 5 of 6: Document Similarity

Finding related documents with cosine similarity

Cosine similarity of D1: Space vs each document:
D1: Space (query)1.000
D3: Space 0.548
shared terms: rocket, space
D2: Ocean 0.000
no shared TF-IDF terms → similarity = 0
D4: Food 0.000
no shared TF-IDF terms → similarity = 0
Query
D1 Space
Most similar
D3 Space
Cos sim
0.548

D1: Space is most similar to D3: Space (cosine = 0.548) because they share TF-IDF dimensions: rocket, space. Documents with no shared terms score 0, because their vectors are orthogonal.

Click each document to make it the query and read the ranked similarities. Cosine similarity is the dot product divided by both lengths, cos θ = a · b / (|a| |b|), and because TF-IDF weights are never negative it ranges from 0 to 1. D1 and D3 score 0.548 through rocket and space. D2 and D4 score 0.073 through deep, a link that is weak and also wrong: a deep ocean and a deep pan mean different things. TF-IDF matches exact strings, so it cannot separate word senses or connect synonyms like ship and vessel.

Step 6 of 6: TF-IDF in Practice

Search ranking and retrieval

Query vector (1 × IDF for each query term):
rocket=0.69space=0.00whale=0.00ocean=0.00deep=0.00astronaut=0.00chef=0.00fish=0.00the=0.00
Click a result to see the cosine similarity computation:
Cosine similarity: Query × D1: Space
termquery×doc=product
rocket0.693×0.154=0.1068
space0.000×0.077=0.0000
dot product = 0.1068
|query| = 0.6931 |doc| = 0.1722
cosine = 0.1068 / (0.6931 × 0.1722) = 0.894

Single-term query "rocket": only documents containing it get a non-zero dot product. The ranking then depends on how much of each document's vector that one term holds.

Click the query words to add or remove each term, then click a result to see its dot-product breakdown. With rocket alone, D1 scores 0.894 and D3 0.408. Adding space, deep, and whale makes the query rocket, space, deep, whale, and D2 jumps to first at 0.742. The query is treated as a tiny document and ranked with the same cosine formula. This is the vector space model that Gerard Salton's SMART retrieval system made standard in the 1970s. Inverted indexes make it fast: only documents containing a query term are ever scored.

What is TF-IDF?

TF-IDF (term frequency times inverse document frequency) is a score for how important a word is to one document within a collection. It rises when a word appears often in that document and falls when the word appears in many documents. Words like "the" end up near zero, and topic words like "whale" or "rocket" stand out.

TF-IDF turns each document into a vector of word weights, which makes text usable by any method that works on numbers. It powered keyword search engines for decades, and its descendant BM25 is still the default ranking function in Lucene and Elasticsearch. It is also a strong baseline for spam filtering, topic classification, document clustering, and keyword extraction, and it handles the keyword side of hybrid search in retrieval-augmented generation (RAG).

Key ideas

  • TF measures how often a term appears in one document, usually divided by the document's length.
  • IDF = log(N / df) is high for terms found in few documents and zero for terms found in all of them.
  • TF-IDF multiplies the two, so a high score needs a term that is frequent locally and rare globally.
  • Each document becomes a sparse vector with one dimension per vocabulary term.
  • Cosine similarity between TF-IDF vectors ranks related documents and search results.
  • TF-IDF matches exact words only, so it misses synonyms and confuses different senses of one word.

Common questions

What is the difference between TF-IDF and word embeddings?

TF-IDF gives every word its own dimension and assigns weights from counts, so "car" and "automobile" are as unrelated as "car" and "banana". Word embeddings such as word2vec learn dense vectors in which related words sit close together. TF-IDF needs no training and is easy to interpret; embeddings capture meaning but need a trained model.

Is TF-IDF still used?

Yes. It is a fast, strong baseline for text classification and clustering, and BM25, its refined form, still runs first-stage retrieval in many search systems. Hybrid search often combines BM25 scores with dense embedding scores because keyword matching catches exact names, codes, and rare terms that embeddings can blur.

How do you compute TF-IDF in Python?

scikit-learn's TfidfVectorizer tokenizes the text, builds the vocabulary, computes smoothed IDF weights, and returns an L2-normalized sparse matrix with one row per document. Calling fit_transform on the training documents and transform on new ones keeps the IDF values fixed to the training corpus.

Can TF-IDF use phrases instead of single words?

Yes. Treating n-grams such as "new york" or "not good" as extra terms lets TF-IDF keep some word order and negation. The vocabulary grows quickly, so n-gram features are usually limited by a minimum document frequency or a maximum feature count.

Related explainers

References

  1. Spärck Jones (1972), "A Statistical Interpretation of Term Specificity and Its Application in Retrieval", Journal of Documentation
  2. Salton, Wong & Yang (1975), "A Vector Space Model for Automatic Indexing", Communications of the ACM
  3. Robertson & Zaragoza (2009), "The Probabilistic Relevance Framework: BM25 and Beyond"
  4. Jurafsky & Martin, "Speech and Language Processing" (3rd ed. draft)

Last updated