Lensa ML
Lensa ML

TF-IDF

Step 1 of 6

Term Frequency

Counting words in a document

Corpus of 4 documents — counting "rocket"
D1: Space2 of 9 tokens → TF = 0.222
therocketlaunchedintospacetherocketorbitedearth
D2: Ocean0 of 10 tokens → TF = 0.000
thewhaleswaminthedeepoceanthewhalesurfaced
D3: Space1 of 8 tokens → TF = 0.125
theastronautfloatedinspaceneartherocket
D4: Food0 of 10 tokens → TF = 0.000
thechefcookedthefreshfishinthedeeppan
TF("rocket", doc) = count("rocket") ÷ total tokens in doc
TF (D1)
0.222
TF (D2)
0.000
TF (D3)
0.125
TF (D4)
0.000

"rocket" has high TF in D1, D3 but zero in the others. Terms concentrated in specific documents are more discriminating.

Step 1 of 6: Term Frequency

Counting words in a document

Corpus of 4 documents — counting "rocket"
D1: Space2 of 9 tokens → TF = 0.222
therocketlaunchedintospacetherocketorbitedearth
D2: Ocean0 of 10 tokens → TF = 0.000
thewhaleswaminthedeepoceanthewhalesurfaced
D3: Space1 of 8 tokens → TF = 0.125
theastronautfloatedinspaceneartherocket
D4: Food0 of 10 tokens → TF = 0.000
thechefcookedthefreshfishinthedeeppan
TF("rocket", doc) = count("rocket") ÷ total tokens in doc
TF (D1)
0.222
TF (D2)
0.000
TF (D3)
0.125
TF (D4)
0.000

"rocket" has high TF in D1, D3 but zero in the others. Terms concentrated in specific documents are more discriminating.

Step 2 of 6: Inverse Document Frequency

Rare words matter more

In how many documents does "rocket" appear?
D1: Spacecontains
therocketlaunchedintospacetherocketorbitedearth
D2: Oceanabsent
thewhaleswaminthedeepoceanthewhalesurfaced
D3: Spacecontains
theastronautfloatedinspaceneartherocket
D4: Foodabsent
thechefcookedthefreshfishinthedeeppan
IDF("rocket") = log(4 / 2) = 0.693
N (DOCS)
4
DF
2
IDF
0.693

"rocket" appears in 2 of 4 documents — IDF = log(4/2) = 0.69. The fewer documents a term appears in, the higher its IDF score.

Step 3 of 6: TF-IDF Score

Combining frequency and rarity

TF-IDF("rocket") = TF × IDF for each document
D1: SpaceTF=0.222 × IDF=0.693 = 0.154
therocketlaunchedintospacetherocketorbitedearth
D2: OceanTF=0.000 × IDF=0.693 = 0.000
thewhaleswaminthedeepoceanthewhalesurfaced
D3: SpaceTF=0.125 × IDF=0.693 = 0.087
theastronautfloatedinspaceneartherocket
D4: FoodTF=0.000 × IDF=0.693 = 0.000
thechefcookedthefreshfishinthedeeppan
TF-IDF = TF × IDF — IDF("rocket") = log(4/2) = 0.693
TF-IDF (D1)
0.154
TF-IDF (D2)
0.000
TF-IDF (D3)
0.087
TF-IDF (D4)
0.000

"rocket" has different TF-IDF scores across documents. Higher scores mean the term is both frequent in that document (high TF) and rare across the corpus (high IDF) — the hallmark of a distinctive term.

Step 4 of 6: Document Vectors

Documents as points in term-space

D1: Space
therocketlaunchedintospacetherocketorbitedearth
TF-IDF vector for D1: Space — one dimension per vocab term:
rocket
0.154
space
0.077
whale
0.000
ocean
0.000
deep
0.000
astronaut
0.000
chef
0.000
fish
0.000
the
0.000
vector = [0.15, 0.08, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00]
DIMENSIONS
9
NON-ZERO
2
MAX VALUE
0.154
SPARSITY
78%

D1: Space has a very sparse vector — only 2 non-zero dimensions. Most terms don't appear in this document, so those dimensions are 0. Sparse vectors are typical in TF-IDF because each document uses only a small fraction of the vocabulary.

Step 5 of 6: Document Similarity

Finding related documents with cosine similarity

Cosine similarity of D1: Space vs each document:
D1: Space (query)1.000
D3: Space 0.548
shared terms: rocket, space
D2: Ocean 0.000
no shared TF-IDF terms → similarity = 0
D4: Food 0.000
no shared TF-IDF terms → similarity = 0
QUERY
Space
MOST SIMILAR
Space
COS SIM
0.548

D1: Space is most similar to D3: Space (cosine = 0.548) because they share TF-IDF dimensions: rocket, space. Documents with no shared terms have cosine similarity = 0 — their vectors are orthogonal.

Step 6 of 6: TF-IDF in Practice

Search ranking and retrieval

Query vector (1 × IDF for each query term):
rocket=0.69space=0.00whale=0.00ocean=0.00deep=0.00astronaut=0.00chef=0.00fish=0.00the=0.00
Click a result to see the cosine similarity computation:
Cosine similarity: Query × D1: Space
termquery×doc=product
rocket0.693×0.154=0.1068
space0.000×0.077=0.0000
dot product = 0.1068
|query| = 0.6931 |doc| = 0.1722
cosine = 0.1068 / (0.6931 × 0.1722) = 0.894

Single-term query "rocket": only terms where both query and document have non-zero values contribute to the dot product. This is exactly how early search engines like AltaVista worked!