TF-IDF, how important is a word to a document?

1.6 TF-IDF, how important is a word to a document?

Tokenization gives us pieces; now we want to weigh them. TF-IDF answers a classic question: how important is a word to a specific document, relative to a whole collection?

The intuition has two halves:

• A word that appears often in one document is probably relevant to it.
• A word that appears in every document ("the", "is") is not distinctive and tells you nothing.

So we multiply the two:

$$\text{TF-IDF} = \underbrace{\text{TF}}_{\text{how frequent } \textit{here}} \times \underbrace{\text{IDF}}_{\text{how rare } \textit{everywhere}}$$

• TF (Term Frequency), how often the word appears in this document.
• IDF (Inverse Document Frequency), the inverse of how many documents contain it, so common words are pushed toward zero and rare words are amplified.

The concrete formulas, for a term $t$ and document $d$ drawn from a collection of $N$ documents:

$$\text{TF}(t, d) = \frac{\text{count of } t \text{ in } d}{\text{total terms in } d}, \qquad \text{IDF}(t) = \log\!\left(\frac{N}{\text{DF}(t)}\right)$$

where $\text{DF}(t)$ is the document frequency: the number of documents that contain $t$ at least once (not how many times). A term present in every document has $\text{DF}(t) = N$, so $\text{IDF}(t) = \log(1) = 0$, exactly the "worthless" case below.

Worked example

Three documents:

	Doc 1 (Cooking)	Doc 2 (Cooking)	Doc 3 (Sports)
Content	"the cat eats fish"	"the dog eats meat"	"the cat plays ball"

"the" appears in all three documents → its IDF is ~0 → its TF-IDF is worthless, exactly as we want. Meanwhile "ball" appears in only one document → high IDF → it becomes a strong, distinctive signal for Doc 3.

Computing it, "the" vs. "cat" vs. "ball" in Doc 1 / Doc 3 (each document has 4 terms, so every TF here is $1/4 = 0.25$; $N = 3$):

Term	DF	IDF $= \log(N / \text{DF})$	TF-IDF in its document
"the"	3 (all docs)	$\log(3/3) = 0$	$0.25 \times 0 = 0.000$
"cat"	2 (Doc 1, Doc 3)	$\log(3/2) \approx 0.405$	$0.25 \times 0.405 \approx 0.101$
"ball"	1 (Doc 3 only)	$\log(3/1) \approx 1.099$	$0.25 \times 1.099 \approx 0.275$

The ranking falls out exactly as intuition predicts: "the" scores zero, "cat" (shared by two of three documents) scores a modest 0.101, and "ball" (unique to Doc 3) scores nearly three times higher at 0.275, the rarer the word across the collection, the more weight it carries wherever it does appear.

Takeaway: TF-IDF automatically down-weights common words and highlights distinctive ones, with no hand-built stop-word list.

From scores to vectors

Score every word against every document and you get a document vector, the row of TF-IDF values:

	ball	cat	dog	…
Doc 1	TF-IDF(ball, D1)	TF-IDF(cat, D1)	TF-IDF(dog, D1)	…
Doc 2	TF-IDF(ball, D2)	TF-IDF(cat, D2)	TF-IDF(dog, D2)	…
Doc 3	TF-IDF(ball, D3)	TF-IDF(cat, D3)	TF-IDF(dog, D3)	…

These vectors are the foundation of classic search engines, document comparison, and keyword extraction, and, as we'll see in the RAG chapters, TF-IDF's cousin BM25 is still a first-class citizen in modern retrieval systems.

---

1.7 N-grams, bringing back word order

TF-IDF (as described) treats a document as a bag of words: it counts words but forgets the order they came in. That loses real meaning:

"The product is not good"

As single words (unigrams), "not" and "good" sit apart and the negation vanishes. As a bigram, "not good" is a single feature that clearly reads as negative.

An n-gram is a contiguous run of n tokens:

Sentence	Unigrams (n=1)	Bigrams (n=2)	Trigrams (n=3)
"the cat eats fish"	the, cat, eats, fish	the cat, cat eats, eats fish	the cat eats, cat eats fish

You then run TF-IDF over n-grams instead of (or alongside) single words, so the document vector gains columns like cat eats:

	cat	eats	fish	cat eats	…
Doc 1	…	TF-IDF(eats, D1)	TF-IDF(fish, D1)	TF-IDF(cat eats, D1)	…

The trade-off: more n-grams = richer features but a much larger vocabulary and slower training. Bigrams and trigrams are common; beyond that the vocabulary explodes and most n-grams appear too rarely to help.

---