TF-IDF

Type a word into a search box and something has to decide which of the million matching documents to show first. For decades, that something was TF-IDF — a formula invented by Gerard Salton and colleagues in the 1970s that is still woven into every modern search system.

The idea starts with a puzzle. The word "the" appears in every English document. If you search for "the", every document matches equally — that is useless. Conversely, the word "eigenvalue" appears in very few documents. If your document uses it a lot, that is a strong signal it is about eigenvalues.

TF-IDF captures this trade-off in two multiplied pieces:

• TF (term frequency): how often the word appears in this document, normalized by document length.
• IDF (inverse document frequency): $\log\!\left(\dfrac{N}{df}\right)$, where $N$ is the total number of documents and $df$ is how many contain the word. Rare words get a high IDF; universal words get near zero.

Multiply them together and you have a score that rises when a word appears often here and rarely elsewhere — exactly the words that characterize a document.

Below are five short documents and a query box. The demo computes a TF-IDF score for each document against your query and ranks them. Try single words like "algorithm" or "matrix", then try rare words versus common ones like "the" and watch the scores collapse.

<!-- {{c_html_intro}} -->
<p class="hint">{{hint_para}}</p>
<div class="query-row">
  <label for="query">{{label_query}}</label>
  <input id="query" type="text" placeholder="{{placeholder_query}}" autocomplete="off" spellcheck="false" />
  <button id="btn-rank" type="button">{{btn_rank}}</button>
  <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
</div>
<div id="results" class="results"></div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.query-row { display: flex; align-items: center; gap: .5rem; flex-wrap: wrap; margin-bottom: .9rem; }
label { font-weight: 600; font-size: .9rem; white-space: nowrap; }
input { flex: 1 1 140px; padding: .4rem .65rem; border: 1px solid #9ab; border-radius: 7px;
        font: 15px system-ui, sans-serif; min-width: 80px; }
button { font: 600 14px system-ui, sans-serif; padding: .42rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; white-space: nowrap; }
button.ghost { background: #fff; color: #1d3557; }
.results { display: flex; flex-direction: column; gap: .55rem; }
.doc-card { border: 1px solid #d0d8e0; border-radius: 10px; padding: .6rem .8rem;
            background: #f7f9fb; transition: border-color .15s; }
.doc-card.top { border-color: #1d6a3a; background: #f0faf4; }
.doc-header { display: flex; align-items: center; gap: .5rem; margin-bottom: .3rem; }
.doc-rank { font: 700 13px ui-monospace, monospace; color: #1d3557; min-width: 22px; }
.doc-title { font-weight: 700; font-size: .92rem; flex: 1; }
.doc-score { font: 600 13px ui-monospace, monospace; color: #1d6a3a; min-width: 60px; text-align: right; }
.doc-score.zero { color: #999; }
.bar-wrap { height: 6px; background: #dde4ea; border-radius: 3px; overflow: hidden; }
.bar { height: 6px; background: #1d3557; border-radius: 3px; transition: width .3s; }
.doc-text { font-size: .82rem; color: #555; margin-top: .35rem; line-height: 1.45; }
.doc-text mark { background: #fef08a; border-radius: 2px; padding: 0 1px; }

// Code not found

Notice how a word that appears in all five documents gets an IDF of zero and contributes nothing to any score. A word that appears in only one document gets a high IDF and a document that uses it repeatedly shoots to the top.

TF-IDF is beautifully cheap to compute. For a corpus of $N$ documents with a total of $T$ tokens, building the full index takes $O(T)$ time and $O(V \cdot N)$ space, where $V$ is the vocabulary size. Scoring a query of $q$ terms against all documents is $O(q \cdot N)$ — essentially linear.

But cheapness hides real weaknesses:

• Bag of words. TF-IDF treats each document as an unordered bag of words. "Dog bites man" and "Man bites dog" are identical to it.
• No synonyms. A document about automobiles scores zero for a query about cars unless the same word appears.
• No context. "Bank" (river) and "bank" (finance) have the same representation.
• Length saturation. A very long document can accumulate high raw TF just by being long. The variant BM25 (used by Elasticsearch and most modern search engines) adds a length-normalization term and a saturation parameter $k_1$ to cap the benefit of repeated occurrences.

The deeper successor is dimensionality reduction: represent words as dense vectors (word2vec, GloVe, then transformer embeddings) so that "car" and "automobile" land near each other in vector space. Modern search pipelines often combine a BM25 first pass with a neural re-ranker — the $O(T)$ formula is still doing the heavy lifting on the first cut.

Half a century after its invention TF-IDF still shows up almost everywhere text is processed:

• Search engines: Google's original PageRank paper treated TF-IDF as the baseline retrieval model it had to beat. Modern engines use BM25 — a direct descendant — for first-stage retrieval.
• Keyword extraction: the highest-TF-IDF terms in a document are its signature words — useful for auto-tagging, summarization, and SEO analysis.
• Document clustering and classification: TF-IDF vectors fed into k-means or a naive Bayes classifier gave early text-mining its first practical legs.
• Spam filtering: an email that talks about "money" and "transfer" much more than your inbox usually does raises a TF-IDF alarm.
• Legal e-discovery: law firms use TF-IDF to sift millions of documents for the handful relevant to a case keyword.
• Research baselines: every pattern matching or information-retrieval paper still reports TF-IDF accuracy as the floor the new method must clear.

TF-IDF is one of those rare ideas that was right on the first try. The intuition — a word matters when it is common here and rare everywhere else — is so fundamental that every more sophisticated model (BM25, word embeddings, transformer re-rankers) is essentially a correction on top of it.

The formula is $O(T)$, fits in a spreadsheet, and still beats far more complex rivals on short queries. That is not an accident. It is what happens when a simple model captures the true structure of the problem.

The next time a search engine returns exactly the document you were looking for, there is a good chance that a weighted logarithm — unchanged since the 1970s — did most of the work.