Introduction

Type a word into a search box and something has to decide which of the million matching documents to show first. For decades, that something was TF-IDF — a formula invented by Gerard Salton and colleagues in the 1970s that is still woven into every modern search system.

The idea starts with a puzzle. The word "the" appears in every English document. If you search for "the", every document matches equally — that is useless. Conversely, the word "eigenvalue" appears in very few documents. If your document uses it a lot, that is a strong signal it is about eigenvalues.

TF-IDF captures this trade-off in two multiplied pieces:

  • TF (term frequency): how often the word appears in this document, normalized by document length.
  • IDF (inverse document frequency): log⁡ ⁣(Ndf)\log\!\left(\dfrac{N}{df}\right), where NN is the total number of documents and dfdf is how many contain the word. Rare words get a high IDF; universal words get near zero.

Multiply them together and you have a score that rises when a word appears often here and rarely elsewhere — exactly the words that characterize a document.

Try It

Below are five short documents and a query box. The demo computes a TF-IDF score for each document against your query and ranks them. Try single words like "algorithm" or "matrix", then try rare words versus common ones like "the" and watch the scores collapse.

<!-- Interactive TF-IDF document ranker — type a query, click Rank. -->
<p class="hint">Enter one or more words and click <b>Rank</b>. Each document gets a TF-IDF score: high when the word appears <em>often in that document</em> and <em>rarely across all documents</em>. Try common words like "the" and watch scores drop to zero.</p>
<div class="query-row">
  <label for="query">Query:</label>
  <input id="query" type="text" placeholder="e.g. algorithm" autocomplete="off" spellcheck="false" />
  <button id="btn-rank" type="button">Rank</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
</div>
<div id="results" class="results"></div>

/* Styles for the TF-IDF ranker demo. */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.query-row { display: flex; align-items: center; gap: .5rem; flex-wrap: wrap; margin-bottom: .9rem; }
label { font-weight: 600; font-size: .9rem; white-space: nowrap; }
input { flex: 1 1 140px; padding: .4rem .65rem; border: 1px solid #9ab; border-radius: 7px;
        font: 15px system-ui, sans-serif; min-width: 80px; }
button { font: 600 14px system-ui, sans-serif; padding: .42rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; white-space: nowrap; }
button.ghost { background: #fff; color: #1d3557; }
.results { display: flex; flex-direction: column; gap: .55rem; }
.doc-card { border: 1px solid #d0d8e0; border-radius: 10px; padding: .6rem .8rem;
            background: #f7f9fb; transition: border-color .15s; }
.doc-card.top { border-color: #1d6a3a; background: #f0faf4; }
.doc-header { display: flex; align-items: center; gap: .5rem; margin-bottom: .3rem; }
.doc-rank { font: 700 13px ui-monospace, monospace; color: #1d3557; min-width: 22px; }
.doc-title { font-weight: 700; font-size: .92rem; flex: 1; }
.doc-score { font: 600 13px ui-monospace, monospace; color: #1d6a3a; min-width: 60px; text-align: right; }
.doc-score.zero { color: #999; }
.bar-wrap { height: 6px; background: #dde4ea; border-radius: 3px; overflow: hidden; }
.bar { height: 6px; background: #1d3557; border-radius: 3px; transition: width .3s; }
.doc-text { font-size: .82rem; color: #555; margin-top: .35rem; line-height: 1.45; }
.doc-text mark { background: #fef08a; border-radius: 2px; padding: 0 1px; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// TF-IDF: score = sum over query terms of tf(term, doc) * idf(term, corpus)
// tf(term, doc) = count(term in doc) / total tokens in doc
// idf(term) = log((N+1)/(df(term)+1)) + 1  (smoothed to avoid zero for rare terms)
// score(doc, query) = sum of tf*idf for each query term

// The five sample documents (each is a short paragraph about a CS topic).
const DOCS = [
  { id: 'doc1', title: 'Algorithms & Complexity', text: 'An algorithm is a finite sequence of well-defined instructions used to solve a problem or perform a computation. Algorithms can be analyzed for time complexity and space complexity. Sorting algorithms, graph algorithms, and dynamic programming are key topics in computer science.' },
  { id: 'doc2', title: 'Machine Learning', text: 'Machine learning algorithms allow computers to learn from data without being explicitly programmed. Gradient descent, backpropagation, and neural networks are core techniques. Training a deep learning model requires large datasets and significant computational resources.' },
  { id: 'doc3', title: 'Linear Algebra', text: 'Linear algebra studies vectors, matrices, and linear transformations. The eigenvalue decomposition of a matrix reveals its intrinsic structure. Matrix multiplication and the dot product are fundamental operations used throughout mathematics and machine learning.' },
  { id: 'doc4', title: 'Databases', text: 'A relational database stores data in tables with rows and columns. SQL is the standard query language for retrieving and manipulating data. Indexes speed up query execution by allowing the engine to locate rows without a full table scan.' },
  { id: 'doc5', title: 'Information Retrieval', text: 'Information retrieval systems rank documents by relevance to a query. TF-IDF weights each term by its frequency in the document and its rarity across the corpus. BM25 extends TF-IDF with length normalization and term saturation to improve ranking quality.' }
];

// Lowercase, strip punctuation, split on whitespace.
function tokenize(str) {
  return str.toLowerCase().replace(/[^a-z\s]/g, ' ').split(/\s+/).filter(Boolean);
}

// Build per-doc token frequency map.
function buildIndex(docs) {
  const index = {};
  docs.forEach(doc => {
    const tokens = tokenize(doc.text);
    const freq = {};
    tokens.forEach(t => { freq[t] = (freq[t] || 0) + 1; });
    index[doc.id] = { tokens, freq, len: tokens.length };
  });
  return index;
}

// Count how many docs contain each term (document frequency).
function computeDF(docs, index) {
  const df = {};
  docs.forEach(doc => {
    Object.keys(index[doc.id].freq).forEach(term => {
      df[term] = (df[term] || 0) + 1;
    });
  });
  return df;
}

// TF-IDF score for one doc against the query terms.
function scoreDoc(docId, queryTerms, index, df, N) {
  const { freq, len } = index[docId];
  if (!len) return 0;
  let score = 0;
  queryTerms.forEach(term => {
    const tf = (freq[term] || 0) / len;
    const idf = Math.log((N + 1) / ((df[term] || 0) + 1)) + 1; // Smoothed IDF keeps rare terms from getting infinite weight.
    score += tf * idf;
  });
  return score;
}

// Wrap matching query terms in <mark> tags.
function highlight(text, terms) {
  const re = new RegExp('\\b(' + terms.map(t => t.replace(/[.*+?^${}()|[\]\\]/g, '\\amp;')).join('|') + ')\\b', 'gi');
  return text.replace(re, '<mark>$1</mark>');
}

const idx = buildIndex(DOCS);
const df = computeDF(DOCS, idx);
const N = DOCS.length;

function render(scored) {
  const container = document.getElementById('results');
  const maxScore = Math.max(...scored.map(d => d.score), 1e-9);
  container.innerHTML = scored.map((d, i) => {
    const pct = maxScore > 0 ? (d.score / maxScore) * 100 : 0;
    const scoreStr = d.score.toFixed(4);
    const isTop = i === 0 && d.score > 0;
    const body = d.queryTerms.length ? highlight(d.doc.text, d.queryTerms) : d.doc.text;
    return `<div class="doc-card${isTop ? ' top' : ''}">
      <div class="doc-header">
        <span class="doc-rank">#${i + 1}</span>
        <span class="doc-title">${d.doc.title}</span>
        <span class="doc-score${d.score === 0 ? ' zero' : ''}">${scoreStr}</span>
      </div>
      <div class="bar-wrap"><div class="bar" style="width:${pct.toFixed(1)}%"></div></div>
      <div class="doc-text">${body}</div>
    </div>`;
  }).join('');
}

function rankDocs(query) {
  const qTerms = tokenize(query);
  const scored = DOCS.map(doc => ({
    doc,
    score: scoreDoc(doc.id, qTerms, idx, df, N),
    queryTerms: qTerms
  }));
  scored.sort((a, b) => b.score - a.score);
  render(scored);
}

// Show all docs in original order with score 0 at startup.
function renderDefault() {
  render(DOCS.map(doc => ({ doc, score: 0, queryTerms: [] })));
}

document.getElementById('btn-rank').addEventListener('click', () => {
  rankDocs(document.getElementById('query').value.trim());
});
document.getElementById('query').addEventListener('keydown', e => {
  if (e.key === 'Enter') rankDocs(document.getElementById('query').value.trim());
});
document.getElementById('btn-reset').addEventListener('click', () => {
  document.getElementById('query').value = '';
  renderDefault();
});

renderDefault();
Result
Console

Notice how a word that appears in all five documents gets an IDF of zero and contributes nothing to any score. A word that appears in only one document gets a high IDF and a document that uses it repeatedly shoots to the top.

The Real Complexity

TF-IDF is beautifully cheap to compute. For a corpus of NN documents with a total of TT tokens, building the full index takes O(T)O(T) time and O(V⋅N)O(V \cdot N) space, where VV is the vocabulary size. Scoring a query of qq terms against all documents is O(q⋅N)O(q \cdot N) — essentially linear.

But cheapness hides real weaknesses:

  • Bag of words. TF-IDF treats each document as an unordered bag of words. "Dog bites man" and "Man bites dog" are identical to it.
  • No synonyms. A document about automobiles scores zero for a query about cars unless the same word appears.
  • No context. "Bank" (river) and "bank" (finance) have the same representation.
  • Length saturation. A very long document can accumulate high raw TF just by being long. The variant BM25 (used by Elasticsearch and most modern search engines) adds a length-normalization term and a saturation parameter k1k_1 to cap the benefit of repeated occurrences.

The deeper successor is dimensionality reduction: represent words as dense vectors (word2vec, GloVe, then transformer embeddings) so that "car" and "automobile" land near each other in vector space. Modern search pipelines often combine a BM25 first pass with a neural re-ranker — the O(T)O(T) formula is still doing the heavy lifting on the first cut.

Where It Matters

Half a century after its invention TF-IDF still shows up almost everywhere text is processed:

  • Search engines: Google's original PageRank paper treated TF-IDF as the baseline retrieval model it had to beat. Modern engines use BM25 — a direct descendant — for first-stage retrieval.
  • Keyword extraction: the highest-TF-IDF terms in a document are its signature words — useful for auto-tagging, summarization, and SEO analysis.
  • Document clustering and classification: TF-IDF vectors fed into k-means or a naive Bayes classifier gave early text-mining its first practical legs.
  • Spam filtering: an email that talks about "money" and "transfer" much more than your inbox usually does raises a TF-IDF alarm.
  • Legal e-discovery: law firms use TF-IDF to sift millions of documents for the handful relevant to a case keyword.
  • Research baselines: every pattern matching or information-retrieval paper still reports TF-IDF accuracy as the floor the new method must clear.

Conclusion

TF-IDF is one of those rare ideas that was right on the first try. The intuition — a word matters when it is common here and rare everywhere else — is so fundamental that every more sophisticated model (BM25, word embeddings, transformer re-rankers) is essentially a correction on top of it.

The formula is O(T)O(T), fits in a spreadsheet, and still beats far more complex rivals on short queries. That is not an accident. It is what happens when a simple model captures the true structure of the problem.

The next time a search engine returns exactly the document you were looking for, there is a good chance that a weighted logarithm — unchanged since the 1970s — did most of the work.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/tf-idf/Content licensed under CC BY-NC 4.0.