Latent Semantic Analysis

Words are slippery. "Car" and "automobile" mean the same thing, yet a keyword search that finds one misses the other. "Bank" appears in both finance articles and river-walk guides, yet clearly signals different topics. Latent Semantic Analysis (LSA) is a technique, introduced by Deerwester et al. in 1990, that sidesteps these problems without any dictionary or grammar rules.

The idea is beautifully simple: collect a large body of text, build a table whose rows are words and whose columns are documents, and fill each cell with how often that word appears in that document. Then factor the table using the Singular Value Decomposition (SVD) — the same algebraic tool that compresses images and powers recommendation systems — and keep only the most important dimensions. What remain are latent topics: directions in a high-dimensional space along which words and documents cluster because they tend to co-occur, not because a human labeled them.

The word latent is the key insight. You never told the algorithm that "engine", "fuel", and "highway" belong to the same concept. It discovered that cluster on its own, purely from patterns of co-occurrence across thousands of documents.

The demo below runs LSA on six short documents drawn from two hidden topics — space exploration and cooking. Each document is represented as a bag of words; the algorithm knows nothing else.

<!-- {{c_html_intro}} -->
<p class="hint">{{hint_para}}</p>
<div class="docs-row" id="docs-row"></div>
<div class="canvas-wrap">
  <canvas id="plot" width="420" height="260"></canvas>
  <div class="axis-label x-label">{{label_axis_x}}</div>
  <div class="axis-label y-label">{{label_axis_y}}</div>
</div>
<div class="status" id="status">{{status_ready}}</div>
<div class="btns">
  <button id="run" type="button">{{btn_run}}</button>
  <button id="reset" type="button" class="ghost">{{btn_reset}}</button>
</div>
<div class="tooltip" id="tooltip"></div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .85rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
.docs-row { display: flex; flex-wrap: wrap; gap: .35rem; margin-bottom: .6rem; }
.doc-chip { font-size: .75rem; padding: .25rem .55rem; border-radius: 99px; cursor: pointer;
            border: 1.5px solid transparent; transition: all .15s; user-select: none; }
.doc-chip.space { background: #dbeafe; border-color: #93c5fd; color: #1e40af; }
.doc-chip.cook  { background: #dcfce7; border-color: #86efac; color: #166534; }
.doc-chip.active { border-width: 2.5px; font-weight: 700; }
.canvas-wrap { position: relative; }
canvas { display: block; background: #f8fafc; border: 1px solid #cbd5e1; border-radius: 8px; max-width: 100%; }
.axis-label { position: absolute; font-size: .7rem; color: #64748b; }
.x-label { bottom: 6px; right: 10px; }
.y-label { top: 6px; left: 10px; }
.status { font-size: .95rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; }
.status.ok { color: #0a7d33; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem;
         border: 1px solid #1d3557; background: #1d3557; color: #fff;
         border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.tooltip { position: fixed; background: #1e293b; color: #f1f5f9; font-size: .78rem;
           padding: .3rem .55rem; border-radius: 6px; pointer-events: none;
           display: none; white-space: pre; z-index: 99; line-height: 1.5; }

// Code not found

Click Run LSA to factorize the term-document matrix with a hand-computed SVD approximation and project every document onto the top two latent dimensions. Then click any document bubble to see its dominant topic. Notice that documents about the same theme land near each other in the concept space, even when they share no words at all — that is the power of latent structure.

Unlike many problems on this site, LSA is solved in the sense that its core computation is efficient and well-understood.

• Building the matrix takes $O(N \cdot d)$ time, where $N$ is the vocabulary size and $d$ is the number of documents. In practice both can be in the millions, but the matrix is extremely sparse.
• Full SVD of an $m \times n$ matrix costs $O(m \cdot n \cdot \min(m, n))$. For a vocabulary of 100,000 words and a corpus of 1,000,000 documents that is plainly infeasible.
• Truncated SVD (keeping only $k$ singular values) can be computed in $O(m \cdot n \cdot k)$ using randomized algorithms, making LSA practical. A typical value is $k = 300$.
• The hard question is choosing $k$. Too small and unrelated concepts bleed together; too large and you overfit, capturing noise instead of meaning. No formula decides $k$ — practitioners cross-validate or use heuristics.

A deeper subtlety: LSA assumes that meaning is a linear structure in word-occurrence space. This is a strong approximation. Words are not independent, word order matters, and negation ("not fast") collapses to the same bag-of-words as "fast". These limits motivated later models such as probabilistic topic models (LDA) and, eventually, neural word embeddings like Word2Vec and neural network training. Yet for many retrieval tasks a 300-dimensional LSA space still competes with far more complex approaches — a testament to how much structure hides in simple co-occurrence counts.

LSA's ability to group semantically related items without hand-crafted rules made it one of the first practical wins for unsupervised machine learning on language:

• Information retrieval: a query about "automobile fuel economy" returns articles that mention "car gas mileage" even without shared keywords, because both project close together in latent space.
• Plagiarism and near-duplicate detection: two reworded passages land near each other in concept space, flagging potential plagiarism even after synonym substitution.
• Automatic essay scoring: the Educational Testing Service deployed LSA in the 1990s to grade short answers by comparing student essays with high-scoring exemplars in latent space.
• Word sense disambiguation: the context documents around an ambiguous word vote on which latent direction it occupies, helping resolve meanings.
• Precursor to modern embeddings: Word2Vec, GloVe, and today's transformer-based contextual embeddings all grew from the same insight — that co-occurrence encodes meaning. LSA is the direct ancestor of the dimensionality reduction pipeline now standard in NLP.

Understanding LSA means understanding why structure emerges from statistics, and that lesson runs through every modern language model.

Latent Semantic Analysis is a quiet triumph of applied linear algebra. By treating a corpus as a matrix and factoring it with SVD, it surfaces the hidden conceptual skeleton of a language — without dictionaries, ontologies, or labeled training data.

Its limits are real: linear geometry misses word order, negation, and polysemy. But those limits drove three decades of innovation, from probabilistic topic models to the transformer architectures that power today's large language models. Every time a search engine finds what you meant rather than what you typed, you are seeing LSA's intellectual descendants at work.

The deeper lesson is that meaning is a pattern, and patterns live in data. You don't need to define "concept" — you just need enough text, a clever factorization, and the patience to look at what the algebra reveals.