Introduction

Words are slippery. "Car" and "automobile" mean the same thing, yet a keyword search that finds one misses the other. "Bank" appears in both finance articles and river-walk guides, yet clearly signals different topics. Latent Semantic Analysis (LSA) is a technique, introduced by Deerwester et al. in 1990, that sidesteps these problems without any dictionary or grammar rules.

The idea is beautifully simple: collect a large body of text, build a table whose rows are words and whose columns are documents, and fill each cell with how often that word appears in that document. Then factor the table using the Singular Value Decomposition (SVD) — the same algebraic tool that compresses images and powers recommendation systems — and keep only the most important dimensions. What remain are latent topics: directions in a high-dimensional space along which words and documents cluster because they tend to co-occur, not because a human labeled them.

The word latent is the key insight. You never told the algorithm that "engine", "fuel", and "highway" belong to the same concept. It discovered that cluster on its own, purely from patterns of co-occurrence across thousands of documents.

Try It: Concept Clustering

The demo below runs LSA on six short documents drawn from two hidden topics — space exploration and cooking. Each document is represented as a bag of words; the algorithm knows nothing else.

<!-- LSA demo: term-document matrix factored by 2-component SVD power iteration. -->
<p class="hint">Six short documents from two hidden topics (space & cooking) are mapped to a word-count matrix. Click <b>Run LSA</b> to project them onto the top 2 latent dimensions via SVD. Click a document chip to highlight it.</p>
<div class="docs-row" id="docs-row"></div>
<div class="canvas-wrap">
  <canvas id="plot" width="420" height="260"></canvas>
  <div class="axis-label x-label">Latent dimension 1</div>
  <div class="axis-label y-label">Latent dimension 2</div>
</div>
<div class="status" id="status">Click Run LSA to start.</div>
<div class="btns">
  <button id="run" type="button">Run LSA</button>
  <button id="reset" type="button" class="ghost">Reset</button>
</div>
<div class="tooltip" id="tooltip"></div>

/* Styles for the LSA concept-space demo. */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .85rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
.docs-row { display: flex; flex-wrap: wrap; gap: .35rem; margin-bottom: .6rem; }
.doc-chip { font-size: .75rem; padding: .25rem .55rem; border-radius: 99px; cursor: pointer;
            border: 1.5px solid transparent; transition: all .15s; user-select: none; }
.doc-chip.space { background: #dbeafe; border-color: #93c5fd; color: #1e40af; }
.doc-chip.cook  { background: #dcfce7; border-color: #86efac; color: #166534; }
.doc-chip.active { border-width: 2.5px; font-weight: 700; }
.canvas-wrap { position: relative; }
canvas { display: block; background: #f8fafc; border: 1px solid #cbd5e1; border-radius: 8px; max-width: 100%; }
.axis-label { position: absolute; font-size: .7rem; color: #64748b; }
.x-label { bottom: 6px; right: 10px; }
.y-label { top: 6px; left: 10px; }
.status { font-size: .95rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; }
.status.ok { color: #0a7d33; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem;
         border: 1px solid #1d3557; background: #1d3557; color: #fff;
         border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.tooltip { position: fixed; background: #1e293b; color: #f1f5f9; font-size: .78rem;
           padding: .3rem .55rem; border-radius: 6px; pointer-events: none;
           display: none; white-space: pre; z-index: 99; line-height: 1.5; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// LSA via truncated SVD (2 components, power iteration) — educational approximation.

// Six documents, two hidden topics: space (blue) and cooking (green).
const DOCS = [
  { id: 0, label: '1 · Rocket launch',   topic: 'space', words: ['rocket','orbit','nasa','launch','satellite'] },
  { id: 1, label: '2 · Planet orbit',   topic: 'space', words: ['planet','mars','orbit','telescope','nasa'] },
  { id: 2, label: '3 · Gravity & moon',  topic: 'space', words: ['gravity','orbit','satellite','moon','launch'] },
  { id: 3, label: '4 · Baking recipe',   topic: 'cook',  words: ['recipe','flour','bake','oven','butter'] },
  { id: 4, label: '5 · Pasta sauce',    topic: 'cook',  words: ['pasta','boil','sauce','flour','cook'] },
  { id: 5, label: '6 · Dessert',  topic: 'cook',  words: ['bake','butter','sugar','oven','recipe'] }
];

// Vocabulary = union of all words, sorted.
const VOCAB = [...new Set(DOCS.flatMap(d => d.words))].sort();

// Build the term-document matrix M[doc][word] = count.
function buildMatrix() {
  return DOCS.map(doc => VOCAB.map(w => doc.words.filter(x => x === w).length));
}

// Truncated SVD via power iteration: extract the dominant singular triplet (u, s, v).
// This is a simplified educational approximation, not full LAPACK SVD.
function powerIter(A, iters) {
  const m = A.length, n = A[0].length;
  let v = Array.from({length: n}, () => Math.random() - 0.5);
  const norm = x => Math.sqrt(x.reduce((s, vi) => s + vi * vi, 0));
  const scale = (x, f) => x.map(xi => xi * f);
  const matVec = (M, x) => M.map(row => row.reduce((s, mij, j) => s + mij * x[j], 0));
  const matTVec = (M, x) => M[0].map((_, j) => M.reduce((s, row, i) => s + row[j] * x[i], 0));
  v = scale(v, 1 / norm(v));
  for (let i = 0; i < iters; i++) {
    let u = matVec(A, v); const su = norm(u); u = scale(u, 1 / su);
    v = matTVec(A, u); const sv = norm(v); v = scale(v, 1 / sv);
  }
  const u = (() => { let u2 = matVec(A, v); const s = norm(u2); return { u: scale(u2, 1/s), s }; })();
  return { u: u.u, s: u.s, v };
}

function deflate(A, u, s, v) {
  // Deflate: remove the first component to expose the second.
  return A.map((row, i) => row.map((val, j) => val - s * u[i] * v[j]));
}

function lsa(matrix) {
  // Run 2-component LSA: two rounds of power iteration + deflation.
  const r1 = powerIter(matrix, 60);
  const A2 = deflate(matrix, r1.u, r1.s, r1.v);
  const r2 = powerIter(A2, 60);
  // Project each document onto components 1 and 2.
  return DOCS.map((_, i) => ({ x: r1.s * r1.u[i], y: r2.s * r2.u[i] }));
}

let points = null;
let activeDoc = null;

// Render document chips above the canvas.
function renderChips() {
  const row = document.getElementById('docs-row');
  row.innerHTML = '';
  DOCS.forEach(doc => {
    const chip = document.createElement('span');
    chip.className = 'doc-chip ' + doc.topic + (activeDoc === doc.id ? ' active' : '');
    chip.textContent = doc.label;
    chip.title = doc.words.join(', ');
    chip.addEventListener('click', () => {
      activeDoc = activeDoc === doc.id ? null : doc.id;
      renderChips();
      if (points) drawPlot();
      showInfo(doc.id);
    });
    row.appendChild(chip);
  });
}

function showInfo(id) {
  const doc = DOCS[id];
  const statusEl = document.getElementById('status');
  if (activeDoc === id) {
    statusEl.textContent = i18nFmt('%label% · %topic% · words: %words%', { label: doc.label, topic: doc.topic === 'space' ? 'Topic: Space exploration' : 'Topic: Cooking', words: doc.words.join(', ') });
    statusEl.className = 'status ok';
  }
}

function drawPlot() {
  const canvas = document.getElementById('plot');
  const ctx = canvas.getContext('2d');
  const W = canvas.width, H = canvas.height;
  ctx.clearRect(0, 0, W, H);

  if (!points) return;

  // Map latent coordinates to canvas pixels.
  const xs = points.map(p => p.x), ys = points.map(p => p.y);
  const xmin = Math.min(...xs), xmax = Math.max(...xs);
  const ymin = Math.min(...ys), ymax = Math.max(...ys);
  const pad = 40;
  const toX = x => pad + (x - xmin) / (xmax - xmin + 1e-9) * (W - 2 * pad);
  const toY = y => H - pad - (y - ymin) / (ymax - ymin + 1e-9) * (H - 2 * pad);

  // Draw faint crosshair axes.
  ctx.strokeStyle = '#cbd5e1'; ctx.lineWidth = 1;
  ctx.beginPath(); ctx.moveTo(pad, H / 2); ctx.lineTo(W - pad, H / 2); ctx.stroke();
  ctx.beginPath(); ctx.moveTo(W / 2, pad); ctx.lineTo(W / 2, H - pad); ctx.stroke();

  // Draw one bubble per document.
  points.forEach((p, i) => {
    const doc = DOCS[i];
    const cx = toX(p.x), cy = toY(p.y);
    const isActive = activeDoc === i;
    ctx.beginPath(); ctx.arc(cx, cy, isActive ? 13 : 9, 0, 2 * Math.PI);
    ctx.fillStyle = doc.topic === 'space' ? '#3b82f6' : '#22c55e';
    ctx.globalAlpha = isActive ? 1 : 0.7;
    ctx.fill();
    ctx.globalAlpha = 1;
    ctx.strokeStyle = isActive ? '#1e3a8a' : '#fff';
    ctx.lineWidth = isActive ? 2.5 : 1.5;
    ctx.stroke();
    ctx.fillStyle = '#fff';
    ctx.font = `${isActive ? 700 : 600} ${isActive ? 11 : 9}px system-ui`;
    ctx.textAlign = 'center'; ctx.textBaseline = 'middle';
    ctx.fillText(i + 1, cx, cy);
  });
}

function run() {
  const matrix = buildMatrix();
  points = lsa(matrix);
  drawPlot();
  document.getElementById('status').textContent = i18nFmt('LSA done — %n%-word vocabulary projected to %k% dimensions.', { n: VOCAB.length, k: 2 });
  document.getElementById('status').className = 'status ok';
}

function reset() {
  points = null;
  activeDoc = null;
  renderChips();
  const ctx = document.getElementById('plot').getContext('2d');
  ctx.clearRect(0, 0, 420, 260);
  document.getElementById('status').textContent = 'Click Run LSA to start.';
  document.getElementById('status').className = 'status';
}

document.getElementById('run').addEventListener('click', run);
document.getElementById('reset').addEventListener('click', reset);

// Show word list on hover over a bubble.
document.getElementById('plot').addEventListener('mousemove', function(e) {
  const tip = document.getElementById('tooltip');
  if (!points) { tip.style.display = 'none'; return; }
  const rect = this.getBoundingClientRect();
  const mx = e.clientX - rect.left, my = e.clientY - rect.top;
  const W = this.width, H = this.height, pad = 40;
  const xs = points.map(p => p.x), ys = points.map(p => p.y);
  const xmin = Math.min(...xs), xmax = Math.max(...xs);
  const ymin = Math.min(...ys), ymax = Math.max(...ys);
  const scaleX = (W - 2 * pad) / (xmax - xmin + 1e-9);
  const scaleY = (H - 2 * pad) / (ymax - ymin + 1e-9);
  let hit = null, best = 18;
  points.forEach((p, i) => {
    const cx = pad + (p.x - xmin) * scaleX;
    const cy = H - pad - (p.y - ymin) * scaleY;
    const dist = Math.hypot(mx - cx, my - cy);
    if (dist < best) { best = dist; hit = i; }
  });
  if (hit !== null) {
    const doc = DOCS[hit];
    tip.style.display = 'block';
    tip.style.left = (e.clientX + 12) + 'px';
    tip.style.top = (e.clientY - 8) + 'px';
    tip.textContent = doc.label + '\n' + (doc.topic === 'space' ? 'Topic: Space exploration' : 'Topic: Cooking') + '\n' + doc.words.join(', ');
  } else {
    tip.style.display = 'none';
  }
});
document.getElementById('plot').addEventListener('mouseleave', () => {
  document.getElementById('tooltip').style.display = 'none';
});

renderChips();
Result
Console

Click Run LSA to factorize the term-document matrix with a hand-computed SVD approximation and project every document onto the top two latent dimensions. Then click any document bubble to see its dominant topic. Notice that documents about the same theme land near each other in the concept space, even when they share no words at all — that is the power of latent structure.

The Real Complexity

Unlike many problems on this site, LSA is solved in the sense that its core computation is efficient and well-understood.

  • Building the matrix takes O(N⋅d)O(N \cdot d) time, where NN is the vocabulary size and dd is the number of documents. In practice both can be in the millions, but the matrix is extremely sparse.
  • Full SVD of an m×nm \times n matrix costs O(m⋅n⋅min⁡(m,n))O(m \cdot n \cdot \min(m, n)). For a vocabulary of 100,000 words and a corpus of 1,000,000 documents that is plainly infeasible.
  • Truncated SVD (keeping only kk singular values) can be computed in O(m⋅n⋅k)O(m \cdot n \cdot k) using randomized algorithms, making LSA practical. A typical value is k=300k = 300.
  • The hard question is choosing kk. Too small and unrelated concepts bleed together; too large and you overfit, capturing noise instead of meaning. No formula decides kk — practitioners cross-validate or use heuristics.

A deeper subtlety: LSA assumes that meaning is a linear structure in word-occurrence space. This is a strong approximation. Words are not independent, word order matters, and negation ("not fast") collapses to the same bag-of-words as "fast". These limits motivated later models such as probabilistic topic models (LDA) and, eventually, neural word embeddings like Word2Vec and neural network training. Yet for many retrieval tasks a 300-dimensional LSA space still competes with far more complex approaches — a testament to how much structure hides in simple co-occurrence counts.

Where It Matters

LSA's ability to group semantically related items without hand-crafted rules made it one of the first practical wins for unsupervised machine learning on language:

  • Information retrieval: a query about "automobile fuel economy" returns articles that mention "car gas mileage" even without shared keywords, because both project close together in latent space.
  • Plagiarism and near-duplicate detection: two reworded passages land near each other in concept space, flagging potential plagiarism even after synonym substitution.
  • Automatic essay scoring: the Educational Testing Service deployed LSA in the 1990s to grade short answers by comparing student essays with high-scoring exemplars in latent space.
  • Word sense disambiguation: the context documents around an ambiguous word vote on which latent direction it occupies, helping resolve meanings.
  • Precursor to modern embeddings: Word2Vec, GloVe, and today's transformer-based contextual embeddings all grew from the same insight — that co-occurrence encodes meaning. LSA is the direct ancestor of the dimensionality reduction pipeline now standard in NLP.

Understanding LSA means understanding why structure emerges from statistics, and that lesson runs through every modern language model.

Conclusion

Latent Semantic Analysis is a quiet triumph of applied linear algebra. By treating a corpus as a matrix and factoring it with SVD, it surfaces the hidden conceptual skeleton of a language — without dictionaries, ontologies, or labeled training data.

Its limits are real: linear geometry misses word order, negation, and polysemy. But those limits drove three decades of innovation, from probabilistic topic models to the transformer architectures that power today's large language models. Every time a search engine finds what you meant rather than what you typed, you are seeing LSA's intellectual descendants at work.

The deeper lesson is that meaning is a pattern, and patterns live in data. You don't need to define "concept" — you just need enough text, a clever factorization, and the patience to look at what the algebra reveals.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/latent-semantic-analysis/Content licensed under CC BY-NC 4.0.