Introduction

Suppose you have a million news articles and no labels. You want to know: what are they about? Which documents share a subject? Which words cluster together into coherent themes?

Latent Dirichlet Allocation (LDA), introduced in 2003 by David Blei, Andrew Ng, and Michael I. Jordan, answers that question without any human annotations. It is a generative probabilistic model: it proposes a simple story for how documents could have been written, then runs the story backwards to discover the hidden topics that best explain the words it observes.

The story goes like this. Before writing a document, an author picks a mixture of topics — say 70% politics, 20% economics, 10% sports. For every word, the author first picks a topic from that mixture, then picks a word that topic likes to use. The model encodes both mixtures using the Dirichlet distribution, a probability distribution over distributions, controlled by a single concentration parameter α\alpha (for document-topic mixes) and β\beta (for topic-word mixes).

The insight is that every document in a corpus was generated by this same latent machinery. LDA's task is inference: given only the words, recover the topic-word distributions ϕk\phi_k and the per-document topic proportions θd\theta_d. That inversion is what makes the problem both powerful and computationally interesting.

Related ideas appear in dimensionality reduction — LDA can be seen as finding a low-dimensional topic space — and in Bayesian inference, since the full model is a hierarchical Bayes network.

Try It: Topic Discovery

Below is a toy corpus of short documents covering two hidden topics: computing and nature. Press Run LDA to watch collapsed Gibbs sampling assign each word token to a topic, then converge. The colored bars show the top words per topic after sampling.

<!-- LDA demo — corpus display + topic word-weight bars -->
<p class="hint">Each document is a bag of colored words. Colors show which <b>topic</b> each word is currently assigned to (blue = computing, green = nature). Click <b>Run LDA</b> to let collapsed Gibbs sampling discover the hidden topics automatically.</p>
<div class="corpus" id="corpus"></div>
<div class="btns">
  <button id="btn-run" type="button">Run LDA</button>
  <button id="btn-step" type="button">Step</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
  <span class="iter-label">Iteration: <span id="iter-count">0</span></span>
</div>
<div class="status" id="status"></div>
<div class="topics-section">
  <div class="topic-block" id="topic0">
    <div class="topic-title topic0-color">Topic A (computing)</div>
    <div class="bars" id="bars0"></div>
  </div>
  <div class="topic-block" id="topic1">
    <div class="topic-title topic1-color">Topic B (nature)</div>
    <div class="bars" id="bars1"></div>
  </div>
</div>

/* Demo styles — tokens colored by topic assignment */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; font-size: 14px; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
.corpus { display: flex; flex-direction: column; gap: 4px; margin-bottom: .6rem; }
.doc { display: flex; flex-wrap: wrap; gap: 3px; align-items: center; padding: 4px 6px;
       background: #f0f3f6; border-radius: 6px; }
.doc-id { font-size: .75rem; color: #888; margin-right: 4px; min-width: 22px; }
/* Token colors reflect current Gibbs topic assignment */
.tok { display: inline-block; padding: 1px 5px; border-radius: 4px; font-size: .82rem;
       font-weight: 600; cursor: default; transition: background .25s; }
.tok.t0 { background: #bde0ff; color: #1a4a7a; }
.tok.t1 { background: #c8f5d0; color: #1a5c2a; }
.tok.unset { background: #ddd; color: #555; }
.btns { display: flex; gap: .4rem; flex-wrap: wrap; align-items: center; margin-bottom: .4rem; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .45; cursor: default; }
.iter-label { font-size: .83rem; color: #666; }
.status { font-size: .88rem; font-weight: 600; min-height: 1.2em; margin-bottom: .5rem; color: #1d3557; }
.topics-section { display: flex; gap: 12px; flex-wrap: wrap; }
.topic-block { flex: 1; min-width: 140px; background: #f8f9fa; border-radius: 8px; padding: 8px 10px; }
.topic-title { font-weight: 700; font-size: .85rem; margin-bottom: 6px; }
.topic0-color { color: #1a4a7a; }
.topic1-color { color: #1a5c2a; }
/* Bar chart for top words per topic */
.bar-row { display: flex; align-items: center; gap: 5px; margin-bottom: 3px; }
.bar-word { width: 64px; font-size: .78rem; text-align: right; color: #444; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.bar-bg { flex: 1; background: #e0e4e8; border-radius: 3px; height: 10px; }
.bar-fill { height: 10px; border-radius: 3px; transition: width .3s; }
.bar-fill.t0 { background: #4a90d9; }
.bar-fill.t1 { background: #34a853; }

// Collapsed Gibbs sampling for LDA with K=2 topics
const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));

// Toy corpus: 8 short documents mixing computing and nature vocabulary
const DOCS = [
  ['computer','algorithm','data','memory','processor','data'],
  ['tree','forest','river','mountain','animal','forest'],
  ['algorithm','network','computer','tree','data','river'],
  ['mountain','animal','river','forest','mountain','animal'],
  ['processor','memory','network','algorithm','computer','data'],
  ['forest','river','tree','mountain','animal','tree'],
  ['computer','network','processor','memory','algorithm','network'],
  ['river','animal','forest','mountain','tree','river']
];

// Vocabulary = sorted unique words across all documents
const VOCAB = [...new Set(DOCS.flat())].sort();
const K = 2; // number of topics
const ALPHA = 0.5; // document-topic Dirichlet concentration
const BETA = 0.1;  // topic-word Dirichlet concentration

// Sampling state arrays
let z = [];          // topic assignment per token z[d][n]
let ndz = [];        // count of tokens in doc d assigned to topic k
let nzw = [];        // count of word w assigned to topic k
let nz = [];         // total tokens assigned to topic k
let iterCount = 0;
let running = false;
let rafId = null;

function wordIdx(w) { return VOCAB.indexOf(w); }

// Initialize: assign each token a random topic and populate count tables
function init() {
  z = [];
  ndz = DOCS.map(() => new Array(K).fill(0));
  nzw = Array.from({length: K}, () => new Array(VOCAB.length).fill(0));
  nz = new Array(K).fill(0);
  iterCount = 0;
  document.getElementById('iter-count').textContent = '0';
  document.getElementById('status').textContent = '';
  for (let d = 0; d < DOCS.length; d++) {
    z[d] = [];
    for (let n = 0; n < DOCS[d].length; n++) {
      const topic = Math.floor(Math.random() * K);
      z[d][n] = topic;
      ndz[d][topic]++;
      nzw[topic][wordIdx(DOCS[d][n])]++;
      nz[topic]++;
    }
  }
}

// Collapsed Gibbs: remove current assignment, recompute conditional, draw new topic
function sampleTopic(d, n) {
  const w = wordIdx(DOCS[d][n]);
  const oldTopic = z[d][n];
  ndz[d][oldTopic]--;
  nzw[oldTopic][w]--;
  nz[oldTopic]--;
  const probs = new Array(K);
  for (let k = 0; k < K; k++) {
    probs[k] = (ndz[d][k] + ALPHA) * (nzw[k][w] + BETA) / (nz[k] + VOCAB.length * BETA);
  }
  // Sample from the unnormalized probability vector
  let sum = 0; for (let k = 0; k < K; k++) sum += probs[k];
  let r = Math.random() * sum;
  let newTopic = K - 1;
  for (let k = 0; k < K; k++) { r -= probs[k]; if (r <= 0) { newTopic = k; break; } }
  z[d][n] = newTopic;
  ndz[d][newTopic]++;
  nzw[newTopic][w]++;
  nz[newTopic]++;
}

// One full Gibbs sweep over all tokens
function sweep() {
  for (let d = 0; d < DOCS.length; d++)
    for (let n = 0; n < DOCS[d].length; n++)
      sampleTopic(d, n);
  iterCount++;
}

// Render the corpus with tokens colored by current topic assignment
function renderCorpus() {
  const el = document.getElementById('corpus');
  el.innerHTML = '';
  for (let d = 0; d < DOCS.length; d++) {
    const row = document.createElement('div');
    row.className = 'doc';
    const id = document.createElement('span');
    id.className = 'doc-id';
    id.textContent = 'D' + (d + 1);
    row.appendChild(id);
    for (let n = 0; n < DOCS[d].length; n++) {
      const span = document.createElement('span');
      const t = (z.length && z[d]) ? z[d][n] : -1;
      span.className = 'tok ' + (t >= 0 ? 't' + t : 'unset');
      span.textContent = DOCS[d][n];
      row.appendChild(span);
    }
    el.appendChild(row);
  }
}

// Render top-5 words per topic as a proportional bar chart
function renderBars() {
  for (let k = 0; k < K; k++) {
    const container = document.getElementById('bars' + k);
    container.innerHTML = '';
    const total = nz[k] + VOCAB.length * BETA;
    const scored = VOCAB.map((w, i) => ({ w, p: (nzw[k][i] + BETA) / total }));
    scored.sort((a, b) => b.p - a.p);
    const top = scored.slice(0, 5);
    const maxP = top[0] ? top[0].p : 1;
    for (const { w, p } of top) {
      const row = document.createElement('div');
      row.className = 'bar-row';
      const label = document.createElement('span');
      label.className = 'bar-word';
      label.textContent = w;
      const bg = document.createElement('div');
      bg.className = 'bar-bg';
      const fill = document.createElement('div');
      fill.className = 'bar-fill t' + k;
      fill.style.width = (100 * p / maxP).toFixed(1) + '%';
      bg.appendChild(fill);
      row.appendChild(label);
      row.appendChild(bg);
      container.appendChild(row);
    }
  }
}

function updateIterLabel() {
  document.getElementById('iter-count').textContent = iterCount;
}

function setStatus(msg) {
  document.getElementById('status').textContent = msg;
}

// Wire up button event listeners
document.getElementById('btn-step').addEventListener('click', () => {
  if (!z.length) init();
  sweep();
  renderCorpus();
  renderBars();
  updateIterLabel();
  setStatus(i18nFmt('Completed iteration %iter%.', { iter: iterCount }));
});

document.getElementById('btn-run').addEventListener('click', () => {
  if (running) { running = false; return; }
  if (!z.length) init();
  running = true;
  // Animate up to 60 iterations using requestAnimationFrame
  function loop() {
    if (!running) return;
    sweep();
    renderCorpus();
    renderBars();
    updateIterLabel();
    if (iterCount < 60) {
      rafId = requestAnimationFrame(loop);
    } else {
      running = false;
      setStatus(i18nFmt('Converged after %iter% iterations. Topics are stable.', { iter: iterCount }));
    }
  }
  loop();
});

document.getElementById('btn-reset').addEventListener('click', () => {
  running = false;
  if (rafId) cancelAnimationFrame(rafId);
  z = [];
  iterCount = 0;
  document.getElementById('iter-count').textContent = '0';
  document.getElementById('status').textContent = '';
  document.getElementById('bars0').innerHTML = '';
  document.getElementById('bars1').innerHTML = '';
  renderCorpus();
});

// Initialize on load and show an initial state
init();
renderCorpus();
renderBars();
setStatus('Ready — press Run LDA or Step to begin sampling.');
Result
Console

Notice how the algorithm separates words even though no document is purely about one topic. The mixing is the key: a document about a forest computer lab touches both themes, and LDA represents that as a genuine mixture rather than forcing a single label.

The Real Complexity

LDA looks deceptively simple on paper, but inference — computing the posterior over topics given the words — is #P-hard in general. The number of possible topic assignments for a corpus with NN words and KK topics is KNK^{N}, a space too vast to sum over exactly.

Two approximate families dominate in practice:

  • Variational EM (the method in the original Blei et al. paper): replace the true posterior with a family of factored distributions, then minimize the KL divergence by alternating between an E-step (update per-document parameters) and an M-step (update global topic-word parameters). Each full pass over the corpus costs O(N⋅K⋅V)O(N \cdot K \cdot V) where VV is the vocabulary size.
  • Collapsed Gibbs sampling: integrate out θ\theta and ϕ\phi analytically, reducing the sampling to just the topic assignment zd,nz_{d,n} for each word token. Each sample costs O(K)O(K) and the chain mixes well in practice, though proving convergence requires care.

Neither method is guaranteed to find the global optimum: the likelihood surface is non-convex, and both algorithms can settle into local modes. This connects LDA to the broader challenge of non-convex optimization and the counting complexity studied in #P.

The number of topics KK must be chosen in advance (or inferred via nonparametric extensions like the Hierarchical Dirichlet Process). Choosing KK is as much art as science: too few topics blur distinct themes; too many fragment coherent ones.

Where It Matters

LDA's influence stretches far beyond its original text-mining home:

  • Academic literature: tools like Google Scholar and Semantic Scholar use topic models to surface related papers and track how research themes evolve over decades.
  • Recommendation systems: a user's reading history is treated as a "document" of item identifiers; LDA recovers latent interest topics to drive personalized suggestions.
  • Bioinformatics: genetic variants and gene expression profiles play the role of "words"; topics correspond to biological pathways or cell types.
  • Social media analysis: tweet streams and reddit threads are modeled as short documents; LDA reveals trending topics, opinion clusters, and how narratives shift over time.
  • Digital humanities: historians use LDA to find thematic patterns across thousands of historical texts that no human team could read in full.

The core idea — that observations arise from mixtures of latent components — also underpins modern neural topic models and the attention mechanism in transformers, connecting LDA to the cutting edge of machine learning.

Conclusion

Latent Dirichlet Allocation proposes a beautifully simple fiction: documents are mixtures of topics, topics are mixtures of words, and the Dirichlet distribution ties everything together. Inverting that fiction — reading the hidden topics out of plain text — is computationally hard in theory but tractable in practice with variational or sampling methods.

Twenty years after its introduction, LDA remains one of the most cited papers in machine learning not because it is the most powerful model available, but because it gives us a legible story: you can look at the top words for each discovered topic and immediately understand what the algorithm found. In a world of black-box deep learning, that interpretability is still worth a great deal.

The next time you search for papers, get a recommendation, or see a news cluster, there is a good chance a descendant of LDA is quietly deciding what belongs together — and all it ever saw was a bag of words.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/latent-dirichlet-allocation/Content licensed under CC BY-NC 4.0.