Latent Dirichlet Allocation

Suppose you have a million news articles and no labels. You want to know: what are they about? Which documents share a subject? Which words cluster together into coherent themes?

Latent Dirichlet Allocation (LDA), introduced in 2003 by David Blei, Andrew Ng, and Michael I. Jordan, answers that question without any human annotations. It is a generative probabilistic model: it proposes a simple story for how documents could have been written, then runs the story backwards to discover the hidden topics that best explain the words it observes.

The story goes like this. Before writing a document, an author picks a mixture of topics — say 70% politics, 20% economics, 10% sports. For every word, the author first picks a topic from that mixture, then picks a word that topic likes to use. The model encodes both mixtures using the Dirichlet distribution, a probability distribution over distributions, controlled by a single concentration parameter $\alpha$ (for document-topic mixes) and $\beta$ (for topic-word mixes).

The insight is that every document in a corpus was generated by this same latent machinery. LDA's task is inference: given only the words, recover the topic-word distributions $\phi_k$ and the per-document topic proportions $\theta_d$. That inversion is what makes the problem both powerful and computationally interesting.

Related ideas appear in dimensionality reduction — LDA can be seen as finding a low-dimensional topic space — and in Bayesian inference, since the full model is a hierarchical Bayes network.

Below is a toy corpus of short documents covering two hidden topics: computing and nature. Press Run LDA to watch collapsed Gibbs sampling assign each word token to a topic, then converge. The colored bars show the top words per topic after sampling.

<!-- {{c_html_intro}} -->
<p class="hint">{{hint_para}}</p>
<div class="corpus" id="corpus"></div>
<div class="btns">
  <button id="btn-run" type="button">{{btn_run}}</button>
  <button id="btn-step" type="button">{{btn_step}}</button>
  <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
  <span class="iter-label">{{label_iter}} <span id="iter-count">0</span></span>
</div>
<div class="status" id="status"></div>
<div class="topics-section">
  <div class="topic-block" id="topic0">
    <div class="topic-title topic0-color">{{topic_a_label}}</div>
    <div class="bars" id="bars0"></div>
  </div>
  <div class="topic-block" id="topic1">
    <div class="topic-title topic1-color">{{topic_b_label}}</div>
    <div class="bars" id="bars1"></div>
  </div>
</div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; font-size: 14px; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
.corpus { display: flex; flex-direction: column; gap: 4px; margin-bottom: .6rem; }
.doc { display: flex; flex-wrap: wrap; gap: 3px; align-items: center; padding: 4px 6px;
       background: #f0f3f6; border-radius: 6px; }
.doc-id { font-size: .75rem; color: #888; margin-right: 4px; min-width: 22px; }
/* {{c_token_style}} */
.tok { display: inline-block; padding: 1px 5px; border-radius: 4px; font-size: .82rem;
       font-weight: 600; cursor: default; transition: background .25s; }
.tok.t0 { background: #bde0ff; color: #1a4a7a; }
.tok.t1 { background: #c8f5d0; color: #1a5c2a; }
.tok.unset { background: #ddd; color: #555; }
.btns { display: flex; gap: .4rem; flex-wrap: wrap; align-items: center; margin-bottom: .4rem; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .45; cursor: default; }
.iter-label { font-size: .83rem; color: #666; }
.status { font-size: .88rem; font-weight: 600; min-height: 1.2em; margin-bottom: .5rem; color: #1d3557; }
.topics-section { display: flex; gap: 12px; flex-wrap: wrap; }
.topic-block { flex: 1; min-width: 140px; background: #f8f9fa; border-radius: 8px; padding: 8px 10px; }
.topic-title { font-weight: 700; font-size: .85rem; margin-bottom: 6px; }
.topic0-color { color: #1a4a7a; }
.topic1-color { color: #1a5c2a; }
/* {{c_bar_style}} */
.bar-row { display: flex; align-items: center; gap: 5px; margin-bottom: 3px; }
.bar-word { width: 64px; font-size: .78rem; text-align: right; color: #444; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.bar-bg { flex: 1; background: #e0e4e8; border-radius: 3px; height: 10px; }
.bar-fill { height: 10px; border-radius: 3px; transition: width .3s; }
.bar-fill.t0 { background: #4a90d9; }
.bar-fill.t1 { background: #34a853; }

// Code not found

Notice how the algorithm separates words even though no document is purely about one topic. The mixing is the key: a document about a forest computer lab touches both themes, and LDA represents that as a genuine mixture rather than forcing a single label.

LDA looks deceptively simple on paper, but inference — computing the posterior over topics given the words — is #P-hard in general. The number of possible topic assignments for a corpus with $N$ words and $K$ topics is $K^{N}$, a space too vast to sum over exactly.

Two approximate families dominate in practice:

• Variational EM (the method in the original Blei et al. paper): replace the true posterior with a family of factored distributions, then minimize the KL divergence by alternating between an E-step (update per-document parameters) and an M-step (update global topic-word parameters). Each full pass over the corpus costs $O(N \cdot K \cdot V)$ where $V$ is the vocabulary size.
• Collapsed Gibbs sampling: integrate out $\theta$ and $\phi$ analytically, reducing the sampling to just the topic assignment $z_{d,n}$ for each word token. Each sample costs $O(K)$ and the chain mixes well in practice, though proving convergence requires care.

Neither method is guaranteed to find the global optimum: the likelihood surface is non-convex, and both algorithms can settle into local modes. This connects LDA to the broader challenge of non-convex optimization and the counting complexity studied in #P.

The number of topics $K$ must be chosen in advance (or inferred via nonparametric extensions like the Hierarchical Dirichlet Process). Choosing $K$ is as much art as science: too few topics blur distinct themes; too many fragment coherent ones.

LDA's influence stretches far beyond its original text-mining home:

• Academic literature: tools like Google Scholar and Semantic Scholar use topic models to surface related papers and track how research themes evolve over decades.
• Recommendation systems: a user's reading history is treated as a "document" of item identifiers; LDA recovers latent interest topics to drive personalized suggestions.
• Bioinformatics: genetic variants and gene expression profiles play the role of "words"; topics correspond to biological pathways or cell types.
• Social media analysis: tweet streams and reddit threads are modeled as short documents; LDA reveals trending topics, opinion clusters, and how narratives shift over time.
• Digital humanities: historians use LDA to find thematic patterns across thousands of historical texts that no human team could read in full.

The core idea — that observations arise from mixtures of latent components — also underpins modern neural topic models and the attention mechanism in transformers, connecting LDA to the cutting edge of machine learning.

Latent Dirichlet Allocation proposes a beautifully simple fiction: documents are mixtures of topics, topics are mixtures of words, and the Dirichlet distribution ties everything together. Inverting that fiction — reading the hidden topics out of plain text — is computationally hard in theory but tractable in practice with variational or sampling methods.

Twenty years after its introduction, LDA remains one of the most cited papers in machine learning not because it is the most powerful model available, but because it gives us a legible story: you can look at the top words for each discovered topic and immediately understand what the algorithm found. In a world of black-box deep learning, that interpretability is still worth a great deal.

The next time you search for papers, get a recommendation, or see a news cluster, there is a good chance a descendant of LDA is quietly deciding what belongs together — and all it ever saw was a bag of words.