Variational Inference

Every time a machine learning model updates its beliefs from data, it faces a hidden problem: computing the posterior distribution. In principle, Bayes' theorem tells you exactly how to do it — multiply the prior by the likelihood and normalize. In practice, that normalization requires integrating over every possible setting of the model's parameters, and for any realistic model that integral is computationally intractable.

The naive answer is to sample from the posterior using Markov Chain Monte Carlo (MCMC). MCMC is exact in the limit, but it is also slow — it can take millions of steps to explore a high-dimensional space well enough to trust the result.

Variational inference (VI) takes a different route: instead of sampling, it optimizes. Pick a family of simple, tractable distributions — Gaussians, for instance. Find the member of that family that is closest to the true posterior, where "closest" is measured by KL divergence. The impossible integral becomes a loss function, and modern gradient descent can minimize it efficiently.

The tradeoff is honesty. Variational inference is fast but approximate: it finds the best simple distribution, not the true one. Understanding that tradeoff — and what "best" even means — is the heart of modern approximate Bayesian inference.

Below, a target distribution (shown in blue) represents the true posterior we want to approximate — imagine it as the real answer hidden behind an impossible integral. The variational distribution (shown in orange) is a Gaussian that we are free to move and reshape.

<p class="hint">{{hint}}</p>
<canvas id="c" width="480" height="200"></canvas>
<div class="info" id="info">{{kl_init}}</div>
<div class="btns">
  <button id="step">{{btn_step}}</button>
  <button id="run">{{btn_run}}</button>
  <button id="stop" disabled>{{btn_stop}}</button>
  <button id="reset" class="ghost">{{btn_reset}}</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
canvas { display: block; width: 100%; max-width: 480px; border: 1px solid #dde3ea; border-radius: 8px; background: #f7f9fb; }
.info { font-size: .92rem; font-weight: 600; margin: .4rem 0; color: #1d3557; min-height: 1.4em; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui; padding: .4rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .4; cursor: default; }

// Code not found

Click Step to perform one gradient-descent update, or Run to let it converge automatically. Watch the orange curve slide toward the blue one, minimizing the KL divergence at each step. Notice that the Gaussian can never perfectly capture a multi-modal target — that gap is the approximation error that variational inference always carries.

Why not just compute the posterior exactly?

• Exact inference is #P-hard in general. Even for discrete graphical models, computing the partition function (the normalization constant) is as hard as counting the solutions to a #P problem — a class believed to be even harder than NP. For continuous models the integral is typically not closed-form at all.
• MCMC is exact but slow. Markov Chain Monte Carlo converges to the right answer, but mixing in high dimensions can require exponentially many samples.
• Variational inference converts the problem. Instead of computing the intractable integral directly, VI maximizes the Evidence Lower BOund (ELBO) — a quantity that is always ≤ log p(data), and equals it when the approximation is perfect. Maximizing the ELBO is equivalent to minimizing KL(q ‖ p), which measures how much information is lost when you use q instead of the true posterior p.
• Mean-field VI assumes the variational family factorizes across latent variables — q(z) = ∏ᵢ $q_{i}(z_{i})$ — making the optimization tractable but ignoring posterior correlations.
• The bias is one-sided. KL(q ‖ p) penalizes q for putting mass where p has none, so mean-field posteriors tend to be overconfident and too narrow — an important failure mode in safety-critical applications.

Modern variants — structured VI, normalizing flows, amortized inference — push back the approximation frontier, but the fundamental tension between tractability and exactness remains.

Variational inference is the engine behind many of the most powerful modern machine learning systems:

• Variational Autoencoders (VAEs): the generative models behind image synthesis and data compression use VI to learn a latent space efficiently. The ELBO objective is trained end-to-end with backpropagation via the reparameterization trick (Kingma & Welling, 2013).
• Topic models (LDA): Latent Dirichlet Allocation — the technique that finds themes in document collections — was one of the first large-scale applications of mean-field VI (Blei, Ng & Jordan, 2003).
• Bayesian neural networks: instead of a single set of weights, a BNN maintains a distribution over weights. VI makes that tractable at scale, enabling uncertainty-aware predictions.
• Probabilistic programming: systems like Pyro, Stan and Edward let researchers write arbitrary probabilistic models and automatically apply VI for inference.
• Medical and scientific AI: wherever a model must communicate how confident it is — drug discovery, climate modeling, anomaly detection — VI provides calibrated uncertainty estimates at a cost that pure MCMC cannot match.

Understanding variational inference is the gateway to the modern probabilistic ML stack, sitting alongside Bayesian inference and dimensionality reduction as a core tool for reasoning under uncertainty.

Variational inference is one of the most pragmatic ideas in all of machine learning: when the exact answer is provably out of reach, turn the question into an optimization problem and settle for the best you can do.

The ELBO is the price tag of that pragmatism — a lower bound that tells you how far your approximation is from the truth. Tightening that bound, whether through richer variational families or smarter optimization, is an active research frontier.

But the core insight endures: hard probabilistic reasoning and optimization are the same thing, just looked at from different angles. Every time a deep generative model or a Bayesian neural network produces a calibrated uncertainty estimate, variational inference is doing the work behind the scenes — optimizing toward a truth it cannot fully reach.