Variational Autoencoders

Imagine you want a machine to not just memorize images of handwritten digits, but to invent new ones — credible digits that never existed. A plain autoencoder can compress data down to a small code and reconstruct it faithfully, but if you ask it to generate a new digit by picking a random code, you get noise. The latent space has holes and sharp cliffs; most points in it decode to garbage.

Variational Autoencoders (VAEs), introduced by Kingma and Welling in 2013, fix this with one elegant idea: instead of mapping each input to a single point in latent space, the encoder maps it to a probability distribution — a small Gaussian cloud centered somewhere, with a learnable spread. Decoding then means sampling a point from that cloud.

This soft encoding, combined with a penalty that pushes every cloud toward the origin, forces the latent space to be smooth and continuous. Nearby points decode to similar outputs, and the entire space is filled — no dead zones. That is what makes generation possible: pick any point, decode it, and you get something coherent.

The idea sits at the intersection of deep learning and Bayesian inference, and it has reshaped how we think about neural network training and representation learning.

The canvas below shows a 2D latent space. Two colored anchor points (A and B) each represent an encoded input. The slider walks a straight path between them, and at each step a small decoder network maps the latent coordinate to a synthetic output (a simple shape pattern).

<p class="hint">{{hint}}</p>
<div class="demo-wrap">
  <canvas id="latent" width="200" height="200"></canvas>
  <div class="right-panel">
    <div class="output-label">{{output_at}} <span id="t-label">t = 0.50</span></div>
    <canvas id="output" width="120" height="120"></canvas>
    <label class="slider-label">{{interpolation_label}} <span id="t-val">0.50</span></label>
    <input type="range" id="slider" min="0" max="100" value="50">
    <button id="reset" type="button" class="ghost">{{reset_btn}}</button>
  </div>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .85rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.demo-wrap { display: flex; gap: 16px; align-items: flex-start; flex-wrap: wrap; }
#latent { border: 1px solid #cdd9e3; border-radius: 8px; background: #f3f6f9; cursor: crosshair; touch-action: none; }
.right-panel { display: flex; flex-direction: column; gap: 8px; min-width: 130px; }
.output-label { font-size: .8rem; color: #555; }
#output { border: 1px solid #cdd9e3; border-radius: 8px; background: #fff; display: block; }
.slider-label { font-size: .8rem; color: #555; }
input[type=range] { width: 100%; accent-color: #1d3557; }
button { font: 600 13px system-ui; padding: .4rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

Drag the anchor points to different regions of the space and watch the generated output change smoothly. This is the core promise of a VAE: because the encoder was trained to produce overlapping Gaussian clouds — not isolated points — the entire space is filled with meaningful representations, and interpolation between any two codes produces coherent intermediate samples.

The genius of VAEs is how they turn an intractable Bayesian inference problem into a trainable neural network objective.

• The goal: learn a model p(x) of data x by introducing a latent variable z. We want the encoder to compute p(z|x) — the distribution over latent codes given an input.
• The problem: computing p(z|x) exactly requires integrating over all possible x, which is intractable for complex data.
• The solution — ELBO: instead of computing the true posterior, the encoder learns an approximation q(z|x), a Gaussian with learnable mean $\mu$ and variance $\sigma^2$. The network then maximizes the Evidence Lower BOund (ELBO):

  $$\mathrm{ELBO} = \mathbb{E}[\log p(x|z)] - \mathrm{KL}(q(z|x) \| p(z))$$

• Two terms, two jobs: the first term is the reconstruction loss — how well the decoder recreates the input from a sampled z. The second term is the KL divergence — a regularizer that pushes q(z|x) toward a standard Gaussian $\mathcal{N}(0,1)$, preventing the encoder from collapsing to a single point per input.
• The reparameterization trick: sampling $z \sim q(z|x)$ is not differentiable — you cannot backpropagate through a random draw. The trick rewrites $z = \mu + \sigma \cdot \varepsilon$ where $\varepsilon \sim \mathcal{N}(0,1)$ is sampled outside the computation graph. Now gradients flow through $\mu$ and $\sigma$ freely.
• Status: VAEs are a solved, well-understood framework — not a hard open problem, but a beautifully engineered intersection of variational Bayesian inference and neural network training.

The KL term is the price of a smooth latent space: it forces the encoder clouds to overlap rather than scatter arbitrarily, which is why interpolation works.

The smooth, generative latent space that VAEs produce turns out to be useful far beyond digit generation:

• Image synthesis: VAEs can generate novel faces, textures, and artwork by sampling from the latent space — a precursor to today's diffusion models.
• Drug discovery: molecules can be encoded into a continuous space, and gradient-based search finds new molecular structures with desired properties — a technique pioneered by Gómez-Bombarelli et al. (2018).
• Anomaly detection: an input that reconstructs poorly (high reconstruction loss) is likely an outlier; VAEs provide a principled score.
• Disentangled representations: variants like β-VAE (Higgins et al., 2017) strengthen the KL penalty to learn latent dimensions that each correspond to a single human-interpretable factor (shape, color, orientation).
• Semi-supervised learning: the latent structure learned from unlabeled data can dramatically reduce the labeled examples needed for classification.
• Data compression: VAEs offer a probabilistic view of lossy compression, connecting to information theory and the ideas behind compression algorithms.

Every application exploits the same property: the latent space is continuous, structured, and searchable — a geometry that plain autoencoders and discrete codebooks cannot provide.

Variational Autoencoders teach a counterintuitive lesson: uncertainty is not a bug, it is a feature. By forcing the encoder to commit to a distribution rather than a single point, VAEs create a latent space where every location carries meaning, transitions are smooth, and generation becomes as natural as decoding.

The two-term ELBO objective — reconstruction accuracy balanced against KL regularity — is one of the most elegant training objectives in modern machine learning. It connects Bayesian reasoning to gradient descent in a single differentiable formula.

Whether you are generating new drug candidates, detecting anomalies in industrial sensors, or simply marveling that a machine can interpolate smoothly between a "3" and a "7", the VAE's core idea remains the same: represent the world not as a point, but as a cloud. Navigate that cloud, and you can imagine things that have never existed — coherently, continuously, and on demand.