Normalizing Flows

Every generative model faces the same challenge: how do you assign a probability to a data point? A photo, a sentence, a molecule — each lives in a high-dimensional space, and computing its exact probability is usually intractable. Transformers predict the next token but never claim to model the full joint distribution. Variational autoencoders optimize a lower bound. GANs skip likelihoods entirely.

Normalizing flows take a different path. Start with a simple distribution you can evaluate exactly — a standard Gaussian, say. Then apply a sequence of invertible, differentiable transformations to warp it into something that matches your data. Because every transformation is invertible, you can map any data point back to the simple space and read off its exact probability using the change-of-variables formula.

The word normalizing refers to this act of "normalizing" a complex density back into a simple (normal) one. The word flow describes the sequence of transformations through which probability mass flows and reshapes. Together, they form one of the few families of generative models that give you exact log-likelihoods — which means you can train directly by maximum likelihood and evaluate precisely how probable any sample is.

The demo below shows the change-of-variables formula in action. We start with a 1-D standard Gaussian $\mathcal{N}(0,1)$ — the base distribution — and apply a single affine (linear + shift) flow $f(z) = az + b$ to it. The new distribution $p_X(x)$ is computed exactly:

$$p_X(x) = p_Z(f^{-1}(x)) \cdot |f^{-1}{}'(x)| = p_Z\!\left(\frac{x-b}{a}\right) \cdot \frac{1}{|a|}$$

Adjust the scale $a$ and shift $b$ sliders and watch the density reshape in real time. Notice that scaling by $a > 1$ stretches the bell curve wider and flattens it so the total area under the curve stays exactly 1.

<!-- {{c_html_comment}} -->
<div class="controls">
  <label>
    <span>{{lbl_scale}} <em>a</em> = <output id="aVal">1.0</output></span>
    <input type="range" id="scaleSlider" min="0.2" max="4" step="0.05" value="1">
  </label>
  <label>
    <span>{{lbl_shift}} <em>b</em> = <output id="bVal">0.0</output></span>
    <input type="range" id="shiftSlider" min="-3" max="3" step="0.1" value="0">
  </label>
</div>
<canvas id="chart" width="500" height="280"></canvas>
<div class="info" id="info"></div>
<div class="btns">
  <button id="btnReset" type="button" class="ghost">{{btn_reset}}</button>
</div>

/* {{c_css_comment}} */
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: system-ui, sans-serif; color: #222; }
.controls { display: flex; flex-direction: column; gap: .5rem; margin-bottom: .8rem; }
label { display: flex; flex-direction: column; gap: .2rem; font-size: .88rem; color: #444; }
label span { display: flex; gap: .4rem; align-items: baseline; }
output { font-weight: 700; color: #1d3557; min-width: 3ch; }
input[type=range] { width: 100%; accent-color: #1d3557; }
canvas { display: block; border: 1px solid #dde3ea; border-radius: 8px;
         max-width: 100%; background: #f8fafc; }
.info { font-size: .85rem; color: #555; margin-top: .5rem; min-height: 2.4em; line-height: 1.55; }
.btns { margin-top: .6rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem;
         border: 1px solid #1d3557; background: #1d3557; color: #fff;
         border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

The $1/|a|$ factor is the Jacobian — it compensates for how the transformation stretches or squishes the probability mass. This is the core bookkeeping that lets normalizing flows compute exact likelihoods without any approximation.

The change-of-variables formula seems clean — but in high dimensions it hides a computational trap.

A general invertible transformation $f: \mathbb{R}^d \to \mathbb{R}^d$ carries a Jacobian matrix $J_f$ of size $d \times d$. Computing $\det(J_f)$ naively costs $O(d^3)$ — catastrophic for images where $d$ can be in the millions.

The central design challenge of normalizing flows is building invertible transformations whose Jacobian determinant is cheap. Several architectures solve this:

• Coupling layers (RealNVP, 2016 — Dinh, Sohl-Dickstein, Bengio): split the input in two halves; pass one half unchanged, and use it to compute an element-wise affine transform of the other. The Jacobian is triangular, so its determinant is just a product of diagonal entries — $O(d)$.
• Autoregressive flows (MAF, IAF): each output dimension depends only on previous ones, giving a triangular Jacobian again.
• 1×1 convolutions (Glow, 2018 — Kingma & Dhariwal): generalize channel permutations with a learned invertible matrix; determinant via LU decomposition in $O(d)$.
• Continuous flows (FFJORD, 2018): parameterize the transformation as an ODE; estimate the trace of the Jacobian (not the full determinant) using Hutchinson's estimator.

The constraint is strict: not every neural-network architecture can be a flow. The network must be exactly invertible, not just approximately, and the Jacobian determinant must be tractable. This distinguishes flows from variational autoencoders, which relax the exact-likelihood requirement via a bound.

Exact log-likelihoods are a superpower in settings where approximations are dangerous:

• Density estimation: because flows give exact probabilities, you can rank how likely each data point is — no approximation needed. Useful anywhere you need a calibrated confidence score.
• Anomaly detection: an input with a very low likelihood under the trained flow is almost certainly out-of-distribution. This powers industrial quality-control and network-intrusion detection.
• Molecular design (drug discovery): flows like GraphNVP and MoFlow generate molecules while computing exact likelihoods over molecular graphs, enabling Bayesian optimization of drug-like properties.
• Speech synthesis: WaveGlow (NVIDIA, 2018) applies flows to mel-spectrograms to synthesize high-fidelity speech in real time with exact likelihoods.
• Variational inference: flows parameterize richer approximate posteriors in Bayesian models, going far beyond the mean-field Gaussian of standard VI.
• Simulation-based inference: in physics and cosmology, flows learn the posterior over simulation parameters when the likelihood itself is intractable.

Wherever a model needs to say "this sample has probability $p$" and actually mean it — not a bound, not a discriminator score — normalizing flows are one of the few tools that deliver.

Normalizing flows make a simple promise: transform noise into data with maps you can invert, and the change-of-variables formula hands you exact probabilities for free. Everything else in the field — coupling layers, autoregressive designs, continuous-time ODEs — is engineering to make that Jacobian determinant cheap enough to compute at scale.

The price of exactness is architectural constraint. You cannot use arbitrary networks; you are limited to those where invertibility and tractable Jacobians coexist. That constraint shapes the entire design space of flows, much as the requirement for exact inference separates easy graphical models from hard ones.

Next time you hear "the model assigns probability $p$ to this sample," ask whether that is exact or an approximation. If the answer is exact, there is almost certainly a normalizing flow — or the same change-of-variables idea — somewhere in the machinery.