Introduction

Every generative model faces the same challenge: how do you assign a probability to a data point? A photo, a sentence, a molecule — each lives in a high-dimensional space, and computing its exact probability is usually intractable. Transformers predict the next token but never claim to model the full joint distribution. Variational autoencoders optimize a lower bound. GANs skip likelihoods entirely.

Normalizing flows take a different path. Start with a simple distribution you can evaluate exactly — a standard Gaussian, say. Then apply a sequence of invertible, differentiable transformations to warp it into something that matches your data. Because every transformation is invertible, you can map any data point back to the simple space and read off its exact probability using the change-of-variables formula.

The word normalizing refers to this act of "normalizing" a complex density back into a simple (normal) one. The word flow describes the sequence of transformations through which probability mass flows and reshapes. Together, they form one of the few families of generative models that give you exact log-likelihoods — which means you can train directly by maximum likelihood and evaluate precisely how probable any sample is.

Warp a Density

The demo below shows the change-of-variables formula in action. We start with a 1-D standard Gaussian N(0,1)\mathcal{N}(0,1) — the base distribution — and apply a single affine (linear + shift) flow f(z)=az+bf(z) = az + b to it. The new distribution pX(x)p_X(x) is computed exactly:

pX(x)=pZ(f−1(x))⋅∣f−1′(x)∣=pZ ⁣(x−ba)⋅1∣a∣p_X(x) = p_Z(f^{-1}(x)) \cdot |f^{-1}{}'(x)| = p_Z\!\left(\frac{x-b}{a}\right) \cdot \frac{1}{|a|}

Adjust the scale aa and shift bb sliders and watch the density reshape in real time. Notice that scaling by a>1a > 1 stretches the bell curve wider and flattens it so the total area under the curve stays exactly 1.

<!-- Flow demo: affine transformation of a 1-D Gaussian with live density update -->
<div class="controls">
  <label>
    <span>Scale <em>a</em> = <output id="aVal">1.0</output></span>
    <input type="range" id="scaleSlider" min="0.2" max="4" step="0.05" value="1">
  </label>
  <label>
    <span>Shift <em>b</em> = <output id="bVal">0.0</output></span>
    <input type="range" id="shiftSlider" min="-3" max="3" step="0.1" value="0">
  </label>
</div>
<canvas id="chart" width="500" height="280"></canvas>
<div class="info" id="info"></div>
<div class="btns">
  <button id="btnReset" type="button" class="ghost">Reset</button>
</div>

/* Normalizing-flows demo styles */
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: system-ui, sans-serif; color: #222; }
.controls { display: flex; flex-direction: column; gap: .5rem; margin-bottom: .8rem; }
label { display: flex; flex-direction: column; gap: .2rem; font-size: .88rem; color: #444; }
label span { display: flex; gap: .4rem; align-items: baseline; }
output { font-weight: 700; color: #1d3557; min-width: 3ch; }
input[type=range] { width: 100%; accent-color: #1d3557; }
canvas { display: block; border: 1px solid #dde3ea; border-radius: 8px;
         max-width: 100%; background: #f8fafc; }
.info { font-size: .85rem; color: #555; margin-top: .5rem; min-height: 2.4em; line-height: 1.55; }
.btns { margin-top: .6rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem;
         border: 1px solid #1d3557; background: #1d3557; color: #fff;
         border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Normalizing flow demo: apply f(z) = a*z + b to a standard Gaussian.
// Change of variables: p_X(x) = p_Z((x-b)/a) * (1/|a|)
const canvas = document.getElementById('chart');
const ctx = canvas.getContext('2d');
const aSlider = document.getElementById('scaleSlider');
const bSlider = document.getElementById('shiftSlider');
const aOut = document.getElementById('aVal');
const bOut = document.getElementById('bVal');
const info = document.getElementById('info');

// Standard Gaussian density N(0,1)
function gaussBase(z) {
  return Math.exp(-0.5 * z * z) / Math.sqrt(2 * Math.PI);
}

// Transformed density via change of variables.
// p_X(x) = (1/|a|) * phi((x-b)/a) where phi = standard Gaussian pdf
function transformedDensity(x, a, b) {
  const z = (x - b) / a;
  return gaussBase(z) / Math.abs(a);
}

const W = canvas.width;
const H = canvas.height;
const PAD = { top: 20, right: 20, bottom: 30, left: 44 };
const plotW = W - PAD.left - PAD.right;
const plotH = H - PAD.top - PAD.bottom;

// Canvas coordinate helpers — maps math coords to pixel coords
const X_MIN = -8, X_MAX = 8;
const Y_MAX = 0.85;

function toCanvasX(x) { return PAD.left + ((x - X_MIN) / (X_MAX - X_MIN)) * plotW; }
function toCanvasY(y) { return PAD.top + (1 - y / Y_MAX) * plotH; }

function drawGrid() {
  ctx.strokeStyle = '#dde3ea';
  ctx.lineWidth = 1;
  // Vertical grid lines every 2 units
  for (let xv = -8; xv <= 8; xv += 2) {
    const cx = toCanvasX(xv);
    ctx.beginPath(); ctx.moveTo(cx, PAD.top); ctx.lineTo(cx, PAD.top + plotH); ctx.stroke();
    ctx.fillStyle = '#999'; ctx.font = '11px system-ui'; ctx.textAlign = 'center';
    ctx.fillText(xv, cx, PAD.top + plotH + 14);
  }
  for (let yv = 0; yv <= Y_MAX; yv += 0.2) {
    const cy = toCanvasY(yv);
    ctx.beginPath(); ctx.moveTo(PAD.left, cy); ctx.lineTo(PAD.left + plotW, cy); ctx.stroke();
    ctx.fillStyle = '#999'; ctx.textAlign = 'right';
    ctx.fillText(yv.toFixed(1), PAD.left - 6, cy + 4);
  }
}

function drawCurve(a, b) {
  ctx.clearRect(0, 0, W, H);
  drawGrid();

  const N = 400;
  const step = (X_MAX - X_MIN) / N;

  // Base distribution N(0,1) shown as dashed gray curve
  ctx.beginPath();
  ctx.strokeStyle = '#adb1b8';
  ctx.lineWidth = 1.5;
  ctx.setLineDash([5, 4]);
  for (let i = 0; i <= N; i++) {
    const x = X_MIN + i * step;
    const y = gaussBase(x);
    const cx = toCanvasX(x); const cy = toCanvasY(y);
    i === 0 ? ctx.moveTo(cx, cy) : ctx.lineTo(cx, cy);
  }
  ctx.stroke();
  ctx.setLineDash([]);

  // Transformed (flow) distribution shown as solid blue curve
  ctx.beginPath();
  ctx.strokeStyle = '#1d3557';
  ctx.lineWidth = 2.4;
  for (let i = 0; i <= N; i++) {
    const x = X_MIN + i * step;
    const y = transformedDensity(x, a, b);
    const cx = toCanvasX(x); const cy = toCanvasY(y);
    i === 0 ? ctx.moveTo(cx, cy) : ctx.lineTo(cx, cy);
  }
  ctx.stroke();

  // Shaded area under the transformed density
  ctx.beginPath();
  ctx.fillStyle = 'rgba(29,53,87,0.08)';
  ctx.moveTo(toCanvasX(X_MIN), toCanvasY(0));
  for (let i = 0; i <= N; i++) {
    const x = X_MIN + i * step;
    const y = transformedDensity(x, a, b);
    ctx.lineTo(toCanvasX(x), toCanvasY(y));
  }
  ctx.lineTo(toCanvasX(X_MAX), toCanvasY(0));
  ctx.closePath(); ctx.fill();

  // Red dashed line at mean = b
  const meanX = toCanvasX(b);
  ctx.strokeStyle = '#e63946'; ctx.lineWidth = 1.5; ctx.setLineDash([4, 3]);
  ctx.beginPath(); ctx.moveTo(meanX, PAD.top); ctx.lineTo(meanX, PAD.top + plotH); ctx.stroke();
  ctx.setLineDash([]);
  ctx.fillStyle = '#e63946'; ctx.font = '12px system-ui'; ctx.textAlign = 'left';
  ctx.fillText('μ=' + b.toFixed(1), meanX + 4, PAD.top + 16);
}

function numericIntegral(a, b) {
  // Numeric integration to verify the density integrates to ~1
  const N = 2000;
  const step = (X_MAX - X_MIN) / N;
  let sum = 0;
  for (let i = 0; i < N; i++) {
    const x = X_MIN + (i + 0.5) * step;
    sum += transformedDensity(x, a, b) * step;
  }
  return sum;
}

function updateInfo(a, b) {
  const sigma = Math.abs(a).toFixed(3);
  const area = numericIntegral(a, b).toFixed(4);
  info.innerHTML =
    fmt('Formula: <b>p(x) = N((x − %b%) / %a%) / %abs%</b><br>Mean: <b>%b2%</b> &nbsp;|&nbsp; Std dev: <b>%sigma%</b> &nbsp;|&nbsp; Area under curve: <b>%area%</b>', { b: b.toFixed(1), a: a.toFixed(2), abs: Math.abs(a).toFixed(2), b2: b.toFixed(2), sigma: sigma, area: area });
}

function update() {
  const a = parseFloat(aSlider.value) || 1;
  const b = parseFloat(bSlider.value) || 0;
  aOut.textContent = a.toFixed(2);
  bOut.textContent = b.toFixed(1);
  drawCurve(a, b);
  updateInfo(a, b);
  // Log current parameters for debugging
  const peak = transformedDensity(b, a, b);
  console.log('scale a =', a.toFixed(3), 'shift b =', b.toFixed(3), 'peak density =', peak.toFixed(4));
}

aSlider.addEventListener('input', update);
bSlider.addEventListener('input', update);
document.getElementById('btnReset').addEventListener('click', () => {
  aSlider.value = 1; bSlider.value = 0; update();
});

update();
Result
Console

The 1/∣a∣1/|a| factor is the Jacobian — it compensates for how the transformation stretches or squishes the probability mass. This is the core bookkeeping that lets normalizing flows compute exact likelihoods without any approximation.

The Real Complexity

The change-of-variables formula seems clean — but in high dimensions it hides a computational trap.

A general invertible transformation f:Rd→Rdf: \mathbb{R}^d \to \mathbb{R}^d carries a Jacobian matrix JfJ_f of size d×dd \times d. Computing det⁡(Jf)\det(J_f) naively costs O(d3)O(d^3) — catastrophic for images where dd can be in the millions.

The central design challenge of normalizing flows is building invertible transformations whose Jacobian determinant is cheap. Several architectures solve this:

  • Coupling layers (RealNVP, 2016 — Dinh, Sohl-Dickstein, Bengio): split the input in two halves; pass one half unchanged, and use it to compute an element-wise affine transform of the other. The Jacobian is triangular, so its determinant is just a product of diagonal entries — O(d)O(d).
  • Autoregressive flows (MAF, IAF): each output dimension depends only on previous ones, giving a triangular Jacobian again.
  • 1×1 convolutions (Glow, 2018 — Kingma & Dhariwal): generalize channel permutations with a learned invertible matrix; determinant via LU decomposition in O(d)O(d).
  • Continuous flows (FFJORD, 2018): parameterize the transformation as an ODE; estimate the trace of the Jacobian (not the full determinant) using Hutchinson's estimator.

The constraint is strict: not every neural-network architecture can be a flow. The network must be exactly invertible, not just approximately, and the Jacobian determinant must be tractable. This distinguishes flows from variational autoencoders, which relax the exact-likelihood requirement via a bound.

Where It Matters

Exact log-likelihoods are a superpower in settings where approximations are dangerous:

  • Density estimation: because flows give exact probabilities, you can rank how likely each data point is — no approximation needed. Useful anywhere you need a calibrated confidence score.
  • Anomaly detection: an input with a very low likelihood under the trained flow is almost certainly out-of-distribution. This powers industrial quality-control and network-intrusion detection.
  • Molecular design (drug discovery): flows like GraphNVP and MoFlow generate molecules while computing exact likelihoods over molecular graphs, enabling Bayesian optimization of drug-like properties.
  • Speech synthesis: WaveGlow (NVIDIA, 2018) applies flows to mel-spectrograms to synthesize high-fidelity speech in real time with exact likelihoods.
  • Variational inference: flows parameterize richer approximate posteriors in Bayesian models, going far beyond the mean-field Gaussian of standard VI.
  • Simulation-based inference: in physics and cosmology, flows learn the posterior over simulation parameters when the likelihood itself is intractable.

Wherever a model needs to say "this sample has probability pp" and actually mean it — not a bound, not a discriminator score — normalizing flows are one of the few tools that deliver.

Conclusion

Normalizing flows make a simple promise: transform noise into data with maps you can invert, and the change-of-variables formula hands you exact probabilities for free. Everything else in the field — coupling layers, autoregressive designs, continuous-time ODEs — is engineering to make that Jacobian determinant cheap enough to compute at scale.

The price of exactness is architectural constraint. You cannot use arbitrary networks; you are limited to those where invertibility and tractable Jacobians coexist. That constraint shapes the entire design space of flows, much as the requirement for exact inference separates easy graphical models from hard ones.

Next time you hear "the model assigns probability pp to this sample," ask whether that is exact or an approximation. If the answer is exact, there is almost certainly a normalizing flow — or the same change-of-variables idea — somewhere in the machinery.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/normalizing-flows/Content licensed under CC BY-NC 4.0.