Introduction

Physics gave us a powerful idea: systems drift toward their lowest energy state. A ball rolls downhill, a molecule settles into its lowest-energy conformation, a magnet aligns its spins. Energy-based models (EBMs) borrow exactly this intuition and hand it to a neural network.

The recipe is simple. Define an energy function Eθ(x)E_\theta(\mathbf{x}) that maps any data point x\mathbf{x} to a real number — a scalar "badness score." The corresponding probability is

pθ(x)=e−Eθ(x)Zθp_\theta(\mathbf{x}) = \frac{e^{-E_\theta(\mathbf{x})}}{Z_\theta}

where Zθ=∫e−Eθ(x)dxZ_\theta = \int e^{-E_\theta(\mathbf{x})} d\mathbf{x} is the partition function — a normalizing constant that makes everything sum to 1. This is the Boltzmann distribution, the same formula statistical mechanics uses for a gas in thermal equilibrium.

The catch: ZθZ_\theta is almost always intractable. Computing it would require integrating over the entire input space, which is exponentially large or even continuous and infinite-dimensional. EBMs sidestep this by never computing ZθZ_\theta directly. Instead they learn to push energy down on real data and push it up everywhere else — and they sample from the model using Markov chain Monte Carlo (MCMC), most elegantly via Langevin dynamics.

The result is a remarkably flexible family of models. Unlike neural network training pipelines that insist on an explicit likelihood, EBMs can impose arbitrary structure on the energy function and still define a valid (if unnormalized) probability distribution over any space.

Try It: Walk the Energy Landscape

Below is a 1-D energy landscape. The curve shows E(x)E(x); the low valleys are where probability mass pools. A particle (the dot) samples from this landscape using Langevin dynamics: at each step it takes a small gradient-descent step (downhill on the energy) and adds a tiny random kick (the "thermal noise" that lets it escape shallow traps and explore).

<!-- Layout: landscape selector + canvas + info row -->
<div class="controls">
  <span class="label">Landscape:</span>
  <div class="btn-group">
    <button id="btn-single" class="active" type="button">Single well</button>
    <button id="btn-double" type="button">Double well</button>
    <button id="btn-flat" type="button">Noisy flat</button>
  </div>
  <button id="btn-reset" class="ghost" type="button">Restart</button>
</div>
<canvas id="canvas" width="560" height="260"></canvas>
<div class="info-row">
  <span id="status-txt" class="status-txt"></span>
  <span id="step-counter" class="step-counter">Steps: 0</span>
</div>
<div class="hint-row">The red dot samples via Langevin dynamics: each step = gradient descent + Gaussian noise. Low-energy valleys attract probability mass — the particle spends most time there.</div>

/* Styles for the EBM Langevin sampling demo */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
canvas { display: block; border-radius: 10px; background: #f0f4f8;
         border: 1px solid #cdd9e3; width: 100%; max-width: 560px; height: auto; }
.controls { display: flex; align-items: center; gap: .5rem; flex-wrap: wrap;
            margin-bottom: .5rem; }
.label { font-size: .85rem; color: #555; }
.btn-group { display: flex; gap: 3px; }
button { font: 600 13px system-ui; padding: .35rem .75rem; border-radius: 7px;
         border: 1px solid #1d3557; background: #fff; color: #1d3557; cursor: pointer; }
button.active { background: #1d3557; color: #fff; }
button.ghost { background: #eef2f6; border-color: #b0bec5; color: #555; }
.info-row { display: flex; justify-content: space-between; align-items: center;
            margin: .4rem 0 .2rem; min-height: 1.5em; }
.status-txt { font-size: .9rem; font-weight: 600; color: #0a7d33; }
.step-counter { font-size: .8rem; color: #666; }
.hint-row { font-size: .82rem; color: #555; line-height: 1.45; }

// Energy-Based Model demo: Langevin dynamics sampling on a 1-D energy landscape.
// Langevin update: x_{t+1} = x_t - step_size * grad E(x_t) + noise
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const W = 560, H = 260;
const X_MIN = -4, X_MAX = 4; // domain of the 1-D energy landscape

// Three preset energy landscapes the user can switch between
const LANDSCAPES = {
  single: x => (x * x - 2) * (x * x - 2) * 0.3 - 0.5 * Math.exp(-x * x),
  double: x => 0.15 * (x * x - 3) * (x * x - 3) * (0.08 * x * x + 0.1),
  flat:   x => 0.1 * Math.sin(2.5 * x) + 0.05 * x * x
};

// Langevin hyperparameters (kept small for visual clarity)
const STEP_SIZE  = 0.04;  // gradient step size
const NOISE_STD  = 0.18;  // std of the Gaussian noise injection
const INTERVAL_MS = 30;   // animation frame interval in ms

let landscape = 'single'; // which preset is active
let x = 0;                // current particle position on the x-axis
let steps = 0;            // total Langevin steps taken
let trail = [];           // recent positions for the fading path
let animId = null;        // requestAnimationFrame handle

// Map real coordinates to canvas pixels
function xToCanvas(xv) { return (xv - X_MIN) / (X_MAX - X_MIN) * W; }
function yToCanvas(yv) { return H * 0.5 - yv * 38; }

// Evaluate the active energy function at xv
function E(xv) { return LANDSCAPES[landscape](xv); }

// Numerical gradient via central differences
function gradE(xv) {
  const h = 1e-4;
  return (E(xv + h) - E(xv - h)) / (2 * h);
}

// One Langevin dynamics step
// x_{t+1} = x_t - step * grad E(x_t) + sqrt(2*step) * N(0,1)
function langevinStep() {
  const noise = randn() * NOISE_STD;
  x = x - STEP_SIZE * gradE(x) - noise;
  // keep particle inside the visible domain
  if (x < X_MIN + 0.1) x = X_MIN + 0.1;
  if (x > X_MAX - 0.1) x = X_MAX - 0.1;
}

// Box-Muller transform for standard normal samples
function randn() {
  const u = 1 - Math.random(), v = Math.random();
  return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
}

// Render the canvas: energy curve, zero line, trail, particle
function draw() {
  ctx.clearRect(0, 0, W, H);

  // Draw the energy landscape curve
  ctx.beginPath();
  ctx.strokeStyle = '#1d3557';
  ctx.lineWidth = 2.5;
  for (let px = 0; px <= W; px++) {
    const xv = X_MIN + (px / W) * (X_MAX - X_MIN);
    const yv = E(xv);
    const cy = yToCanvas(yv);
    if (px === 0) ctx.moveTo(px, cy); else ctx.lineTo(px, cy);
  }
  ctx.stroke();

  // Dashed zero-energy reference line
  ctx.beginPath();
  ctx.strokeStyle = '#b0bec5';
  ctx.lineWidth = 1;
  ctx.setLineDash([4, 4]);
  ctx.moveTo(0, yToCanvas(0));
  ctx.lineTo(W, yToCanvas(0));
  ctx.stroke();
  ctx.setLineDash([]);

  // Faded trail of recent particle positions
  if (trail.length > 1) {
    ctx.beginPath();
    ctx.strokeStyle = 'rgba(230,57,70,0.25)';
    ctx.lineWidth = 1.2;
    for (let i = 0; i < trail.length; i++) {
      const cx = xToCanvas(trail[i]);
      const cy = yToCanvas(E(trail[i]));
      if (i === 0) ctx.moveTo(cx, cy); else ctx.lineTo(cx, cy);
    }
    ctx.stroke();
  }

  // Red dot for the current particle position
  const cx = xToCanvas(x);
  const cy = yToCanvas(E(x));
  ctx.beginPath();
  ctx.arc(cx, cy, 7, 0, 2 * Math.PI);
  ctx.fillStyle = '#e63946';
  ctx.fill();
  ctx.strokeStyle = '#fff';
  ctx.lineWidth = 2;
  ctx.stroke();

  // Axis labels
  ctx.font = '12px system-ui';
  ctx.fillStyle = '#555';
  ctx.fillText('E(x)', 8, 18);
  ctx.fillText('x', W - 14, yToCanvas(0) + 14);
}

// Main animation loop
function animate() {
  langevinStep();
  trail.push(x);
  if (trail.length > 120) trail.shift(); // keep only the last 120 positions
  steps++;
  document.getElementById('step-counter').textContent =
    document.getElementById('step-counter').textContent.replace(/\d+$/, steps);
  draw();
  animId = requestAnimationFrame(animate);
}

// Reset particle, trail and step counter
function reset() {
  if (animId) cancelAnimationFrame(animId);
  x = (Math.random() - 0.5) * 4; // random starting position in [-2, 2]
  steps = 0;
  trail = [];
  document.getElementById('step-counter').textContent =
    document.getElementById('step-counter').textContent.replace(/\d+$/, 0);
  document.getElementById('status-txt').textContent = '';
  draw();
  animId = requestAnimationFrame(animate);
}

// Wire up the landscape preset buttons
['single', 'double', 'flat'].forEach(id => {
  document.getElementById('btn-' + id).addEventListener('click', () => {
    landscape = id;
    document.querySelectorAll('.btn-group button').forEach(b => b.classList.remove('active'));
    document.getElementById('btn-' + id).classList.add('active');
    reset();
  });
});
document.getElementById('btn-reset').addEventListener('click', reset);

// Kick off the animation
reset();
Result
Console

Use the buttons to change the landscape and watch how the sampler behaves. Notice that the particle spends most of its time near the energy minima — that is exactly what p(x)∝e−E(x)p(x) \propto e^{-E(x)} predicts. A steep, narrow valley traps the chain; a wide, shallow valley is explored quickly. Multiple modes separated by high barriers can starve one mode entirely — the classic mixing problem that haunts all MCMC samplers.

The Real Complexity

EBMs look simple — just a scalar output — but training them is a subtle art.

The gradient of the log-likelihood is

∇θlog⁡pθ(x)=−∇θEθ(x)+Epθ[∇θEθ(x~)]\nabla_\theta \log p_\theta(\mathbf{x}) = -\nabla_\theta E_\theta(\mathbf{x}) + \mathbb{E}_{p_\theta}[\nabla_\theta E_\theta(\tilde{\mathbf{x}})]

The first term is easy: push the energy of real data down. The second term requires sampling from the model itself — the expectation under pθp_\theta, which involves ZθZ_\theta. This circular dependency is the core difficulty.

  • Contrastive Divergence (CD): run a short Markov chain initialized at a data point, then treat where it lands as a "negative sample." Fast, biased, but good enough to train Restricted Boltzmann Machines — a key ingredient in early deep networks (Hinton & Salakhutdinov, 2006).
  • Score Matching (Hyvärinen, 2005): avoid ZθZ_\theta entirely by matching the gradient of the log-density. Works beautifully on continuous data and is the theoretical backbone of modern diffusion models.
  • Noise-Contrastive Estimation (NCE): treat the EBM as a classifier between data and noise; cleverly estimates log⁡Zθ\log Z_\theta as an extra parameter.
  • MCMC mixing: even with a perfect gradient, if the energy landscape has well-separated modes, the Markov chain may never cross the high-energy barrier between them. A sampler can be trapped in one mode for millions of steps — metastability, the bane of EBM training on complex distributions.

The situation is not hopeless. Persistent Contrastive Divergence (PCD), parallel tempering, and modern short-run MCMC methods all help. And score matching completely avoids the mixing problem by never sampling from pθp_\theta at all. But the fundamental tension — a model that defines probability implicitly cannot be trained without sampling — has no fully satisfying solution, and this is an active research frontier.

Where It Matters

Energy-based models are not a curiosity — they are the conceptual core of some of the most powerful generative systems today:

  • Diffusion models (DALL·E, Stable Diffusion, Sora): the score network sθ(x,t)=−∇xEθ(x,t)s_\theta(\mathbf{x},t) = -\nabla_\mathbf{x} E_\theta(\mathbf{x},t) is literally the gradient of an energy function, trained via denoising score matching. Sampling is Langevin dynamics run backwards in time.
  • Restricted Boltzmann Machines: a bipartite EBM that powered the first wave of deep pre-training (Hinton & Salakhutdinov, 2006). Today superseded but foundational.
  • Molecular design: proteins and small molecules are scored by an energy function (physical or learned). Protein folding solvers use energy minimization at their core.
  • Structured prediction: when the output space is structured (parse trees, segmentation masks, 3-D poses), an EBM scores joint (input, output) pairs and inference is energy minimization.
  • Anomaly detection: a trained EBM assigns high energy to out-of-distribution inputs; anything with energy above a threshold is flagged as anomalous.

The unifying thread is the energy function as universal comparator: you don't need to enumerate all possibilities, you just need to say which ones are more or less plausible. That is a weaker requirement than a normalized probability, and it opens the door to richer, more expressive models.

Conclusion

Energy-based models offer a liberating trade-off: give up the normalizing constant, gain the freedom to shape the energy landscape however you like. The price is that sampling and training become genuinely hard problems — intractable partition functions, slow-mixing Markov chains, biased gradients.

Yet the core insight — probability is just upside-down energy — has proven irresistibly powerful. Score matching turned the intractability into an advantage, giving us diffusion models that generate images and audio with extraordinary fidelity. The particle you watched walk the landscape in the demo above is the same Langevin sampler that, scaled up and run in reverse, powers the best image generators in the world.

If you want to go deeper, the natural next steps are neural network training (for how gradients flow through the energy function) and non-convex optimization (for why energy minimization is so subtle when the landscape has many local minima).

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/energy-based-models/Content licensed under CC BY-NC 4.0.