Energy-Based Models

Physics gave us a powerful idea: systems drift toward their lowest energy state. A ball rolls downhill, a molecule settles into its lowest-energy conformation, a magnet aligns its spins. Energy-based models (EBMs) borrow exactly this intuition and hand it to a neural network.

The recipe is simple. Define an energy function $E_\theta(\mathbf{x})$ that maps any data point $\mathbf{x}$ to a real number — a scalar "badness score." The corresponding probability is

$$p_\theta(\mathbf{x}) = \frac{e^{-E_\theta(\mathbf{x})}}{Z_\theta}$$

where $Z_\theta = \int e^{-E_\theta(\mathbf{x})} d\mathbf{x}$ is the partition function — a normalizing constant that makes everything sum to 1. This is the Boltzmann distribution, the same formula statistical mechanics uses for a gas in thermal equilibrium.

The catch: $Z_\theta$ is almost always intractable. Computing it would require integrating over the entire input space, which is exponentially large or even continuous and infinite-dimensional. EBMs sidestep this by never computing $Z_\theta$ directly. Instead they learn to push energy down on real data and push it up everywhere else — and they sample from the model using Markov chain Monte Carlo (MCMC), most elegantly via Langevin dynamics.

The result is a remarkably flexible family of models. Unlike neural network training pipelines that insist on an explicit likelihood, EBMs can impose arbitrary structure on the energy function and still define a valid (if unnormalized) probability distribution over any space.

Below is a 1-D energy landscape. The curve shows $E(x)$; the low valleys are where probability mass pools. A particle (the dot) samples from this landscape using Langevin dynamics: at each step it takes a small gradient-descent step (downhill on the energy) and adds a tiny random kick (the "thermal noise" that lets it escape shallow traps and explore).

<!-- {{c_html_intro}} -->
<div class="controls">
  <span class="label">{{lbl_landscape}}</span>
  <div class="btn-group">
    <button id="btn-single" class="active" type="button">{{btn_single}}</button>
    <button id="btn-double" type="button">{{btn_double}}</button>
    <button id="btn-flat" type="button">{{btn_flat}}</button>
  </div>
  <button id="btn-reset" class="ghost" type="button">{{btn_reset}}</button>
</div>
<canvas id="canvas" width="560" height="260"></canvas>
<div class="info-row">
  <span id="status-txt" class="status-txt"></span>
  <span id="step-counter" class="step-counter">{{lbl_steps}} 0</span>
</div>
<div class="hint-row">{{hint_text}}</div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
canvas { display: block; border-radius: 10px; background: #f0f4f8;
         border: 1px solid #cdd9e3; width: 100%; max-width: 560px; height: auto; }
.controls { display: flex; align-items: center; gap: .5rem; flex-wrap: wrap;
            margin-bottom: .5rem; }
.label { font-size: .85rem; color: #555; }
.btn-group { display: flex; gap: 3px; }
button { font: 600 13px system-ui; padding: .35rem .75rem; border-radius: 7px;
         border: 1px solid #1d3557; background: #fff; color: #1d3557; cursor: pointer; }
button.active { background: #1d3557; color: #fff; }
button.ghost { background: #eef2f6; border-color: #b0bec5; color: #555; }
.info-row { display: flex; justify-content: space-between; align-items: center;
            margin: .4rem 0 .2rem; min-height: 1.5em; }
.status-txt { font-size: .9rem; font-weight: 600; color: #0a7d33; }
.step-counter { font-size: .8rem; color: #666; }
.hint-row { font-size: .82rem; color: #555; line-height: 1.45; }

// Code not found

Use the buttons to change the landscape and watch how the sampler behaves. Notice that the particle spends most of its time near the energy minima — that is exactly what $p(x) \propto e^{-E(x)}$ predicts. A steep, narrow valley traps the chain; a wide, shallow valley is explored quickly. Multiple modes separated by high barriers can starve one mode entirely — the classic mixing problem that haunts all MCMC samplers.

EBMs look simple — just a scalar output — but training them is a subtle art.

The gradient of the log-likelihood is

$$\nabla_\theta \log p_\theta(\mathbf{x}) = -\nabla_\theta E_\theta(\mathbf{x}) + \mathbb{E}_{p_\theta}[\nabla_\theta E_\theta(\tilde{\mathbf{x}})]$$

The first term is easy: push the energy of real data down. The second term requires sampling from the model itself — the expectation under $p_\theta$, which involves $Z_\theta$. This circular dependency is the core difficulty.

• Contrastive Divergence (CD): run a short Markov chain initialized at a data point, then treat where it lands as a "negative sample." Fast, biased, but good enough to train Restricted Boltzmann Machines — a key ingredient in early deep networks (Hinton & Salakhutdinov, 2006).
• Score Matching (Hyvärinen, 2005): avoid $Z_\theta$ entirely by matching the gradient of the log-density. Works beautifully on continuous data and is the theoretical backbone of modern diffusion models.
• Noise-Contrastive Estimation (NCE): treat the EBM as a classifier between data and noise; cleverly estimates $\log Z_\theta$ as an extra parameter.
• MCMC mixing: even with a perfect gradient, if the energy landscape has well-separated modes, the Markov chain may never cross the high-energy barrier between them. A sampler can be trapped in one mode for millions of steps — metastability, the bane of EBM training on complex distributions.

The situation is not hopeless. Persistent Contrastive Divergence (PCD), parallel tempering, and modern short-run MCMC methods all help. And score matching completely avoids the mixing problem by never sampling from $p_\theta$ at all. But the fundamental tension — a model that defines probability implicitly cannot be trained without sampling — has no fully satisfying solution, and this is an active research frontier.

Energy-based models are not a curiosity — they are the conceptual core of some of the most powerful generative systems today:

• Diffusion models (DALL·E, Stable Diffusion, Sora): the score network $s_\theta(\mathbf{x},t) = -\nabla_\mathbf{x} E_\theta(\mathbf{x},t)$ is literally the gradient of an energy function, trained via denoising score matching. Sampling is Langevin dynamics run backwards in time.
• Restricted Boltzmann Machines: a bipartite EBM that powered the first wave of deep pre-training (Hinton & Salakhutdinov, 2006). Today superseded but foundational.
• Molecular design: proteins and small molecules are scored by an energy function (physical or learned). Protein folding solvers use energy minimization at their core.
• Structured prediction: when the output space is structured (parse trees, segmentation masks, 3-D poses), an EBM scores joint (input, output) pairs and inference is energy minimization.
• Anomaly detection: a trained EBM assigns high energy to out-of-distribution inputs; anything with energy above a threshold is flagged as anomalous.

The unifying thread is the energy function as universal comparator: you don't need to enumerate all possibilities, you just need to say which ones are more or less plausible. That is a weaker requirement than a normalized probability, and it opens the door to richer, more expressive models.

Energy-based models offer a liberating trade-off: give up the normalizing constant, gain the freedom to shape the energy landscape however you like. The price is that sampling and training become genuinely hard problems — intractable partition functions, slow-mixing Markov chains, biased gradients.

Yet the core insight — probability is just upside-down energy — has proven irresistibly powerful. Score matching turned the intractability into an advantage, giving us diffusion models that generate images and audio with extraordinary fidelity. The particle you watched walk the landscape in the demo above is the same Langevin sampler that, scaled up and run in reverse, powers the best image generators in the world.

If you want to go deeper, the natural next steps are neural network training (for how gradients flow through the energy function) and non-convex optimization (for why energy minimization is so subtle when the landscape has many local minima).