Classifier-Free Guidance

Every time you type a prompt and a stunning image appears, a simple arithmetic trick is doing most of the work. It is called classifier-free guidance (CFG), and it was introduced by Jonathan Ho and Tim Salimans in 2022.

A diffusion model learns to reverse noise: given a noisy image, predict the noise that was added. Doing that conditioned on a text prompt should push the result toward your description — but in practice the model drifts. Without extra pressure it hedges, averaging over everything consistent with the words and producing flat, unconvincing samples.

CFG fixes this with a remarkably small change. At each denoising step the model runs twice: once with your prompt and once with an empty prompt (no condition at all). Then it extrapolates — pushes further in the direction the conditional score differs from the unconditional one:

$$\tilde{\epsilon}_\theta(x_t, c) = \epsilon_\theta(x_t) + w \cdot \bigl(\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t)\bigr)$$

where $w$ is the guidance scale. At $w = 1$ you get the raw conditional prediction. Push $w$ higher and the output becomes sharper and more faithful to the prompt — at the cost of less variety and eventually over-saturation.

That single number $w$ is the creative control sitting behind the slider in every major image generator today.

The simulation below runs a toy 1-D diffusion process over a mixture of two Gaussians (think of them as two "concepts" in the model's world). The target concept is the right-hand peak. Drag the guidance scale and press Sample to see where the denoised particles land.

<div class="hint">{{hint_para}}</div>
<div class="controls">
  <label for="guidance">{{lbl_guidance}} <span id="wval">1.0</span></label>
  <input type="range" id="guidance" min="0" max="20" step="0.5" value="1">
</div>
<div class="controls">
  <label for="nsamples">{{lbl_samples}} <span id="nsval">40</span></label>
  <input type="range" id="nsamples" min="5" max="80" step="5" value="40">
</div>
<canvas id="canvas" width="480" height="220"></canvas>
<div class="status" id="status"></div>
<div class="btns">
  <button id="btn-sample" type="button">{{btn_sample}}</button>
  <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.controls { display: flex; align-items: center; gap: .6rem; margin-bottom: .4rem; font-size: .9rem; }
.controls label { min-width: 160px; }
input[type=range] { flex: 1; accent-color: #1d3557; }
canvas { display: block; border: 1px solid #cdd9e3; border-radius: 8px;
         background: #f8fafc; width: 100%; max-width: 480px; margin: .5rem 0; }
.status { font-size: .95rem; font-weight: 600; min-height: 1.4em; margin: .3rem 0; }
.status.ok { color: #0a7d33; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem;
         border: 1px solid #1d3557; background: #1d3557; color: #fff;
         border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

At $w = 0$ (no guidance) samples scatter across both peaks — the model is unguided. As you raise $w$ the particles migrate to the target peak and cluster tightly. Push $w$ too high and they overshoot: the distribution collapses to a single spike and diversity vanishes. This is exactly the quality-diversity trade-off that practitioners tune when generating images.

The formula is short, but the reasoning behind it runs deep.

Score functions and Bayes' theorem. A diffusion model implicitly estimates the score $\nabla_{x_t} \log p(x_t)$ — the gradient pointing toward higher-probability images. A conditional model estimates $\nabla_{x_t} \log p(x_t \mid c)$. By Bayes' theorem:

$$\nabla_{x_t} \log p(x_t \mid c) = \nabla_{x_t} \log p(x_t) + \nabla_{x_t} \log p(c \mid x_t)$$

The second term is exactly what a classifier trained on noisy images would provide — pushing the score toward images that a classifier would label as class $c$. The original classifier guidance (Dhariwal & Nichol, 2021) literally trained such a classifier and added its gradient. The problem: you need a separate noise-robust classifier for every concept.

The classifier-free trick. Ho & Salimans realized you can implicitly compute $\nabla_{x_t} \log p(c \mid x_t)$ without any classifier: just subtract the unconditional score from the conditional score. Scale that difference by $w$ and add it back:

$$\tilde{\epsilon} = \epsilon_\theta(x_t) + w \cdot \bigl(\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t)\bigr)$$

This is equivalent to classifier guidance with a "virtual" classifier whose gradient is approximated by the model's own conditional-vs-unconditional gap.

The cost. Every step needs two forward passes instead of one — roughly doubling inference time. Practitioners commonly set $w$ between 7 and 15 for photorealistic models; going much higher produces artifacts (the model enters regions of the distribution it was never trained on).

Why it works so well. The guidance is applied at every denoising step, so the small nudge compounds: each step reinforces the previous ones and the trajectory bends steadily toward the target region. It is the same principle as gradient ascent — repeated small pushes in the right direction.

CFG is not a research curiosity — it is the workhorse behind nearly every modern generative system:

• Text-to-image: Stable Diffusion, DALL-E 3, Imagen, Flux — all use CFG or a close variant. The guidance scale is usually the first parameter a user is told to tweak.
• Text-to-video: Sora, RunwayML, and similar models extend the same trick to the temporal dimension. A high guidance scale keeps shots coherent to the prompt across frames.
• Audio generation: Diffusion-based audio models (MusicLM variants, AudioLDM) apply CFG on mel-spectrograms to steer music or speech toward a text description.
• Protein and molecule design: Protein folding researchers use guidance to bias diffusion trajectories toward structures with desired properties — binding sites, thermostability — without retraining the backbone model.
• Conditional scientific simulation: climate and fluid models trained as diffusion processes can be guided toward physically meaningful states using the same extrapolation.

The key insight — steer by extrapolating away from the unconstrained distribution — is so clean and general that it has been adopted far beyond image synthesis. Any generative model that can run both conditioned and unconditioned can gain controllability with CFG.

Classifier-free guidance is a beautiful example of a small idea with enormous reach. Two forward passes, a subtraction, a scalar — and a diffusion model suddenly listens to your prompt.

The deeper lesson is geometric: the unconditioned model knows the shape of the whole distribution, and the conditioned model knows the direction of your target. Extrapolating between them is just a way of saying "go further in the direction the condition points." Every image generator with a "CFG scale" slider is running that arithmetic thousands of times per second.

The trade-off it encodes — fidelity vs. diversity — is not a bug but a feature of how probability distributions work. Push $w$ to infinity and you get the most likely single sample, perfectly faithful but utterly brittle. Stay near $w = 1$ and you roam the full distribution, creative but unfocused. The art is in choosing $w$ wisely, which is why it remains one of the most-discussed numbers in neural network training and generative AI.