Introduction

Modern GPUs are celebrated for running thousands of threads at once. A high-end GPU may launch a million threads for a single kernel — yet hiding behind that number is a strict organizational rule that shapes everything about how fast your code runs.

Threads on a GPU do not run independently. They are grouped into warps of exactly 32 threads (on NVIDIA hardware; AMD calls them wavefronts of 64). Every thread in a warp shares one instruction pointer and executes the same instruction at the same clock cycle — a model called SIMT (Single Instruction, Multiple Threads). Think of a warp as a row of 32 swimmers who must all stroke in unison: no one can stop while the others keep going.

This lockstep design makes GPUs extraordinarily efficient when all 32 threads take the same path. But the moment a branch — an if, a while, any conditional — causes some threads to go left and others to go right, the warp cannot split. Instead it must serialize: first disable the threads on the right and run the left path; then disable the left threads and run the right path. Parallelism collapses.

That collapse is called warp divergence, and it is one of the most important performance concepts in GPU programming.

Watch Divergence in Action

Below is a warp of 32 lanes. Each lane holds a value from 0 to 31. Use the threshold slider to set a branch condition — lanes whose value is below the threshold take the if-branch (shown in blue); the rest take the else-branch (shown in orange).

<!-- Warp divergence demo — 32 lanes, one branch, watch serialization happen -->
<div class="controls">
  <label for="threshold">Branch threshold: <span id="thresh-val">16</span></label>
  <input type="range" id="threshold" min="0" max="32" value="16" />
  <button id="btn-run" type="button">Run warp</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
</div>
<div class="legend">
  <span class="dot if-dot"></span> if-branch (lane < threshold)
  <span class="dot else-dot"></span> else-branch (lane >= threshold)
  <span class="dot idle-dot"></span> masked (idle)
</div>
<div id="warp-grid" class="warp-grid" title="Warp lanes colored by branch direction"></div>
<div id="passes" class="passes"></div>
<div id="summary" class="summary"></div>

/* Styles for the warp divergence demo */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: .5rem; }
.controls { display: flex; align-items: center; gap: .6rem; flex-wrap: wrap; margin-bottom: .5rem; }
label { font-size: .85rem; font-weight: 600; white-space: nowrap; }
input[type=range] { flex: 1; min-width: 120px; max-width: 240px; }
button { font: 600 13px system-ui; padding: .35rem .75rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.legend { display: flex; align-items: center; gap: .8rem; font-size: .8rem; margin-bottom: .4rem; flex-wrap: wrap; }
.dot { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 3px; vertical-align: middle; }
.if-dot { background: #3a86ff; }
.else-dot { background: #ff6b35; }
.idle-dot { background: #c9ccd1; }
/* Lane grid — 16 per row so all 32 fit in two compact rows */
.warp-grid { display: grid; grid-template-columns: repeat(16, 1fr); gap: 3px; margin-bottom: .6rem; }
.lane { height: 28px; border-radius: 5px; display: flex; align-items: center; justify-content: center;
        font: 700 10px ui-monospace, monospace; color: #fff; transition: background .2s; cursor: default; }
.lane.if-lane { background: #3a86ff; }
.lane.else-lane { background: #ff6b35; }
.lane.idle-lane { background: #c9ccd1; color: #555; }
/* Serial pass breakdown panels */
.passes { display: flex; flex-direction: column; gap: .4rem; margin-bottom: .5rem; }
.pass-row { border: 1px solid #cdd9e3; border-radius: 8px; padding: .35rem .55rem; }
.pass-title { font: 700 12px system-ui; margin-bottom: .25rem; }
.pass-grid { display: grid; grid-template-columns: repeat(16, 1fr); gap: 2px; }
.pl { height: 20px; border-radius: 3px; display: flex; align-items: center; justify-content: center;
      font: 600 9px ui-monospace, monospace; }
.pl.active { color: #fff; }
.pl.masked { background: #ebebeb; color: #bbb; }
.pl.if-active { background: #3a86ff; }
.pl.else-active { background: #ff6b35; }
.pass-stat { font-size: .78rem; margin-top: .2rem; color: #555; }
/* Summary message styling */
.summary { font-size: .9rem; font-weight: 600; min-height: 1.3em; }
.summary.good { color: #0a7d33; }
.summary.warn { color: #c07000; }

// GPU warp divergence simulator — SIMT serialization model
const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
const LANES = 32;

const sliderEl = document.getElementById('threshold');
const threshValEl = document.getElementById('thresh-val');
const gridEl = document.getElementById('warp-grid');
const passesEl = document.getElementById('passes');
const summaryEl = document.getElementById('summary');

// Current threshold and whether the simulation has run
let threshold = 16;
let simRan = false;

function buildGrid() {
  // Build the 32-lane visual with if/else coloring
  gridEl.innerHTML = '';
  for (let i = 0; i < LANES; i++) {
    const d = document.createElement('div');
    if (!simRan) {
      d.className = 'lane ' + (i < threshold ? 'if-lane' : 'else-lane');
    } else {
      d.className = 'lane ' + (i < threshold ? 'if-lane' : 'else-lane');
    }
    d.textContent = i;
    d.title = i18nFmt('Lane %id% — value: %val%', { id: i, val: i });
    gridEl.appendChild(d);
  }
}

function computePasses() {
  // Compute which lanes take each path
  const ifLanes = [];
  const elseLanes = [];
  for (let i = 0; i < LANES; i++) {
    if (i < threshold) ifLanes.push(i); else elseLanes.push(i);
  }
  const passes = [];
  if (ifLanes.length > 0) passes.push({ label: 'if-branch', active: new Set(ifLanes), cls: 'if-active' });
  if (elseLanes.length > 0) passes.push({ label: 'else-branch', active: new Set(elseLanes), cls: 'else-active' });
  return passes;
}

function renderPasses(passes) {
  // Render serial pass breakdown rows
  passesEl.innerHTML = '';
  passes.forEach((pass, pi) => {
    const row = document.createElement('div');
    row.className = 'pass-row';
    const title = document.createElement('div');
    title.className = 'pass-title';
    title.textContent = i18nFmt('Pass %n%: %label%', { n: pi + 1, label: pass.label });
    row.appendChild(title);
    const pg = document.createElement('div');
    pg.className = 'pass-grid';
    for (let i = 0; i < LANES; i++) {
      const pl = document.createElement('div');
      if (pass.active.has(i)) {
        pl.className = 'pl active ' + pass.cls;
      } else {
        pl.className = 'pl masked';
      }
      pl.textContent = i;
      pg.appendChild(pl);
    }
    row.appendChild(pg);
    const stat = document.createElement('div');
    stat.className = 'pass-stat';
    stat.textContent = i18nFmt('%active% / %total% lanes active', { active: pass.active.size, total: LANES });
    row.appendChild(stat);
    passesEl.appendChild(row);
  });
}

function renderSummary(passes) {
  // Show final divergence summary
  const k = passes.length;
  if (k === 0) {
    summaryEl.textContent = 'Move the slider and click Run warp.';
    summaryEl.className = 'summary';
    return;
  }
  if (k === 1) {
    summaryEl.textContent = i18nFmt('No divergence — all %active% lanes take the same path. One pass, full efficiency.', { active: LANES });
    summaryEl.className = 'summary good';
  } else {
    const ifCount = threshold;
    const elseCount = LANES - threshold;
    summaryEl.textContent = i18nFmt('Warp diverged into %k% serial passes: %if_count% if-lanes, %else_count% else-lanes.', { k: k, if_count: ifCount, else_count: elseCount });
    summaryEl.className = 'summary warn';
  }
}

function runSim() {
  // Execute the simulated warp with current threshold
  simRan = true;
  buildGrid();
  const passes = computePasses();
  renderPasses(passes);
  renderSummary(passes);
}

function reset() {
  // Clear pass panels and return to initial state
  simRan = false;
  passesEl.innerHTML = '';
  summaryEl.textContent = '';
  summaryEl.className = 'summary';
  buildGrid();
}

sliderEl.addEventListener('input', () => {
  threshold = parseInt(sliderEl.value, 10);
  threshValEl.textContent = threshold;
  if (simRan) runSim(); else buildGrid();
});

document.getElementById('btn-run').addEventListener('click', runSim);
document.getElementById('btn-reset').addEventListener('click', reset);

// Initial render on page load
buildGrid();
Result
Console

When all lanes take the same path the warp stays perfectly parallel — one pass, full efficiency. The moment even one lane disagrees, the hardware must run two serial passes: first the if-lanes (else-lanes masked off), then the else-lanes (if-lanes masked off). The active lane count in each pass tells you exactly how much parallelism you have left. Maximizing it is what GPU programmers mean by "avoiding divergence."

The Real Cost

The numbers behind divergence are stark.

A warp of 32 lanes executes in passes — one pass per distinct path taken by at least one lane. In each pass, only the lanes on that path do useful work; the rest are masked (they consume time but produce nothing). If the branch splits the warp into kk non-empty groups, execution takes kk passes. Throughput for that branch region drops to 1k\frac{1}{k} of what it would be with no divergence.

The worst case is an if (lane_id == 0) that singles out exactly one lane: k=2k = 2, one pass for lane 0, one pass for lanes 1–31. Only 132\frac{1}{32} of the warp's capacity does productive work per pass on average — a 32× slowdown in that section. For a shader or kernel where such branches appear in a tight loop, the overall slowdown can be severe.

Divergence is not a bug — it is an architectural trade-off. SIMT hardware is simpler and more area-efficient than MIMD (Multiple Instruction, Multiple Data), where each thread has its own instruction pointer. The price is that irregular control flow is penalized. NVIDIA's Volta and Turing architectures introduced independent thread scheduling, which adds per-lane program counters and lets the scheduler reconverge warps more flexibly, but the fundamental cost of divergent paths remains.

Strategies to minimize divergence include sorting input data so that nearby threads make the same choice, restructuring loops to hoist conditionals outside, and using branchless programming tricks that replace if with arithmetic.

Where It Matters

Warp divergence is not an academic concern — it drives design decisions across the most demanding GPU workloads:

  • Ray tracing: rays scatter in different directions when they hit surfaces; adjacent rays quickly land in different materials and take wildly different shading paths. Managing divergence is one of the hardest engineering problems in real-time ray tracing.
  • Sparse neural networks: pruned models skip zero weights, but which weights are zero varies per thread. Without careful data layout, every sparse operation creates heavy divergence.
  • BVH traversal and spatial queries: tree traversal branches on bounding-box tests, and nearby threads can end up deep in different subtrees.
  • Physics and fluid simulation: particle systems often branch on particle type, phase, or collision state — a natural source of divergence in every time step.
  • Rasterization pipelines: fragment shaders diverge when pixels hit different surfaces, transparency layers, or early-out conditions.

Understanding warp divergence is the first step toward writing GPU code that actually saturates the hardware. It connects directly to scheduling — the GPU's warp scheduler is constantly choosing which of many resident warps to issue next, hiding latency by keeping the execution units fed even when some warps stall.

Conclusion

The GPU's promise of massive parallelism comes with a quiet asterisk: threads are not truly independent. They travel in warps of 32, locked in step, and any branch that makes them disagree extracts a serialization tax.

Warp divergence is not a flaw to be patched — it is the price of the SIMT model that makes GPUs so efficient in the common case. Every 32 lanes that always agree are essentially free; every branch that splits them costs you passes. The best GPU programmers learn to read their code through the lens of the warp scheduler: will these 32 threads agree, or am I about to pay the divergence toll?

Next time you reach for an if inside a kernel, pause. Check whether adjacent threads will make the same choice. If they will, your warp stays parallel. If they won't, you've just invited scheduling complexity into your silicon — and the hardware will dutifully serialize every last disagreement.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/gpu-warps/Content licensed under CC BY-NC 4.0.