Introduction

Every prediction model gives you a number. A neural network says a patient's risk score is 0.73. A random forest says a house will sell for $312,000. But how confident should you be? And what does "confident" even mean when you only have a finite dataset?

Conformal prediction — developed by Vladimir Vovk, Alexander Gammerman, and Glenn Shafer in the late 1990s — solves this without assuming anything about the data distribution. Instead of a point estimate, it produces a prediction set (or interval) that is guaranteed to contain the true label at least a fraction 1−α1 - \alpha of the time, for any α\alpha you choose.

The guarantee is not asymptotic, not approximate, not model-dependent. It holds for any model, any data distribution, in finite samples — as long as the calibration data and the new point are exchangeable (a mild assumption weaker than i.i.d.).

The key insight is beautifully simple: use a held-out calibration set to measure how "surprising" new predictions are compared to past errors, then set a threshold that controls how often you are wrong.

Try It

The demo below simulates a simple regression task. A model has already been trained; the calibration set shows residuals (how wrong the model was on held-out points). Drag the coverage slider to set your target — say 90% — and watch the prediction interval automatically expand or shrink so it covers that fraction of calibration points.

<!-- Conformal prediction demo: calibration scores and adaptive threshold -->
<div class="controls">
  <label for="coverage">Coverage target: <span id="cov-val">90</span>%</label>
  <input type="range" id="coverage" min="50" max="99" value="90" step="1">
</div>
<canvas id="chart" width="560" height="240"></canvas>
<div class="info-row">
  <span id="threshold-info"></span>
  <span id="coverage-info"></span>
</div>
<p class="hint-text">Green dots fall within the prediction interval; red dots miss it. The threshold (blue line) is set by the calibration quantile to guarantee the chosen coverage.</p>

/* Styles for the conformal prediction coverage demo */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: .5rem; }
.controls { display: flex; align-items: center; gap: .8rem; margin-bottom: .5rem; font-size: .9rem; }
label { font-weight: 600; white-space: nowrap; }
input[type=range] { flex: 1; accent-color: #1d6fa6; }
canvas { display: block; width: 100%; height: auto; border: 1px solid #dce3ea; border-radius: 8px; background: #f7f9fb; }
.info-row { display: flex; gap: 1rem; flex-wrap: wrap; margin-top: .45rem; font-size: .88rem; font-weight: 600; }
#threshold-info { color: #1d6fa6; }
#coverage-info { color: #0a7d33; }
.hint-text { font-size: .82rem; color: #555; margin: .5rem 0 0; line-height: 1.45; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Conformal prediction: set the threshold to the (1-alpha)(n+1)/n quantile of calibration scores.

// Deterministic pseudo-random number generator for reproducible calibration scores
function seededRng(seed) {
  let s = seed;
  return () => { s = (s * 1664525 + 1013904223) & 0xffffffff; return (s >>> 0) / 4294967296; };
}
const rng = seededRng(42);

// Generate N_CALIB nonconformity scores (|y - y_hat| on a held-out calibration set)
const N_CALIB = 40;
const scores = Array.from({ length: N_CALIB }, () => {
  // Skewed scores to resemble real residuals: small values common, large values rare
  const base = rng() * 1.4 + 0.1;
  return base * base;
}).sort((a, b) => a - b);

const canvas = document.getElementById('chart');
const ctx = canvas.getContext('2d');
const W = canvas.width, H = canvas.height;
const PAD = { top: 20, right: 20, bottom: 30, left: 44 };
const innerW = W - PAD.left - PAD.right;
const innerH = H - PAD.top - PAD.bottom;

const maxScore = scores[scores.length - 1] * 1.1;

function scoreToY(s) {
  return PAD.top + innerH - (s / maxScore) * innerH;
}
function indexToX(i) {
  return PAD.left + (i / (N_CALIB - 1)) * innerW;
}

function getQuantile(alpha) {
  // Conformal quantile: index = ceil((1-alpha)*(n+1)) - 1 (1-indexed formula, converted to 0-indexed)
  const idx = Math.ceil((1 - alpha) * (N_CALIB + 1)) - 1;
  const clamped = Math.min(Math.max(idx, 0), N_CALIB - 1);
  return scores[clamped];
}

function draw(alpha) {
  const q = getQuantile(alpha);
  const coveredCount = scores.filter(s => s <= q).length;
  const actualCoverage = Math.round((coveredCount / N_CALIB) * 100);

  ctx.clearRect(0, 0, W, H);

  // Draw Y-axis gridlines and value labels
  ctx.strokeStyle = '#dce3ea';
  ctx.lineWidth = 1;
  for (let i = 0; i <= 4; i++) {
    const y = PAD.top + (i / 4) * innerH;
    ctx.beginPath(); ctx.moveTo(PAD.left, y); ctx.lineTo(PAD.left + innerW, y); ctx.stroke();
    const label = ((1 - i / 4) * maxScore).toFixed(2);
    ctx.fillStyle = '#888'; ctx.font = '10px system-ui'; ctx.textAlign = 'right';
    ctx.fillText(label, PAD.left - 4, y + 3.5);
  }

  // Draw the conformal threshold line
  const threshY = scoreToY(q);
  ctx.save();
  ctx.strokeStyle = '#1d6fa6';
  ctx.lineWidth = 2;
  ctx.setLineDash([6, 3]);
  ctx.beginPath(); ctx.moveTo(PAD.left, threshY); ctx.lineTo(PAD.left + innerW, threshY); ctx.stroke();
  ctx.restore();

  // Label the threshold line
  ctx.fillStyle = '#1d6fa6'; ctx.font = 'bold 11px system-ui'; ctx.textAlign = 'left';
  ctx.fillText(i18nFmt('q̂ = %q%', { q: q.toFixed(3) }), PAD.left + 4, threshY - 5);

  // Draw each calibration score; green = covered, red = missed
  scores.forEach((s, i) => {
    const x = indexToX(i);
    const y = scoreToY(s);
    const covered = s <= q;
    ctx.beginPath();
    ctx.arc(x, y, 5, 0, Math.PI * 2);
    ctx.fillStyle = covered ? '#0a7d33' : '#e63946';
    ctx.fill();
    ctx.strokeStyle = covered ? '#075c26' : '#c92f3c';
    ctx.lineWidth = 1;
    ctx.stroke();
  });

  // Draw axis borders
  ctx.strokeStyle = '#aaa'; ctx.lineWidth = 1;
  ctx.beginPath(); ctx.moveTo(PAD.left, PAD.top); ctx.lineTo(PAD.left, PAD.top + innerH); ctx.stroke();
  ctx.beginPath(); ctx.moveTo(PAD.left, PAD.top + innerH); ctx.lineTo(PAD.left + innerW, PAD.top + innerH); ctx.stroke();

  // Axis title labels
  ctx.fillStyle = '#666'; ctx.font = '11px system-ui'; ctx.textAlign = 'center';
  ctx.fillText('Calibration points (sorted by score)', PAD.left + innerW / 2, H - 4);
  ctx.save(); ctx.rotate(-Math.PI / 2);
  ctx.fillText('Nonconformity score', -(PAD.top + innerH / 2), 12);
  ctx.restore();

  // Update threshold and coverage info below the chart
  document.getElementById('threshold-info').textContent =
    i18nFmt('Threshold q̂ = %q%', { q: q.toFixed(3) });
  document.getElementById('coverage-info').textContent =
    i18nFmt('Empirical coverage: %pct%% (%n% / %total% points)', { pct: actualCoverage, n: coveredCount, total: N_CALIB });
}

const slider = document.getElementById('coverage');
const covVal = document.getElementById('cov-val');

slider.addEventListener('input', () => {
  const pct = parseInt(slider.value, 10);
  covVal.textContent = pct;
  draw((100 - pct) / 100);
});

// Initial draw at 90% coverage (alpha=0.10)
draw(0.10);
Result
Console

Notice the asymmetry: a tight interval (low coverage) occasionally misses the true value; a wide interval (high coverage) is always honest but less informative. The interval width is not a design choice — it is forced by the data to deliver the promised guarantee.

The Real Guarantee

How does conformal prediction deliver a guaranteed interval without knowing the data distribution? The argument fits on one page.

The setup. You have a trained model and a calibration set of nn labeled points (x1,y1),…,(xn,yn)(x_1, y_1), \dots, (x_n, y_n) held out from training. For each calibration point, compute a nonconformity score sis_i — any measure of how wrong the model was; a common choice for regression is si=∣yi−y^i∣s_i = |y_i - \hat{y}_i|.

The threshold. Given a target error rate α\alpha, let q^\hat{q} be the ⌈(1−α)(n+1)⌉/n\lceil (1 - \alpha)(n + 1) \rceil / n empirical quantile of {s1,…,sn}\{s_1, \dots, s_n\}. Then the prediction interval for a new point xn+1x_{n+1} is:

C(xn+1)={y:s(xn+1,y)≤q^}\mathcal{C}(x_{n+1}) = \{ y : s(x_{n+1}, y) \le \hat{q} \}

The proof. If the n+1n + 1 points are exchangeable (any permutation is equally likely), then the rank of sn+1s_{n+1} among all n+1n + 1 scores is uniform on {1,…,n+1}\{1, \dots, n+1\}. A score below q^\hat{q} means the new point is not an outlier, so:

P(yn+1∈C(xn+1))≥1−αP(y_{n+1} \in \mathcal{C}(x_{n+1})) \ge 1 - \alpha

That is the entire proof. No assumptions on the model, no assumptions on the distribution — just the combinatorial fact that a uniform rank falls below a quantile with the right probability.

  • Marginal coverage is exact: the interval is wrong at most α\alpha of the time, averaged over data.
  • Conditional coverage (the interval is right for this specific xx) is harder and requires additional assumptions — an active research area.
  • Efficiency (interval width) depends on the model quality: a better underlying model produces narrower intervals for the same coverage.

The technique connects to PAC learning: conformal prediction is one concrete way to turn a learning algorithm into a provably reliable predictor without the distribution-dependence that PAC bounds often carry.

Where It Matters

"My model predicts X" is rarely enough. In high-stakes settings you need "and I am 95% confident the truth is within Y of X." Conformal prediction provides that rigorously:

  • Medical diagnosis: a classifier wrapping a diagnostic model outputs a set of candidate diagnoses guaranteed to include the true one 95% of the time — safety-critical and auditable.
  • Drug discovery: prediction intervals on molecular property estimates tell chemists which candidates are worth synthesizing before running expensive lab assays.
  • Natural language processing: a large language model can be wrapped to produce sets of possible answers; any answer outside the set was flagged as unreliable with a formal guarantee.
  • Anomaly detection: a point whose nonconformity score exceeds the calibration threshold is flagged as an outlier — the false-alarm rate is controlled by α\alpha.
  • Autonomous systems: a perception model outputs not "object at position X" but "object is in this bounding box with 99% coverage" — a direct input to a safety controller.

The technique pairs naturally with PAC learning theory and with Bayesian inference: where Bayesian credible intervals require a correct prior, conformal intervals need only exchangeability.

Conclusion

Conformal prediction achieves something that sounds impossible: it wraps any model with guaranteed coverage using only a calibration set and a single quantile — no distributional assumptions, no asymptotic hand-waving, no retraining.

The price is modest: you spend some labeled data on calibration, and you get marginal (not conditional) coverage. But that guarantee is real, finite-sample, and model-agnostic. For a field that has struggled to make neural networks say "I don't know" in a trustworthy way, conformal prediction is one of the clearest answers we have.

The next time a model hands you a confident prediction on a life-or-death decision, ask whether its confidence comes with a proof — or just a feeling. With conformal prediction, you can have the proof.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/conformal-prediction/Content licensed under CC BY-NC 4.0.