Conformal Prediction

Every prediction model gives you a number. A neural network says a patient's risk score is 0.73. A random forest says a house will sell for $312,000. But how confident should you be? And what does "confident" even mean when you only have a finite dataset?

Conformal prediction — developed by Vladimir Vovk, Alexander Gammerman, and Glenn Shafer in the late 1990s — solves this without assuming anything about the data distribution. Instead of a point estimate, it produces a prediction set (or interval) that is guaranteed to contain the true label at least a fraction $1 - \alpha$ of the time, for any $\alpha$ you choose.

The guarantee is not asymptotic, not approximate, not model-dependent. It holds for any model, any data distribution, in finite samples — as long as the calibration data and the new point are exchangeable (a mild assumption weaker than i.i.d.).

The key insight is beautifully simple: use a held-out calibration set to measure how "surprising" new predictions are compared to past errors, then set a threshold that controls how often you are wrong.

The demo below simulates a simple regression task. A model has already been trained; the calibration set shows residuals (how wrong the model was on held-out points). Drag the coverage slider to set your target — say 90% — and watch the prediction interval automatically expand or shrink so it covers that fraction of calibration points.

<!-- {{c_html_comment}} -->
<div class="controls">
  <label for="coverage">{{lbl_coverage}} <span id="cov-val">90</span>%</label>
  <input type="range" id="coverage" min="50" max="99" value="90" step="1">
</div>
<canvas id="chart" width="560" height="240"></canvas>
<div class="info-row">
  <span id="threshold-info"></span>
  <span id="coverage-info"></span>
</div>
<p class="hint-text">{{hint_text}}</p>

/* {{c_css_comment}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: .5rem; }
.controls { display: flex; align-items: center; gap: .8rem; margin-bottom: .5rem; font-size: .9rem; }
label { font-weight: 600; white-space: nowrap; }
input[type=range] { flex: 1; accent-color: #1d6fa6; }
canvas { display: block; width: 100%; height: auto; border: 1px solid #dce3ea; border-radius: 8px; background: #f7f9fb; }
.info-row { display: flex; gap: 1rem; flex-wrap: wrap; margin-top: .45rem; font-size: .88rem; font-weight: 600; }
#threshold-info { color: #1d6fa6; }
#coverage-info { color: #0a7d33; }
.hint-text { font-size: .82rem; color: #555; margin: .5rem 0 0; line-height: 1.45; }

// Code not found

Notice the asymmetry: a tight interval (low coverage) occasionally misses the true value; a wide interval (high coverage) is always honest but less informative. The interval width is not a design choice — it is forced by the data to deliver the promised guarantee.

How does conformal prediction deliver a guaranteed interval without knowing the data distribution? The argument fits on one page.

The setup. You have a trained model and a calibration set of $n$ labeled points $(x_1, y_1), \dots, (x_n, y_n)$ held out from training. For each calibration point, compute a nonconformity score $s_i$ — any measure of how wrong the model was; a common choice for regression is $s_i = |y_i - \hat{y}_i|$.

The threshold. Given a target error rate $\alpha$, let $\hat{q}$ be the $\lceil (1 - \alpha)(n + 1) \rceil / n$ empirical quantile of $\{s_1, \dots, s_n\}$. Then the prediction interval for a new point $x_{n+1}$ is:

$$\mathcal{C}(x_{n+1}) = \{ y : s(x_{n+1}, y) \le \hat{q} \}$$

The proof. If the $n + 1$ points are exchangeable (any permutation is equally likely), then the rank of $s_{n+1}$ among all $n + 1$ scores is uniform on $\{1, \dots, n+1\}$. A score below $\hat{q}$ means the new point is not an outlier, so:

$$P(y_{n+1} \in \mathcal{C}(x_{n+1})) \ge 1 - \alpha$$

That is the entire proof. No assumptions on the model, no assumptions on the distribution — just the combinatorial fact that a uniform rank falls below a quantile with the right probability.

• Marginal coverage is exact: the interval is wrong at most $\alpha$ of the time, averaged over data.
• Conditional coverage (the interval is right for this specific $x$) is harder and requires additional assumptions — an active research area.
• Efficiency (interval width) depends on the model quality: a better underlying model produces narrower intervals for the same coverage.

The technique connects to PAC learning: conformal prediction is one concrete way to turn a learning algorithm into a provably reliable predictor without the distribution-dependence that PAC bounds often carry.

"My model predicts X" is rarely enough. In high-stakes settings you need "and I am 95% confident the truth is within Y of X." Conformal prediction provides that rigorously:

• Medical diagnosis: a classifier wrapping a diagnostic model outputs a set of candidate diagnoses guaranteed to include the true one 95% of the time — safety-critical and auditable.
• Drug discovery: prediction intervals on molecular property estimates tell chemists which candidates are worth synthesizing before running expensive lab assays.
• Natural language processing: a large language model can be wrapped to produce sets of possible answers; any answer outside the set was flagged as unreliable with a formal guarantee.
• Anomaly detection: a point whose nonconformity score exceeds the calibration threshold is flagged as an outlier — the false-alarm rate is controlled by $\alpha$.
• Autonomous systems: a perception model outputs not "object at position X" but "object is in this bounding box with 99% coverage" — a direct input to a safety controller.

The technique pairs naturally with PAC learning theory and with Bayesian inference: where Bayesian credible intervals require a correct prior, conformal intervals need only exchangeability.

Conformal prediction achieves something that sounds impossible: it wraps any model with guaranteed coverage using only a calibration set and a single quantile — no distributional assumptions, no asymptotic hand-waving, no retraining.

The price is modest: you spend some labeled data on calibration, and you get marginal (not conditional) coverage. But that guarantee is real, finite-sample, and model-agnostic. For a field that has struggled to make neural networks say "I don't know" in a trustworthy way, conformal prediction is one of the clearest answers we have.

The next time a model hands you a confident prediction on a life-or-death decision, ask whether its confidence comes with a proof — or just a feeling. With conformal prediction, you can have the proof.