Gaussian Processes

Most machine-learning models hand you a single number: "the temperature tomorrow will be 22 °C." A Gaussian process hands you something richer: "the temperature will be 22 °C — but I'm only ±1 °C confident near your training data, and ±8 °C out where you've given me nothing."

That extra information — the confidence band — is not decoration. It tells you where the model is trustworthy and where it is merely extrapolating, which is exactly what you need before acting on a prediction.

The idea traces back to Carl Friedrich Gauss and was formalized for spatial statistics by Danie Krige in the 1950s (the technique is still called kriging in geostatistics). The modern machine-learning treatment was consolidated by Carl Edward Rasmussen and Christopher Williams in their 2006 textbook Gaussian Processes for Machine Learning, now the standard reference.

At heart a GP is a prior over functions: instead of placing a distribution over parameters (intercept, slope, …), you place a distribution directly over the space of all smooth functions. Add data and the prior narrows into a posterior — a whole family of functions that are consistent with what you observed, together with how likely each one is.

Click anywhere on the canvas to add an observation. The blue curve is the GP posterior mean — the model's best guess for every x. The shaded band is ±2 standard deviations: the region the true function almost certainly passes through. Points outside existing data widen the band; points close together narrow it.

<p class="hint">{{hint}}</p>
<canvas id="gp" width="560" height="320"></canvas>
<div class="controls">
  <label>{{label_ls}} <input type="range" id="ls" min="5" max="120" value="40"> <span id="ls-val">40</span></label>
  <button id="clear" type="button" class="ghost">{{btn_clear}}</button>
</div>
<div class="status" id="status">{{status_initial}}</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
canvas { border: 1px solid #cdd9e3; border-radius: 10px; cursor: crosshair;
         max-width: 100%; display: block; background: #f8fafc; }
.controls { display: flex; align-items: center; gap: 1rem; flex-wrap: wrap;
            margin: .6rem 0 .3rem; font-size: .88rem; }
label { display: flex; align-items: center; gap: .4rem; }
input[type=range] { width: 90px; accent-color: #1d3557; }
.status { font-size: .9rem; font-weight: 600; min-height: 1.3em; color: #1d3557; }
button { font: 600 13px system-ui; padding: .35rem .75rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

Notice what happens far from your points: the band widens back to its prior width. The model admits ignorance instead of pretending confidence — a property that ordinary polynomial or neural-network regression does not have by default.

A Gaussian process is defined by two things: a mean function $m(x)$ (often taken to be zero) and a kernel (covariance function) $k(x, x')$ that encodes how similar two inputs are. The most common kernel is the squared-exponential (RBF):

$$k(x, x') = \sigma^2 \exp\!\left(-\frac{\|x - x'\|^2}{2\ell^2}\right)$$

Here $\sigma^2$ controls the overall variance and $\ell$ is the length-scale — how far apart two inputs must be before their outputs become nearly independent.

Given n training points, inference builds the n × n covariance matrix K and inverts it. That inversion costs $O(n^{3})$ time and $O(n^{2})$ memory — exact, but it means naive GPs hit a wall around n ≈ 10 000 points on a modern laptop.

• Why cubic? Gaussian elimination on an n × n matrix takes $n^{3}$ multiplications. There is no known way to sidestep this for a general dense matrix.
• Approximations rescue scale. Sparse GPs with m ≪ n inducing points reduce cost to $O(nm^{2})$ training and $O(m^{2})$ prediction. Random Fourier features (Rahimi & Recht, 2007) approximate the kernel with a low-rank expansion for even cheaper inference.
• The payoff is calibrated uncertainty. Unlike neural networks, a well-tuned GP posterior is provably calibrated under its kernel assumptions: the 95% band really does cover the true function 95% of the time (in expectation over the prior).

This places GP regression in a sweet spot: it is a solved, polynomial-time algorithm ($O(n^{3})$) for exact inference, but one whose cost makes it a research-active area for scaling to modern dataset sizes. See also Bayesian inference for the broader probabilistic framework that GPs inhabit.

Calibrated uncertainty is valuable wherever decisions have consequences:

• Bayesian optimization: finding the maximum of an expensive black-box function (tuning neural-network hyperparameters, designing experiments) uses a GP as a surrogate model. The uncertainty band tells the optimizer where to sample next — explore the wide band or exploit the high mean.
• Geostatistics (kriging): predicting mineral concentrations, rainfall, or pollution levels between sparse measurement stations. The original application that gave GPs their name in spatial statistics.
• Drug discovery and materials science: predicting molecular properties from limited lab assays; the GP band flags which candidates are worth testing next.
• Robotics and control: learning the dynamics of a robot arm from sensor data, with uncertainty bounds that feed directly into safe planning algorithms.
• Climate and weather modeling: emulating expensive climate simulations at untried parameter settings, with honest error bars on the emulator output.
• Active learning: any setting where labeling data is costly and you want to query the most informative point — the GP uncertainty tells you where you know the least.

The common thread: wherever you need a model that says "I don't know" rather than confidently extrapolating, a GP is a natural first choice. See also PAC learning for the contrast with worst-case learning theory.

A Gaussian process is a rare thing in machine learning: a model that is simultaneously exact (given its assumptions), interpretable (every parameter has a physical meaning), and honest (it widens its band where it should). The cost is cubic in training size, which is why sparse and approximate GPs are an active research area — but for small-to-medium datasets the exact version remains one of the most principled tools available.

The deeper lesson is philosophical. Most models give you a point estimate and hide their uncertainty in held-out metrics. A GP makes uncertainty a first-class citizen of the prediction itself. That shift — from "here is the answer" to "here is the answer, and here is what I don't know" — is exactly what Bayesian inference asks of every model.