Bayesian Optimization

Imagine you are training a neural network and you need to pick the right learning rate, the number of hidden layers, and the dropout probability. Each combination requires a full training run that can take hours — sometimes days — on expensive hardware. Trying every combination is simply impossible.

This is the setting Bayesian optimization was built for: find the best input to an expensive black-box function using as few evaluations as possible. The word black-box means you can measure the output for any input you choose, but you cannot see a formula or gradient — only the result.

The key idea is to be smart about where you look next. Instead of searching at random (which wastes evaluations) or following a gradient (which doesn't exist), Bayesian optimization builds a probabilistic model of the unknown function, uses that model to decide the most promising next point, evaluates the function there, then updates the model and repeats. Every evaluation makes the model sharper.

Introduced in its modern form by Jonas Mockus in the 1970s and popularized for machine learning by Snoek, Larochelle, and Adams (2012), Bayesian optimization is now the backbone of hyperparameter tuning systems like Google Vizier and Optuna.

The demo below runs Bayesian optimization on a noisy one-dimensional function. The blue curve is the true (hidden) function; the orange curve is the Gaussian process surrogate the algorithm has built from the evaluations so far. The green band shows uncertainty — wider means less explored.

<p class="hint">{{hint}}</p>
<canvas id="chart" width="560" height="220"></canvas>
<div class="info" id="info">{{press_to_start}}</div>
<div class="btns">
  <button id="btn-next" type="button">{{btn_next}}</button>
  <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
canvas { display: block; width: 100%; max-width: 560px; border: 1px solid #dde3ea; border-radius: 8px; background: #f8fafc; }
.info { font-size: .92rem; font-weight: 600; min-height: 1.4em; margin: .5rem 0; color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

Click Next evaluation to let the algorithm pick the most promising point (highest Expected Improvement) and sample the true function there. Watch how the orange surrogate gets closer to the blue truth as evaluations accumulate — and how the algorithm stops wasting effort in regions it has already mapped.

Bayesian optimization is not a hard open problem — it is a well-understood, practical algorithm. Its status: a solved heuristic with provable regret bounds (Srinivas et al., 2010).

Two costs dominate:

• Surrogate fitting: the standard Gaussian process (GP) stores an n × n covariance matrix and inverts it at every step. That inversion is $O(n^{3})$ in the number of past evaluations. In practice this caps usage at a few thousand points; beyond that, sparse or deep-learning surrogates are substituted.
• Acquisition function optimization: choosing the next point requires maximizing a cheap closed-form function (like Expected Improvement or Upper Confidence Bound). Because this inner loop is cheap and smooth, standard gradient methods handle it efficiently.

The total evaluation budget n is usually tiny (tens to a few hundred) because each real evaluation is expensive. So the $O(n^{3})$ GP cost is usually irrelevant compared to the cost of running the actual experiment.

Regret bounds give theoretical guarantees: with a squared-exponential kernel, the cumulative gap from the true optimum grows only as O(√(n log n)), meaning the algorithm converges, just not at an exponential rate.

Compare to alternatives: grid search wastes exponentially many evaluations in high dimensions; random search is often surprisingly competitive but has no memory; gradient descent needs a differentiable function. Bayesian optimization sits in the sweet spot when evaluations are costly and the budget is small.

Related ideas appear in non-convex optimization and the bandit framework used by recommendation systems.

Any domain where each experiment is expensive and the search space is moderate is a natural home for Bayesian optimization:

• Machine learning hyperparameters: learning rate, architecture depth, regularization strength. Systems like Google Vizier, Optuna, and Ax are built on it. A single well-placed run can beat a dozen randomly chosen ones.
• Drug discovery and molecular design: each lab synthesis is costly. BO guides which molecule to test next, steering toward desirable properties like binding affinity or solubility.
• Materials science and chemistry: finding alloy compositions or battery electrolyte formulas. Each physical experiment can take days; BO reduces the trial count dramatically.
• Robotics policy search: tuning the control parameters of a physical robot without breaking it. Each rollout is a real-world experiment.
• A/B testing and product optimization: when running a new variant is costly or risky, BO selects the most informative variant to test rather than a random one.
• Simulation calibration: fitting simulator parameters to match real-world observations, where each simulation run is slow.

The field sits at the intersection of statistics (Gaussian processes, Bayesian inference) and optimization (acquisition maximization). Understanding it unlocks a family of "smart experiment design" methods used across science and engineering.

Bayesian optimization is the answer to a deceptively simple question: how do you find the best configuration when you can only afford a handful of experiments?

The answer is elegant: keep a running model of what you believe the function looks like, express uncertainty honestly, and always spend your next experiment where the payoff is highest given what you already know. The Gaussian process surrogate does the bookkeeping; the acquisition function makes the decision.

The algorithm is practical today in everything from AutoML pipelines to drug discovery labs. Its costs are real — the $O(n^{3})$ GP and the need for a sensible prior — but for the regimes it was designed for, nothing beats it.

Next time you see a well-tuned neural network that took only twenty training runs to configure, there is a good chance Bayesian optimization was the quiet strategist behind the scenes.