Introduction

Imagine fitting a curve to a handful of noisy data points. Use a straight line and you'll miss the real shape — the line is too rigid, and it's wrong in the same systematic way no matter what data you feed it. That stubborn, built-in error is called bias.

Now swing the other way. Use a wildly wiggly curve that threads through every point exactly. On the data you trained on, it looks perfect. But it has memorized the random noise, not the signal — and on fresh data it lurches all over the place. That jumpiness, the way the model changes drastically with each new sample, is called variance.

Here is the catch that defines machine learning: you cannot drive both to zero at once. Make the model simpler and bias rises; make it more complex and variance rises. The whole game is finding the sweet spot in between — and that tug-of-war is the bias-variance tradeoff.

Slide the Complexity

Below is a small set of noisy points generated from a hidden smooth curve. A polynomial model tries to fit them. Drag the complexity slider to raise the polynomial's degree and watch what happens to two numbers: the error on the training points the model saw, and the error on fresh test points it didn't.

<p class="hint">Noisy points come from a hidden smooth curve. A polynomial model fits them. Drag <b>complexity</b> to raise the degree and watch the two errors: training (points the model saw) and test (fresh points it didn't).</p>
<div class="row">
  <label for="deg">Complexity (polynomial degree): <b id="degv">4</b></label>
  <input id="deg" type="range" min="1" max="10" value="4" step="1">
</div>
<svg id="plot" viewBox="0 0 360 220" preserveAspectRatio="xMidYMid meet"></svg>
<div class="stats">
  <div class="stat"><span class="k">Training error</span><span id="trErr" class="v tr">–</span></div>
  <div class="stat"><span class="k">Test error</span><span id="teErr" class="v te">–</span></div>
  <div class="stat"><span class="k">Regime</span><span id="regime" class="v">–</span></div>
</div>
<div class="btns">
  <button id="best" type="button">Jump to the sweet spot</button>
  <button id="newdata" type="button" class="ghost">New noisy sample</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.row { margin: .3rem 0 .5rem; font-size: .92rem; }
.row input { width: 100%; }
svg { width: 100%; height: auto; background: #f6f8fa; border: 1px solid #dde3ea; border-radius: 8px; }
.pt { fill: #1d3557; }
.pt.test { fill: #e67e22; }
.fit { fill: none; stroke: #2a9d8f; stroke-width: 2; }
.truth { fill: none; stroke: #adb5bd; stroke-width: 1.5; stroke-dasharray: 4 3; }
.stats { display: flex; gap: .6rem; flex-wrap: wrap; margin: .6rem 0; }
.stat { flex: 1 1 100px; background: #eef2f6; border: 1px solid #dde3ea; border-radius: 8px; padding: .45rem .6rem; }
.k { display: block; font-size: .72rem; color: #5a7088; text-transform: uppercase; letter-spacing: .03em; }
.v { font: 700 1.1rem ui-monospace, monospace; }
.v.tr { color: #1d3557; }
.v.te { color: #e67e22; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Bias-variance tradeoff, made visible. A hidden smooth curve f(x) generates
    // noisy training points. We fit a polynomial of adjustable DEGREE by least
    // squares. Training error always falls as degree rises (more flexibility fits
    // the seen points). Test error (on fresh points from the SAME curve) traces a
    // U: too-low degree = high bias (underfit); too-high degree = high variance
    // (overfit, chasing noise). The bottom of the U is the sweet spot.
const W = 360, H = 220, PADL = 8, PADR = 8, PADT = 10, PADB = 10;
const XMIN = 0, XMAX = 1;
const truth = x => 0.5 + 0.4 * Math.sin(2 * Math.PI * x); // hidden curve in [0,1]
const NOISE = 0.18;
const NTRAIN = 9, NTEST = 60, MAXDEG = 10;

// simple seeded RNG so "New noisy sample" is reproducible-ish but varied.
    // seed 4 gives a clean default U: sweet spot near degree 4.
let seed = 4;
function rnd(){ seed = (seed * 1103515245 + 12345) & 0x7fffffff; return seed / 0x7fffffff; }
function gauss(){ return (rnd() + rnd() + rnd() + rnd() - 2) * 0.7; }

let train = [], test = [];
function sample(){
  train = []; test = [];
  for (let i = 0; i < NTRAIN; i++){ const x = (i + 0.5) / NTRAIN; train.push([x, truth(x) + NOISE * gauss()]); }
  for (let i = 0; i < NTEST; i++){ const x = rnd(); test.push([x, truth(x) + NOISE * gauss()]); }
}

// Least-squares polynomial fit: solve (X^T X) w = X^T y via Gaussian elimination.
function polyfit(pts, degree){
  const n = degree + 1;
  const A = Array.from({length:n}, () => new Array(n).fill(0));
  const b = new Array(n).fill(0);
  for (const [x, y] of pts){
    const p = []; let xp = 1;
    for (let k = 0; k < n; k++){ p.push(xp); xp *= x; }
    for (let i = 0; i < n; i++){ for (let j = 0; j < n; j++) A[i][j] += p[i] * p[j]; b[i] += p[i] * y; }
  }
  // tiny ridge term for numerical stability at high degree (does not change the lesson)
  for (let i = 0; i < n; i++) A[i][i] += 1e-6;
  // Gaussian elimination with partial pivoting
  for (let c = 0; c < n; c++){
    let piv = c; for (let r = c + 1; r < n; r++) if (Math.abs(A[r][c]) > Math.abs(A[piv][c])) piv = r;
    [A[c], A[piv]] = [A[piv], A[c]]; [b[c], b[piv]] = [b[piv], b[c]];
    const d = A[c][c] || 1e-12;
    for (let j = c; j < n; j++) A[c][j] /= d; b[c] /= d;
    for (let r = 0; r < n; r++){ if (r === c) continue; const f = A[r][c];
      for (let j = c; j < n; j++) A[r][j] -= f * A[c][j]; b[r] -= f * b[c]; }
  }
  return b; // coefficients w0 + w1 x + w2 x^2 + ...
}
function evalp(w, x){ let y = 0, xp = 1; for (const c of w){ y += c * xp; xp *= x; } return y; }
function mse(w, pts){ let s = 0; for (const [x, y] of pts){ const e = evalp(w, x) - y; s += e * e; } return s / pts.length; }

// screen mapping
const sx = x => PADL + (x - XMIN) / (XMAX - XMIN) * (W - PADL - PADR);
const sy = y => H - PADB - (y - 0) / (1 - 0) * (H - PADT - PADB);

function path(fn){ let d = ''; for (let i = 0; i <= 120; i++){ const x = i / 120;
  const y = Math.max(-0.3, Math.min(1.3, fn(x))); d += (i ? 'L' : 'M') + sx(x).toFixed(1) + ' ' + sy(y).toFixed(1); } return d; }

function render(){
  const deg = +document.getElementById('deg').value;
  document.getElementById('degv').textContent = deg;
  const w = polyfit(train, deg);
  const tr = mse(w, train), te = mse(w, test);
  let svg = '';
  svg += '<path class="truth" d="' + path(truth) + '"/>';
  svg += '<path class="fit" d="' + path(x => evalp(w, x)) + '"/>';
  for (const [x, y] of test) svg += '<circle class="pt test" cx="' + sx(x).toFixed(1) + '" cy="' + sy(y).toFixed(1) + '" r="2"/>';
  for (const [x, y] of train) svg += '<circle class="pt" cx="' + sx(x).toFixed(1) + '" cy="' + sy(y).toFixed(1) + '" r="3.2"/>';
  document.getElementById('plot').innerHTML = svg;
  document.getElementById('trErr').textContent = tr.toFixed(4);
  document.getElementById('teErr').textContent = te.toFixed(4);
  const reg = document.getElementById('regime');
  if (deg <= 2){ reg.textContent = 'Underfit (high bias)'; reg.style.color = '#c0392b'; }
  else if (deg >= 7){ reg.textContent = 'Overfit (high variance)'; reg.style.color = '#c0392b'; }
  else { reg.textContent = 'Near the sweet spot'; reg.style.color = '#0a7d33'; }
}
// Find the degree minimizing TEST error — the empirical bottom of the U.
function bestDegree(){
  let bd = 1, bv = Infinity;
  for (let d = 1; d <= MAXDEG; d++){ const v = mse(polyfit(train, d), test); if (v < bv){ bv = v; bd = d; } }
  return bd;
}
document.getElementById('deg').addEventListener('input', render);
document.getElementById('best').addEventListener('click', () => {
  const bd = bestDegree(); document.getElementById('deg').value = bd; render();
  console.log('sweet spot (min test error) at degree', bd);
});
document.getElementById('newdata').addEventListener('click', () => { sample(); render(); });
sample(); render();
Result
Console

Watch the asymmetry. Training error falls steadily — more flexibility always fits the seen points better. But test error traces a U: it drops as the model escapes underfitting, bottoms out at the sweet spot, then shoots up as the curve starts chasing noise. The lowest point of that U, not the lowest training error, is the model you actually want.

The Real Complexity

How real is this tradeoff? It is not a vague rule of thumb — it is a proven decomposition, and its status is settled, not open.

  • The exact equation. For squared-error loss, the expected error on a new point splits cleanly into three pieces: bias2bias^{2} (how far the average prediction sits from the truth) + variance (how much the prediction wobbles across different training sets) + irreducible noise (the part no model can ever remove). This is a classical identity in statistical learning theory.
  • Why you can't beat it. Lowering bias means a more flexible model, which raises variance; lowering variance means a stiffer model, which raises bias. The two terms pull in opposite directions, so the sum has a minimum somewhere in the middle.
  • It's not undecidable, not NP-hard — it's a theorem. Unlike the questions behind P vs NP, there is no mystery here about why the curve is U-shaped. The mystery is purely practical: where exactly is the bottom, for your data?
  • The modern twist: double descent. For very large models, recent work shows test error can fall, rise (the classic U), and then fall again as the model grows past the point of perfectly fitting the data. The tradeoff still holds within each regime, but the global picture is richer than the textbook U.

So the tradeoff is mathematically airtight. The hard part is engineering — estimating that sweet spot from finite, noisy data, which connects it to questions of learnability and how much data a model truly needs.

Where It Matters

Almost every practical trick in machine learning is, underneath, a way to steer the bias-variance balance:

  • Regularization (ridge, lasso, weight decay) deliberately adds a little bias to cut variance — pulling a too-flexible model back toward simplicity.
  • Cross-validation is how you find the sweet spot empirically: hold out data, measure the U-curve, pick the dip.
  • Ensembles like random forests and bagging average many high-variance models to cancel their wobble, lowering variance without much added bias.
  • Early stopping in neural network training halts before the model starts memorizing noise — the U-curve in time rather than in degree.
  • More data is the one move that shrinks variance without paying in bias, which is why data is so valuable.

Understand the tradeoff and these stop being a grab-bag of tricks: they are all answers to one question — how flexible should the model be?

Conclusion

The bias-variance tradeoff is one of the few ideas in machine learning that is both deeply practical and mathematically exact. The decomposition into bias2bias^{2}, variance and irreducible noise is a proven identity — there is no open problem about whether the tradeoff exists.

What stays hard is the everyday craft: estimating, from finite and noisy data, exactly how flexible your model should be. The next time a fancier model scores worse, you'll know it isn't broken — it has simply slid past the bottom of the U, trading away bias only to drown in variance. The art of learning is finding that bottom, and it connects straight to deeper questions of what is even learnable.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/bias-variance/Content licensed under CC BY-NC 4.0.