Introduction

A neural network is, deep down, a tower of tunable knobs called weights. You feed numbers in the front, they get multiplied and added and squashed layer by layer, and a prediction comes out the back. At first the weights are random, so the prediction is wrong, and the gap between the answer and the truth is the error.

Learning means nudging every weight a little so that next time the error is smaller. The obvious question is brutal: a real network has millions of weights, so which way should each one move, and by how much?

Backpropagation answers that question for every weight at once. Published in its famous form by Rumelhart, Hinton and Williams in 1986, it is not a deep mystery — it is the humble chain rule from calculus, applied backward through the layers. One forward pass to make a prediction, one backward pass to assign blame, and every knob learns the direction to turn.

Watch It Learn

Here is a tiny network with two inputs, one hidden layer and one output, trying to learn a simple target. Press Step to run one forward pass (the prediction) followed by one backward pass (the error flowing back through every weight). Watch the connections glow as blame travels backward, and the error number shrink.

<p class="hint">A tiny network: 2 inputs &rarr; 2 hidden neurons &rarr; 1 output. It is trying to predict the <b>target</b> from the fixed input. Press <b>Step</b> for one forward pass (prediction) then one backward pass (error flowing back through every weight). Glowing edges carry the most blame. Watch the error shrink.</p>
<svg id="net" viewBox="0 0 320 200" aria-label="neural network"></svg>
<div class="row">
  <span class="pill">prediction <b id="pred">–</b></span>
  <span class="pill">target <b id="targ">0.90</b></span>
  <span class="pill err">error <b id="err">–</b></span>
</div>
<div class="btns">
  <button id="step" type="button">Step</button>
  <button id="train" type="button">Train 200&times;</button>
  <button id="reset" type="button" class="ghost">Reset weights</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
#net { width: 100%; max-width: 420px; height: auto; display: block; margin: 0 auto; }
.edge { stroke: #9aa3ad; transition: stroke .15s, stroke-width .15s; }
.node { fill: #e8eef3; stroke: #1d3557; stroke-width: 1.5; }
.node.out { fill: #d8e8df; }
.nlabel { font: 600 11px ui-monospace, monospace; fill: #1d3557; text-anchor: middle; }
.row { display: flex; gap: .5rem; flex-wrap: wrap; justify-content: center; margin: .5rem 0; }
.pill { font-size: .85rem; background: #eef2f6; border: 1px solid #d3dce4; border-radius: 999px;
        padding: .25rem .7rem; }
.pill b { font: 700 .9rem ui-monospace, monospace; color: #1d3557; }
.pill.err b { color: #c92f3c; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; justify-content: center; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// A 2-2-1 sigmoid network trained by BACKPROPAGATION (chain rule run backward).
// Forward: h = sig(W1.x + b1); y = sig(W2.h + b2). Loss = 1/2 (y - t)^2.
// Backward (the whole point): one sweep gives the gradient of EVERY weight.
const x = [1, 0];          // fixed input
const t = 0.90;            // target
const lr = 0.5;            // learning rate
const sig = z => 1 / (1 + Math.exp(-z));

let W1, b1, W2, b2;        // W1[j][i] hidden j from input i; W2[j] output from hidden j
function reset() {
  W1 = [[0.30, -0.20], [-0.10, 0.40]];
  b1 = [0.0, 0.0];
  W2 = [0.20, -0.30];
  b2 = 0.0;
  lastGrad = { W1: [[0,0],[0,0]], W2: [0,0] };
  forward(); draw(); setReadout();
}

let h = [0, 0], y = 0, lastGrad;
function forward() {
  for (let j = 0; j < 2; j++) h[j] = sig(W1[j][0]*x[0] + W1[j][1]*x[1] + b1[j]);
  y = sig(W2[0]*h[0] + W2[1]*h[1] + b2);
}

// One backward pass: chain rule from the loss back to every weight.
function backward() {
  const dL_dy = (y - t);          // d loss / d y
  const dy_dz = y * (1 - y);      // sigmoid derivative at output
  const dOut = dL_dy * dy_dz;     // "blame" at the output neuron
  const gW2 = [dOut * h[0], dOut * h[1]];
  const gb2 = dOut;
  const gW1 = [[0,0],[0,0]], gb1 = [0,0];
  for (let j = 0; j < 2; j++) {
    const dh = dOut * W2[j] * h[j] * (1 - h[j]);  // blame pushed back to hidden j
    gW1[j][0] = dh * x[0];
    gW1[j][1] = dh * x[1];
    gb1[j] = dh;
  }
  // gradient descent step
  for (let j = 0; j < 2; j++) {
    W2[j] -= lr * gW2[j];
    for (let i = 0; i < 2; i++) W1[j][i] -= lr * gW1[j][i];
    b1[j] -= lr * gb1[j];
  }
  b2 -= lr * gb2;
  lastGrad = { W1: gW1, W2: gW2 };
}

function step() { forward(); backward(); forward(); draw(); setReadout(); }
function train() { for (let k = 0; k < 200; k++) { forward(); backward(); } forward(); draw(); setReadout(); }

function setReadout() {
  const err = 0.5 * (y - t) * (y - t);
  document.getElementById('pred').textContent = y.toFixed(3);
  document.getElementById('err').textContent = err.toFixed(4);
}

// ── drawing ──────────────────────────────────────────────────────────
const IN = [[40,60],[40,140]], HID = [[160,55],[160,145]], OUT = [[280,100]];
function edgeWidth(g) { return 1 + Math.min(6, Math.abs(g) * 40); }
function edgeColor(g) { const a = Math.min(1, Math.abs(g) * 6);
  return g >= 0 ? `rgba(230,57,70,${0.25+0.75*a})` : `rgba(29,53,87,${0.25+0.75*a})`; }
function draw() {
  const svg = document.getElementById('net'); let s = '';
  for (let j = 0; j < 2; j++) for (let i = 0; i < 2; i++) {
    const g = lastGrad.W1[j][i];
    s += `<line class="edge" x1="${IN[i][0]}" y1="${IN[i][1]}" x2="${HID[j][0]}" y2="${HID[j][1]}" stroke="${edgeColor(g)}" stroke-width="${edgeWidth(g)}"/>`;
  }
  for (let j = 0; j < 2; j++) {
    const g = lastGrad.W2[j];
    s += `<line class="edge" x1="${HID[j][0]}" y1="${HID[j][1]}" x2="${OUT[0][0]}" y2="${OUT[0][1]}" stroke="${edgeColor(g)}" stroke-width="${edgeWidth(g)}"/>`;
  }
  const node = (p, lbl, cls='') => `<circle class="node ${cls}" cx="${p[0]}" cy="${p[1]}" r="16"/><text class="nlabel" x="${p[0]}" y="${p[1]+4}">${lbl}</text>`;
  s += node(IN[0], x[0]); s += node(IN[1], x[1]);
  s += node(HID[0], h[0].toFixed(2)); s += node(HID[1], h[1].toFixed(2));
  s += node(OUT[0], y.toFixed(2), 'out');
  svg.innerHTML = s;
}

document.getElementById('step').addEventListener('click', step);
document.getElementById('train').addEventListener('click', train);
document.getElementById('reset').addEventListener('click', reset);
reset();
Result
Console

Notice what backpropagation buys you. The forward pass just computes a prediction. The backward pass reuses those same numbers to hand every single weight its own gradient — its personal "move me this way" instruction — in one sweep, no matter how many weights there are. Press Train 200x and watch the error collapse as the gradients are followed by gradient descent.

The Real Complexity

How expensive is it to learn? The naive answer is terrifying — and that is exactly what backpropagation defeats.

  • The naive way would estimate each weight's effect by nudging it and re-running the whole network. With W weights that is W forward passes per step — utterly hopeless for millions of weights.
  • Backpropagation is reverse-mode automatic differentiation. It computes the gradient for all weights in a single backward pass whose cost is about the same as one forward pass — roughly O(W)O(W) total, not O(W2)O(W^{2}).
  • The engine is the chain rule. Each layer locally knows how its output depends on its input. Multiply those local derivatives backward from the error, caching shared terms, and every weight's gradient drops out for free.
  • It is exact, not approximate. Unlike numerical nudging, backpropagation returns the true derivative, limited only by floating-point arithmetic.

That is the punchline: backpropagation is solved and cheap. Computing the gradients was never the hard part. The genuinely hard part is what you do with them — and finding weights that minimize the error is a non-convex search related to neural-network training being NP-hard.

Where It Matters

"Compute every gradient cheaply" turned out to be the lever that lifted modern AI off the ground:

  • Deep learning, everywhere: language models, image recognizers, speech systems and recommenders are all trained by backpropagation feeding gradient descent.
  • Autograd frameworks: PyTorch, TensorFlow and JAX are, at heart, automatic-differentiation engines that run backpropagation over any computation you write.
  • Beyond neural nets: reverse-mode differentiation now optimizes physics simulations, financial models and engineering designs — anywhere you need a gradient of a big function.
  • Why scale was possible: because the backward pass costs the same as the forward pass, doubling a model's size only doubles training cost, not squares it. That linearity is why billion-parameter models are even trainable.

Understand backpropagation and you understand the metabolism of deep learning — the same gradients feed the learning theory behind PAC learning.

Conclusion

Backpropagation hides a beautiful economy: the same forward computation that makes a prediction can be run backward to hand every weight, however many millions there are, its own marching order — all for the price of one extra pass. It is not magic and it is not approximate; it is the chain rule, applied with care.

So the next time a model "learns," picture the error flowing backward through the layers, each weight quietly adjusting. The computing of gradients is a solved problem (Rumelhart, Hinton & Williams, 1986). What remains genuinely hard is steering those gradients to a good answer — a search that brushes up against P vs NP and the limits of optimization itself.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/backpropagation/Content licensed under CC BY-NC 4.0.