Backpropagation

A neural network is, deep down, a tower of tunable knobs called weights. You feed numbers in the front, they get multiplied and added and squashed layer by layer, and a prediction comes out the back. At first the weights are random, so the prediction is wrong, and the gap between the answer and the truth is the error.

Learning means nudging every weight a little so that next time the error is smaller. The obvious question is brutal: a real network has millions of weights, so which way should each one move, and by how much?

Backpropagation answers that question for every weight at once. Published in its famous form by Rumelhart, Hinton and Williams in 1986, it is not a deep mystery — it is the humble chain rule from calculus, applied backward through the layers. One forward pass to make a prediction, one backward pass to assign blame, and every knob learns the direction to turn.

Here is a tiny network with two inputs, one hidden layer and one output, trying to learn a simple target. Press Step to run one forward pass (the prediction) followed by one backward pass (the error flowing back through every weight). Watch the connections glow as blame travels backward, and the error number shrink.

<p class="hint">{{hint}}</p>
<svg id="net" viewBox="0 0 320 200" aria-label="{{aria_net}}"></svg>
<div class="row">
  <span class="pill">{{label_prediction}} <b id="pred">–</b></span>
  <span class="pill">{{label_target}} <b id="targ">0.90</b></span>
  <span class="pill err">{{label_error}} <b id="err">–</b></span>
</div>
<div class="btns">
  <button id="step" type="button">{{btn_step}}</button>
  <button id="train" type="button">{{btn_train}}</button>
  <button id="reset" type="button" class="ghost">{{btn_reset}}</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
#net { width: 100%; max-width: 420px; height: auto; display: block; margin: 0 auto; }
.edge { stroke: #9aa3ad; transition: stroke .15s, stroke-width .15s; }
.node { fill: #e8eef3; stroke: #1d3557; stroke-width: 1.5; }
.node.out { fill: #d8e8df; }
.nlabel { font: 600 11px ui-monospace, monospace; fill: #1d3557; text-anchor: middle; }
.row { display: flex; gap: .5rem; flex-wrap: wrap; justify-content: center; margin: .5rem 0; }
.pill { font-size: .85rem; background: #eef2f6; border: 1px solid #d3dce4; border-radius: 999px;
        padding: .25rem .7rem; }
.pill b { font: 700 .9rem ui-monospace, monospace; color: #1d3557; }
.pill.err b { color: #c92f3c; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; justify-content: center; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

Notice what backpropagation buys you. The forward pass just computes a prediction. The backward pass reuses those same numbers to hand every single weight its own gradient — its personal "move me this way" instruction — in one sweep, no matter how many weights there are. Press Train 200x and watch the error collapse as the gradients are followed by gradient descent.

How expensive is it to learn? The naive answer is terrifying — and that is exactly what backpropagation defeats.

• The naive way would estimate each weight's effect by nudging it and re-running the whole network. With W weights that is W forward passes per step — utterly hopeless for millions of weights.
• Backpropagation is reverse-mode automatic differentiation. It computes the gradient for all weights in a single backward pass whose cost is about the same as one forward pass — roughly $O(W)$ total, not $O(W^{2})$.
• The engine is the chain rule. Each layer locally knows how its output depends on its input. Multiply those local derivatives backward from the error, caching shared terms, and every weight's gradient drops out for free.
• It is exact, not approximate. Unlike numerical nudging, backpropagation returns the true derivative, limited only by floating-point arithmetic.

That is the punchline: backpropagation is solved and cheap. Computing the gradients was never the hard part. The genuinely hard part is what you do with them — and finding weights that minimize the error is a non-convex search related to neural-network training being NP-hard.

"Compute every gradient cheaply" turned out to be the lever that lifted modern AI off the ground:

• Deep learning, everywhere: language models, image recognizers, speech systems and recommenders are all trained by backpropagation feeding gradient descent.
• Autograd frameworks: PyTorch, TensorFlow and JAX are, at heart, automatic-differentiation engines that run backpropagation over any computation you write.
• Beyond neural nets: reverse-mode differentiation now optimizes physics simulations, financial models and engineering designs — anywhere you need a gradient of a big function.
• Why scale was possible: because the backward pass costs the same as the forward pass, doubling a model's size only doubles training cost, not squares it. That linearity is why billion-parameter models are even trainable.

Understand backpropagation and you understand the metabolism of deep learning — the same gradients feed the learning theory behind PAC learning.

Backpropagation hides a beautiful economy: the same forward computation that makes a prediction can be run backward to hand every weight, however many millions there are, its own marching order — all for the price of one extra pass. It is not magic and it is not approximate; it is the chain rule, applied with care.

So the next time a model "learns," picture the error flowing backward through the layers, each weight quietly adjusting. The computing of gradients is a solved problem (Rumelhart, Hinton & Williams, 1986). What remains genuinely hard is steering those gradients to a good answer — a search that brushes up against P vs NP and the limits of optimization itself.