Introduction

Behind almost every model that "learns" — from a tiny line fit to a giant language model — sits the same humble idea. Imagine the model's error as a landscape: every possible setting of its knobs is a point on the ground, and the height at that point is how wrong the model is. Training means finding the lowest valley.

You can't see the whole landscape; it can have millions of dimensions. But at the spot where you're standing you can feel which way is downhill — that direction is the gradient. So you take a small step that way, look again, step again. Roll the ball downhill until the ground flattens out, and you've found a minimum.

That single recipe — follow the slope, step, repeat — is gradient descent. It is old (the method dates to Cauchy, 1847) and it is everywhere. The only real decision is how big each step should be, and as you'll see, that one number can make or break the whole journey.

Roll the Ball

Below is a bumpy 1-D loss curve. Drop a ball anywhere and it rolls downhill by gradient descent: each step moves it against the slope, scaled by the learning rate. Press Step to take one step at a time, or Run to animate the whole descent.

<p class="hint">The curve is a <b>loss</b>: low is good. The ball rolls downhill by gradient descent — each step moves it against the slope, scaled by the <b>learning rate</b>. Small rate = slow but smooth; too large = overshoot, oscillate, even diverge. Click the curve to drop the ball anywhere.</p>
<canvas id="plot" width="460" height="240"></canvas>
<div class="row">
  <label>Learning rate <span id="lrval">0.10</span></label>
  <input id="lr" type="range" min="1" max="220" value="10">
</div>
<div class="status" id="status">Drop the ball, then step.</div>
<div class="btns">
  <button id="step" type="button">Step</button>
  <button id="run" type="button">Run</button>
  <button id="reset" type="button" class="ghost">Reset</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
canvas { width: 100%; max-width: 460px; height: auto; border: 1px solid #cdd9e3;
         border-radius: 8px; background: #f7fafc; display: block; touch-action: none; }
.row { display: flex; align-items: center; gap: .7rem; margin: .7rem 0 .2rem; }
.row label { font-size: .9rem; font-weight: 600; color: #1d3557; white-space: nowrap; }
.row label span { font-family: ui-monospace, monospace; color: #c92f3c; }
.row input[type=range] { flex: 1; accent-color: #1d3557; }
.status { font-size: 1rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; }
.status.ok { color: #0a7d33; }
.status.warn { color: #c77700; }
.status.bad { color: #c92f3c; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Gradient descent on a 1-D loss f(x): repeatedly do  x <- x - lr * f'(x).
// The ball moves AGAINST the slope. The learning rate (lr) sets the step size:
// too small -> slow; too large -> overshoots the minimum, oscillates, diverges.
// The curve has two valleys, so where you start decides which LOCAL minimum you reach.
const cv = document.getElementById('plot'), ctx = cv.getContext('2d');
const W = cv.width, H = cv.height;
const XMIN = -6, XMAX = 6;           // domain shown
// Loss with two wells: a smooth double-well plus a tilt. Minima near x≈-2.1 and x≈2.7.
function f(x){ return 0.05*x*x*x*x - 0.55*x*x + 0.35*x + 3.2; }
function df(x){ return 0.2*x*x*x - 1.1*x + 0.35; }   // exact derivative of f

// ----- scaling between math coords and canvas pixels -----
let YMIN = Infinity, YMAX = -Infinity;
for(let x=XMIN; x<=XMAX; x+=0.02){ const y=f(x); if(y<YMIN)YMIN=y; if(y>YMAX)YMAX=y; }
YMIN -= 0.5; YMAX += 0.8;
const sx = x => (x-XMIN)/(XMAX-XMIN)*W;
const sy = y => H - (y-YMIN)/(YMAX-YMIN)*H;

let x = -4.5, lr = 0.10, steps = 0, running = null, lastY = null;

function drawCurve(){
  ctx.clearRect(0,0,W,H);
  ctx.beginPath();
  for(let px=0; px<=W; px++){ const xx = XMIN + px/W*(XMAX-XMIN); const py = sy(f(xx));
    px===0 ? ctx.moveTo(px,py) : ctx.lineTo(px,py); }
  ctx.strokeStyle = '#1d3557'; ctx.lineWidth = 2; ctx.stroke();
}
function drawBall(){
  const px = sx(x), py = sy(f(x));
  // slope arrow (downhill tangent direction)
  const g = df(x), dir = g>0 ? -1 : 1;
  ctx.beginPath(); ctx.moveTo(px,py); ctx.lineTo(px+dir*26, py); ctx.strokeStyle='#c92f3c';
  ctx.lineWidth=2; ctx.stroke();
  ctx.beginPath(); ctx.moveTo(px+dir*26,py); ctx.lineTo(px+dir*18,py-5);
  ctx.lineTo(px+dir*18,py+5); ctx.closePath(); ctx.fillStyle='#c92f3c'; ctx.fill();
  ctx.beginPath(); ctx.arc(px,py,7,0,7); ctx.fillStyle='#e63946';
  ctx.strokeStyle='#fff'; ctx.lineWidth=2; ctx.fill(); ctx.stroke();
}
function render(){ drawCurve(); if(isFinite(x) && x>=XMIN-2 && x<=XMAX+2) drawBall(); }
function setStatus(t,cls){ const s=document.getElementById('status'); s.textContent=t; s.className='status '+cls; }

function step(){
  const g = df(x);
  const nx = x - lr*g;                       // the gradient-descent update
  const ny = f(nx);
  if(!isFinite(nx) || nx < XMIN-2 || nx > XMAX+2){
    setStatus('Diverged! The step was too big — the ball flew off the surface.','bad');
    stopRun(); x = nx; render(); return false;
  }
  const overshot = (lastY!=null && ny > lastY + 1e-9);  // loss went UP = overshoot
  x = nx; steps++; lastY = ny; render();
  if(Math.abs(g) < 1e-3){
    setStatus(fmt('Converged at x≈%x% after %steps% steps (slope ≈ 0).', { x: x.toFixed(2), steps: steps }),'ok');
    stopRun(); return false;
  }
  if(overshot) setStatus(fmt('Step %steps%: overshooting — loss went up. Lower the rate.', { steps: steps }),'warn');
  else setStatus(fmt('Step %steps%: rolling downhill. slope=%g%', { steps: steps, g: g.toFixed(3) }),'');
  return true;
}
function stopRun(){ if(running){ clearInterval(running); running=null;
  document.getElementById('run').textContent='Run'; } }
function toggleRun(){
  if(running){ stopRun(); return; }
  document.getElementById('run').textContent='Stop';
  running = setInterval(()=>{ if(!step()) stopRun(); }, 120);
}
function reset(){ stopRun(); x=-4.5; steps=0; lastY=null; render(); setStatus('Drop the ball, then step.',''); }

document.getElementById('lr').addEventListener('input', e=>{
  lr = (+e.target.value)/100; document.getElementById('lrval').textContent = lr.toFixed(2);
});
document.getElementById('step').addEventListener('click', step);
document.getElementById('run').addEventListener('click', toggleRun);
document.getElementById('reset').addEventListener('click', reset);
cv.addEventListener('pointerdown', e=>{
  stopRun(); const r=cv.getBoundingClientRect();
  const px = (e.clientX-r.left)/r.width*W;
  x = XMIN + px/W*(XMAX-XMIN); steps=0; lastY=null; render();
  setStatus(fmt('Ball dropped at x=%x%. Now step.', { x: x.toFixed(2) }),'');
});
render();
Result
Console

Now play with the learning rate slider. Make it small and the ball creeps down smoothly but slowly. Make it bigger and it descends faster — until, past a point, it starts to overshoot and oscillate, and if you push further it diverges, flying off the surface entirely. Drop the ball on different sides and you'll also see it settle into different local minima: gradient descent only ever finds the bottom of the valley it happens to be in, not the deepest valley overall.

The Real Complexity

How hard is it to roll downhill? It depends entirely on the shape of the landscape.

  • On a bowl (convex loss) it is easy. There is exactly one valley, every downhill step is progress, and gradient descent provably converges to the single global minimum. Fitting a line or a logistic classifier lives here.
  • The learning rate is everything. Too small and it crawls; too large and steps overshoot the bottom and oscillate or diverge. The safe range depends on how sharply the surface curves (its "smoothness").
  • On a bumpy landscape (non-convex loss) — neural networks — there are no global guarantees. You can get stuck in a local minimum, stall on a flat saddle point, or land somewhere that depends on where you started.
  • The honest status: plain gradient descent is not solved as a route to the best answer. Even just finding an approximate stationary point of a smooth non-convex function is PPAD-hard (Daskalakis, Skoulakis, Zampetakis, 2021) — there is no known efficient method guaranteed to reach the bottom.

The astonishing fact of modern AI is that the gamble pays off anyway: on the giant loss surfaces of deep networks, local minima tend to be good enough, and the simple downhill walk keeps working. That is closer to luck and empirical landscape structure than to a proof — a cousin of the open questions around P vs NP and hard non-convex optimization.

Where It Matters

"Adjust the knobs to reduce the error" is the shape of an enormous fraction of real machine learning, and gradient descent is the engine that does it:

  • Training neural networks: every deep model — vision, speech, language — is trained by some flavor of gradient descent (SGD, Adam, RMSProp), with gradients supplied by backpropagation.
  • Classic statistics and ML: linear and logistic regression, support vector machines and many others are just convex loss surfaces being rolled down.
  • Recommendation and ranking: matrix-factorization and embedding models nudge millions of parameters downhill to predict what you'll click.
  • Beyond machine learning: control, robotics, physics simulations and engineering design all minimize differentiable cost functions the same way.

Understand gradient descent and you understand the common thread under nearly all of modern AI — and why so much research goes into its smarter siblings and into taming hard non-convex optimization.

Conclusion

Gradient descent is almost embarrassingly simple: feel the slope, take a step, repeat. From that one move comes nearly all of machine learning — and also all of its drama. The learning rate decides whether the ball glides to the bottom or flies off the surface, and on a bumpy landscape there is no promise the valley you reach is the deepest one.

So the next time a model "learns," picture a ball rolling downhill in the dark, feeling its way one step at a time. It works astonishingly well, but it is no oracle: like the great open problems behind P vs NP, reaching the true bottom of a hard landscape may have no shortcut at all.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/gradient-descent/Content licensed under CC BY-NC 4.0.