Where SGD Matters

Introduction

Every modern neural network — the language model answering your questions, the image recognizer on your phone — was trained by minimizing a loss function. Loss measures how wrong the model is; training means finding the model weights that make loss as small as possible.

The textbook answer is gradient descent: compute the slope of the loss at your current weights, step downhill, repeat. The slope is the gradient, a vector pointing in the direction of steepest increase, so stepping in the opposite direction decreases the loss.

The catch: computing the exact gradient means summing over the entire training dataset — millions of examples — every single step. For even a modest deep network that is ruinously expensive.

Stochastic Gradient Descent (SGD) makes one radical substitution: instead of the full dataset, use a random mini-batch of, say, 32 or 256 examples. The mini-batch gradient is a noisy estimate of the true gradient, but it is cheap, and with enough steps the noise averages out. The algorithm arrived — in statistical form — with Herbert Robbins and Sutton Monro in 1951, and its modern deep-learning incarnation was systematized by Léon Bottou through the 1990s and 2000s.

The surprising truth: the noise is not merely tolerated — it actively helps. Noisy updates escape shallow local minima that would trap the exact gradient, and they implicitly regularize the model by preventing it from memorizing every quirk of the training set.

Try It: Race on the Loss Surface

The canvas below shows a 2D loss surface (warm = high loss, cool = low loss). Each optimizer starts from the same point and tries to reach the minimum. Press Run to start, then adjust the learning rate and batch size to see how trajectory and speed change.

<div class="controls">
  <label>Learning rate: <span id="lrVal">0.05</span>
    <input type="range" id="lr" min="0.005" max="0.15" step="0.005" value="0.05">
  </label>
  <label>Batch size:
    <select id="bs">
      <option value="1">1 (pure SGD)</option>
      <option value="8" selected>8 (mini-batch)</option>
      <option value="200">200 (full batch)</option>
    </select>
  </label>
  <div class="btn-row">
    <button id="runBtn">Run</button>
    <button id="resetBtn" class="ghost">Reset</button>
  </div>
</div>
<canvas id="cv" width="380" height="280"></canvas>
<div id="legend" class="legend">
  <span class="dot" style="background:#e63946"></span> Full-batch (B=200)
  <span class="dot" style="background:#457b9d"></span> Mini-batch (B=8)
  <span class="dot" style="background:#2a9d8f"></span> Pure SGD (B=1)
</div>
<div id="info" class="info">Press Run to start all three optimizers.</div>

*{box-sizing:border-box;margin:0;padding:0}
body{font-family:system-ui,sans-serif;color:#222;background:#f8f9fa;padding:.6rem}
.controls{display:flex;flex-wrap:wrap;gap:.5rem .9rem;align-items:center;margin-bottom:.5rem}
label{display:flex;align-items:center;gap:.35rem;font-size:.85rem;font-weight:600}
input[type=range]{width:100px;cursor:pointer}
select{font-size:.85rem;padding:.2rem .4rem;border:1px solid #adb5bd;border-radius:6px;cursor:pointer}
.btn-row{display:flex;gap:.4rem}
button{font:600 13px system-ui;padding:.35rem .8rem;border:1px solid #1d3557;
       background:#1d3557;color:#fff;border-radius:7px;cursor:pointer}
button.ghost{background:#fff;color:#1d3557}
canvas{display:block;border-radius:10px;border:1px solid #dee2e6;width:100%;max-width:380px}
.legend{display:flex;gap:.8rem;flex-wrap:wrap;margin-top:.4rem;font-size:.8rem;align-items:center}
.dot{display:inline-block;width:10px;height:10px;border-radius:50%;margin-right:2px}
.info{font-size:.82rem;color:#495057;margin-top:.3rem;min-height:1.2em}

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// ── Loss surface: Rosenbrock-like 2D function ───────────────────────────────────────────
// We use a scaled bowl with a ridge to make optimizer differences visible.
// True minimum at (wx=0.6, wy=0.4) in [0,1]^2.
const W = 380, H = 280;
const cv = document.getElementById('cv');
const ctx = cv.getContext('2d');

// Loss function in parameter space [0,1]^2
// Minimum at (wx=0.6, wy=0.4). Elongated bowl with a sin*sin ridge
// that is zero at the minimum (preserving it) but adds interesting topology.
function loss(wx, wy) {
  const a = wx - 0.6, b = wy - 0.4;
  return 3*a*a + 7*b*b + 1.5*a*b + 0.25*Math.sin(5*a)*Math.sin(5*b);
}

// Numerical gradient
function grad(wx, wy) {
  const eps = 1e-4;
  return [
    (loss(wx+eps, wy) - loss(wx-eps, wy)) / (2*eps),
    (loss(wx, wy+eps) - loss(wx, wy-eps)) / (2*eps)
  ];
}

// Canvas coordinate transforms
const PAD = 20;
function toCanvas(wx, wy) {
  return [PAD + wx*(W-2*PAD), PAD + (1-wy)*(H-2*PAD)];
}

// ── Render the heatmap ──────────────────────────────────────────────────────────
let heatmapImg = null;
function buildHeatmap() {
  const id = ctx.createImageData(W, H);
  const d = id.data;
  // Find min/max
  let mn=1e9, mx=-1e9;
  for(let py=0;py<H;py++) for(let px=0;px<W;px++){
    const wx=(px-PAD)/(W-2*PAD), wy=1-(py-PAD)/(H-2*PAD);
    if(wx<0||wx>1||wy<0||wy>1) continue;
    const v=loss(wx,wy); if(v<mn)mn=v; if(v>mx)mx=v;
  }
  // Color: cool blue → warm red
  for(let py=0;py<H;py++) for(let px=0;px<W;px++){
    const i=(py*W+px)*4;
    const wx=(px-PAD)/(W-2*PAD), wy=1-(py-PAD)/(H-2*PAD);
    if(wx<0||wx>1||wy<0||wy>1){
      d[i]=248;d[i+1]=249;d[i+2]=250;d[i+3]=255; continue;
    }
    const t=Math.pow((loss(wx,wy)-mn)/(mx-mn),0.55); // gamma
    // cool=dark-blue(29,53,87) to warm=red(230,57,70)
    d[i]   = Math.round(29 + t*(230-29));
    d[i+1] = Math.round(53 + t*(57-53));
    d[i+2] = Math.round(87 + t*(70-87));
    d[i+3] = 255;
  }
  heatmapImg = id;
}

function drawHeatmap() {
  if(!heatmapImg) buildHeatmap();
  ctx.putImageData(heatmapImg, 0, 0);
  // Draw minimum star
  const [mx,my] = toCanvas(0.6, 0.4);
  ctx.save();
  ctx.strokeStyle='#fff';ctx.lineWidth=1.5;
  ctx.beginPath();
  for(let i=0;i<5;i++){
    const a1=i*Math.PI*2/5-Math.PI/2, a2=a1+Math.PI/5;
    const x1=mx+6*Math.cos(a1),y1=my+6*Math.sin(a1);
    const x2=mx+2.5*Math.cos(a2),y2=my+2.5*Math.sin(a2);
    i===0?ctx.moveTo(x1,y1):ctx.lineTo(x1,y1);
    ctx.lineTo(x2,y2);
  }
  ctx.closePath(); ctx.stroke(); ctx.restore();
}

// ── Optimizer state ─────────────────────────────────────────────────────────────────────
// Simulated "dataset": 200 2D points with noise that shifts the apparent gradient
const N_DATA = 200;
const DATA = [];
const rng = (() => { let s=12345; return ()=>{ s=(s*1664525+1013904223)&0xffffffff; return (s>>>0)/4294967296; }; })();
for(let i=0;i<N_DATA;i++) DATA.push([rng(), rng()]);

// Noisy loss gradient for a batch: true gradient + noise proportional to 1/sqrt(B)
function batchGrad(wx, wy, batchSize) {
  // True gradient
  const [gx, gy] = grad(wx, wy);
  if(batchSize >= N_DATA) return [gx, gy]; // full batch = exact
  // Mini-batch: sample batchSize indices, compute gradient mean + scaled noise
  const scale = 1/Math.sqrt(batchSize);
  // Noise derived from variation between sampled points (simplified)
  const nx = (rng()-0.5)*scale*0.8;
  const ny = (rng()-0.5)*scale*0.8;
  return [gx+nx, gy+ny];
}

const COLORS = ['#e63946','#457b9d','#2a9d8f'];
const LABELS = ['Full-batch','Mini-batch','Pure SGD'];
const BATCH_SIZES = [200, 8, 1];
const START = [0.1, 0.85]; // starting point

let optimizers = [];
let running = false;
let animId = null;
let stepCount = 0;

function initOptimizers(lr) {
  optimizers = BATCH_SIZES.map((b, i) => ({
    wx: START[0], wy: START[1],
    color: COLORS[i], label: LABELS[i], batch: b,
    path: [[START[0], START[1]]],
    done: false
  }));
  stepCount = 0;
}

function drawPaths() {
  drawHeatmap();
  for(const opt of optimizers) {
    if(opt.path.length < 2) continue;
    ctx.save();
    ctx.strokeStyle = opt.color;
    ctx.lineWidth = 2;
    ctx.globalAlpha = 0.85;
    ctx.beginPath();
    const [sx,sy] = toCanvas(opt.path[0][0], opt.path[0][1]);
    ctx.moveTo(sx,sy);
    for(let i=1;i<opt.path.length;i++){
      const [cx,cy]=toCanvas(opt.path[i][0],opt.path[i][1]);
      ctx.lineTo(cx,cy);
    }
    ctx.stroke();
    // Draw current head
    const [hx,hy]=toCanvas(opt.wx,opt.wy);
    ctx.globalAlpha=1;
    ctx.fillStyle=opt.color;
    ctx.beginPath();
    ctx.arc(hx,hy,4.5,0,Math.PI*2);
    ctx.fill();
    ctx.restore();
  }
}

function step() {
  const lr = parseFloat(document.getElementById('lr').value);
  let allDone = true;
  for(const opt of optimizers) {
    if(opt.done) continue;
    allDone = false;
    const [gx,gy] = batchGrad(opt.wx, opt.wy, opt.batch);
    opt.wx -= lr * gx;
    opt.wy -= lr * gy;
    // Clamp to [0,1]
    opt.wx = Math.max(0,Math.min(1,opt.wx));
    opt.wy = Math.max(0,Math.min(1,opt.wy));
    opt.path.push([opt.wx, opt.wy]);
    // Stop if close to minimum
    const d = Math.sqrt((opt.wx-0.6)**2+(opt.wy-0.4)**2);
    if(d < 0.015 || opt.path.length > 600) opt.done = true;
  }
  stepCount++;
  drawPaths();
  const info = document.getElementById('info');
  const doneLabels = optimizers.filter(o=>o.done).map(o=>`${o.label}(${o.path.length-1})`);
  info.textContent = doneLabels.length
    ? fmt('Converged: %doneLabels% steps to reach minimum.', { doneLabels: doneLabels.join(', ') })
    : fmt('Step %stepCount% — optimizers running…', { stepCount: stepCount });
  if(allDone) { running=false; info.textContent=fmt('All converged. %map%', { map: optimizers.map(o=>`${o.label}: ${o.path.length-1} steps`).join(' | ') }); return; }
  if(running) animId = requestAnimationFrame(step);
}

document.getElementById('lr').addEventListener('input', function(){
  document.getElementById('lrVal').textContent = (+this.value).toFixed(3);
});

document.getElementById('runBtn').addEventListener('click', function(){
  if(running){ running=false; cancelAnimationFrame(animId); this.textContent='Run'; return; }
  const bs = parseInt(document.getElementById('bs').value);
  // Override middle optimizer batch size with user choice
  const lr = parseFloat(document.getElementById('lr').value);
  optimizers = [
    {wx:START[0],wy:START[1],color:COLORS[0],label:'Full-batch',batch:200,path:[[START[0],START[1]]],done:false},
    {wx:START[0],wy:START[1],color:COLORS[1],label:'Mini-batch',batch:bs,path:[[START[0],START[1]]],done:false},
    {wx:START[0],wy:START[1],color:COLORS[2],label:'Pure SGD',batch:1,path:[[START[0],START[1]]],done:false}
  ];
  stepCount=0;
  running=true;
  this.textContent='Pause';
  step();
});

document.getElementById('resetBtn').addEventListener('click', function(){
  running=false;
  cancelAnimationFrame(animId);
  document.getElementById('runBtn').textContent='Run';
  document.getElementById('info').textContent='Press Run to start all three optimizers.';
  initOptimizers(parseFloat(document.getElementById('lr').value));
  drawHeatmap();
});

// Init
buildHeatmap();
initOptimizers(0.05);
drawHeatmap();

// Verify core algorithm in console
// Gradient at (0.6,0.4): sin(5*0)=0 so perturb gradient is 0 there
const [gx0,gy0] = grad(0.6,0.4);
console.assert(Math.abs(gx0)<1e-4 && Math.abs(gy0)<1e-4,
  fmt('Gradient at minimum should be ~0, got %gx0%,%gy0%', { gx0: gx0, gy0: gy0 }));
const l0 = loss(0.6,0.4);
const l1 = loss(0.1,0.85);
console.assert(l0 < l1, 'Minimum loss should be less than starting loss');
// Gradient descent from (0.5,0.5) should converge near (0.6,0.4)
let wx=0.5, wy=0.5;
for(let i=0;i<200;i++){const [gx,gy]=grad(wx,wy);wx-=0.04*gx;wy-=0.04*gy;wx=Math.max(0,Math.min(1,wx));wy=Math.max(0,Math.min(1,wy));}
console.assert(Math.abs(wx-0.6)<0.05 && Math.abs(wy-0.4)<0.05, fmt('GD should reach minimum region, got %wx%,%wy%', { wx: wx.toFixed(3), wy: wy.toFixed(3) }));
console.log(fmt('Demo verify OK: min loss=%l0%, start loss=%l1%, GD converged to wx=%wx% wy=%wy%', { l0: l0.toFixed(4), l1: l1.toFixed(4), wx: wx.toFixed(4), wy: wy.toFixed(4) }));

<div class="controls">
  <label>Learning rate: <span id="lrVal">0.05</span>
    <input type="range" id="lr" min="0.005" max="0.15" step="0.005" value="0.05">
  </label>
  <label>Batch size:
    <select id="bs">
      <option value="1">1 (pure SGD)</option>
      <option value="8" selected>8 (mini-batch)</option>
      <option value="200">200 (full batch)</option>
    </select>
  </label>
  <div class="btn-row">
    <button id="runBtn">Run</button>
    <button id="resetBtn" class="ghost">Reset</button>
  </div>
</div>
<canvas id="cv" width="380" height="280"></canvas>
<div id="legend" class="legend">
  <span class="dot" style="background:#e63946"></span> Full-batch (B=200)
  <span class="dot" style="background:#457b9d"></span> Mini-batch (B=8)
  <span class="dot" style="background:#2a9d8f"></span> Pure SGD (B=1)
</div>
<div id="info" class="info">Press Run to start all three optimizers.</div>

*{box-sizing:border-box;margin:0;padding:0}
body{font-family:system-ui,sans-serif;color:#222;background:#f8f9fa;padding:.6rem}
.controls{display:flex;flex-wrap:wrap;gap:.5rem .9rem;align-items:center;margin-bottom:.5rem}
label{display:flex;align-items:center;gap:.35rem;font-size:.85rem;font-weight:600}
input[type=range]{width:100px;cursor:pointer}
select{font-size:.85rem;padding:.2rem .4rem;border:1px solid #adb5bd;border-radius:6px;cursor:pointer}
.btn-row{display:flex;gap:.4rem}
button{font:600 13px system-ui;padding:.35rem .8rem;border:1px solid #1d3557;
       background:#1d3557;color:#fff;border-radius:7px;cursor:pointer}
button.ghost{background:#fff;color:#1d3557}
canvas{display:block;border-radius:10px;border:1px solid #dee2e6;width:100%;max-width:380px}
.legend{display:flex;gap:.8rem;flex-wrap:wrap;margin-top:.4rem;font-size:.8rem;align-items:center}
.dot{display:inline-block;width:10px;height:10px;border-radius:50%;margin-right:2px}
.info{font-size:.82rem;color:#495057;margin-top:.3rem;min-height:1.2em}

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// ── Loss surface: Rosenbrock-like 2D function ───────────────────────────────────────────
// We use a scaled bowl with a ridge to make optimizer differences visible.
// True minimum at (wx=0.6, wy=0.4) in [0,1]^2.
const W = 380, H = 280;
const cv = document.getElementById('cv');
const ctx = cv.getContext('2d');

// Loss function in parameter space [0,1]^2
// Minimum at (wx=0.6, wy=0.4). Elongated bowl with a sin*sin ridge
// that is zero at the minimum (preserving it) but adds interesting topology.
function loss(wx, wy) {
  const a = wx - 0.6, b = wy - 0.4;
  return 3*a*a + 7*b*b + 1.5*a*b + 0.25*Math.sin(5*a)*Math.sin(5*b);
}

// Numerical gradient
function grad(wx, wy) {
  const eps = 1e-4;
  return [
    (loss(wx+eps, wy) - loss(wx-eps, wy)) / (2*eps),
    (loss(wx, wy+eps) - loss(wx, wy-eps)) / (2*eps)
  ];
}

// Canvas coordinate transforms
const PAD = 20;
function toCanvas(wx, wy) {
  return [PAD + wx*(W-2*PAD), PAD + (1-wy)*(H-2*PAD)];
}

// ── Render the heatmap ──────────────────────────────────────────────────────────
let heatmapImg = null;
function buildHeatmap() {
  const id = ctx.createImageData(W, H);
  const d = id.data;
  // Find min/max
  let mn=1e9, mx=-1e9;
  for(let py=0;py<H;py++) for(let px=0;px<W;px++){
    const wx=(px-PAD)/(W-2*PAD), wy=1-(py-PAD)/(H-2*PAD);
    if(wx<0||wx>1||wy<0||wy>1) continue;
    const v=loss(wx,wy); if(v<mn)mn=v; if(v>mx)mx=v;
  }
  // Color: cool blue → warm red
  for(let py=0;py<H;py++) for(let px=0;px<W;px++){
    const i=(py*W+px)*4;
    const wx=(px-PAD)/(W-2*PAD), wy=1-(py-PAD)/(H-2*PAD);
    if(wx<0||wx>1||wy<0||wy>1){
      d[i]=248;d[i+1]=249;d[i+2]=250;d[i+3]=255; continue;
    }
    const t=Math.pow((loss(wx,wy)-mn)/(mx-mn),0.55); // gamma
    // cool=dark-blue(29,53,87) to warm=red(230,57,70)
    d[i]   = Math.round(29 + t*(230-29));
    d[i+1] = Math.round(53 + t*(57-53));
    d[i+2] = Math.round(87 + t*(70-87));
    d[i+3] = 255;
  }
  heatmapImg = id;
}

function drawHeatmap() {
  if(!heatmapImg) buildHeatmap();
  ctx.putImageData(heatmapImg, 0, 0);
  // Draw minimum star
  const [mx,my] = toCanvas(0.6, 0.4);
  ctx.save();
  ctx.strokeStyle='#fff';ctx.lineWidth=1.5;
  ctx.beginPath();
  for(let i=0;i<5;i++){
    const a1=i*Math.PI*2/5-Math.PI/2, a2=a1+Math.PI/5;
    const x1=mx+6*Math.cos(a1),y1=my+6*Math.sin(a1);
    const x2=mx+2.5*Math.cos(a2),y2=my+2.5*Math.sin(a2);
    i===0?ctx.moveTo(x1,y1):ctx.lineTo(x1,y1);
    ctx.lineTo(x2,y2);
  }
  ctx.closePath(); ctx.stroke(); ctx.restore();
}

// ── Optimizer state ─────────────────────────────────────────────────────────────────────
// Simulated "dataset": 200 2D points with noise that shifts the apparent gradient
const N_DATA = 200;
const DATA = [];
const rng = (() => { let s=12345; return ()=>{ s=(s*1664525+1013904223)&0xffffffff; return (s>>>0)/4294967296; }; })();
for(let i=0;i<N_DATA;i++) DATA.push([rng(), rng()]);

// Noisy loss gradient for a batch: true gradient + noise proportional to 1/sqrt(B)
function batchGrad(wx, wy, batchSize) {
  // True gradient
  const [gx, gy] = grad(wx, wy);
  if(batchSize >= N_DATA) return [gx, gy]; // full batch = exact
  // Mini-batch: sample batchSize indices, compute gradient mean + scaled noise
  const scale = 1/Math.sqrt(batchSize);
  // Noise derived from variation between sampled points (simplified)
  const nx = (rng()-0.5)*scale*0.8;
  const ny = (rng()-0.5)*scale*0.8;
  return [gx+nx, gy+ny];
}

const COLORS = ['#e63946','#457b9d','#2a9d8f'];
const LABELS = ['Full-batch','Mini-batch','Pure SGD'];
const BATCH_SIZES = [200, 8, 1];
const START = [0.1, 0.85]; // starting point

let optimizers = [];
let running = false;
let animId = null;
let stepCount = 0;

function initOptimizers(lr) {
  optimizers = BATCH_SIZES.map((b, i) => ({
    wx: START[0], wy: START[1],
    color: COLORS[i], label: LABELS[i], batch: b,
    path: [[START[0], START[1]]],
    done: false
  }));
  stepCount = 0;
}

function drawPaths() {
  drawHeatmap();
  for(const opt of optimizers) {
    if(opt.path.length < 2) continue;
    ctx.save();
    ctx.strokeStyle = opt.color;
    ctx.lineWidth = 2;
    ctx.globalAlpha = 0.85;
    ctx.beginPath();
    const [sx,sy] = toCanvas(opt.path[0][0], opt.path[0][1]);
    ctx.moveTo(sx,sy);
    for(let i=1;i<opt.path.length;i++){
      const [cx,cy]=toCanvas(opt.path[i][0],opt.path[i][1]);
      ctx.lineTo(cx,cy);
    }
    ctx.stroke();
    // Draw current head
    const [hx,hy]=toCanvas(opt.wx,opt.wy);
    ctx.globalAlpha=1;
    ctx.fillStyle=opt.color;
    ctx.beginPath();
    ctx.arc(hx,hy,4.5,0,Math.PI*2);
    ctx.fill();
    ctx.restore();
  }
}

function step() {
  const lr = parseFloat(document.getElementById('lr').value);
  let allDone = true;
  for(const opt of optimizers) {
    if(opt.done) continue;
    allDone = false;
    const [gx,gy] = batchGrad(opt.wx, opt.wy, opt.batch);
    opt.wx -= lr * gx;
    opt.wy -= lr * gy;
    // Clamp to [0,1]
    opt.wx = Math.max(0,Math.min(1,opt.wx));
    opt.wy = Math.max(0,Math.min(1,opt.wy));
    opt.path.push([opt.wx, opt.wy]);
    // Stop if close to minimum
    const d = Math.sqrt((opt.wx-0.6)**2+(opt.wy-0.4)**2);
    if(d < 0.015 || opt.path.length > 600) opt.done = true;
  }
  stepCount++;
  drawPaths();
  const info = document.getElementById('info');
  const doneLabels = optimizers.filter(o=>o.done).map(o=>`${o.label}(${o.path.length-1})`);
  info.textContent = doneLabels.length
    ? fmt('Converged: %doneLabels% steps to reach minimum.', { doneLabels: doneLabels.join(', ') })
    : fmt('Step %stepCount% — optimizers running…', { stepCount: stepCount });
  if(allDone) { running=false; info.textContent=fmt('All converged. %map%', { map: optimizers.map(o=>`${o.label}: ${o.path.length-1} steps`).join(' | ') }); return; }
  if(running) animId = requestAnimationFrame(step);
}

document.getElementById('lr').addEventListener('input', function(){
  document.getElementById('lrVal').textContent = (+this.value).toFixed(3);
});

document.getElementById('runBtn').addEventListener('click', function(){
  if(running){ running=false; cancelAnimationFrame(animId); this.textContent='Run'; return; }
  const bs = parseInt(document.getElementById('bs').value);
  // Override middle optimizer batch size with user choice
  const lr = parseFloat(document.getElementById('lr').value);
  optimizers = [
    {wx:START[0],wy:START[1],color:COLORS[0],label:'Full-batch',batch:200,path:[[START[0],START[1]]],done:false},
    {wx:START[0],wy:START[1],color:COLORS[1],label:'Mini-batch',batch:bs,path:[[START[0],START[1]]],done:false},
    {wx:START[0],wy:START[1],color:COLORS[2],label:'Pure SGD',batch:1,path:[[START[0],START[1]]],done:false}
  ];
  stepCount=0;
  running=true;
  this.textContent='Pause';
  step();
});

document.getElementById('resetBtn').addEventListener('click', function(){
  running=false;
  cancelAnimationFrame(animId);
  document.getElementById('runBtn').textContent='Run';
  document.getElementById('info').textContent='Press Run to start all three optimizers.';
  initOptimizers(parseFloat(document.getElementById('lr').value));
  drawHeatmap();
});

// Init
buildHeatmap();
initOptimizers(0.05);
drawHeatmap();

// Verify core algorithm in console
// Gradient at (0.6,0.4): sin(5*0)=0 so perturb gradient is 0 there
const [gx0,gy0] = grad(0.6,0.4);
console.assert(Math.abs(gx0)<1e-4 && Math.abs(gy0)<1e-4,
  fmt('Gradient at minimum should be ~0, got %gx0%,%gy0%', { gx0: gx0, gy0: gy0 }));
const l0 = loss(0.6,0.4);
const l1 = loss(0.1,0.85);
console.assert(l0 < l1, 'Minimum loss should be less than starting loss');
// Gradient descent from (0.5,0.5) should converge near (0.6,0.4)
let wx=0.5, wy=0.5;
for(let i=0;i<200;i++){const [gx,gy]=grad(wx,wy);wx-=0.04*gx;wy-=0.04*gy;wx=Math.max(0,Math.min(1,wx));wy=Math.max(0,Math.min(1,wy));}
console.assert(Math.abs(wx-0.6)<0.05 && Math.abs(wy-0.4)<0.05, fmt('GD should reach minimum region, got %wx%,%wy%', { wx: wx.toFixed(3), wy: wy.toFixed(3) }));
console.log(fmt('Demo verify OK: min loss=%l0%, start loss=%l1%, GD converged to wx=%wx% wy=%wy%', { l0: l0.toFixed(4), l1: l1.toFixed(4), wx: wx.toFixed(4), wy: wy.toFixed(4) }));

Result

Console

Notice that the full-batch path is smooth but each step costs a full sweep of the data. The mini-batch path is jagged but much cheaper per step and often reaches a good solution faster in wall-clock time. Single-sample SGD bounces most — but sometimes crosses barriers that the others cannot.

The Real Complexity

SGD is deceptively simple to state; understanding why it converges is more subtle.

Convex case. When the loss surface is a bowl (convex), Robbins and Monro proved in 1951 that SGD converges to the global minimum if the learning rate schedule satisfies two conditions: the steps must sum to infinity (so the optimizer can reach anywhere) and the sum of squares must be finite (so the noise eventually dies). A harmonic schedule

\eta_t = \eta_0 / t

satisfies both.

Non-convex case. Deep networks have wildly non-convex loss landscapes — riddled with saddle points, flat plateaus, and countless local minima. Here the theory only guarantees convergence to a stationary point (where the gradient is zero), not a global minimum. Yet in practice, the minima that SGD finds in deep networks generalize remarkably well. A 2020 result by Chatterjee and others suggests that neural network loss landscapes are more benign than worst-case theory implies.

Momentum (Polyak, 1964) accumulates a velocity term: instead of stepping along the raw gradient, the optimizer builds momentum in directions of consistent gradient. This damps oscillations across steep ravines and accelerates progress along shallow valleys. PyTorch's default SGD uses momentum=0.9.

Adam (Kingma & Ba, 2014) goes further: it maintains per-parameter adaptive learning rates, dividing each gradient by a running estimate of its magnitude. Rare parameters get larger effective steps; common parameters get smaller ones. Adam is the default in almost every modern deep learning pipeline.

The cost per step is $O(B \cdot d)$ , where $B$ is the batch size and $d$ is the number of parameters. Full-batch gradient descent uses $B = N$ (all examples); pure SGD uses $B = 1$ ; mini-batch typically uses $B \in \{32, 256, 1024\}$ . Modern hardware (GPUs) parallelizes over the batch, so going from B=32 to B=256 costs little extra time while dramatically reducing gradient noise. See also non-convex optimization for the broader landscape of difficult optimization problems.

Where It Matters

SGD and its adaptive variants are the universal training algorithm for virtually every large-scale machine-learning system:

Large language models: GPT-4, LLaMA, Gemini and every other transformer are trained with Adam or a close variant over trillions of tokens. The sheer scale makes full-batch gradients inconceivable; mini-batches of a few thousand tokens run in parallel across thousands of GPUs.
Image recognition: AlexNet (2012) famously used SGD with momentum to win ImageNet, triggering the modern deep-learning era. Every subsequent vision model — ResNet, ViT, CLIP — follows the same recipe.
Recommendation systems: Netflix, YouTube and Spotify optimize embedding tables with billions of parameters using variants of Adam. Mini-batches correspond to random user sessions.
Reinforcement learning: policy-gradient methods like PPO (used to train ChatGPT's RLHF stage) apply SGD-style updates to reward signals, not supervised labels.
Scientific computing: neural network training for physics simulations (PINNs) and protein-structure prediction (AlphaFold) all rely on Adam to navigate high-dimensional loss surfaces with complex geometry.

The mini-batch idea has spread beyond neural networks too: online learning algorithms, Bayesian stochastic variational inference, and large-scale linear models all borrow the core SGD trick of replacing the full expectation with a cheap sample.

Conclusion

Stochastic gradient descent made one substitution — swap the exact gradient for a noisy mini-batch estimate — and that substitution unlocked an era. Training on millions of examples became tractable; noise that looked like a liability turned into a feature; and variants like Adam fine-tuned the idea into the universal workhorse of modern AI.

The deeper lesson is that approximate is often better than exact when the problem is large enough. Full-batch gradient descent is provably correct at each step but practically useless at scale. SGD is provably wrong at each step but practically unbeatable. Understanding why the wrong algorithm wins is one of the most productive questions in machine learning theory — and one we are still answering. Explore non-convex optimization if you want to dive into the landscape SGD is navigating.

Comments

Loading comments...

https://www.kipuhub.com/en/article/stochastic-gradient-descent/Content licensed under CC BY-NC 4.0.

Cookies & privacy

Stochastic Gradient Descent

Why the noisiest optimizer wins