Introduction

Every time you use a large language model, ask an AI to generate an image, or get a machine translation, you are seeing the result of optimization. Training a neural network means adjusting millions of numbers — the weights — until the model's predictions match the training data. The algorithm doing that adjusting is called an optimizer, and for the last decade one optimizer has dominated: Adam.

Adam was introduced in 2014 by Diederik Kingma and Jimmy Ba in a paper that has since become one of the most cited in all of computer science. The name stands for Adaptive Moment Estimation. The core insight is elegantly simple: instead of nudging every weight by the same step size, Adam maintains a separate running estimate for each weight — tracking both the typical size of its gradients and how noisy those gradients are — and uses those estimates to set an individual step for that weight at every training iteration.

The result is an optimizer that is fast, almost hyperparameter-free in practice, and robust enough to work across an enormous range of architectures and tasks without hand-tuning.

Try It: Race Down a Loss Surface

The surface below is a 2-D loss landscape — think of it as the error your model makes as two weights vary. The goal is to reach the minimum (the darkest valley). Click anywhere on the surface to place a starting point, then press Run to watch both optimizers descend simultaneously.

<p class="hint">Click anywhere on the loss surface to place a starting point, then press <b>Run</b>. Watch Adam (orange) and plain gradient descent (blue) race to the minimum (white dot at the banana's tip).</p>
<canvas id="canvas" width="380" height="280"></canvas>
<div class="info-row">
  <span class="dot adam-dot"></span><span>Adam</span>
  <span class="dot gd-dot"></span><span>Gradient Descent</span>
</div>
<div class="stats" id="stats">Click the surface above to start.</div>
<div class="btns">
  <button id="runBtn" type="button" disabled>Run</button>
  <button id="resetBtn" type="button" class="ghost">Reset</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .85rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
canvas { display: block; border-radius: 10px; cursor: crosshair; max-width: 100%; }
.info-row { display: flex; align-items: center; gap: .45rem; font-size: .82rem; margin: .45rem 0 .15rem; }
.dot { width: 12px; height: 12px; border-radius: 50%; display: inline-block; }
.adam-dot { background: #e07b00; }
.gd-dot { background: #1d6fa8; }
.stats { font-size: .82rem; color: #555; min-height: 2.8em; line-height: 1.5; }
.btns { display: flex; gap: .5rem; margin-top: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .42rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button:disabled { opacity: .45; cursor: default; }
button.ghost { background: #fff; color: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Loss surface: Rosenbrock "banana" function
//   f(x,y) = (1-x)^2 + 2*(y-x^2)^2
// Global minimum: f(1,1) = 0, gradients both 0 at (1,1).
// This surface has a long curved narrow valley that plain GD
// oscillates inside, while Adam's adaptive steps glide along it.
const W = 380, H = 280;
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');

// Parameter space: x in [-2, 2], y in [-1, 3]
const X0 = -2, X1 = 2, Y0 = -1, Y1 = 3;
const px2x = px => X0 + (px / W) * (X1 - X0);
const px2y = py => Y1 - (py / H) * (Y1 - Y0);  // y flipped: top=Y1
const x2px = x  => ((x - X0) / (X1 - X0)) * W;
const y2py = y  => ((Y1 - y) / (Y1 - Y0)) * H;

function loss(x, y) {
  return (1 - x) * (1 - x) + 2 * (y - x * x) * (y - x * x);
}
function gradX(x, y) {
  return -2 * (1 - x) - 8 * x * (y - x * x);
}
function gradY(x, y) {
  return 4 * (y - x * x);
}

// Pre-render the surface using log-scaled loss for better visual contrast
const imgData = ctx.createImageData(W, H);
for (let py = 0; py < H; py++) for (let px = 0; px < W; px++) {
  const lv = Math.log1p(loss(px2x(px), px2y(py)));
  // map [0, log(~25)] to colour: deep purple (low) -> teal -> yellow (high)
  const maxLog = Math.log1p(25);
  const t = Math.min(1, lv / maxLog);
  const r = Math.round(20  + t * 220);
  const g = Math.round(30  + t * 140);
  const b = Math.round(100 + (1 - t) * 110);
  const i = (py * W + px) * 4;
  imgData.data[i] = r; imgData.data[i+1] = g; imgData.data[i+2] = b; imgData.data[i+3] = 255;
}

// Mark the global minimum at (1, 1)
const MIN_X = 1, MIN_Y = 1;
function drawSurface() {
  ctx.putImageData(imgData, 0, 0);
  ctx.beginPath();
  ctx.arc(x2px(MIN_X), y2py(MIN_Y), 5, 0, 2 * Math.PI);
  ctx.fillStyle = '#fff'; ctx.fill();
  ctx.strokeStyle = '#333'; ctx.lineWidth = 1.5; ctx.stroke();
  // Label
  ctx.fillStyle = '#fff'; ctx.font = '10px system-ui';
  ctx.fillText('min', x2px(MIN_X) + 7, y2py(MIN_Y) + 4);
}
drawSurface();

let startX = null, startY = null;
let running = false, animId = null;

function setStatus(t) { document.getElementById('stats').innerHTML = t; }

canvas.addEventListener('click', e => {
  if (running) return;
  const r = canvas.getBoundingClientRect();
  const px = (e.clientX - r.left) * (W / r.width);
  const py = (e.clientY - r.top)  * (H / r.height);
  startX = px2x(px); startY = px2y(py);
  drawSurface();
  ctx.beginPath();
  ctx.arc(x2px(startX), y2py(startY), 5, 0, 2 * Math.PI);
  ctx.fillStyle = '#eee'; ctx.fill();
  ctx.strokeStyle = '#555'; ctx.lineWidth = 1.5; ctx.stroke();
  document.getElementById('runBtn').disabled = false;
  setStatus(fmt('Start: (%startX%, %startY%)  loss=%loss%. Press Run.', { startX: startX.toFixed(2), startY: startY.toFixed(2), loss: loss(startX, startY).toFixed(2) }));
});

document.getElementById('runBtn').addEventListener('click', () => {
  if (startX === null || running) return;
  running = true;
  document.getElementById('runBtn').disabled = true;
  runAnimation();
});

document.getElementById('resetBtn').addEventListener('click', () => {
  if (animId) cancelAnimationFrame(animId);
  running = false; startX = null; startY = null;
  drawSurface();
  document.getElementById('runBtn').disabled = true;
  setStatus('Click the surface above to start.');
});

function runAnimation() {
  // Adam hyperparameters (same as paper defaults, larger alpha for visibility)
  let ax = startX, ay = startY;
  let am1x = 0, am1y = 0, am2x = 0, am2y = 0;
  const aAlpha = 0.005, aBeta1 = 0.9, aBeta2 = 0.999, aEps = 1e-8;

  // Plain gradient descent
  let gx = startX, gy = startY;
  const gAlpha = 0.002;

  const STEPS = 2000;
  let step = 0;
  const adamPath = [[ax, ay]];
  const gdPath   = [[gx, gy]];
  // Subsample paths for drawing (keep every Nth point)
  const DRAW_EVERY = 5;

  function tick() {
    for (let i = 0; i < 20 && step < STEPS; i++, step++) {
      const t = step + 1;
      // Adam update
      const agx = gradX(ax, ay), agy = gradY(ax, ay);
      am1x = aBeta1 * am1x + (1 - aBeta1) * agx;
      am1y = aBeta1 * am1y + (1 - aBeta1) * agy;
      am2x = aBeta2 * am2x + (1 - aBeta2) * agx * agx;
      am2y = aBeta2 * am2y + (1 - aBeta2) * agy * agy;
      const b1t = 1 - Math.pow(aBeta1, t);
      const b2t = 1 - Math.pow(aBeta2, t);
      ax -= aAlpha * (am1x / b1t) / (Math.sqrt(am2x / b2t) + aEps);
      ay -= aAlpha * (am1y / b1t) / (Math.sqrt(am2y / b2t) + aEps);
      if (step % DRAW_EVERY === 0) adamPath.push([ax, ay]);

      // GD update
      gx -= gAlpha * gradX(gx, gy);
      gy -= gAlpha * gradY(gx, gy);
      if (step % DRAW_EVERY === 0) gdPath.push([gx, gy]);
    }

    drawSurface();
    // Start marker
    ctx.beginPath();
    ctx.arc(x2px(startX), y2py(startY), 4, 0, 2 * Math.PI);
    ctx.fillStyle = '#eee'; ctx.fill();
    ctx.strokeStyle = '#666'; ctx.lineWidth = 1.2; ctx.stroke();

    // GD path (blue)
    if (gdPath.length > 1) {
      ctx.beginPath();
      ctx.moveTo(x2px(gdPath[0][0]), y2py(gdPath[0][1]));
      for (let i = 1; i < gdPath.length; i++) ctx.lineTo(x2px(gdPath[i][0]), y2py(gdPath[i][1]));
      ctx.strokeStyle = '#3a9bd5'; ctx.lineWidth = 2; ctx.stroke();
    }
    ctx.beginPath();
    ctx.arc(x2px(gx), y2py(gy), 5, 0, 2 * Math.PI);
    ctx.fillStyle = '#1d6fa8'; ctx.fill();

    // Adam path (orange)
    if (adamPath.length > 1) {
      ctx.beginPath();
      ctx.moveTo(x2px(adamPath[0][0]), y2py(adamPath[0][1]));
      for (let i = 1; i < adamPath.length; i++) ctx.lineTo(x2px(adamPath[i][0]), y2py(adamPath[i][1]));
      ctx.strokeStyle = '#f5a34a'; ctx.lineWidth = 2; ctx.stroke();
    }
    ctx.beginPath();
    ctx.arc(x2px(ax), y2py(ay), 5, 0, 2 * Math.PI);
    ctx.fillStyle = '#e07b00'; ctx.fill();

    const aL = loss(ax, ay), gL = loss(gx, gy);
    setStatus(fmt('Step %step% / %STEPS%<br>Adam loss: <b>%aL%</b> at (%ax%, %ay%)<br>GD&nbsp;&nbsp; loss: <b>%gL%</b> at (%gx%, %gy%)', { step: step, STEPS: STEPS, aL: aL.toFixed(5), ax: ax.toFixed(3), ay: ay.toFixed(3), gL: gL.toFixed(5), gx: gx.toFixed(3), gy: gy.toFixed(3) }));

    if (step < STEPS) {
      animId = requestAnimationFrame(tick);
    } else {
      running = false;
      document.getElementById('runBtn').disabled = false;
      console.log('Adam final loss:', loss(ax, ay), 'at', ax.toFixed(4), ay.toFixed(4));
      console.log('GD   final loss:', loss(gx, gy), 'at', gx.toFixed(4), gy.toFixed(4));
    }
  }
  animId = requestAnimationFrame(tick);
}
Result
Console

Notice how plain gradient descent (blue) takes uniform steps and can stall on shallow plateaus or oscillate in narrow ravines. Adam (orange) adapts: it takes larger steps where gradients have been consistently small (flat regions) and smaller steps where gradients swing wildly (steep ravines). Adam almost always finds the valley faster — and that speed advantage multiplies enormously when the loss surface has millions of dimensions instead of two.

How Adam Really Works

Adam's update rule uses three ideas woven together. At each training step t, for a parameter θ\theta with gradient g:

1. First moment (momentum). Adam keeps a running average of the gradient itself:

m=β1m+(1−β1)gm = \beta_1 m + (1 - \beta_1) g

This smooths out noise — if the gradient keeps pointing in one direction, m builds up and the step accelerates. β1\beta_1 is usually 0.9 (forget only 10% of history each step).

2. Second moment (adaptive scale). Adam also keeps a running average of the squared gradient:

v=β2v+(1−β2)g2v = \beta_2 v + (1 - \beta_2) g^2

Large v means the gradient for this parameter has been large or noisy; small v means the gradient has been small and consistent. β2\beta_2 is usually 0.999.

3. Bias correction. Because both averages start at zero, they are biased toward zero in the early steps. Adam corrects this:

m^=m1−β1tv^=v1−β2t\hat{m} = \frac{m}{1 - \beta_1^t} \qquad \hat{v} = \frac{v}{1 - \beta_2^t}

4. The update.

θ=θ−α⋅m^v^+ϵ\theta = \theta - \alpha \cdot \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}

The learning rate α\alpha (typically 0.001) is divided by v^\sqrt{\hat{v}}, so parameters with large or noisy gradients get smaller steps, and parameters with tiny, consistent gradients get larger steps. ϵ\epsilon (≈10−8\approx 10^{-8}) prevents division by zero.

Status: solved algorithm. Adam is not a complexity-theory puzzle — it is a practical method with proven convergence guarantees under mild conditions (Kingma & Ba, 2014). Each step costs O(n)O(n) time and memory where n is the number of parameters — the same as plain gradient descent — so the adaptive bookkeeping is essentially free.

Adam can be seen as combining two earlier ideas: momentum (the first moment) and RMSProp (the second moment normalization, proposed by Geoff Hinton in 2012). Kingma and Ba showed that adding bias correction and combining both moments gave a single method that matched or outperformed every predecessor on the tasks they tested. See also how neural network training fits the wider picture of non-convex optimization.

Where It Matters

Adam's adaptive per-parameter steps matter most precisely where modern deep learning operates:

  • Large language models: Transformers have attention heads, embedding matrices, and feed-forward layers that span wildly different gradient scales. Adam's normalization keeps all of them moving at a sensible pace simultaneously.
  • Image generation (diffusion models, GANs): training is notoriously unstable. Adam's momentum smoothing dampens the oscillations that cause vanilla gradient descent to diverge.
  • Sparse gradients: in natural language, most word embeddings receive zero gradient for most batches — only the words actually seen are updated. Adam's second moment accumulates slowly for rare parameters and gives them a large effective step when they do appear.
  • Recommendation systems: embedding tables have billions of entries with highly imbalanced update frequencies. Adam handles this naturally where SGD struggles.
  • Reinforcement learning: reward signals are noisy and non-stationary. Adam's running estimates act as a form of signal smoothing that keeps training stable.

Variants of Adam — AdaGrad, RMSProp, AdamW, Nadam, Lion — address specific weaknesses (weight decay, generalization, memory) but all share the same core idea: track gradient history per parameter and adapt the step accordingly. AdamW in particular, which decouples weight decay from the adaptive update, is the standard choice for training large language models today.

Conclusion

Three equations, two hyperparameters you almost never need to tune, and O(n)O(n) cost per step — Adam's elegance is that it turns a single insight (track gradient history per parameter) into a method that works out of the box on nearly every deep learning problem.

It is not magic. Adam does not guarantee finding the global minimum of a non-convex loss surface, and on some problems carefully tuned SGD with momentum can generalize better. But as a robust default that gets models trained quickly and reliably, Adam has earned its place as the most widely used optimizer in the history of machine learning.

The next time a model surprises you — a chatbot that reasons, an image generator that dreams — remember that behind it, running quietly for millions of steps, were three moving averages doing arithmetic on gradients.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/adam-optimizer/Content licensed under CC BY-NC 4.0.