The Adam Optimizer

Every time you use a large language model, ask an AI to generate an image, or get a machine translation, you are seeing the result of optimization. Training a neural network means adjusting millions of numbers — the weights — until the model's predictions match the training data. The algorithm doing that adjusting is called an optimizer, and for the last decade one optimizer has dominated: Adam.

Adam was introduced in 2014 by Diederik Kingma and Jimmy Ba in a paper that has since become one of the most cited in all of computer science. The name stands for Adaptive Moment Estimation. The core insight is elegantly simple: instead of nudging every weight by the same step size, Adam maintains a separate running estimate for each weight — tracking both the typical size of its gradients and how noisy those gradients are — and uses those estimates to set an individual step for that weight at every training iteration.

The result is an optimizer that is fast, almost hyperparameter-free in practice, and robust enough to work across an enormous range of architectures and tasks without hand-tuning.

The surface below is a 2-D loss landscape — think of it as the error your model makes as two weights vary. The goal is to reach the minimum (the darkest valley). Click anywhere on the surface to place a starting point, then press Run to watch both optimizers descend simultaneously.

<p class="hint">{{hint}}</p>
<canvas id="canvas" width="380" height="280"></canvas>
<div class="info-row">
  <span class="dot adam-dot"></span><span>{{legend_adam}}</span>
  <span class="dot gd-dot"></span><span>{{legend_gd}}</span>
</div>
<div class="stats" id="stats">{{stats_initial}}</div>
<div class="btns">
  <button id="runBtn" type="button" disabled>{{btn_run}}</button>
  <button id="resetBtn" type="button" class="ghost">{{btn_reset}}</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .85rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
canvas { display: block; border-radius: 10px; cursor: crosshair; max-width: 100%; }
.info-row { display: flex; align-items: center; gap: .45rem; font-size: .82rem; margin: .45rem 0 .15rem; }
.dot { width: 12px; height: 12px; border-radius: 50%; display: inline-block; }
.adam-dot { background: #e07b00; }
.gd-dot { background: #1d6fa8; }
.stats { font-size: .82rem; color: #555; min-height: 2.8em; line-height: 1.5; }
.btns { display: flex; gap: .5rem; margin-top: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .42rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button:disabled { opacity: .45; cursor: default; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

Notice how plain gradient descent (blue) takes uniform steps and can stall on shallow plateaus or oscillate in narrow ravines. Adam (orange) adapts: it takes larger steps where gradients have been consistently small (flat regions) and smaller steps where gradients swing wildly (steep ravines). Adam almost always finds the valley faster — and that speed advantage multiplies enormously when the loss surface has millions of dimensions instead of two.

Adam's update rule uses three ideas woven together. At each training step t, for a parameter $\theta$ with gradient g:

1. First moment (momentum). Adam keeps a running average of the gradient itself:

$$m = \beta_1 m + (1 - \beta_1) g$$

This smooths out noise — if the gradient keeps pointing in one direction, m builds up and the step accelerates. $\beta_1$ is usually 0.9 (forget only 10% of history each step).

2. Second moment (adaptive scale). Adam also keeps a running average of the squared gradient:

$$v = \beta_2 v + (1 - \beta_2) g^2$$

Large v means the gradient for this parameter has been large or noisy; small v means the gradient has been small and consistent. $\beta_2$ is usually 0.999.

3. Bias correction. Because both averages start at zero, they are biased toward zero in the early steps. Adam corrects this:

$$\hat{m} = \frac{m}{1 - \beta_1^t} \qquad \hat{v} = \frac{v}{1 - \beta_2^t}$$

4. The update.

$$\theta = \theta - \alpha \cdot \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}$$

The learning rate $\alpha$ (typically 0.001) is divided by $\sqrt{\hat{v}}$, so parameters with large or noisy gradients get smaller steps, and parameters with tiny, consistent gradients get larger steps. $\epsilon$ ($\approx 10^{-8}$) prevents division by zero.

Status: solved algorithm. Adam is not a complexity-theory puzzle — it is a practical method with proven convergence guarantees under mild conditions (Kingma & Ba, 2014). Each step costs $O(n)$ time and memory where n is the number of parameters — the same as plain gradient descent — so the adaptive bookkeeping is essentially free.

Adam can be seen as combining two earlier ideas: momentum (the first moment) and RMSProp (the second moment normalization, proposed by Geoff Hinton in 2012). Kingma and Ba showed that adding bias correction and combining both moments gave a single method that matched or outperformed every predecessor on the tasks they tested. See also how neural network training fits the wider picture of non-convex optimization.

Adam's adaptive per-parameter steps matter most precisely where modern deep learning operates:

• Large language models: Transformers have attention heads, embedding matrices, and feed-forward layers that span wildly different gradient scales. Adam's normalization keeps all of them moving at a sensible pace simultaneously.
• Image generation (diffusion models, GANs): training is notoriously unstable. Adam's momentum smoothing dampens the oscillations that cause vanilla gradient descent to diverge.
• Sparse gradients: in natural language, most word embeddings receive zero gradient for most batches — only the words actually seen are updated. Adam's second moment accumulates slowly for rare parameters and gives them a large effective step when they do appear.
• Recommendation systems: embedding tables have billions of entries with highly imbalanced update frequencies. Adam handles this naturally where SGD struggles.
• Reinforcement learning: reward signals are noisy and non-stationary. Adam's running estimates act as a form of signal smoothing that keeps training stable.

Variants of Adam — AdaGrad, RMSProp, AdamW, Nadam, Lion — address specific weaknesses (weight decay, generalization, memory) but all share the same core idea: track gradient history per parameter and adapt the step accordingly. AdamW in particular, which decouples weight decay from the adaptive update, is the standard choice for training large language models today.

Three equations, two hyperparameters you almost never need to tune, and $O(n)$ cost per step — Adam's elegance is that it turns a single insight (track gradient history per parameter) into a method that works out of the box on nearly every deep learning problem.

It is not magic. Adam does not guarantee finding the global minimum of a non-convex loss surface, and on some problems carefully tuned SGD with momentum can generalize better. But as a robust default that gets models trained quickly and reliably, Adam has earned its place as the most widely used optimizer in the history of machine learning.

The next time a model surprises you — a chatbot that reasons, an image generator that dreams — remember that behind it, running quietly for millions of steps, were three moving averages doing arithmetic on gradients.