Neural Scaling Laws

In 2020, a team at OpenAI noticed something strange: when they plotted the test loss of language models against model size, dataset size, or training compute, the result was not a messy cloud of points. It was a straight line on a log-log graph — a perfect power law extending over many orders of magnitude.

The finding, published by Kaplan et al., is now called the neural scaling law: for a broad class of tasks, the loss L falls as

$L \propto N^{-\alpha}$

where N is model size (or data, or compute) and α is a small positive exponent (typically 0.05–0.09 for language). Double the parameters, and loss drops by a fixed fraction — reliably, smoothly, without any phase transition.

This is not an obvious result. Deep learning is notoriously hard to predict. Yet the scaling law has held across seven or more orders of magnitude of model size. It turned AI research from an empirical guessing game into something closer to engineering: given a fixed compute budget, you can now calculate the optimal model size and token count before training begins.

Two years later, Hoffmann et al. (DeepMind, 2022) sharpened the picture with the Chinchilla models, showing the 2020 laws slightly under-counted the value of data. Their revised prescription: for every doubling of parameters, you should also roughly double the training tokens. This insight directly influenced the design of every major model since.

The chart below shows empirical cross-entropy loss versus model size for a family of language models. The x-axis is model parameters (log scale), the y-axis is held-out loss (log scale).

<div class="controls">
  <label>{{noise_label}} <input type="range" id="noise" min="0" max="0.15" step="0.005" value="0.04"></label>
  <span id="noiseVal">0.04</span>
  <button id="fitBtn" type="button">{{fit_btn}}</button>
  <button id="resetBtn" type="button" class="ghost">{{reset_btn}}</button>
</div>
<canvas id="chart" width="480" height="300"></canvas>
<div class="info" id="info">{{adjust_hint}}</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; align-items: center; gap: .7rem; flex-wrap: wrap; margin-bottom: .6rem; font-size: .9rem; }
label { display: flex; align-items: center; gap: .4rem; }
input[type=range] { width: 90px; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 7px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
canvas { display: block; border: 1px solid #dde3ea; border-radius: 10px; max-width: 100%; }
.info { margin-top: .5rem; font-size: .92rem; font-weight: 600; min-height: 1.3em; color: #1d3557; }

// Code not found

Click Fit power law to overlay the best-fit line and read off the exponent α. Adjust the noise slider to add measurement scatter and see how robustly the law holds. Notice that the relationship is almost perfectly linear in log-log space — that is the scaling law. The slope is −α, the intercept encodes the irreducible entropy of the dataset.

The elegance of scaling laws hides several deep caveats.

What the law predicts well:

• Loss on the training distribution decreases smoothly as a power law in N, D, and C.
• Compute-optimal allocation: given a fixed FLOP budget C, the Chinchilla law says set N ≈ $C^{0.5}$ and D ≈ $C^{0.5}$ tokens. This was a practical revelation — earlier models were massively under-trained relative to their size.
• Extrapolation: a law fit on 10M-parameter models can predict 70B-parameter loss with surprising accuracy.

What the law does not capture:

• Emergent capabilities — some abilities (chain-of-thought reasoning, in-context learning) appear sharply at certain scales and are invisible in smooth loss curves. These phase transitions are not predicted by the power law.
• Domain shift: the exponent α varies across modalities (images, code, proteins) and even across tasks within language. There is no universal constant.
• The irreducible floor: loss approaches the true entropy of the data from above but never crosses it. Near that floor, more compute yields diminishing returns invisible in log-log space.
• Alignment and behavior: loss is a proxy. A model can have lower loss and worse values, more capable reasoning, or more dangerous outputs. Scaling laws say nothing about any of these.

Scaling laws are empirical regularities, not theorems. They have been wrong before — the 2020 Kaplan et al. exponents slightly over-valued model size, which is why Chinchilla models are smaller and better-trained than their predecessors. Future architectural innovations could bend or break the current curves entirely.

From a complexity standpoint, scaling laws are fascinating because they suggest that the optimization landscape of neural network training has an unexpected regularity — the hard, non-convex search nonetheless traces a smooth frontier. Why this is true remains an open theoretical question.

Scaling laws have moved from academic curiosity to operational doctrine at every major AI lab:

• Hardware and budget planning: if you know α and your target loss, you can back-calculate the compute needed, then decide how many GPUs to rent and for how long — before writing a single line of model code.
• Data curation: Chinchilla showed that data is as valuable as parameters. Labs now invest heavily in dataset quality and quantity, not just model architecture. The "tokens per parameter" ratio is a first-class engineering variable.
• Architecture search: new architectures are evaluated partly by whether they shift the scaling curve — a lower loss at the same compute is a genuine improvement, independent of any specific benchmark.
• AI safety and forecasting: scaling laws are used to predict when a model might cross capability thresholds. Organizations like Anthropic and DeepMind use them to anticipate risks before deployment and to schedule evaluations for dangerous capabilities.
• Scientific modeling: the same power-law behavior appears in protein language models (e.g., ESM-2), code models, and multimodal systems. Scaling laws are becoming a cross-domain tool for scientific prediction.

Understanding scaling laws is understanding why the AI industry looks the way it does: a race to accumulate compute and data, because the math says bigger almost always wins — until it doesn't, and someone finds a better architecture.

Neural scaling laws are one of the most practically powerful empirical discoveries in modern science. A single relationship — loss falls as a power law in model size, data, and compute — has guided the construction of every major language model of the last five years, from GPT-3 to Gemini.

They are not magic. They break near the entropy floor, they miss emergent capabilities, and the exponents shift with architecture and domain. But within their domain of validity they are startlingly predictive, turning a field famous for surprises into something that can be planned.

The deeper mystery is why they hold at all. The optimization problem behind neural network training is non-convex and astronomically high-dimensional. Yet somehow, across architectures, tasks, and modalities, loss traces a clean power law. That regularity is waiting for a theory to explain it — and finding that theory may be the next great step in understanding intelligence itself.