Neural Tangent Kernel

Every time you train a neural network, something mysterious happens. Millions of parameters shift slightly with each gradient step, tangled together by the chain rule. The network is nonlinear, the loss landscape is a high-dimensional maze, and yet — somehow — things converge, generalize, and work.

In 1998, Arthur Jacot, Franck Gabriel, and Clément Hongler (NeurIPS 2018) asked what happens if you make the network infinitely wide. The answer was startling: in that limit, the network stops being mysterious. Its training dynamics become exactly equivalent to kernel regression with a single fixed kernel — the Neural Tangent Kernel (NTK).

That kernel is determined entirely by the network's architecture before any training begins. Once you know it, you can predict how any infinitely wide version of that architecture will learn on any dataset, without simulating a single gradient step. The NTK turned a black box into an equation.

The demo below compares two learners on the same 1-D regression task: gradient descent on a finite-width network and the NTK kernel predictor (the closed-form solution the infinite-width theory predicts).

<!-- {{c_layout}} -->
<p class="hint">{{hint_para}}</p>
<div class="controls">
  <label>{{lbl_width}} <strong id="width-val">64</strong>
    <input type="range" id="width-slider" min="4" max="512" step="4" value="64">
  </label>
  <label>{{lbl_steps}} <strong id="steps-val">200</strong>
    <input type="range" id="steps-slider" min="20" max="600" step="20" value="200">
  </label>
</div>
<canvas id="chart" width="560" height="260"></canvas>
<div class="status" id="status"></div>
<div class="btns">
  <button id="run" type="button">{{btn_run}}</button>
  <button id="reset" type="button" class="ghost">{{btn_reset}}</button>
</div>
<div class="legend">
  <span class="dot data"></span>{{legend_data}}
  <span class="dot ntk"></span>{{legend_ntk}}
  <span class="dot gd"></span>{{legend_gd}}
</div>

/* {{c_style}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.controls { display: flex; flex-direction: column; gap: .4rem; margin-bottom: .6rem; }
label { font-size: .88rem; color: #333; }
input[type=range] { width: 200px; margin-left: .4rem; vertical-align: middle; }
canvas { display: block; border: 1px solid #cdd9e3; border-radius: 8px; background: #f8fafc; max-width: 100%; }
.status { font-size: .9rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; }
.status.ok { color: #0a7d33; }
.status.info { color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-bottom: .5rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.legend { display: flex; align-items: center; gap: .8rem; font-size: .82rem; flex-wrap: wrap; }
.dot { display: inline-block; width: 12px; height: 12px; border-radius: 50%; margin-right: 3px; }
.dot.data { background: #1d3557; }
.dot.ntk  { background: #e63946; }
.dot.gd   { background: #2a9d8f; }

// Code not found

At small widths the two curves differ — gradient descent is noisy and sensitive to initialization. As you increase the width, the gap closes: the finite network's output converges toward the NTK prediction. The kernel predictor never trains at all; it just solves a linear system once.

Why does infinite width make training solvable? Here is the core argument.

The NTK is defined by the Jacobian. If a network outputs $f(\theta, x)$ and $\theta$ are its parameters, the NTK between two inputs $x$ and $x'$ is:

$$\Theta(x, x') = \sum_{k} \frac{\partial f(\theta, x)}{\partial \theta_k} \cdot \frac{\partial f(\theta, x')}{\partial \theta_k}$$

This is just the dot product of the gradient vectors. In a finite network it changes as $\theta$ changes. The key insight: as width $\to \infty$, the NTK freezes at its initial value and stays constant throughout training.

Why? Each individual weight contributes $O(1/\sqrt{n})$ to the output (standard random initialization). With $n$ neurons, the NTK sums $n$ such terms, which by a law-of-large-numbers argument converges to a deterministic limit. And because every weight moves only $O(1/\sqrt{n})$ during training (the "lazy training" regime), their collective contribution to the kernel doesn't shift.

Training becomes a linear ODE. With a fixed kernel $\Theta$, the loss dynamics under gradient descent on a mean-squared loss become:

$$\frac{d\hat{y}}{dt} = -\Theta(X, X)\bigl(\hat{y}(t) - y\bigr)$$

This is a linear system with an explicit solution:

$$\hat{y}(t) = \bigl(I - e^{-\Theta t}\bigr) y$$

The infinite-time prediction is kernel regression. As $t \to \infty$, this converges to the kernel regression solution $\hat{y}_{\infty} = \Theta(x, X)\,\Theta(X, X)^{-1}\,y$: the standard result for a Gaussian process with covariance $\Theta$.

This was proved rigorously by Jacot et al. (2018), with follow-ups by Du et al. and Li & Liang (2019) extending convergence guarantees to finite over-parameterized networks.

The NTK is not just a theoretical curiosity. It has reshaped how researchers think about deep learning:

• Generalization theory: in the kernel regime, standard statistical learning theory applies. The NTK's eigenvalue spectrum controls which functions are learned first and how well the network generalizes — giving a rigorous handle on a question that was previously opaque.
• Gaussian processes: an infinitely wide network's output is a Gaussian process whose covariance is the NTK. This links neural network training directly to Bayesian inference.
• Architecture comparison: two architectures can be compared by their NTKs before any training, predicting which one will learn faster or generalize better on structured data.
• Explaining lazy training: many large models in practice barely move their parameters (relative to initialization). The NTK explains why this "lazy" regime can still reach near-zero training loss.
• Limitations and the feature-learning gap: the NTK regime describes networks that do not learn useful internal representations — they essentially memorize through the kernel. Real networks that generalize well (like transformers on language) often escape the kernel regime, which is itself an active research frontier.

The Neural Tangent Kernel gave theorists something rare: an exact analytical solution to a deep learning problem. By pushing width to infinity, a tangled nonlinear optimizer becomes a clean kernel regression — predictable, analyzable, and connected to classical statistics.

The lesson cuts two ways. On one side, the NTK shows that over-parameterized networks are not as mysterious as they look: in the right limit they reduce to well-understood mathematics. On the other side, the most powerful networks we build today — transformers, diffusion models — work precisely because they escape the kernel regime and learn rich internal representations that no static kernel can capture.

Understanding where the NTK applies and where it breaks down is one of the central questions in modern neural network training theory, and the boundary between the two regimes is still being drawn.