Introduction

Every time you train a neural network, something mysterious happens. Millions of parameters shift slightly with each gradient step, tangled together by the chain rule. The network is nonlinear, the loss landscape is a high-dimensional maze, and yet — somehow — things converge, generalize, and work.

In 1998, Arthur Jacot, Franck Gabriel, and Clément Hongler (NeurIPS 2018) asked what happens if you make the network infinitely wide. The answer was startling: in that limit, the network stops being mysterious. Its training dynamics become exactly equivalent to kernel regression with a single fixed kernel — the Neural Tangent Kernel (NTK).

That kernel is determined entirely by the network's architecture before any training begins. Once you know it, you can predict how any infinitely wide version of that architecture will learn on any dataset, without simulating a single gradient step. The NTK turned a black box into an equation.

Try It

The demo below compares two learners on the same 1-D regression task: gradient descent on a finite-width network and the NTK kernel predictor (the closed-form solution the infinite-width theory predicts).

<!-- Main layout — hint, controls, canvas, status, buttons, legend -->
<p class="hint">The <strong>red dashed curve</strong> is the NTK kernel prediction — computed once from the data, no training needed. The <strong>teal curve</strong> is a finite-width network trained with gradient descent. Increase the width and watch the two converge.</p>
<div class="controls">
  <label>Network width: <strong id="width-val">64</strong>
    <input type="range" id="width-slider" min="4" max="512" step="4" value="64">
  </label>
  <label>GD steps: <strong id="steps-val">200</strong>
    <input type="range" id="steps-slider" min="20" max="600" step="20" value="200">
  </label>
</div>
<canvas id="chart" width="560" height="260"></canvas>
<div class="status" id="status"></div>
<div class="btns">
  <button id="run" type="button">Run gradient descent</button>
  <button id="reset" type="button" class="ghost">Reset</button>
</div>
<div class="legend">
  <span class="dot data"></span>Training points
  <span class="dot ntk"></span>NTK prediction (infinite width)
  <span class="dot gd"></span>Gradient descent (finite width)
</div>

/* Styles for layout, canvas, controls, buttons, legend and status */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
.controls { display: flex; flex-direction: column; gap: .4rem; margin-bottom: .6rem; }
label { font-size: .88rem; color: #333; }
input[type=range] { width: 200px; margin-left: .4rem; vertical-align: middle; }
canvas { display: block; border: 1px solid #cdd9e3; border-radius: 8px; background: #f8fafc; max-width: 100%; }
.status { font-size: .9rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; }
.status.ok { color: #0a7d33; }
.status.info { color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; margin-bottom: .5rem; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
.legend { display: flex; align-items: center; gap: .8rem; font-size: .82rem; flex-wrap: wrap; }
.dot { display: inline-block; width: 12px; height: 12px; border-radius: 50%; margin-right: 3px; }
.dot.data { background: #1d3557; }
.dot.ntk  { background: #e63946; }
.dot.gd   { background: #2a9d8f; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// NTK demo: compare kernel regression (infinite-width theory) vs gradient descent
// on a shallow tanh network. The RBF kernel approximates the NTK for this architecture.
// At large width the GD curve should match the NTK curve closely.

const canvas = document.getElementById('chart');
const ctx = canvas.getContext('2d');
const W = canvas.width, H = canvas.height;
const PAD = { l: 36, r: 12, t: 12, b: 28 };
const PW = W - PAD.l - PAD.r, PH = H - PAD.t - PAD.b;

// Five fixed training points (x, y) in [-1, 1]
const XS = [-0.9, -0.6, -0.2, 0.3, 0.7];
const YS = [0.6, -0.5, 0.2, 0.9, -0.3];
const N_DATA = XS.length;

// Dense grid of x values for plotting predictions
const N_EVAL = 80;
const XGRID = Array.from({ length: N_EVAL }, (_, i) => -1 + 2 * i / (N_EVAL - 1));

// Kernel function approximating the NTK: RBF (radial basis function) with sigma=0.5.
// The true NTK for a tanh network is an arc-cosine kernel; RBF is a close stand-in for 1-D demo clarity.
function rbfKernel(a, b, sigma) {
  const d = a - b;
  return Math.exp(-d * d / (2 * sigma * sigma));
}
const SIGMA = 0.5;

function ntkPredict(xq) {
  // Solve kernel regression: alpha = (K+lam*I)^-1 y, then predict with k(xq,X) alpha
  const K = [];
  for (let i = 0; i < N_DATA; i++) {
    K.push([]);
    for (let j = 0; j < N_DATA; j++) K[i].push(rbfKernel(XS[i], XS[j], SIGMA));
  }
  // Add small ridge regularization for numerical stability
  const lam = 1e-4;
  for (let i = 0; i < N_DATA; i++) K[i][i] += lam;
  const alpha = solveLinear(K, YS.slice());
  let out = 0;
  for (let i = 0; i < N_DATA; i++) out += alpha[i] * rbfKernel(xq, XS[i], SIGMA);
  return out;
}

// Gaussian elimination with partial pivoting to solve the linear system
function solveLinear(A, b) {
  const n = b.length;
  const M = A.map((r, i) => [...r, b[i]]);
  for (let col = 0; col < n; col++) {
    let maxR = col;
    for (let r = col + 1; r < n; r++) if (Math.abs(M[r][col]) > Math.abs(M[maxR][col])) maxR = r;
    [M[col], M[maxR]] = [M[maxR], M[col]];
    const piv = M[col][col];
    if (Math.abs(piv) < 1e-12) continue;
    for (let r = 0; r < n; r++) {
      if (r === col) continue;
      const f = M[r][col] / piv;
      for (let c = col; c <= n; c++) M[r][c] -= f * M[col][c];
    }
    for (let c = col; c <= n; c++) M[col][c] /= piv;
  }
  return M.map(r => r[n]);
}

// Single hidden layer tanh network: in=1, hidden=width, out=1
// Weights scaled by 1/sqrt(width) so variance stays O(1) as width grows (NTK parameterization)
function runGD(width, steps) {
  const lr = 0.05;
  // Input-to-hidden weights; shape [width, 1]
  const W1 = Array.from({ length: width }, () =>
    Array.from({ length: 1 }, () => (Math.random() * 2 - 1) / Math.sqrt(1))
  );
  // Hidden-to-output weights; shape [width]
  const W2 = Array.from({ length: width }, () => (Math.random() * 2 - 1) / Math.sqrt(width));

  function forward(x) {
    // Forward pass through tanh hidden layer
    let out = 0;
    for (let h = 0; h < width; h++) {
      const pre = W1[h][0] * x;
      const act = Math.tanh(pre);
      out += W2[h] * act;
    }
    return out / Math.sqrt(width);
  }

  for (let step = 0; step < steps; step++) {
    // Gradient step over all training points (full-batch)
    const gW1 = Array.from({ length: width }, () => [0]);
    const gW2 = new Array(width).fill(0);
    let loss = 0;
    for (let i = 0; i < N_DATA; i++) {
      const x = XS[i], y = YS[i];
      const hPre = W1.map(w => w[0] * x);
      const hAct = hPre.map(Math.tanh);
      const pred = hAct.reduce((s, a, h) => s + W2[h] * a, 0) / Math.sqrt(width);
      const err = pred - y;
      loss += err * err;
      const dOut = 2 * err / Math.sqrt(width);
      for (let h = 0; h < width; h++) {
        gW2[h] += dOut * hAct[h];
        const dAct = W2[h] * dOut;
        const dPre = dAct * (1 - hAct[h] * hAct[h]);
        gW1[h][0] += dPre * x;
      }
    }
    for (let h = 0; h < width; h++) {
      W2[h] -= lr * gW2[h];
      W1[h][0] -= lr * gW1[h][0];
    }
  }
  return XGRID.map(x => forward(x));
}

const NTK_Y = XGRID.map(x => ntkPredict(x));
let gdY = null;

function toCanvasX(x) { return PAD.l + (x + 1) / 2 * PW; }
function toCanvasY(y) { return PAD.t + (1 - (y + 1) / 2) * PH; }

function drawAxes() {
  ctx.clearRect(0, 0, W, H);
  ctx.strokeStyle = '#aaa'; ctx.lineWidth = 0.8;
  // Draw x-axis at y=0
  ctx.beginPath(); ctx.moveTo(PAD.l, PAD.t + PH / 2); ctx.lineTo(PAD.l + PW, PAD.t + PH / 2); ctx.stroke();
  // Draw y-axis at x=-1
  ctx.beginPath(); ctx.moveTo(PAD.l, PAD.t); ctx.lineTo(PAD.l, PAD.t + PH); ctx.stroke();
  ctx.fillStyle = '#888'; ctx.font = '10px system-ui';
  ctx.fillText('-1', PAD.l - 18, PAD.t + PH + 14);
  ctx.fillText('1', PAD.l + PW - 4, PAD.t + PH + 14);
  ctx.fillText('1', PAD.l - 14, PAD.t + 4);
  ctx.fillText('-1', PAD.l - 18, PAD.t + PH);
}

function drawCurve(ys, color, dash) {
  ctx.strokeStyle = color; ctx.lineWidth = 2;
  ctx.setLineDash(dash || []);
  ctx.beginPath();
  XGRID.forEach((x, i) => {
    const cx = toCanvasX(x), cy = toCanvasY(Math.max(-1.2, Math.min(1.2, ys[i])));
    i === 0 ? ctx.moveTo(cx, cy) : ctx.lineTo(cx, cy);
  });
  ctx.stroke();
  ctx.setLineDash([]);
}

function drawPoints() {
  // Draw training data as filled circles
  XS.forEach((x, i) => {
    const cx = toCanvasX(x), cy = toCanvasY(YS[i]);
    ctx.beginPath(); ctx.arc(cx, cy, 5, 0, 2 * Math.PI);
    ctx.fillStyle = '#1d3557'; ctx.fill();
  });
}

function render() {
  drawAxes();
  drawCurve(NTK_Y, '#e63946', [6, 3]);
  if (gdY) drawCurve(gdY, '#2a9d8f', []);
  drawPoints();
}

function setStatus(t, cls) {
  const s = document.getElementById('status');
  s.textContent = t; s.className = 'status ' + cls;
}

const widthSlider = document.getElementById('width-slider');
const stepsSlider = document.getElementById('steps-slider');
const widthVal   = document.getElementById('width-val');
const stepsVal   = document.getElementById('steps-val');

widthSlider.addEventListener('input', () => { widthVal.textContent = widthSlider.value; });
stepsSlider.addEventListener('input', () => { stepsVal.textContent = stepsSlider.value; });

document.getElementById('run').addEventListener('click', () => {
  const width = parseInt(widthSlider.value, 10);
  const steps = parseInt(stepsSlider.value, 10);
  setStatus('Running gradient descent…', 'info');
  setTimeout(() => {
    gdY = runGD(width, steps);
    render();
    setStatus(i18nFmt('Done — width %width%, %steps% steps.', { width, steps }), 'ok');
  }, 20);
});

document.getElementById('reset').addEventListener('click', () => {
  gdY = null;
  render();
  setStatus('', '');
});

render();
setStatus('Adjust width and steps, then click Run.', 'info');
Result
Console

At small widths the two curves differ — gradient descent is noisy and sensitive to initialization. As you increase the width, the gap closes: the finite network's output converges toward the NTK prediction. The kernel predictor never trains at all; it just solves a linear system once.

The Real Theory

Why does infinite width make training solvable? Here is the core argument.

The NTK is defined by the Jacobian. If a network outputs f(θ,x)f(\theta, x) and θ\theta are its parameters, the NTK between two inputs xx and x′x' is:

Θ(x,x′)=∑k∂f(θ,x)∂θk⋅∂f(θ,x′)∂θk\Theta(x, x') = \sum_{k} \frac{\partial f(\theta, x)}{\partial \theta_k} \cdot \frac{\partial f(\theta, x')}{\partial \theta_k}

This is just the dot product of the gradient vectors. In a finite network it changes as θ\theta changes. The key insight: as width →∞\to \infty, the NTK freezes at its initial value and stays constant throughout training.

Why? Each individual weight contributes O(1/n)O(1/\sqrt{n}) to the output (standard random initialization). With nn neurons, the NTK sums nn such terms, which by a law-of-large-numbers argument converges to a deterministic limit. And because every weight moves only O(1/n)O(1/\sqrt{n}) during training (the "lazy training" regime), their collective contribution to the kernel doesn't shift.

Training becomes a linear ODE. With a fixed kernel Θ\Theta, the loss dynamics under gradient descent on a mean-squared loss become:

dy^dt=−Θ(X,X)(y^(t)−y)\frac{d\hat{y}}{dt} = -\Theta(X, X)\bigl(\hat{y}(t) - y\bigr)

This is a linear system with an explicit solution:

y^(t)=(I−e−Θt)y\hat{y}(t) = \bigl(I - e^{-\Theta t}\bigr) y

The infinite-time prediction is kernel regression. As t→∞t \to \infty, this converges to the kernel regression solution y^∞=Θ(x,X) Θ(X,X)−1 y\hat{y}_{\infty} = \Theta(x, X)\,\Theta(X, X)^{-1}\,y: the standard result for a Gaussian process with covariance Θ\Theta.

This was proved rigorously by Jacot et al. (2018), with follow-ups by Du et al. and Li & Liang (2019) extending convergence guarantees to finite over-parameterized networks.

Where It Matters

The NTK is not just a theoretical curiosity. It has reshaped how researchers think about deep learning:

  • Generalization theory: in the kernel regime, standard statistical learning theory applies. The NTK's eigenvalue spectrum controls which functions are learned first and how well the network generalizes — giving a rigorous handle on a question that was previously opaque.
  • Gaussian processes: an infinitely wide network's output is a Gaussian process whose covariance is the NTK. This links neural network training directly to Bayesian inference.
  • Architecture comparison: two architectures can be compared by their NTKs before any training, predicting which one will learn faster or generalize better on structured data.
  • Explaining lazy training: many large models in practice barely move their parameters (relative to initialization). The NTK explains why this "lazy" regime can still reach near-zero training loss.
  • Limitations and the feature-learning gap: the NTK regime describes networks that do not learn useful internal representations — they essentially memorize through the kernel. Real networks that generalize well (like transformers on language) often escape the kernel regime, which is itself an active research frontier.

Conclusion

The Neural Tangent Kernel gave theorists something rare: an exact analytical solution to a deep learning problem. By pushing width to infinity, a tangled nonlinear optimizer becomes a clean kernel regression — predictable, analyzable, and connected to classical statistics.

The lesson cuts two ways. On one side, the NTK shows that over-parameterized networks are not as mysterious as they look: in the right limit they reduce to well-understood mathematics. On the other side, the most powerful networks we build today — transformers, diffusion models — work precisely because they escape the kernel regime and learn rich internal representations that no static kernel can capture.

Understanding where the NTK applies and where it breaks down is one of the central questions in modern neural network training theory, and the boundary between the two regimes is still being drawn.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/neural-tangent-kernel/Content licensed under CC BY-NC 4.0.