Introduction

In 2018, Ricky Chen and colleagues at the University of Toronto made a deceptively simple observation: a residual network is just an Euler integrator in disguise.

Every ResNet layer computes ht+1=ht+f(ht,θt)\mathbf{h}_{t+1} = \mathbf{h}_t + f(\mathbf{h}_t, \theta_t). That is exactly one step of Euler's method for the ODE dhdt=f(h(t),t,θ)\frac{d\mathbf{h}}{dt} = f(\mathbf{h}(t), t, \theta). Push the step size toward zero and the discrete stack of layers becomes a continuous flow — a neural ordinary differential equation.

Instead of deciding how many layers to stack, you now ask an ODE solver to integrate from t=0t=0 to t=1t=1. The solver adaptively chooses its own step count, spending more steps where the dynamics are complex and fewer where they are smooth. The network's depth becomes a run-time decision, not a design-time one.

The trick sounds academic, but it buys something real: constant memory during backpropagation, because you never have to store intermediate layer activations. That is what the adjoint method delivers — and it is the heart of why Neural ODEs matter. Learn about related continuous representations in transformers.

Try It: Follow the Flow

The demo below shows a learned vector field — a grid of arrows, each pointing in the direction a Neural ODE would push a particle at that location. Choose a starting point and watch numerical integration carry it forward.

<!-- Canvas-based Neural ODE demo: vector field + numerical integration paths -->
<div class="controls">
  <label for="field-select">Field:</label>
  <select id="field-select">
    <option value="spiral">Spiral sink</option>
    <option value="saddle">Saddle</option>
    <option value="vortex">Vortex</option>
  </select>
  <label for="method-select">Integrator:</label>
  <select id="method-select">
    <option value="euler">Euler</option>
    <option value="rk4">Euler vs RK4</option>
  </select>
  <button id="btn-reset" type="button">Reset</button>
</div>
<canvas id="canvas" width="420" height="300"></canvas>
<div id="info" class="info">Click anywhere on the canvas to place a starting point.</div>

/* Styles for the Neural ODE vector field demo */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; background: #fff; }
.controls { display: flex; align-items: center; flex-wrap: wrap; gap: .5rem; margin-bottom: .5rem; font-size: .85rem; }
label { font-weight: 600; }
select { padding: .25rem .4rem; border: 1px solid #aaa; border-radius: 6px; font-size: .85rem; }
button { padding: .3rem .7rem; font: 600 .85rem system-ui; border: 1px solid #1d3557; background: #1d3557; color: #fff; border-radius: 6px; cursor: pointer; }
button:hover { background: #14253e; }
canvas { display: block; border: 1px solid #cdd9e3; border-radius: 8px; cursor: crosshair; max-width: 100%; }
.info { margin-top: .4rem; font-size: .85rem; color: #555; min-height: 1.3em; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Neural ODE demo: draw a vector field and integrate a trajectory through it
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const W = canvas.width, H = canvas.height;

// World coordinate domain and canvas mapping helpers
const XMIN = -2, XMAX = 2, YMIN = -1.5, YMAX = 1.5;
const toCanvasX = x => (x - XMIN) / (XMAX - XMIN) * W;
const toCanvasY = y => H - (y - YMIN) / (YMAX - YMIN) * H;
const fromCanvas = (px, py) => [
  XMIN + px / W * (XMAX - XMIN),
  YMIN + (H - py) / H * (YMAX - YMIN)
];

// Three preset vector fields — each is a simple 2D ODE right-hand side
const FIELDS = {
  spiral: (x, y) => [-x - y, x - y],
  saddle:  (x, y) => [x, -y],
  vortex:  (x, y) => [-y, x]
};

// Euler method: constant step size, first-order accuracy
function eulerSteps(f, x0, y0, t1, dt) {
  const path = [[x0, y0]];
  let x = x0, y = y0;
  for (let t = 0; t < t1; t += dt) {
    const [dx, dy] = f(x, y);
    x += dx * dt;
    y += dy * dt;
    path.push([x, y]);
    // stop if trajectory leaves the visible domain
    if (Math.abs(x) > 5 || Math.abs(y) > 5) break;
  }
  return path;
}

// Runge-Kutta 4: 4 evaluations per step, fourth-order accuracy
function rk4Steps(f, x0, y0, t1, dt) {
  const path = [[x0, y0]];
  let x = x0, y = y0;
  for (let t = 0; t < t1; t += dt) {
    const [k1x, k1y] = f(x, y);
    const [k2x, k2y] = f(x + k1x * dt / 2, y + k1y * dt / 2);
    const [k3x, k3y] = f(x + k2x * dt / 2, y + k2y * dt / 2);
    const [k4x, k4y] = f(x + k3x * dt, y + k3y * dt);
    x += (k1x + 2 * k2x + 2 * k3x + k4x) * dt / 6;
    y += (k1y + 2 * k2y + 2 * k3y + k4y) * dt / 6;
    path.push([x, y]);
    if (Math.abs(x) > 5 || Math.abs(y) > 5) break;
  }
  return path;
}

// Draw the vector field as a grid of arrows
function drawArrows(f) {
  const COLS = 18, ROWS = 13;
  const stepX = W / COLS, stepY = H / ROWS;
  ctx.strokeStyle = '#aab8c8';
  ctx.lineWidth = 0.8;
  for (let i = 0; i < COLS; i++) {
    for (let j = 0; j < ROWS; j++) {
      const px = (i + 0.5) * stepX, py = (j + 0.5) * stepY;
      const [wx, wy] = fromCanvas(px, py);
      const [dvx, dvy] = f(wx, wy);
      const mag = Math.sqrt(dvx * dvx + dvy * dvy) + 1e-9;
      const scale = Math.min(stepX, stepY) * 0.38 / Math.sqrt(mag + 0.3);
      const ex = px + dvx * scale, ey = py - dvy * scale;
      ctx.beginPath();
      ctx.moveTo(px, py);
      ctx.lineTo(ex, ey);
      // small arrowhead lines
      const angle = Math.atan2(ey - py, ex - px);
      ctx.lineTo(ex - 5 * Math.cos(angle - 0.4), ey - 5 * Math.sin(angle - 0.4));
      ctx.moveTo(ex, ey);
      ctx.lineTo(ex - 5 * Math.cos(angle + 0.4), ey - 5 * Math.sin(angle + 0.4));
      ctx.stroke();
    }
  }
}

// Draw an integrated trajectory path on the canvas
function drawPath(path, color, lineWidth) {
  if (path.length < 2) return;
  ctx.strokeStyle = color;
  ctx.lineWidth = lineWidth;
  ctx.lineJoin = 'round';
  ctx.lineCap = 'round';
  ctx.beginPath();
  ctx.moveTo(toCanvasX(path[0][0]), toCanvasY(path[0][1]));
  for (let i = 1; i < path.length; i++) {
    ctx.lineTo(toCanvasX(path[i][0]), toCanvasY(path[i][1]));
  }
  ctx.stroke();
}

// Draw a filled circle (starting point marker)
function drawDot(x, y, color, r) {
  ctx.fillStyle = color;
  ctx.beginPath();
  ctx.arc(toCanvasX(x), toCanvasY(y), r, 0, 2 * Math.PI);
  ctx.fill();
}

let startPt = null;

function getField() {
  return FIELDS[document.getElementById('field-select').value];
}
function getMethod() {
  return document.getElementById('method-select').value;
}

function redraw() {
  ctx.clearRect(0, 0, W, H);
  const f = getField();
  drawArrows(f);
  if (!startPt) return;
  const [x0, y0] = startPt;
  const T = 4, DT_EULER = 0.05, DT_RK4 = 0.1;
  const method = getMethod();
  if (method === 'euler') {
    // single Euler path in orange
    const path = eulerSteps(f, x0, y0, T, DT_EULER);
    drawPath(path, '#e07b39', 2.5);
    // report how many Euler steps were taken
    const steps = path.length - 1;
    document.getElementById('info').textContent =
      fmt('Euler: %steps% steps to integrate T=4.', { steps: steps });
  } else {
    // show Euler (faded) and RK4 (bright) side by side
    const pathE = eulerSteps(f, x0, y0, T, DT_EULER);
    const pathR = rk4Steps(f, x0, y0, T, DT_RK4);
    drawPath(pathE, '#e0a039', 1.5);
    drawPath(pathR, '#c8a000', 2.5);
    const stepsE = pathE.length - 1;
    const stepsR = pathR.length - 1;
    document.getElementById('info').textContent =
      fmt('RK4: %stepsR% steps (gold) vs Euler: %stepsE% steps (faded). Same field, better accuracy per step.', { stepsR: stepsR, stepsE: stepsE });
  }
  // red dot marks the starting point
  drawDot(x0, y0, '#e63946', 5);
}

canvas.addEventListener('click', e => {
  const rect = canvas.getBoundingClientRect();
  const px = (e.clientX - rect.left) * (W / rect.width);
  const py = (e.clientY - rect.top) * (H / rect.height);
  startPt = fromCanvas(px, py);
  redraw();
});

document.getElementById('field-select').addEventListener('change', redraw);
document.getElementById('method-select').addEventListener('change', redraw);
document.getElementById('btn-reset').addEventListener('click', () => {
  startPt = null;
  redraw();
  document.getElementById('info').textContent = 'Click anywhere on the canvas to place a starting point.';
});

// Draw the initial field on load (no trajectory yet)
redraw();
Result
Console

Notice that Euler (small orange steps) can drift away from the true path, while RK4 (the gold curve) stays accurate with fewer evaluations. A Neural ODE solver makes this choice automatically — spending more function evaluations where the field curves sharply and fewer where it is nearly straight. The total number of steps is not fixed in advance; it is decided at run time by the solver's error estimate.

The Real Complexity

Training a Neural ODE means differentiating through the ODE solver — and that is where the real cleverness (and cost) lives.

  • Naïve backprop would store every intermediate state the solver computed, then differentiate through each step. Memory usage scales with the number of solver steps, which can be large for stiff ODEs.
  • The adjoint method sidesteps this. Instead of storing the forward trajectory, you solve a second ODE backwards in time that computes the gradient of the loss with respect to both the parameters θ\theta and the initial state. Memory cost: O(1)O(1) in the number of solver steps — only the current adjoint state is kept.
  • The price: two ODE solves instead of one, plus numerical error in the adjoint itself. If the forward solve is inaccurate, the gradient estimate will be too. Stiff problems (fast-changing dynamics) can make the backward solve very slow or unstable.
  • Depth is implicit: you never choose the number of layers directly. The solver decides, making the model adaptive in depth — but also harder to reason about in terms of classical non-convex optimization landscapes.

The trade-off is sharp: Neural ODEs are memory-efficient but compute-intensive, and correctness of gradients depends on the accuracy of the ODE solver itself.

Where It Matters

The continuous-depth framing is not just elegant — it fits naturally into several important problem classes:

  • Irregularly sampled time series: medical records, financial tick data, sensor logs. Classic RNNs demand evenly spaced steps; a Neural ODE integrates to exactly the time of each observation.
  • Normalizing flows: to build expressive probability distributions you need invertible transformations. ODE flows are invertible by design (run the solver backward), and the instantaneous change-of-variables formula lets you compute exact log-likelihoods cheaply.
  • Physics-informed learning: when you know the governing equation is an ODE or PDE, embedding that structure into the network makes it dramatically data-efficient and physically consistent.
  • Continuous dimensionality reduction: latent ODE models learn a smooth trajectory in latent space, making interpolation and extrapolation well-defined operations.

The Chen et al. (2018) paper won the NeurIPS Best Paper award and launched a wave of follow-ups — Latent ODEs, Neural CDEs, Neural SDEs — each extending the idea to new regimes of data and noise.

Conclusion

Neural ODEs are a beautiful example of a unifying idea hiding in plain sight: every ResNet was already integrating an ODE, it just didn't know it yet.

By making the connection explicit, Chen et al. unlocked adaptive-depth computation, constant-memory training via the adjoint method, and natural handling of continuous-time data — all from one conceptual step.

The price is real: you trade the predictability of a fixed layer count for the flexibility (and numerical sensitivity) of an ODE solver. But when the problem is continuous by nature — physics, time series, generative modeling — fitting a discrete architecture to it was always a compromise. Neural ODEs let you stop pretending the world is a finite list of layers and start treating it as the flow it actually is.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/neural-odes/Content licensed under CC BY-NC 4.0.