Neural ODEs

In 2018, Ricky Chen and colleagues at the University of Toronto made a deceptively simple observation: a residual network is just an Euler integrator in disguise.

Every ResNet layer computes $\mathbf{h}_{t+1} = \mathbf{h}_t + f(\mathbf{h}_t, \theta_t)$. That is exactly one step of Euler's method for the ODE $\frac{d\mathbf{h}}{dt} = f(\mathbf{h}(t), t, \theta)$. Push the step size toward zero and the discrete stack of layers becomes a continuous flow — a neural ordinary differential equation.

Instead of deciding how many layers to stack, you now ask an ODE solver to integrate from $t=0$ to $t=1$. The solver adaptively chooses its own step count, spending more steps where the dynamics are complex and fewer where they are smooth. The network's depth becomes a run-time decision, not a design-time one.

The trick sounds academic, but it buys something real: constant memory during backpropagation, because you never have to store intermediate layer activations. That is what the adjoint method delivers — and it is the heart of why Neural ODEs matter. Learn about related continuous representations in transformers.

The demo below shows a learned vector field — a grid of arrows, each pointing in the direction a Neural ODE would push a particle at that location. Choose a starting point and watch numerical integration carry it forward.

<!-- {{c_html_intro}} -->
<div class="controls">
  <label for="field-select">{{lbl_field}}</label>
  <select id="field-select">
    <option value="spiral">{{opt_spiral}}</option>
    <option value="saddle">{{opt_saddle}}</option>
    <option value="vortex">{{opt_vortex}}</option>
  </select>
  <label for="method-select">{{lbl_method}}</label>
  <select id="method-select">
    <option value="euler">{{opt_euler}}</option>
    <option value="rk4">{{opt_rk4}}</option>
  </select>
  <button id="btn-reset" type="button">{{btn_reset}}</button>
</div>
<canvas id="canvas" width="420" height="300"></canvas>
<div id="info" class="info">{{msg_click}}</div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; background: #fff; }
.controls { display: flex; align-items: center; flex-wrap: wrap; gap: .5rem; margin-bottom: .5rem; font-size: .85rem; }
label { font-weight: 600; }
select { padding: .25rem .4rem; border: 1px solid #aaa; border-radius: 6px; font-size: .85rem; }
button { padding: .3rem .7rem; font: 600 .85rem system-ui; border: 1px solid #1d3557; background: #1d3557; color: #fff; border-radius: 6px; cursor: pointer; }
button:hover { background: #14253e; }
canvas { display: block; border: 1px solid #cdd9e3; border-radius: 8px; cursor: crosshair; max-width: 100%; }
.info { margin-top: .4rem; font-size: .85rem; color: #555; min-height: 1.3em; }

// Code not found

Notice that Euler (small orange steps) can drift away from the true path, while RK4 (the gold curve) stays accurate with fewer evaluations. A Neural ODE solver makes this choice automatically — spending more function evaluations where the field curves sharply and fewer where it is nearly straight. The total number of steps is not fixed in advance; it is decided at run time by the solver's error estimate.

Training a Neural ODE means differentiating through the ODE solver — and that is where the real cleverness (and cost) lives.

• Naïve backprop would store every intermediate state the solver computed, then differentiate through each step. Memory usage scales with the number of solver steps, which can be large for stiff ODEs.
• The adjoint method sidesteps this. Instead of storing the forward trajectory, you solve a second ODE backwards in time that computes the gradient of the loss with respect to both the parameters $\theta$ and the initial state. Memory cost: $O(1)$ in the number of solver steps — only the current adjoint state is kept.
• The price: two ODE solves instead of one, plus numerical error in the adjoint itself. If the forward solve is inaccurate, the gradient estimate will be too. Stiff problems (fast-changing dynamics) can make the backward solve very slow or unstable.
• Depth is implicit: you never choose the number of layers directly. The solver decides, making the model adaptive in depth — but also harder to reason about in terms of classical non-convex optimization landscapes.

The trade-off is sharp: Neural ODEs are memory-efficient but compute-intensive, and correctness of gradients depends on the accuracy of the ODE solver itself.

The continuous-depth framing is not just elegant — it fits naturally into several important problem classes:

• Irregularly sampled time series: medical records, financial tick data, sensor logs. Classic RNNs demand evenly spaced steps; a Neural ODE integrates to exactly the time of each observation.
• Normalizing flows: to build expressive probability distributions you need invertible transformations. ODE flows are invertible by design (run the solver backward), and the instantaneous change-of-variables formula lets you compute exact log-likelihoods cheaply.
• Physics-informed learning: when you know the governing equation is an ODE or PDE, embedding that structure into the network makes it dramatically data-efficient and physically consistent.
• Continuous dimensionality reduction: latent ODE models learn a smooth trajectory in latent space, making interpolation and extrapolation well-defined operations.

The Chen et al. (2018) paper won the NeurIPS Best Paper award and launched a wave of follow-ups — Latent ODEs, Neural CDEs, Neural SDEs — each extending the idea to new regimes of data and noise.

Neural ODEs are a beautiful example of a unifying idea hiding in plain sight: every ResNet was already integrating an ODE, it just didn't know it yet.

By making the connection explicit, Chen et al. unlocked adaptive-depth computation, constant-memory training via the adjoint method, and natural handling of continuous-time data — all from one conceptual step.

The price is real: you trade the predictability of a fixed layer count for the flexibility (and numerical sensitivity) of an ODE solver. But when the problem is continuous by nature — physics, time series, generative modeling — fitting a discrete architecture to it was always a compromise. Neural ODEs let you stop pretending the world is a finite list of layers and start treating it as the flow it actually is.