Optical Flow Interactive Demo

Introduction

Every time you watch a video, your brain effortlessly decides which parts of the scene are moving and which are still. Computers cannot afford that luxury — they get only a grid of brightness values changing from frame to frame. Optical flow is the name for the apparent velocity field that explains those changes: for every pixel, how far did it shift between two consecutive frames?

The question sounds simple, but it is fundamentally ill-posed. Imagine a perfectly uniform gray disk sliding across a white background. Looking at a single pixel in isolation you cannot tell whether it moved up or sideways — a problem known as the aperture problem. Some additional assumption is always needed to pin down a unique answer.

In 1981, Bruce D. Lucas and Takeo Kanade proposed the cleanest possible assumption: brightness constancy. A surface patch keeps roughly the same brightness as it moves a short distance in a short time. This turns motion estimation into a small system of linear equations — solvable in milliseconds for every pixel in the image.

The Lucas-Kanade method has been refined, extended, and combined with deep-learning pipelines for forty years. It remains the foundation that anyone studying dimensionality reduction or pattern matching in vision should know first.

Try It

The canvas below shows two consecutive synthetic frames. A bright blob moves a few pixels each step — press Play to animate it, or use Step to advance one frame at a time. The arrows drawn on each pixel show the velocity $(u, v)$ that Lucas-Kanade estimates at that location.

<!-- Canvas demo: Lucas-Kanade optical flow on a synthetic moving blob -->
<p class="hint">A bright blob moves across the canvas. The golden arrows show the velocity <b>(u,v)</b> that Lucas-Kanade estimates at each grid point — arrows appear only where the image has gradients in two directions (the blob edge), not on the flat background.</p>
<canvas id="canvas" width="320" height="240"></canvas>
<div class="status" id="status">Ready — press Play or Step.</div>
<div class="btns">
  <button id="btn-play" type="button">Play</button>
  <button id="btn-step" type="button" class="ghost">Step</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
</div>

/* Minimal dark-canvas layout with control buttons */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .85rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
canvas { display: block; border: 1px solid #cdd9e3; border-radius: 8px; background: #000; max-width: 100%; }
.status { font-size: .95rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .45; cursor: default; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Lucas-Kanade optical flow: assumes brightness constancy and constant flow
// over a local window, then solves the 2x2 least-squares normal equations.

const W = 320, H = 240;
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');

// Blob position and velocity (pixels per frame)
let bx = 80, by = 120, vx = 1.4, vy = 0.6;
let frame = 0;
let playing = false;
let rafId = null;

// Render a single grayscale-ish frame with a soft Gaussian-ish blob
function makeFrame(cx, cy) {
  const img = ctx.createImageData(W, H);
  const d = img.data;
  // Dark background
  for (let i = 0; i < d.length; i += 4) {
    d[i] = 18; d[i + 1] = 18; d[i + 2] = 28; d[i + 3] = 255;
  }
  // Radial gradient blob: brightness falls off with distance squared
  const R = 36;
  for (let y = 0; y < H; y++) {
    for (let x = 0; x < W; x++) {
      const dx = x - cx, dy = y - cy;
      const dist2 = dx * dx + dy * dy;
      if (dist2 < R * R) {
        const bright = Math.round(200 * (1 - dist2 / (R * R)));
        const idx = (y * W + x) * 4;
        d[idx] = bright; d[idx + 1] = bright + 20; d[idx + 2] = bright + 60; d[idx + 3] = 255;
      }
    }
  }
  return img;
}

// Compute Lucas-Kanade flow vectors on a sparse grid
// Returns an array of {cx, cy, u, v} arrow descriptors
function computeFlow(imgA, imgB) {
  const a = imgA.data, b = imgB.data;
  const STEP = 8;  // sample every 8 pixels
  const WIN = 7;   // half-window size
  const arrows = [];
  for (let cy = WIN + 1; cy < H - WIN - 1; cy += STEP) {
    for (let cx = WIN + 1; cx < W - WIN - 1; cx += STEP) {
      // Accumulate the structure tensor and the right-hand side over the window
      let sIxx = 0, sIxy = 0, sIyy = 0, sIxt = 0, sIyt = 0;
      for (let wy = -WIN; wy <= WIN; wy++) {
        for (let wx = -WIN; wx <= WIN; wx++) {
          const x = cx + wx, y = cy + wy;
          // Convert RGB to grayscale (luminance formula)
          const idxC = (y * W + x) * 4;
          const gA = 0.299 * a[idxC] + 0.587 * a[idxC + 1] + 0.114 * a[idxC + 2];
          const gB = 0.299 * b[idxC] + 0.587 * b[idxC + 1] + 0.114 * b[idxC + 2];
          // Central-difference spatial gradients and temporal gradient
          const rp = (y * W + x + 1) * 4, rm = (y * W + x - 1) * 4;
          const dp = ((y + 1) * W + x) * 4, dm = ((y - 1) * W + x) * 4;
          const Ix = (0.299 * (a[rp] - a[rm]) + 0.587 * (a[rp + 1] - a[rm + 1]) + 0.114 * (a[rp + 2] - a[rm + 2])) / 2;
          const Iy = (0.299 * (a[dp] - a[dm]) + 0.587 * (a[dp + 1] - a[dm + 1]) + 0.114 * (a[dp + 2] - a[dm + 2])) / 2;
          const It = gB - gA;
          sIxx += Ix * Ix; sIxy += Ix * Iy; sIyy += Iy * Iy;
          sIxt += Ix * It; sIyt += Iy * It;
        }
      }
      // Solve 2x2 system: [Ixx Ixy; Ixy Iyy][u;v] = -[Ixt; Iyt]
      const det = sIxx * sIyy - sIxy * sIxy;
      if (Math.abs(det) < 1e-3) continue; // skip singular or near-singular windows (flat region)
      const u = -(sIyy * sIxt - sIxy * sIyt) / det;
      const v = -(sIxx * sIyt - sIxy * sIxt) / det;
      // Skip very small or unrealistically large vectors
      const mag = Math.sqrt(u * u + v * v);
      if (mag < 0.2 || mag > 8) continue;
      arrows.push({ cx, cy, u, v });
    }
  }
  return arrows;
}

// Draw a single flow arrow at canvas position (x,y) with direction (u,v)
function drawArrow(x, y, u, v) {
  const scale = 4;
  const ex = x + u * scale, ey = y + v * scale;
  ctx.beginPath(); ctx.moveTo(x, y); ctx.lineTo(ex, ey);
  // Draw arrowhead lines
  const ang = Math.atan2(ey - y, ex - x);
  ctx.lineTo(ex - 5 * Math.cos(ang - 0.4), ey - 5 * Math.sin(ang - 0.4));
  ctx.moveTo(ex, ey);
  ctx.lineTo(ex - 5 * Math.cos(ang + 0.4), ey - 5 * Math.sin(ang + 0.4));
  ctx.stroke();
}

function render() {
  // Synthesize previous frame from current blob position minus one velocity step
  const prevX = bx - vx, prevY = by - vy;
  const imgA = makeFrame(prevX, prevY);
  const imgB = makeFrame(bx, by);
  ctx.putImageData(imgB, 0, 0);
  const arrows = computeFlow(imgA, imgB);
  ctx.strokeStyle = 'rgba(255,200,60,0.85)';
  ctx.lineWidth = 1.5;
  arrows.forEach(({ cx, cy, u, v }) => drawArrow(cx, cy, u, v));
  document.getElementById('status').textContent =
    i18nFmt('Frame %frame% · %arrows% flow arrows', { frame: frame, arrows: arrows.length });
}

function step() {
  frame++;
  // Bounce the blob off canvas edges
  if (bx + vx > W - 40 || bx + vx < 40) vx = -vx;
  if (by + vy > H - 40 || by + vy < 40) vy = -vy;
  bx += vx; by += vy;
  render();
}

function startPlay() {
  if (playing) return;
  playing = true;
  document.getElementById('btn-play').textContent = 'Pause';
  document.getElementById('btn-play').disabled = false;
  function loop() { step(); rafId = requestAnimationFrame(loop); }
  rafId = requestAnimationFrame(loop);
}
function stopPlay() {
  playing = false;
  if (rafId) { cancelAnimationFrame(rafId); rafId = null; }
  document.getElementById('btn-play').textContent = 'Play';
}

document.getElementById('btn-play').addEventListener('click', () => {
  if (playing) stopPlay(); else startPlay();
});
document.getElementById('btn-step').addEventListener('click', () => { stopPlay(); step(); });
document.getElementById('btn-reset').addEventListener('click', () => {
  stopPlay(); bx = 80; by = 120; vx = 1.4; vy = 0.6; frame = 0; render();
  document.getElementById('status').textContent = 'Ready — press Play or Step.';
});

// Draw the first frame
render();

<!-- Canvas demo: Lucas-Kanade optical flow on a synthetic moving blob -->
<p class="hint">A bright blob moves across the canvas. The golden arrows show the velocity <b>(u,v)</b> that Lucas-Kanade estimates at each grid point — arrows appear only where the image has gradients in two directions (the blob edge), not on the flat background.</p>
<canvas id="canvas" width="320" height="240"></canvas>
<div class="status" id="status">Ready — press Play or Step.</div>
<div class="btns">
  <button id="btn-play" type="button">Play</button>
  <button id="btn-step" type="button" class="ghost">Step</button>
  <button id="btn-reset" type="button" class="ghost">Reset</button>
</div>

/* Minimal dark-canvas layout with control buttons */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .85rem; color: #444; margin: 0 0 .6rem; line-height: 1.45; }
canvas { display: block; border: 1px solid #cdd9e3; border-radius: 8px; background: #000; max-width: 100%; }
.status { font-size: .95rem; font-weight: 600; margin: .5rem 0; min-height: 1.4em; color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }
button:disabled { opacity: .45; cursor: default; }

const i18nFmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Lucas-Kanade optical flow: assumes brightness constancy and constant flow
// over a local window, then solves the 2x2 least-squares normal equations.

const W = 320, H = 240;
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');

// Blob position and velocity (pixels per frame)
let bx = 80, by = 120, vx = 1.4, vy = 0.6;
let frame = 0;
let playing = false;
let rafId = null;

// Render a single grayscale-ish frame with a soft Gaussian-ish blob
function makeFrame(cx, cy) {
  const img = ctx.createImageData(W, H);
  const d = img.data;
  // Dark background
  for (let i = 0; i < d.length; i += 4) {
    d[i] = 18; d[i + 1] = 18; d[i + 2] = 28; d[i + 3] = 255;
  }
  // Radial gradient blob: brightness falls off with distance squared
  const R = 36;
  for (let y = 0; y < H; y++) {
    for (let x = 0; x < W; x++) {
      const dx = x - cx, dy = y - cy;
      const dist2 = dx * dx + dy * dy;
      if (dist2 < R * R) {
        const bright = Math.round(200 * (1 - dist2 / (R * R)));
        const idx = (y * W + x) * 4;
        d[idx] = bright; d[idx + 1] = bright + 20; d[idx + 2] = bright + 60; d[idx + 3] = 255;
      }
    }
  }
  return img;
}

// Compute Lucas-Kanade flow vectors on a sparse grid
// Returns an array of {cx, cy, u, v} arrow descriptors
function computeFlow(imgA, imgB) {
  const a = imgA.data, b = imgB.data;
  const STEP = 8;  // sample every 8 pixels
  const WIN = 7;   // half-window size
  const arrows = [];
  for (let cy = WIN + 1; cy < H - WIN - 1; cy += STEP) {
    for (let cx = WIN + 1; cx < W - WIN - 1; cx += STEP) {
      // Accumulate the structure tensor and the right-hand side over the window
      let sIxx = 0, sIxy = 0, sIyy = 0, sIxt = 0, sIyt = 0;
      for (let wy = -WIN; wy <= WIN; wy++) {
        for (let wx = -WIN; wx <= WIN; wx++) {
          const x = cx + wx, y = cy + wy;
          // Convert RGB to grayscale (luminance formula)
          const idxC = (y * W + x) * 4;
          const gA = 0.299 * a[idxC] + 0.587 * a[idxC + 1] + 0.114 * a[idxC + 2];
          const gB = 0.299 * b[idxC] + 0.587 * b[idxC + 1] + 0.114 * b[idxC + 2];
          // Central-difference spatial gradients and temporal gradient
          const rp = (y * W + x + 1) * 4, rm = (y * W + x - 1) * 4;
          const dp = ((y + 1) * W + x) * 4, dm = ((y - 1) * W + x) * 4;
          const Ix = (0.299 * (a[rp] - a[rm]) + 0.587 * (a[rp + 1] - a[rm + 1]) + 0.114 * (a[rp + 2] - a[rm + 2])) / 2;
          const Iy = (0.299 * (a[dp] - a[dm]) + 0.587 * (a[dp + 1] - a[dm + 1]) + 0.114 * (a[dp + 2] - a[dm + 2])) / 2;
          const It = gB - gA;
          sIxx += Ix * Ix; sIxy += Ix * Iy; sIyy += Iy * Iy;
          sIxt += Ix * It; sIyt += Iy * It;
        }
      }
      // Solve 2x2 system: [Ixx Ixy; Ixy Iyy][u;v] = -[Ixt; Iyt]
      const det = sIxx * sIyy - sIxy * sIxy;
      if (Math.abs(det) < 1e-3) continue; // skip singular or near-singular windows (flat region)
      const u = -(sIyy * sIxt - sIxy * sIyt) / det;
      const v = -(sIxx * sIyt - sIxy * sIxt) / det;
      // Skip very small or unrealistically large vectors
      const mag = Math.sqrt(u * u + v * v);
      if (mag < 0.2 || mag > 8) continue;
      arrows.push({ cx, cy, u, v });
    }
  }
  return arrows;
}

// Draw a single flow arrow at canvas position (x,y) with direction (u,v)
function drawArrow(x, y, u, v) {
  const scale = 4;
  const ex = x + u * scale, ey = y + v * scale;
  ctx.beginPath(); ctx.moveTo(x, y); ctx.lineTo(ex, ey);
  // Draw arrowhead lines
  const ang = Math.atan2(ey - y, ex - x);
  ctx.lineTo(ex - 5 * Math.cos(ang - 0.4), ey - 5 * Math.sin(ang - 0.4));
  ctx.moveTo(ex, ey);
  ctx.lineTo(ex - 5 * Math.cos(ang + 0.4), ey - 5 * Math.sin(ang + 0.4));
  ctx.stroke();
}

function render() {
  // Synthesize previous frame from current blob position minus one velocity step
  const prevX = bx - vx, prevY = by - vy;
  const imgA = makeFrame(prevX, prevY);
  const imgB = makeFrame(bx, by);
  ctx.putImageData(imgB, 0, 0);
  const arrows = computeFlow(imgA, imgB);
  ctx.strokeStyle = 'rgba(255,200,60,0.85)';
  ctx.lineWidth = 1.5;
  arrows.forEach(({ cx, cy, u, v }) => drawArrow(cx, cy, u, v));
  document.getElementById('status').textContent =
    i18nFmt('Frame %frame% · %arrows% flow arrows', { frame: frame, arrows: arrows.length });
}

function step() {
  frame++;
  // Bounce the blob off canvas edges
  if (bx + vx > W - 40 || bx + vx < 40) vx = -vx;
  if (by + vy > H - 40 || by + vy < 40) vy = -vy;
  bx += vx; by += vy;
  render();
}

function startPlay() {
  if (playing) return;
  playing = true;
  document.getElementById('btn-play').textContent = 'Pause';
  document.getElementById('btn-play').disabled = false;
  function loop() { step(); rafId = requestAnimationFrame(loop); }
  rafId = requestAnimationFrame(loop);
}
function stopPlay() {
  playing = false;
  if (rafId) { cancelAnimationFrame(rafId); rafId = null; }
  document.getElementById('btn-play').textContent = 'Play';
}

document.getElementById('btn-play').addEventListener('click', () => {
  if (playing) stopPlay(); else startPlay();
});
document.getElementById('btn-step').addEventListener('click', () => { stopPlay(); step(); });
document.getElementById('btn-reset').addEventListener('click', () => {
  stopPlay(); bx = 80; by = 120; vx = 1.4; vy = 0.6; frame = 0; render();
  document.getElementById('status').textContent = 'Ready — press Play or Step.';
});

// Draw the first frame
render();

Result

Console

Notice that arrows inside the blob track its true motion while arrows on the featureless background stay near zero — exactly what the algorithm predicts: it can only recover motion where the image has both horizontal and vertical intensity gradients (i.e., corners or edges, not flat regions).

The Real Complexity

The brightness-constancy constraint

Let $I(x, y, t)$ be the brightness of pixel $(x, y)$ at time $t$ . If pixel $(x, y)$ moves to $(x + u, y + v)$ by time $t + 1$ , brightness constancy says:

I(x, y, t) = I(x + u, y + v, t + 1)

Taylor-expanding the right side and keeping only first-order terms gives the optical flow constraint equation:

I_x \, u + I_y \, v + I_t = 0

where $I_x$ , $I_y$ are spatial gradients and $I_t$ is the temporal gradient at that pixel. One equation, two unknowns — that is the aperture problem in mathematical form.

The Lucas-Kanade fix: a local window

Lucas and Kanade assume the flow $(u, v)$ is constant over a small $w \times w$ window around each pixel. With $w^2$ pixels in the window you have $w^2$ equations and still only two unknowns — an over-determined system. They solve it in the least-squares sense:

\begin{pmatrix} \sum I_x^2 & \sum I_x I_y \\ \sum I_x I_y & \sum I_y^2 \end{pmatrix} \begin{pmatrix} u \\ v \end{pmatrix} = -\begin{pmatrix} \sum I_x I_t \\ \sum I_y I_t \end{pmatrix}

The $2 \times 2$ matrix on the left is called the structure tensor (or Harris matrix). The system has a unique solution exactly when the tensor is invertible — which happens when the window contains gradients in at least two independent directions, i.e., a corner or textured region.

Flat regions: both eigenvalues near zero — flow is undefined.
Edges: one large eigenvalue — flow is constrained only perpendicular to the edge (aperture problem).
Corners: both eigenvalues large — flow is fully determined.

Computational cost

At each pixel: compute three gradients, accumulate a $2 \times 2$ matrix over the window, invert it. That is $O(w^2)$ per pixel and $O(N w^2)$ total for an $N$ -pixel image — a constant multiplied by the image size, fully parallelizable on a GPU.

Where It Matters

Per-pixel motion is one of the most useful signals in visual computing:

Video compression: MPEG and H.264/H.265 use motion vectors — computed by variants of this idea — to encode only the difference between frames. Without optical flow, streaming video would be 10–50× larger.
Autonomous driving: cameras on self-driving cars track how the road, pedestrians, and obstacles appear to move. Combined with dimensionality reduction and depth estimation, flow reveals the 3-D structure of the scene.
Medical imaging: echocardiography and MRI motion analysis use optical flow to track heart-wall motion across time, quantifying cardiac function without invasive procedures.
Action recognition: instead of classifying raw pixels, many systems first compute flow and classify the motion pattern — dramatically reducing the complexity and improving generalization.
Augmented reality: anchoring virtual objects to a real scene requires knowing exactly how every surface point moves; optical flow provides the dense estimate that sparse feature trackers miss.

Modern deep-learning approaches (FlowNet, RAFT) replace the structure tensor with learned filters but still train on the same brightness-constancy intuition. Understanding Lucas-Kanade is the fastest path into that literature.

Conclusion

Lucas-Kanade optical flow is a masterclass in turning a hard, ill-posed problem into a tractable one with a single physical assumption: pixels keep their brightness as they move. That one sentence becomes a linear system, the linear system becomes a $2 \times 2$ matrix inverse, and the matrix inverse becomes a real-time algorithm used in billions of devices today.

The limits are just as instructive as the successes: flat regions give no information, large motions break the Taylor expansion, and occlusions violate brightness constancy entirely. Each limitation has spawned decades of follow-on research — pyramidal Lucas-Kanade, robust estimation, deep flow networks — but every one of them traces its roots to the elegant constraint equation $I_x u + I_y v + I_t = 0$ .

Next time you pause a video and notice the streaks of motion blur, remember: a computer is solving thousands of $2 \times 2$ systems to understand exactly what you are seeing.

Comments

Loading comments...

https://www.kipuhub.com/en/article/optical-flow-lucas-kanade/Content licensed under CC BY-NC 4.0.

Cookies & privacy

Optical Flow (Lucas-Kanade)

The equation that lets computers read motion from light