Harris Corner Detection

Look at any photograph and pick a point at random. If it lies on a smooth wall, the color barely changes in any direction — it carries almost no information. If it lies on a straight edge, intensity changes sharply across the edge but not along it, so you can slide along the edge without knowing where you are. But if it sits at a corner — where two edges meet — intensity changes strongly in every direction. You are pinned.

That simple insight drives Harris Corner Detection, introduced by Chris Harris and Mike Stephens in 1988. By measuring how much brightness changes when you shift a small window in any direction, the algorithm assigns every pixel a corner response score. High score means corner; near-zero means flat; a score that is high in one direction but low in another means edge.

The result is a set of stable, repeatable landmark points — the same physical corner lights up whenever the camera sees it, regardless of small shifts in viewpoint or lighting. Those landmarks are the foundation of almost everything modern computer vision does: stitching panoramas, tracking objects, reconstructing 3-D scenes from video, and matching features across images taken from completely different angles.

The canvas below draws a checkerboard pattern and runs the Harris detector on every pixel. Red dots mark the corners that scored above the threshold; the color of each pixel's background shows its raw corner response (brighter = stronger response).

<!-- {{c_html_desc}} -->
<div class="controls">
  <label>{{lbl_threshold}} <span id="threshVal">40</span>%
    <input type="range" id="thresh" min="1" max="99" value="40">
  </label>
  <label>{{lbl_cell_size}}
    <select id="cellSz">
      <option value="16">16 px</option>
      <option value="24" selected>24 px</option>
      <option value="32">32 px</option>
    </select>
  </label>
  <button id="btnRun" type="button">{{btn_run}}</button>
</div>
<div class="canvas-wrap">
  <canvas id="cvs"></canvas>
</div>
<div class="legend">
  <span class="dot corner"></span> {{legend_corner}}
  <span class="dot edge"></span> {{legend_edge}}
  <span class="dot flat"></span> {{legend_flat}}
</div>
<div id="status" class="status"></div>

/* {{c_css_desc}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; gap: .7rem; flex-wrap: wrap; align-items: center; margin-bottom: .6rem; }
label { font-size: .88rem; display: flex; align-items: center; gap: .35rem; }
input[type=range] { width: 110px; cursor: pointer; }
select { font-size: .88rem; border: 1px solid #aaa; border-radius: 6px; padding: .15rem .3rem; }
button { font: 600 13px system-ui; padding: .4rem .85rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
.canvas-wrap { overflow: auto; max-width: 100%; }
canvas { display: block; image-rendering: pixelated; border: 1px solid #ccc; border-radius: 4px; }
.legend { font-size: .82rem; margin-top: .5rem; display: flex; gap: .9rem; align-items: center; }
.dot { display: inline-block; width: 12px; height: 12px; border-radius: 50%; }
.dot.corner { background: #e63946; }
.dot.edge    { background: #457b9d; }
.dot.flat    { background: #ccc; }
.status { font-size: .88rem; font-weight: 600; margin-top: .4rem; min-height: 1.2em; color: #1d3557; }

// Code not found

Drag the threshold slider to control how selective the detector is. Lower the threshold and more corners light up — including some that are barely distinguishable from edges. Raise it and only the sharpest corners survive. This trade-off between recall (finding everything) and precision (finding only true corners) is the same one that every feature detector faces.

The core idea is the structure tensor (also called the second-moment matrix). For each pixel, compute the image gradients $I_x$ and $I_y$ — roughly, how fast brightness changes horizontally and vertically. Then form the $2 \times 2$ matrix

$$M = \begin{pmatrix} \sum I_x^2 & \sum I_x I_y \\ \sum I_x I_y & \sum I_y^2 \end{pmatrix}$$

where the sums run over a small window (typically $3 \times 3$ or $5 \times 5$ pixels) around the pixel of interest. The matrix $M$ encodes how the local image patch responds to shifts in each direction.

The eigenvalues $\lambda_1, \lambda_2$ of $M$ tell the whole story:

• Both small ($\lambda_1 \approx \lambda_2 \approx 0$): flat region — intensity barely changes anywhere.
• One large, one small ($\lambda_1 \gg \lambda_2$): edge — strong change in one direction, weak in another.
• Both large ($\lambda_1 \approx \lambda_2 \gg 0$): corner — strong change in every direction.

Computing eigenvalues explicitly is expensive. Harris and Stephens observed that you can score a pixel cheaply without them:

$$R = \det(M) - k \cdot \text{tr}(M)^2 = \lambda_1 \lambda_2 - k(\lambda_1 + \lambda_2)^2$$

with $k$ typically around $0.04$–$0.06$. A large positive $R$ means corner; $R$ near zero means flat; large negative $R$ means edge. The whole computation is $O(1)$ per pixel (one pass for gradients, one convolution for window sums, one arithmetic expression for $R$), so the full image runs in $O(n)$ time — linear in the number of pixels.

The detector was solved by Harris and Stephens in 1988 and remains in active use today, both directly and as the mathematical backbone of many modern pattern matching and dimensionality reduction pipelines.

Stable, repeatable landmark points are the currency of computer vision. Harris corners appear wherever a system needs to track or match what it sees:

• Panorama stitching: find matching corners in two overlapping photos, compute the homography between them, warp and blend. Apps like Google Photos and iPhone do this in milliseconds.
• Structure from Motion (SfM): track corners across frames of a moving camera to reconstruct a 3-D scene from a flat video — the engine behind photogrammetry drones and city-scale 3-D maps.
• SLAM (Simultaneous Localization and Mapping): robots and AR headsets use corners as landmarks to build a map while tracking their own position inside it.
• Optical flow: matching corners between consecutive video frames gives the apparent motion of objects — used in video stabilization, sports analytics, and autonomous driving.
• Augmented reality: stick a virtual object on a real-world corner and it stays anchored even as the camera moves.

Modern descriptors like SIFT, SURF, and ORB still detect corners at their core, just with more scale-space machinery around the Harris idea. The 1988 paper remains one of the most cited in all of computer vision.

Harris Corner Detection is a masterclass in turning a geometric intuition into a fast, practical algorithm. The observation that corners are "pinned" in every direction becomes a $2 \times 2$ matrix; the matrix becomes a scalar score; the score runs in linear time on the full image.

The corners it finds are not arbitrary pixels — they are the places where the image carries the most geometric information. Stitch them across two photographs and you can align the world. Track them across a video and you can reconstruct a 3-D scene from flat frames. That is the quiet power of a well-chosen mathematical structure: the right abstraction unlocks a whole family of hard problems at once.

Next time you open a panorama stitched perfectly by your phone, or watch an AR label stick to a real object through a moving camera, remember: it is anchored to a set of corners found by asking, for each tiny patch of pixels, how much does this place resist being confused with its neighbors?