Gram-Schmidt Orthogonalization

Imagine you are describing a location in a city. You say: "three blocks north, two blocks east." Those two directions — north and east — are orthogonal: they share nothing, they are independent, and together they span the whole plane. Most coordinate systems we use in practice share this pleasant property.

But raw data rarely arrives in such a tidy form. A dataset of measurements might give you vectors that point in similar directions, partially overlapping. To do geometry, solve equations, or train a machine-learning model, you almost always need a cleaner basis — one where the vectors are perpendicular to each other (orthogonal) and each has length one (normal). Together that is called an orthonormal basis.

The Gram-Schmidt process, developed independently by Jørgen Pedersen Gram (1883) and Erhard Schmidt (1907), is the classical recipe for building exactly that. The idea is beautifully simple: take each vector in turn, subtract the shadow it casts onto all previous vectors, and what remains is a new direction that is perpendicular to everything before it. Normalize it to length one, and you have your next basis vector.

The process always works for any set of linearly independent vectors, and it produces an orthonormal basis for the subspace they span. But there is a catch: when implemented on a computer, the classical version can lose accuracy due to floating-point errors. A small tweak — the modified Gram-Schmidt algorithm — fixes this without changing the mathematical result.

The demo below shows two vectors in the plane. Drag the tip of each arrow to reposition it, then press Orthogonalize to run Gram-Schmidt step by step.

<!-- {{c_html_intro}} -->
<div class="hint">{{hint_para}}</div>
<canvas id="gs-canvas" width="380" height="280"></canvas>
<div class="status" id="gs-status">{{status_initial}}</div>
<div class="btns">
  <button id="btn-run" type="button">{{btn_run}}</button>
  <button id="btn-reset" type="button" class="ghost">{{btn_reset}}</button>
</div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .88rem; color: #444; margin: 0 0 .6rem; line-height: 1.5; }
canvas { display: block; background: #f4f7fa; border-radius: 10px; border: 1px solid #dce3eb;
         cursor: crosshair; touch-action: none; max-width: 100%; }
.status { font-size: .95rem; font-weight: 600; margin: .45rem 0; min-height: 1.4em; }
.status.ok  { color: #0a7d33; }
.status.bad { color: #c92f3c; }
.status.info { color: #1d4e8f; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

Step 1 keeps $\mathbf{v}_1$ as-is and normalizes it to $\mathbf{e}_1$. Step 2 subtracts from $\mathbf{v}_2$ its projection onto $\mathbf{e}_1$ — whatever they share — leaving a remainder that is exactly perpendicular. Normalizing that remainder gives $\mathbf{e}_2$. The result is always a right angle, regardless of where you placed the original vectors — as long as they are not parallel.

Mathematically, classical and modified Gram-Schmidt are identical — both produce the same orthonormal basis. On a real computer, they behave very differently.

Classical Gram-Schmidt computes the full projection of vector $\mathbf{v}_k$ onto all previous basis vectors at once:

$$\mathbf{u}_k = \mathbf{v}_k - \sum_{j=1}^{k-1} \langle \mathbf{v}_k, \mathbf{e}_j \rangle \, \mathbf{e}_j$$

When vectors are nearly parallel, $\mathbf{u}_k$ is the difference of nearly equal numbers. Floating-point arithmetic then magnifies tiny rounding errors, and the resulting vectors can drift far from orthogonal. In bad cases, $\langle \mathbf{e}_i, \mathbf{e}_j \rangle$ can be $10^{-6}$ or worse instead of zero.

Modified Gram-Schmidt subtracts one projection at a time, updating the working vector after each step:

$$\mathbf{u}_k^{(j)} = \mathbf{u}_k^{(j-1)} - \langle \mathbf{u}_k^{(j-1)}, \mathbf{e}_j \rangle \, \mathbf{e}_j$$

Each subtraction uses the most up-to-date version of the vector, so errors do not compound. The result is dramatically more orthogonal in practice.

Even modified Gram-Schmidt is not the last word. For the most demanding applications, Householder reflections (used in standard QR decompositions) provide even better numerical guarantees. But modified Gram-Schmidt remains the workhorse of iterative methods like GMRES and the Lanczos algorithm, where vectors arrive one at a time and you cannot afford to start over.

Orthonormal bases are the backbone of numerical computation, and Gram-Schmidt is how you build them from scratch:

• QR decomposition: the factorization $A = QR$ (where $Q$ has orthonormal columns) underlies least-squares solvers, eigenvalue algorithms, and statistical regression. Gram-Schmidt is one of the three standard ways to compute $Q$.
• Principal Component Analysis (PCA): the principal components of a dataset must be orthogonal. After computing the covariance matrix, an eigenvector decomposition already produces orthogonal directions — but Gram-Schmidt is used in incremental and streaming variants where data arrives in batches.
• Signal processing: orthonormal bases like the Fourier basis or wavelet bases let you represent signals without redundancy. Gram-Schmidt is the tool for building custom orthonormal frames tailored to a specific dataset.
• Iterative linear solvers: Krylov-subspace methods (GMRES, Arnoldi, Lanczos) build an orthonormal basis for a growing subspace one vector at a time. Modified Gram-Schmidt keeps this basis numerically clean without recomputing everything from scratch.
• Computer graphics: orthonormal coordinate frames are needed for camera orientations, surface normals, and tangent-space calculations. The Gram-Schmidt process (often a quick two-vector version) is a standard primitive in 3D engines.

Learn Gram-Schmidt and you have unlocked a tool that quietly runs inside nearly every serious numerical algorithm, from dimensionality reduction to differential equation solvers.

The Gram-Schmidt process distills a powerful idea into a single rule: subtract what two vectors share, keep what is new. Repeat that for each vector in your set, normalize at each step, and you are left with a family of directions that are perfectly perpendicular and independent — an orthonormal basis.

The classical version is easy to understand; the modified version is easy to run on a computer. That small reordering of subtractions, which changes nothing mathematically, is the difference between a basis that drifts and one that stays true.

The next time you solve a least-squares problem, run PCA on a dataset, or watch a CGI camera pivot smoothly, somewhere in that computation a set of vectors was straightened into right angles — and Gram-Schmidt, in one form or another, is why they ended up at exactly 90 degrees.