Introduction

Describe one cup of coffee and you might list its acidity, sweetness, body, aroma, bitterness, aftertaste. That is six numbers for a single cup — a point in six-dimensional space. Add a few more notes and you are in a space no human eye can ever picture.

This is the everyday reality of data. A photo is thousands of pixels; a customer is dozens of behaviors; a gene profile is tens of thousands of measurements. Each new feature is another axis, and past three axes our intuition simply stops.

Dimensionality reduction is the art of folding all those axes down to two or three — few enough to draw — while keeping the part that matters: which points are close, which clusters apart, what the data is shaped like. The trick is deciding what to keep and what to throw away.

Project the Cloud

Below is a cloud of 3D points seen from above. You have to flatten it onto a single line (a 1D shadow). The question PCA answers: which direction keeps the most spread — the most information?

<p class="hint">A cloud of points lives in 2D. To "reduce" it we cast every point onto a single line — a 1D shadow. The goal: pick the line direction that keeps the <b>most spread</b> (variance). Checking a direction is instant; finding the best one is what PCA does for you.</p>
<canvas id="cv" width="320" height="320"></canvas>
<div class="ctl">
  <label>Projection angle: <span id="ang">0</span>&deg;</label>
  <input id="slider" type="range" min="0" max="179" value="0">
</div>
<div class="status" id="status">Variance kept: &mdash;</div>
<div class="btns">
  <button id="find" type="button">Find best axis (PCA)</button>
  <button id="reset" type="button" class="ghost">New cloud</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
canvas { background: #f3f6f9; border: 1px solid #cdd9e3; border-radius: 10px; display: block; touch-action: none; }
.ctl { margin: .7rem 0 .3rem; font-size: .9rem; color: #1d3557; }
.ctl label { font-weight: 600; }
input[type=range] { width: 100%; margin-top: .3rem; }
.status { font-size: 1rem; font-weight: 700; margin: .4rem 0; min-height: 1.4em; color: #1d3557; }
.status.best { color: #0a7d33; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Dimensionality reduction, the smallest case: project a 2D cloud onto a 1D line.
// For a unit direction u = (cos t, sin t), the variance kept is uᵀ·C·u where C is
// the covariance matrix of the (centered) points. PCA's best line is the top
// eigenvector of C — the angle that MAXIMIZES that variance. Checking one angle is O(n); the exact best angle here comes from a closed form for the 2x2 eigenvector.
const cv = document.getElementById('cv'), ctx = cv.getContext('2d');
const W = cv.width, H = cv.height, CX = W/2, CY = H/2;
let pts = [];      // centered points {x,y}
let angle = 0;     // current projection angle in degrees

function makeCloud(){
  // An elongated, tilted Gaussian-ish blob so one direction clearly dominates.
  const n = 80, tilt = 35 * Math.PI/180, sx = 70, sy = 18;
  const raw = [];
  for(let i=0;i<n;i++){
    // crude gaussian via sum of uniforms
    let g1=0,g2=0; for(let k=0;k<6;k++){g1+=Math.random();g2+=Math.random();}
    g1=(g1-3)/1.5; g2=(g2-3)/1.5;
    const ex = g1*sx, ey = g2*sy;
    raw.push({ x: ex*Math.cos(tilt) - ey*Math.sin(tilt),
               y: ex*Math.sin(tilt) + ey*Math.cos(tilt) });
  }
  // center
  const mx = raw.reduce((s,p)=>s+p.x,0)/n, my = raw.reduce((s,p)=>s+p.y,0)/n;
  pts = raw.map(p=>({x:p.x-mx, y:p.y-my}));
}

// Covariance entries of the centered cloud.
function cov(){
  let cxx=0,cyy=0,cxy=0; const n=pts.length;
  for(const p of pts){ cxx+=p.x*p.x; cyy+=p.y*p.y; cxy+=p.x*p.y; }
  return { cxx:cxx/n, cyy:cyy/n, cxy:cxy/n };
}
// Variance of the cloud after projecting onto unit direction at angle deg.
function varianceAt(deg){
  const t = deg*Math.PI/180, ux=Math.cos(t), uy=Math.sin(t);
  const {cxx,cyy,cxy} = cov();
  return ux*ux*cxx + uy*uy*cyy + 2*ux*uy*cxy; // uᵀ C u
}
function totalVar(){ const {cxx,cyy}=cov(); return cxx+cyy; }
// PCA: best angle = top eigenvector of 2x2 covariance (closed form).
function bestAngleDeg(){
  const {cxx,cyy,cxy} = cov();
  let t = 0.5 * Math.atan2(2*cxy, cxx-cyy); // principal axis
  let deg = t*180/Math.PI; deg=((deg%180)+180)%180; // normalize 0..180
  return deg;
}

function draw(){
  ctx.clearRect(0,0,W,H);
  const t = angle*Math.PI/180, ux=Math.cos(t), uy=Math.sin(t);
  // projection line through center
  ctx.strokeStyle='#9aa7b4'; ctx.lineWidth=2;
  ctx.beginPath(); ctx.moveTo(CX-ux*200, CY+uy*200); ctx.lineTo(CX+ux*200, CY-uy*200); ctx.stroke();
  // points + their shadows on the line
  for(const p of pts){
    const px=CX+p.x, py=CY-p.y;
    const s = p.x*ux + p.y*uy; // scalar projection
    const sx=CX+s*ux, sy=CY-s*uy;
    ctx.strokeStyle='rgba(29,53,87,.15)'; ctx.lineWidth=1;
    ctx.beginPath(); ctx.moveTo(px,py); ctx.lineTo(sx,sy); ctx.stroke();
    ctx.fillStyle='#1d3557'; ctx.beginPath(); ctx.arc(px,py,2.6,0,7); ctx.fill();
    ctx.fillStyle='#e63946'; ctx.beginPath(); ctx.arc(sx,sy,2.6,0,7); ctx.fill();
  }
}
function pct(deg){ return Math.round(100*varianceAt(deg)/totalVar()); }
function update(best){
  draw();
  const s=document.getElementById('status');
  s.textContent=fmt('Variance kept: %pct%%', { pct: pct(angle) });
  s.className='status'+(best?' best':'');
}
document.getElementById('slider').addEventListener('input',e=>{
  angle=+e.target.value; document.getElementById('ang').textContent=angle; update(false);
});
document.getElementById('find').addEventListener('click',()=>{
  angle=Math.round(bestAngleDeg());
  document.getElementById('slider').value=angle;
  document.getElementById('ang').textContent=angle;
  update(true);
  console.log('PCA best angle =', angle, 'deg, variance kept =,', 'variance kept =', pct(angle)+'%');
});
document.getElementById('reset').addEventListener('click',()=>{
  makeCloud(); angle=0; document.getElementById('slider').value=0;
  document.getElementById('ang').textContent=0; update(false);
});
makeCloud(); update(false);
Result
Console

Spin the projection line and watch the variance kept rise and fall. Checking a given direction is cheap: project every point, measure the spread. Finding the best direction by hand is fiddly — but press Find best axis (PCA) and the computer lands on the principal component, the line that preserves the most variance. That single best direction is exactly what PCA computes from the data's covariance.

The Real Complexity

How hard is it to compress dimensions well?

  • Linear reduction (PCA) is tractable. Introduced by Karl Pearson (1901) and formalized by Harold Hotelling (1933), PCA finds the directions of greatest variance as the eigenvectors of the covariance matrix. Computing them is an eigenvalue / singular-value-decomposition problem solvable in polynomial time — about O(n⋅d2+d3)O(n \cdot d^{2} + d^{3}) for n points in d dimensions. Cheap, exact, and unique up to sign.
  • The catch is the curse of dimensionality. As d grows, points spread out until distances all look the same and "nearest neighbor" loses meaning. Keeping more axes preserves information but costs storage, noise and compute; keeping fewer is readable but lossy. That information-versus-cost tradeoff is the heart of the problem.
  • Nonlinear reduction is harder. Methods like t-SNE (van der Maaten & Hinton, 2008) and UMAP (McInnes et al., 2018) can unfold curved structure PCA misses, but they optimize a non-convex objective — no single exact answer, results depend on initialization and parameters, and naive versions cost O(n2)O(n^{2}).

So the polite version (linear, variance-preserving) is a clean polynomial-time eigenvalue computation, while the faithful version (nonlinear, neighbor-preserving) is an approximate optimization with all the trouble that brings — the same kind of landscape you meet in non-convex optimization.

Where It Matters

Reducing dimensions is one of the quietest workhorses in all of data science:

  • Visualization: biologists squeeze tens of thousands of gene measurements per cell into a 2D map where cell types fall into visible clusters.
  • Compression and denoising: keep the top components of an image or signal and you store far less while discarding mostly noise.
  • Faster, leaner models: fewer input axes mean less overfitting and quicker training — reduction is a standard preprocessing step before classifiers and neural-network training.
  • Embeddings everywhere: word, image and user embeddings are high-dimensional vectors routinely projected down for search, clustering and exploration.

Whenever you've seen a tidy 2D scatter of something that obviously had way more than two properties, dimensionality reduction did the folding — and the choice of what to keep shaped what you saw.

Conclusion

Dimensionality reduction is a bargain we strike with our own eyes: we cannot see ten axes, so we fold them into two and accept a little loss in exchange for a picture we can actually read. PCA makes that bargain precise and cheap — the best linear view is just the top eigenvectors of the covariance, found in polynomial time.

But "best" always means "best at preserving something," and the moment we want curved structure faithfully unrolled, we step into non-convex optimization with no perfect answer. The next time a complicated dataset collapses into a clean, readable plot, remember: a lot was thrown away on purpose — and choosing what to keep was the whole game. It is a close cousin of compression, where keeping the signal and dropping the rest is the entire point.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/dimensionality-reduction/Content licensed under CC BY-NC 4.0.