Introduction

A decision tree is the most human model in machine learning: a flowchart of yes/no questions — is income above 40k? is age under 30? — that ends in a prediction. It is fast, readable, and needs no scaling or tuning. It is also, on its own, terrible: grow it deep enough to fit the training data and it memorizes the noise, drawing a jagged boundary that collapses on anything new. This is overfitting, and a lone tree is its poster child.

The fix that took over applied machine learning is almost embarrassingly simple: don't trust one tree — grow a crowd of them and let them vote. If each tree is wrong in a different way, their mistakes cancel and what survives the average is the real signal.

Two recipes dominate. Random forests (Leo Breiman, 2001) grow hundreds of trees independently, each on a random slice of the data and features, then average. Boosting grows trees one at a time, each one trained to fix the errors the previous ones made. Both turn weak, shaky trees into the single most reliable predictor we have for tabular data — the spreadsheets and databases that run the real world.

Grow the Forest

Below are two classes of points (blue and orange) that you must separate. A single tree trained on a random subsample draws the boundary you see — notice how blocky and overconfident it is. Press Add tree to train another one on a fresh random subsample and average its vote into the ensemble.

<p class="hint">Two classes of points must be separated. Each <b>tree</b> is trained on a random subsample and votes for a class everywhere on the grid. One tree is jagged and overconfident — average many and the boundary smooths into a calibrated one.</p>
<canvas id="plot" width="320" height="320"></canvas>
<div class="status" id="status">Forest: 1 tree.</div>
<div class="btns">
  <button id="add" type="button">Add tree</button>
  <button id="grow" type="button">Grow 25</button>
  <button id="reset" type="button" class="ghost">Reset</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
canvas { border: 1px solid #cdd9e3; border-radius: 8px; display: block; image-rendering: pixelated; width: 320px; height: 320px; }
.status { font-size: 1rem; font-weight: 600; margin: .6rem 0; min-height: 1.4em; color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Random forest by BAGGING. Each tree is a small CART grown on a bootstrap
// sample (rows drawn with replacement). A tree alone overfits its sample and
// draws a blocky boundary; averaging many decorrelated trees' votes smooths it.
// This is Breiman's 2001 idea in miniature — variance reduction by averaging.
const SIZE = 320, GRID = 64;          // GRID x GRID prediction raster
const MAX_DEPTH = 4, FEAT_TRY = 1;    // shallow trees, 1 feature/split (decorrelate)
let rng = 12345;
const rand = () => (rng = (rng * 1103515245 + 12345) & 0x7fffffff) / 0x7fffffff;

// Two interleaving blobs (a soft XOR-ish pattern) — class 0 vs class 1.
function makeData(){
  const pts = [];
  for(let i=0;i<160;i++){
    const x = rand(), y = rand();
    // true label: a curved boundary plus noise
    const f = Math.sin(3.2*x) * 0.5 + 0.5;
    let label = y > f ? 1 : 0;
    if(rand() < 0.08) label = 1 - label;   // label noise -> single trees overfit
    pts.push({x, y, label});
  }
  return pts;
}
let DATA = makeData();

// ---- CART: grow a tree on a sample, predicting class probability for label 1
function gini(rows){
  if(!rows.length) return 0;
  let p = rows.reduce((s,r)=>s+r.label,0)/rows.length;
  return 2*p*(1-p);
}
function buildTree(rows, depth){
  const n = rows.length;
  const p1 = n ? rows.reduce((s,r)=>s+r.label,0)/n : 0.5;
  if(depth>=MAX_DEPTH || n<6 || p1===0 || p1===1) return {leaf:true, p:p1};
  let best=null;
  const feats=['x','y'];
  // random feature subset (here pick FEAT_TRY of 2) to decorrelate trees
  const tryFeats = rand()<0.5 ? feats : [...feats].reverse();
  for(let fi=0; fi<FEAT_TRY; fi++){
    const f = tryFeats[fi];
    for(let t=1;t<8;t++){
      const thr = t/8;
      const L=[],R=[];
      for(const r of rows){ (r[f]<thr?L:R).push(r); }
      if(!L.length||!R.length) continue;
      const score = (L.length*gini(L)+R.length*gini(R))/n;
      if(!best||score<best.score) best={score,f,thr,L,R};
    }
  }
  if(!best) return {leaf:true,p:p1};
  return {leaf:false,f:best.f,thr:best.thr,
    left:buildTree(best.L,depth+1), right:buildTree(best.R,depth+1)};
}
function predictTree(node,x,y){
  while(!node.leaf) node = (({x,y})[node.f] < node.thr) ? node.left : node.right;
  return node.p;
}
function bootstrap(){
  const s=[]; for(let i=0;i<DATA.length;i++) s.push(DATA[(rand()*DATA.length)|0]); return s;
}

let forest=[];
function addTrees(k){ for(let i=0;i<k;i++) forest.push(buildTree(bootstrap(),0)); }

// ---- render: averaged probability over the grid + the data points
const cv=document.getElementById('plot'), ctx=cv.getContext('2d');
function render(){
  const img=ctx.createImageData(SIZE,SIZE);
  const cell=SIZE/GRID;
  for(let gy=0;gy<GRID;gy++)for(let gx=0;gx<GRID;gx++){
    const x=(gx+0.5)/GRID, y=1-(gy+0.5)/GRID;
    let p=0; for(const t of forest) p+=predictTree(t,x,y); p/=forest.length;
    // blue (class0) <-> orange (class1)
    const r=Math.round(70+p*(245-70)), g=Math.round(110+p*(150-110)), b=Math.round(200+p*(60-200));
    for(let dy=0;dy<cell;dy++)for(let dx=0;dx<cell;dx++){
      const px=((gy*cell+dy)*SIZE+(gx*cell+dx))*4;
      img.data[px]=r; img.data[px+1]=g; img.data[px+2]=b; img.data[px+3]=255;
    }
  }
  ctx.putImageData(img,0,0);
  for(const pt of DATA){
    ctx.beginPath();
    ctx.arc(pt.x*SIZE,(1-pt.y)*SIZE,3.5,0,7);
    ctx.fillStyle=pt.label? '#e8852a':'#2f6bd0';
    ctx.fill(); ctx.lineWidth=1; ctx.strokeStyle='#fff'; ctx.stroke();
  }
  const s=document.getElementById('status');
  s.textContent=fmt('Forest: %length% %v%', { length: forest.length, v: forest.length===1?'tree.':'trees — boundary averaged across them.' });
}
function reset(){ forest=[]; addTrees(1); render(); }
document.getElementById('add').addEventListener('click',()=>{ addTrees(1); render(); });
document.getElementById('grow').addEventListener('click',()=>{ addTrees(25); render(); });
document.getElementById('reset').addEventListener('click',reset);
reset();
Result
Console

Watch the boundary. With one tree it is jagged and brittle. As you add more, the averaged boundary smooths out and the shaded confidence becomes graded instead of all-or-nothing — that soft transition is hundreds of slightly different trees disagreeing politely at the edges. Press Grow 25 to jump ahead, and Reset to start from a single tree again. No single tree got better; the crowd did.

The Real Complexity

A single deep tree has low bias (it can fit anything) but high variance (shuffle the data slightly and the tree changes wildly). Ensembles are two different attacks on that variance.

  • Bagging / random forests — train each tree on a bootstrap sample (random rows, drawn with replacement) and, at each split, only let it consider a random subset of features. The trees end up decorrelated, so averaging their votes slashes variance without raising bias. Status: a solved, practical method, introduced by Leo Breiman in 2001 (building on his 1996 bagging work).
  • Boosting — grow trees sequentially. Each new tree is fit to the residual errors of the ensemble so far, nudging the prediction toward the points it keeps getting wrong. This drives down bias. Gradient boosting (Jerome Friedman, 2001) framed it as gradient descent in function space; modern libraries like XGBoost, LightGBM and CatBoost are its industrial form.
  • The cost. Training is O(T⋅nlog⁡n⋅d)O(T \cdot n \log n \cdot d) for TT trees on nn rows and dd features — linear in the number of trees, so you simply pay for accuracy. Prediction walks each tree top to bottom: cheap and parallel.

Neither method is mysterious or open — there is no P vs NP lurking here. The deep result is statistical: averaging many decorrelated weak learners provably reduces error, and that single idea quietly beats far fancier models on most real tables.

Where It Matters

Whenever your data is a table — rows of records, columns of features — an ensemble of trees is usually the first thing to reach for and often the last thing you need:

  • Fraud and risk scoring: banks and payment networks rank transactions with boosted trees, which handle mixed numeric and categorical features and missing values gracefully.
  • Search and recommendation ranking: gradient-boosted trees (LambdaMART and friends) power learning-to-rank in major search engines.
  • Medicine and science: clinical risk models and genomics pipelines lean on random forests for their robustness and feature-importance scores.
  • Competitions and baselines: for years, XGBoost-style models won the majority of Kaggle contests on structured data — and remain the baseline that neural networks still struggle to beat on plain tables.

The lesson generalizes beyond trees: diversity plus aggregation beats individual brilliance. It is the same instinct behind asking a crowd to estimate a number, and it shows up across machine learning wherever combining many imperfect guesses outperforms trusting one.

Conclusion

Random forests and boosting carry a lesson worth more than any single algorithm: a single expert can be confidently wrong, but a diverse crowd, each member wrong in its own way, lands close to the truth. Bagging builds that crowd in parallel; boosting builds it one careful correction at a time. Both are solved, well-understood methods — no open mysteries, just decades of reliable wins.

In an era loud with billion-parameter networks, the quiet fact remains: hand most teams a spreadsheet and ask for a prediction, and a forest of unremarkable trees will still be hard to beat. Sometimes the strongest model is just a great many weak ones, agreeing.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/random-forests/Content licensed under CC BY-NC 4.0.