Random Forests & Boosting

A decision tree is the most human model in machine learning: a flowchart of yes/no questions — is income above 40k? is age under 30? — that ends in a prediction. It is fast, readable, and needs no scaling or tuning. It is also, on its own, terrible: grow it deep enough to fit the training data and it memorizes the noise, drawing a jagged boundary that collapses on anything new. This is overfitting, and a lone tree is its poster child.

The fix that took over applied machine learning is almost embarrassingly simple: don't trust one tree — grow a crowd of them and let them vote. If each tree is wrong in a different way, their mistakes cancel and what survives the average is the real signal.

Two recipes dominate. Random forests (Leo Breiman, 2001) grow hundreds of trees independently, each on a random slice of the data and features, then average. Boosting grows trees one at a time, each one trained to fix the errors the previous ones made. Both turn weak, shaky trees into the single most reliable predictor we have for tabular data — the spreadsheets and databases that run the real world.

Below are two classes of points (blue and orange) that you must separate. A single tree trained on a random subsample draws the boundary you see — notice how blocky and overconfident it is. Press Add tree to train another one on a fresh random subsample and average its vote into the ensemble.

<p class="hint">{{hint}}</p>
<canvas id="plot" width="320" height="320"></canvas>
<div class="status" id="status">{{status_one}}</div>
<div class="btns">
  <button id="add" type="button">{{btn_add}}</button>
  <button id="grow" type="button">{{btn_grow}}</button>
  <button id="reset" type="button" class="ghost">{{btn_reset}}</button>
</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; }
.hint { font-size: .9rem; color: #444; margin: 0 0 .7rem; line-height: 1.45; }
canvas { border: 1px solid #cdd9e3; border-radius: 8px; display: block; image-rendering: pixelated; width: 320px; height: 320px; }
.status { font-size: 1rem; font-weight: 600; margin: .6rem 0; min-height: 1.4em; color: #1d3557; }
.btns { display: flex; gap: .5rem; flex-wrap: wrap; }
button { font: 600 14px system-ui, sans-serif; padding: .45rem .9rem; border: 1px solid #1d3557;
         background: #1d3557; color: #fff; border-radius: 8px; cursor: pointer; }
button.ghost { background: #fff; color: #1d3557; }

// Code not found

Watch the boundary. With one tree it is jagged and brittle. As you add more, the averaged boundary smooths out and the shaded confidence becomes graded instead of all-or-nothing — that soft transition is hundreds of slightly different trees disagreeing politely at the edges. Press Grow 25 to jump ahead, and Reset to start from a single tree again. No single tree got better; the crowd did.

A single deep tree has low bias (it can fit anything) but high variance (shuffle the data slightly and the tree changes wildly). Ensembles are two different attacks on that variance.

• Bagging / random forests — train each tree on a bootstrap sample (random rows, drawn with replacement) and, at each split, only let it consider a random subset of features. The trees end up decorrelated, so averaging their votes slashes variance without raising bias. Status: a solved, practical method, introduced by Leo Breiman in 2001 (building on his 1996 bagging work).
• Boosting — grow trees sequentially. Each new tree is fit to the residual errors of the ensemble so far, nudging the prediction toward the points it keeps getting wrong. This drives down bias. Gradient boosting (Jerome Friedman, 2001) framed it as gradient descent in function space; modern libraries like XGBoost, LightGBM and CatBoost are its industrial form.
• The cost. Training is $O(T \cdot n \log n \cdot d)$ for $T$ trees on $n$ rows and $d$ features — linear in the number of trees, so you simply pay for accuracy. Prediction walks each tree top to bottom: cheap and parallel.

Neither method is mysterious or open — there is no P vs NP lurking here. The deep result is statistical: averaging many decorrelated weak learners provably reduces error, and that single idea quietly beats far fancier models on most real tables.

Whenever your data is a table — rows of records, columns of features — an ensemble of trees is usually the first thing to reach for and often the last thing you need:

• Fraud and risk scoring: banks and payment networks rank transactions with boosted trees, which handle mixed numeric and categorical features and missing values gracefully.
• Search and recommendation ranking: gradient-boosted trees (LambdaMART and friends) power learning-to-rank in major search engines.
• Medicine and science: clinical risk models and genomics pipelines lean on random forests for their robustness and feature-importance scores.
• Competitions and baselines: for years, XGBoost-style models won the majority of Kaggle contests on structured data — and remain the baseline that neural networks still struggle to beat on plain tables.

The lesson generalizes beyond trees: diversity plus aggregation beats individual brilliance. It is the same instinct behind asking a crowd to estimate a number, and it shows up across machine learning wherever combining many imperfect guesses outperforms trusting one.

Random forests and boosting carry a lesson worth more than any single algorithm: a single expert can be confidently wrong, but a diverse crowd, each member wrong in its own way, lands close to the truth. Bagging builds that crowd in parallel; boosting builds it one careful correction at a time. Both are solved, well-understood methods — no open mysteries, just decades of reliable wins.

In an era loud with billion-parameter networks, the quiet fact remains: hand most teams a spreadsheet and ask for a prediction, and a forest of unremarkable trees will still be hard to beat. Sometimes the strongest model is just a great many weak ones, agreeing.