Ridge and Lasso Regression

Suppose you want to predict a house price from ten measurements — size, age, distance to a school, and seven more. Linear regression fits a weight to each measurement so the predicted prices match the training set as closely as possible. The trouble is, "as closely as possible" can mean the model chases every bump and wiggle in the data — overfitting — and then fails badly on new houses.

The root cause is that the algorithm is free to make any weight as large as it likes. A huge positive weight on one feature can cancel a huge negative weight on another, producing a perfectly fitted training curve that is pure noise.

The solution is surprisingly clean: add a penalty to the objective. Instead of minimising only the prediction error, also penalise large weights. The model still wants to fit the data, but now every large weight comes at a cost.

• Ridge (also called L2 regularisation), introduced in its modern form by Hoerl and Kennard in 1970, adds the sum of squared weights to the loss. All weights shrink toward zero, but none ever reach it exactly.
• Lasso (Least Absolute Shrinkage and Selection Operator), formalised by Tibshirani in 1996, adds the sum of absolute weights instead. The key difference: Lasso's geometry forces some weights all the way to exactly zero, effectively deleting those features from the model.

The result is a solved and practical technique — Ridge and Lasso are not open problems but mature, well-understood tools that sit at the core of modern machine learning. The interesting question is not if they work, but how the choice of penalty shape changes the solution.

The demo below trains Ridge and Lasso on a synthetic dataset with six features: two are genuinely useful, two are weakly correlated, and two are pure noise. Drag the $\lambda$ slider to change the regularisation strength and watch what happens to the six coefficients.

<div class="controls">
  <label for="lam">{{reg_strength}} <span id="lamVal">0.00</span></label>
  <input type="range" id="lam" min="0" max="200" value="0" step="1">
  <div class="legend">
    <span class="dot ridge"></span> {{legend_ridge}}
    <span class="dot lasso"></span> {{legend_lasso}}
  </div>
</div>
<div id="chart" class="chart"></div>
<div id="info" class="info">{{info_zero}}</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; flex-direction: column; gap: .4rem; margin-bottom: .8rem; }
label { font-size: .9rem; font-weight: 600; }
input[type=range] { width: 100%; accent-color: #1d3557; }
.legend { display: flex; gap: 1.2rem; font-size: .85rem; align-items: center; }
.dot { display: inline-block; width: 12px; height: 12px; border-radius: 50%; }
.dot.ridge { background: #457b9d; }
.dot.lasso { background: #e76f51; }
.chart { display: flex; flex-direction: column; gap: 6px; }
.row { display: flex; align-items: center; gap: 8px; }
.label { width: 72px; font-size: .8rem; text-align: right; flex-shrink: 0; color: #555; }
.bars { flex: 1; display: flex; gap: 4px; flex-direction: column; }
.bar-wrap { display: flex; align-items: center; height: 14px; }
.bar { height: 14px; border-radius: 3px; transition: width .15s; min-width: 2px; }
.bar.ridge { background: #457b9d; }
.bar.lasso { background: #e76f51; }
.val { width: 44px; font-size: .75rem; text-align: right; color: #444; flex-shrink: 0; }
.info { margin-top: .7rem; font-size: .85rem; color: #444; min-height: 1.4em; }
.zero { opacity: .3; }

// Code not found

Notice the difference in shape. Ridge (blue) shrinks all coefficients smoothly and proportionally — they approach zero but never touch it. Lasso (orange) behaves differently: as $\lambda$ grows, weak coefficients snap to exactly zero, one by one. At high $\lambda$, only the truly useful features survive. That automatic feature selection is Lasso's superpower.

Both methods start from ordinary least squares but add a regularisation term controlled by a strength parameter $\lambda \ge 0$:

Ridge minimises: $\|y - Xw\|^{2} + \lambda\|w\|^{2}$

The squared penalty is smooth and strictly convex, so it has a single closed-form solution:

$$\hat{w}_{\text{Ridge}} = (X^\top X + \lambda I)^{-1} X^\top y$$

Adding $\lambda I$ to the matrix makes it invertible even when features are correlated or when there are more features than data points — historically, Ridge was invented precisely to fix that instability.

Lasso minimises: $\|y - Xw\|^{2} + \lambda\|w\|_{1}$

The absolute-value penalty is not differentiable at zero, so no closed form exists. The standard solver uses coordinate descent: cycle through weights one at a time and apply a soft-threshold operation — if the unconstrained optimal weight is smaller in magnitude than $\lambda/2$, set it to exactly zero; otherwise shrink it by $\lambda/2$. That threshold is what produces exact zeros.

Why the geometry explains the sparsity. Picture the feasible region defined by the penalty constraint. For Ridge (L2) it is a sphere — the solution sits where the loss ellipsoid first touches the sphere, typically not at a corner. For Lasso (L1) the constraint region is a diamond, with sharp corners on the axes. The loss ellipsoid almost always first touches a corner, where one or more weights are exactly zero.

Bias–variance trade-off. At $\lambda = 0$ both methods recover ordinary least squares: low bias, potentially high variance. As $\lambda$ grows, variance falls (the model can no longer chase noise) while bias rises (the model is pulled away from the true signal). The sweet spot — chosen by cross-validation — minimises total prediction error on unseen data.

A natural extension, Elastic Net (Zou and Hastie, 2005), mixes both penalties: $\lambda_{1}\|w\|_{1} + \lambda_{2}\|w\|^{2}$. It keeps Lasso's sparsity while inheriting Ridge's stability when features are correlated. See also dimensionality reduction for related methods that compress the feature space geometrically.

Regularised regression appears wherever the number of features is large relative to the amount of data, or whenever some features are suspected to be irrelevant:

• Genomics: a genome-wide association study may have thousands of genetic variants (features) and only hundreds of patients. Lasso automatically selects the handful of variants most associated with a disease, producing an interpretable model. Ridge is used when many variants are expected to share a small effect.
• Finance: predicting stock returns from hundreds of technical indicators. Lasso prunes the list to a small set of predictors; Ridge handles the case where many indicators carry correlated information.
• Natural language processing: bag-of-words models for text classification can have millions of word features. L1 or L2 penalties prevent memorising the training corpus while keeping the model fast.
• Medical imaging: fitting a regression model to thousands of voxel intensities. Ridge stabilises the fit when intensities are highly correlated; Lasso finds sparse brain regions most predictive of an outcome.
• Engineering calibration: when sensor readings are multicollinear, Ridge's matrix-inversion fix makes the estimate numerically stable where ordinary least squares would blow up.

The connection to dimensionality reduction is direct: Lasso performs embedded feature selection during training, while methods like PCA reduce dimensions before fitting. Both fight the curse of dimensionality, just from different angles. The k-means and PAC learning articles explore adjacent questions about learning from high-dimensional data.

Plain linear regression is a fitting machine: given enough freedom, it will memorise your training data perfectly and fail on everything else. Ridge and Lasso each fix this with one extra term — a cost for large coefficients — but the choice of penalty shape has surprisingly different consequences.

Ridge shrinks everything smoothly and is easy to solve analytically. Lasso enforces sparsity through a non-smooth penalty and demands an iterative solver, but rewards you with a model that names its sources: only the features that genuinely matter survive.

For most practitioners, the recipe is: try Lasso when you suspect only a few features matter and want an interpretable model; try Ridge when you believe many features each contribute a little; try Elastic Net when you want both properties. In all three cases, cross-validate the regularisation strength $\lambda$ rather than guessing.

The deeper lesson is a general one: adding structure to an optimisation problem — even a constraint that makes the problem harder to solve — can make the solution far more useful in practice. That idea echoes through all of machine learning, from neural network training to PAC learning.