Ridge vs Lasso

Where the Hookean analogy breaks, and why the break gives you exact zeros.

Take a single noisy estimate $y$ of mass $\phi$ and shrink it toward zero. Which estimator you get depends on the shape of the penalty. There are two textbook choices. Both are a device pulling the bead toward the origin. They differ in what kind of device.

Ridge is an ordinary Hookean spring of stiffness $\lambda$ anchored at $0$. Its force is $-\lambda z$ and its energy is $\tfrac12\lambda z^2$. The pull weakens as the bead nears zero. So the equilibrium $z^\ast_{\text{ridge}} = \phi y/(\phi+\lambda)$ approaches zero only in the limit $\lambda \to \infty$. That is shrinkage, never sparsity. Hoerl and Kennard gave the method its name in 1970, in a paper on steadying least squares when the predictors are collinear.

Lasso replaces the spring with a constant-tension device. Mechanically it is a cable over a pulley with a hanging weight $\lambda$. It always pulls toward $0$ with the same force regardless of distance. Its potential is the cone $\lambda|z|$, non-Hookean at the kink the introduction's caveat warns about. Once the data-spring's pull weakens below $\lambda$ the constant-tension device wins outright. The bead sticks at zero. The estimate is $z^\ast_{\text{lasso}} = \mathrm{sign}(y)\max(|y| - \lambda/\phi,\, 0)$, which is soft-thresholding. Tibshirani named the lasso in 1996 and showed that the $L_1$ penalty is what drives coefficients to zero.

data $y$ data mass $\phi$ penalty $\lambda$

ridge estimate lasso estimate dead zone

Raise $\lambda$ past $\phi|y|$ and watch. The ridge bead keeps sliding. It slows down but always moves. The lasso bead stops dead at $0$ and stays there for every larger $\lambda$. That is the shaded dead zone on its rail. This one demo does not run on raw $F=ma$. The kink at $z=0$ makes naive force integration chatter. So the lasso bead is animated by proximal gradient flow rather than the site's usual damped spring integrator. That is an exact implicit step through the kink each frame. The equilibrium it lands on is the one above. Only the telling of the transient differs.

The dictionary calls Gaussian priors Hookean for a reason. A Laplace prior on $z$ is what gives the penalty $\lambda|z|$. Its density has a corner at zero that a Gaussian's never does. Every other demo on this site works because the log-density is a parabola everywhere. This is the one demo about what happens when it is not.

The usual explanation the full derivation, ridge then lasso

Both estimators minimise a data term plus a penalty. For a single coordinate, with data mass $\phi$ and observation $y$:

$$ f(z) = \tfrac12\,\phi\,(z-y)^2 + \operatorname{pen}(z). $$

Ridge takes $\operatorname{pen}(z) = \tfrac12\lambda z^2$. The objective is smooth, so set the derivative to zero:

$$ f'(z) = \phi\,(z-y) + \lambda z = 0. $$

Collect the terms in $z$ and solve. This is the ridge estimate quoted above:

$$ (\phi+\lambda)\,z = \phi y \;\Longrightarrow\; z^\ast = \frac{\phi y}{\phi+\lambda}. $$

Lasso takes $\operatorname{pen}(z) = \lambda|z|$, which has no derivative at zero. Away from zero the penalty derivative is $+\lambda$ for $z>0$ and $-\lambda$ for $z<0$. Handle the two sides first. For $z>0$:

$$ \phi\,(z-y) + \lambda = 0 \;\Longrightarrow\; z = y - \lambda/\phi, \quad \text{valid when } y > \lambda/\phi. $$

For $z<0$ the penalty derivative flips sign:

$$ \phi\,(z-y) - \lambda = 0 \;\Longrightarrow\; z = y + \lambda/\phi, \quad \text{valid when } y < -\lambda/\phi. $$

At the kink $z=0$ the derivative is a subgradient, the whole interval $[-\lambda,\lambda]$. The stationarity condition $0 \in \phi(0-y) + [-\lambda,\lambda]$ holds exactly when the data pull is small:

$$ |\phi y| \le \lambda \;\Longleftrightarrow\; |y| \le \lambda/\phi. $$

The three cases fold into one expression. This is soft-thresholding, the lasso estimate quoted above:

$$ z^\ast = \operatorname{sign}(y)\,\max\!\big(|y| - \lambda/\phi,\, 0\big). $$

The physics proof read off the spring and the pulley

Ridge is the fusion picture with one spring added. It carries the data spring of stiffness $\phi$ anchored at $y$, plus a penalty spring of stiffness $\lambda$ anchored at $0$. The bead stops at their centre of mass.

$$ z^\ast_{\text{ridge}} = \frac{\phi\,y + \lambda\cdot 0}{\phi + \lambda} = \frac{\phi y}{\phi+\lambda}. $$

A spring goes slack as the bead nears its anchor, so that centre of mass slides toward zero and never lands on it. The lasso swaps the penalty spring for the pulley on the lower rail, a constant force $\lambda$ toward zero at any distance. The bead cannot leave zero until the data spring out-pulls the cable, and at the origin that pull is $\phi|y|$. Below the threshold the cable wins and the bead stays at zero. Exact zeros are what a constant force buys you that a spring never can.