Huber loss

A spring that gives up trying, past a threshold.

Five observations, one shared parameter $z$, and three rails. Each observation pulls on $z$ with the same spring shape on a given rail. The three rails differ only in that shape:

$$ F_{\text{L2}}(y,z) = \phi(y-z), \qquad F_{\text{L1}}(y,z) = \phi\,\mathrm{sign}(y-z), \qquad F_{\text{Huber}}(y,z) = \begin{cases} \phi(y-z) & |y-z|\le\delta \\ \phi\,\delta\,\mathrm{sign}(y-z) & |y-z|>\delta \end{cases} $$

L2 is the top rail. It is an ordinary Hookean spring per point. The equilibrium is the plain mean, and one wild point drags it arbitrarily far. L1 is the bottom rail: five constant-tension pulleys, like the Lasso demo's device. The equilibrium is the median, provably immovable by any single point however extreme. Huber is the middle rail, the practical compromise. It is an ordinary spring for small residuals. Past a threshold $\delta$ it switches to the same constant pull as L1. The spring notices it is dealing with an outlier and stops trying so hard.

Drag the rightmost point far away and watch. The L2 hub follows it linearly, without limit. The Huber hub moves, then flattens out once the residual passes $\delta$. The L1 hub barely notices. Slide $\delta$ toward $0$ and the Huber rail approaches L1. Slide it past the data's spread and it becomes L2. One threshold reaches the same two extremes as the exponent dial, which gets there by reshaping the whole curve instead of capping it.

threshold $\delta$ mass $\phi$

L2 (mean) Huber L1 (median)

Peter Huber introduced this loss in 1964, in a paper on robust estimation of a location parameter. He wanted an estimator with a bounded influence function that did not throw away the efficiency of the mean when the data really are clean. His estimators sit between the sample mean and the sample median. That is the compromise this page makes mechanical.

The usual explanation the textbook derivation, in full

Estimating one location $z$ from the points $y_i$ is an M-estimation problem. The estimate minimises a total loss $\sum_i \rho(y_i - z)$. For the Huber loss the summand is quadratic near zero and linear past the threshold $\delta$:

$$ \rho_\delta(r) = \begin{cases} \tfrac12\,r^2 & |r| \le \delta \\[2pt] \delta\big(|r| - \tfrac12\delta\big) & |r| > \delta. \end{cases} $$

Differentiate and set the derivative to zero. The mass $\phi$ scales every term alike and drops out, so the minimiser solves the estimating equation

$$ \sum_i \psi_\delta(y_i - z) = 0, \qquad \psi_\delta(r) = \begin{cases} r & |r| \le \delta \\[2pt] \delta\,\mathrm{sign}(r) & |r| > \delta, \end{cases} $$

where $\psi_\delta$ is the influence function. The two ends are the familiar estimators. Take $\delta$ large and $\psi(r) = r$, so $\sum_i (y_i - z) = 0$ and $z$ is the mean. Take $\delta$ small and $\psi(r) = \mathrm{sign}(r)$, so $\sum_i \mathrm{sign}(y_i - z) = 0$ and $z$ is the median:

$$ z_{\text{L2}} = \frac1n \sum_i y_i, \qquad z_{\text{L1}} = \mathrm{median}(y_i). $$

Huber sits between them, and in general it has no closed form. Which residuals exceed $\delta$ decides the equation, and that set depends on the very $z$ being solved for. So solve it by iteration, Newton or reweighted least squares, until the split of points into inside and outside $\delta$ stops changing.

The physics proof the same estimator, read off three rails

Each hub is a force balance. It rests where the pulls from the five points cancel. The fusion page proved a plain spring balances at the centre of mass, so on the L2 rail, where every point is a plain spring, the hub sits at the mean:

$$ \sum_i F(y_i, z) = 0. $$

On the L1 rail the pulls are constant-tension pulleys of equal force. They cancel only when as many pull up as pull down, and that point is the median. The Huber rail caps each pull past $\delta$, so it settles between the two. Read the answer off whichever rail you are watching.