Least squares as springs

Regression is a rigid rod that stopped arguing with the data.

Each data point is tied to a rigid rod by a spring of natural length zero and unit stiffness. The spring runs in a frictionless vertical guide, so it can only stretch up and down. The stored energy is then $\tfrac12\sum_i r_i^2$, half the residual sum of squares. Release the rod and it relaxes until the spring forces and torques balance. That balance is the normal equations. The resting position is the ordinary least squares line, drawn dashed and computed in closed form. Drag the points and the rod chases the new fit. Pluck it to see it oscillate back. The picture follows Joshua Loftus's “Least Squares as Springs”. Loftus found it hard to turn up early references. The oldest is T. W. Dwight, a forester, who fitted regression lines by least squares this way in The Forestry Chronicle in 1937. For drawing springs properly, see Hadley Wickham's spring geom case study in the ggplot2 book.

energy (½ RSS) rod slope OLS slope weighting a point = a stiffer spring = higher precision

The dictionary has the rest. Stiffness is precision. Weighted least squares is just unequal springs. A ridge penalty is one more spring tying the rod to the prior. Generalise from a rod to a network of beads and you are back at the fusion demo and the Kalman filter.

The usual explanation the textbook derivation, in full

To fit a line by least squares, write the residual of each point and add up the squares. With intercept $a$ and slope $b$ the objective is:

$$ S(a,b) \;=\; \sum_i \big(y_i - a - b\,x_i\big)^2. $$

This is a smooth convex function of the two parameters, so its minimum is the point where both partial derivatives vanish. Differentiate with respect to the intercept first:

$$ \frac{\partial S}{\partial a} = -2\sum_i \big(y_i - a - b\,x_i\big) = 0. $$

Then with respect to the slope:

$$ \frac{\partial S}{\partial b} = -2\sum_i x_i\big(y_i - a - b\,x_i\big) = 0. $$

Divide out the $-2$ and rearrange. The two conditions become a pair of linear equations in $a$ and $b$, the normal equations:

$$ \begin{aligned} n\,a + \Big(\textstyle\sum_i x_i\Big)\,b &= \sum_i y_i, \\[2pt] \Big(\textstyle\sum_i x_i\Big)\,a + \Big(\textstyle\sum_i x_i^2\Big)\,b &= \sum_i x_i\,y_i. \end{aligned} $$

Stack the data into a design matrix $X$ with a column of ones and a column of the $x_i$, gather the responses into a vector $y$, and collect the coefficients into $\beta=(a,b)^{\mathsf T}$. The same pair is one matrix equation:

$$ X^{\mathsf T} X\, \beta \;=\; X^{\mathsf T} y. $$

The Gram matrix $X^{\mathsf T} X$ is invertible whenever the $x_i$ are not all equal, so solve for the coefficients:

$$ \beta \;=\; \big(X^{\mathsf T} X\big)^{-1} X^{\mathsf T} y. $$

That is the ordinary least squares line, the dashed fit on the canvas. Two derivatives and a matrix inverse, and you have it.

The physics proof the same result, read off the rod

The rod carries nothing more than those conditions, so look at what holds it still. Each spring pulls with a force equal to its residual $r_i$. For the rod to stop, the forces must sum to zero and their torques about the centroid must sum to zero:

$$ \sum_i r_i = 0, \qquad \sum_i r_i\,(x_i - \bar x) = 0. $$

Those are the two normal equations, already in balance. The first fixes the height and the second fixes the slope. No derivatives were taken. The rod is the least-squares line at rest, and it found the fit by the only thing it knows how to do, which is stop pulling.