PCA as a rod on sliding collars

Total least squares is the same rod as regression, with different spring attachments.

In the least squares demo each spring is pinned to the rod directly above its data point. It stretches vertically, and the rod relaxes to the regression line. Attach each spring instead through a frictionless collar that slides freely along the rod. Now the spring swings perpendicular. It stores energy $\tfrac12\sum_i d_i^2$ in the squared orthogonal distances. The rod settles with its centre at the centroid and its axis along the direction of maximal variance. That is the first principal component. It is Pearson's 1901 “line of closest fit.”

The rod's torque balance is pendulum-like with period $\pi$. There are two equilibria. The first principal component is the stable one. The second is a balance point. Try the button that stands the rod on PC2 and watch it teeter before swinging down to PC1. It falls slowly when the cloud is nearly round, because the restoring torque is weak and the eigenvalues are nearly tied. It falls violently when the cloud is elongated. The rod librates about PC1 as a normal mode, and its stiffness is the eigenvalue gap $\lambda_1-\lambda_2$. Drag points, reshape the cloud with the sliders, or spin the rod.

cloud tilt cloud flatness

rod angle ° PC1 angle ° energy eigenvalues /

Vertical springs give regression. Sliding collars give PCA. A spring with anisotropic stiffness would interpolate the two, the errors-in-variables case. The energy here is the negative log-likelihood of an isotropic Gaussian around the line. That is why the collar construction finds the top eigenvector of the covariance. It is also why a nearly round cloud barely fixes the axis. The bowl is nearly flat in $\theta$, and the rod wanders. Pearson set this down in 1901. Harold Hotelling carried the method further and named the principal components in 1933. See Pearson (1901) in the reading list.

The usual explanation the textbook derivation, in full

Centre the data and form the covariance matrix $\Sigma$. Project every point onto a unit direction $v$. The variance captured along that direction is a quadratic form in $v$:

$$ \operatorname{Var}(v) = v^{\mathsf T}\Sigma\,v, \qquad v^{\mathsf T} v = 1. $$

The first principal component is the direction that captures the most. So maximise $v^{\mathsf T}\Sigma\,v$ subject to the unit-norm constraint. Fold the constraint in with a Lagrange multiplier $\lambda$:

$$ \mathcal{L}(v,\lambda) = v^{\mathsf T}\Sigma\,v - \lambda\big(v^{\mathsf T} v - 1\big). $$

Set the gradient with respect to $v$ to zero. The constrained problem turns into an eigenproblem:

$$ 2\,\Sigma v - 2\lambda v = 0 \;\Longrightarrow\; \Sigma v = \lambda v. $$

So the stationary directions are the eigenvectors of the covariance, and each multiplier is an eigenvalue. Multiply on the left by $v^{\mathsf T}$ and use the constraint to see what the objective is worth at each one:

$$ v^{\mathsf T}\Sigma\,v = \lambda\,v^{\mathsf T} v = \lambda. $$

The captured variance equals the eigenvalue. To maximise it, take the largest. The first principal component is the top eigenvector of $\Sigma$, and the next axis is the eigenvector below it.

The physics proof the same result, read off the rod

The rod does not solve an eigenproblem. It rides on frictionless collars, so each spring can only pull perpendicular to it. Left alone the rod turns until those sideways pulls $d_i$, acting at their collar offsets $s_i$, no longer twist it:

$$ \sum_i d_i\,s_i = 0. $$

That torque balance is stationary on a principal axis, and only the top one is stable. The rod lies down along PC1 because that is where the cloud lets it rest. Nudge it and it librates as a normal mode whose stiffness is the eigenvalue gap $\lambda_1 - \lambda_2$. When the cloud is round that gap closes, the restoring torque fades, and the rod wanders. That is the eigenproblem going degenerate in plain sight.