Least Squares & Projection - The Statistics Notebook

Ordinary least squares chooses the linear combination of design columns closest to the observed response vector. Its geometry explains the normal equations, and its sampling properties explain both its strengths and the limits of its optimality.

The Linear Mean Model¶

Condition on a fixed design $X\in\mathbb R^{n\times p}$ and suppose

Y=X\beta+\varepsilon,\qquad E[\varepsilon\mid X]=0,\qquad \operatorname{Cov}(\varepsilon\mid X)=\sigma^2I_n.

(1)

The rows are $x_i^\top$ , so $E[Y_i\mid X]=x_i^\top\beta$ . An intercept is represented by a column of ones. Assume full column rank for the inverse formulas below.

These mean and covariance assumptions do not require Gaussian errors. Normality will be added only when stating an exact normal sampling distribution.

Deriving the Normal Equations¶

Minimize

S(\beta)=\lVert y-X\beta\rVert^2 =y^\top y-2\beta^\top X^\top y+\beta^\top X^\top X\beta.

(2)

Differentiate:

\nabla S(\beta)=-2X^\top(y-X\beta),\qquad \nabla^2S(\beta)=2X^\top X.

(3)

The stationary condition is

X^\top X\widehat\beta=X^\top y.

(4)

If $\operatorname{rank}(X)=p$ , the Hessian is positive definite and the unique minimizer is

\boxed{\widehat\beta=(X^\top X)^{-1}X^\top y.}

(5)

Computation: use QR, SVD, or a least-squares solver rather than explicitly calculating the inverse. Forming $X^\top X$ squares the condition number and can amplify numerical difficulties.

Likelihood & Moment Interpretations

Under $Y\mid X\sim N(X\beta,\sigma^2I)$ ,

\ell(\beta,\sigma^2)=-\frac n2\log(2\pi\sigma^2) -\frac{\lVert y-X\beta\rVert^2}{2\sigma^2}.

(6)

For fixed positive variance, maximizing over $\beta$ minimizes squared error. When variance is also estimated and RSS is positive, the Gaussian MLE is $\widehat\sigma^2_{\mathrm{MLE}}=\operatorname{RSS}/n$ .

The mean model also implies the moment condition

E[X^\top(Y-X\beta)\mid X]=0.

(7)

Setting its observed counterpart to zero gives the same normal equations. This moment argument only needs the mean restriction; Gaussianity is unnecessary.

Geometry: Orthogonal Projection¶

The fitted vector and residual are

\widehat y=Hy,\qquad H=X(X^\top X)^{-1}X^\top,\qquad e=(I-H)y.

(8)

Property	Meaning
$\widehat y\in\operatorname{col}(X)$	The fit is a linear combination of the design columns.
$X^\top e=0$	The residual is orthogonal to every design column.
$H^\top=H$	The hat matrix is symmetric.
$H^2=H$	Projecting an already fitted vector does nothing.
$He=0$	The projection removes the residual component.

If an intercept is included, $\mathbf1^\top e=0$ , so residuals sum to zero and the fitted responses have mean $\bar y$ . These properties need not hold for a fit without an intercept or for a penalized intercept.

Rank-Deficient Designs¶

Least-squares solutions always exist. If $X$ is rank deficient, there are infinitely many coefficient solutions differing by vectors in the null space of $X$ . They all produce the same fitted vector:

\widehat y=XX^+y,

(11)

where $X^+$ is the Moore-Penrose pseudoinverse. The minimum-norm coefficient solution is $\widehat\beta=X^+y$ . The fitted subspace has dimension $r=\operatorname{rank}(X)$ , so $\operatorname{tr}(XX^+)=r$ .

Sampling Distribution¶

Write $A=(X^\top X)^{-1}X^\top$ . Since $\widehat\beta=AY$ and $AX=I$ ,

E[\widehat\beta\mid X]=\beta,\qquad \operatorname{Cov}(\widehat\beta\mid X)=A(\sigma^2I)A^\top =\sigma^2(X^\top X)^{-1}.

(12)

With Gaussian errors,

\boxed{\widehat\beta\mid X\sim N\!\left(\beta,\sigma^2(X^\top X)^{-1}\right).}

(13)

For any fixed vector $a$ ,

E[a^\top\widehat\beta\mid X]=a^\top\beta,\qquad \operatorname{Var}(a^\top\widehat\beta\mid X)=\sigma^2a^\top(X^\top X)^{-1}a.

(14)

At a new design row $x_0^\top$ , this gives the variance of the estimated mean $x_0^\top\widehat\beta$ . Predicting an independent future response adds its noise variance $\sigma^2$ under the same model.

For the fitted vector,

\operatorname{Cov}(\widehat Y\mid X)=\sigma^2H,

(15)

and, when $n>p$ , $\widehat\sigma^2=\operatorname{RSS}/(n-p)$ is unbiased under the stated mean and covariance assumptions.

Gauss-Markov: Best Linear Unbiased Estimation¶

What the theorem guarantees

Smallest covariance among estimators that are linear in the response and unbiased for every coefficient vector, under the stated assumptions.

What it leaves open

A biased estimator can have lower mean squared error. Ridge is outside the class compared by Gauss-Markov.

Among estimators $\widetilde\beta=CY$ that are linear in $Y$ and unbiased for every $\beta$ , OLS has the smallest covariance matrix in the positive-semidefinite ordering:

\operatorname{Cov}(\widetilde\beta\mid X)-\operatorname{Cov}(\widehat\beta\mid X)\succeq0.

(16)

This means every fixed linear combination $a^\top\widehat\beta$ has variance no larger than its counterpart $a^\top\widetilde\beta$ .

Scope: “best” means smallest variance among linear unbiased estimators under the given covariance assumptions. It does not compare OLS with biased ridge estimators by mean squared error. It also does not establish the same result under heteroscedastic or correlated errors.

Worked Example¶

Fit an intercept and slope to $x=(-1,0,1)$ and $y=(1,0,1)$ . Then

X^\top X=\begin{pmatrix}3&0\\0&2\end{pmatrix},\qquad X^\top y=\begin{pmatrix}2\\0\end{pmatrix},\qquad \widehat\beta=\begin{pmatrix}2/3\\0\end{pmatrix}.

(19)

The fitted vector is $(2/3,2/3,2/3)^\top$ and $e=(1/3,-2/3,1/3)^\top$ . Both residual sums $\sum_i e_i$ and $\sum_i x_i e_i$ equal zero. RSS is $2/3$ , so the residual variance estimate is $2/3$ with one residual degree of freedom.