Spectral Analysis & Collinearity - The Statistics Notebook

The singular value decomposition separates the design into orthogonal directions. It reveals which combinations of coefficients are well informed by the data and which amplify noise.

Singular Value Decomposition¶

For a rank- $r$ design, the compact SVD is

X=U_rD_rV_r^\top,\qquad D_r=\operatorname{diag}(s_1,\ldots,s_r),\qquad s_j>0,

(1)

where $U_r^\top U_r=V_r^\top V_r=I_r$ . Use $s_1\geq\cdots\geq s_r>0$ .

Object	Shape	Interpretation
$U_r$	$n\times r$	Orthonormal directions in the space of fitted response vectors.
$D_r$	$r\times r$	Strength of the design in each direction.
$V_r$	$p\times r$	Orthonormal combinations of coefficient coordinates.

X^\top X=V_rD_r^2V_r^\top.

(2)

The nonzero eigenvalues of $X^\top X$ are $s_j^2$ . For full column rank, $r=p$ and

\widehat\beta=VD^{-1}U^\top y =\sum_{j=1}^p v_j\frac{u_j^\top y}{s_j},\qquad \widehat y=UU^\top y=\sum_{j=1}^p u_j u_j^\top y.

(3)

With deficient rank, use only the $r$ positive singular values; $V_rD_r^{-1}U_r^\top y$ is the minimum-norm solution.

Small Singular Values Amplify Coefficient Noise¶

Under the linear mean model with covariance $\sigma^2I$ , and full column rank,

\operatorname{Cov}(\widehat\beta\mid X) =\sigma^2VD^{-2}V^\top,

(4)

so

\boxed{\operatorname{Var}(v_j^\top\widehat\beta\mid X)=\frac{\sigma^2}{s_j^2}.}

(5)

A small $s_j$ means the data change little when coefficients move in direction $v_j$ . Many different coefficient vectors then make nearly the same fitted predictions, so noise can move the estimated coefficients a long distance in that direction.

For full column rank, the spectral condition number is

\kappa_2(X)=s_1/s_p,\qquad \kappa_2(X^\top X)=\kappa_2(X)^2.

(6)

A large ratio signals sensitivity to perturbations. At exact rank deficiency, the full-column condition number is infinite; the problem is nonidentification, rather than a uniquely defined coefficient estimator with literal infinite variance.

Coefficients, Fitted Values & New Predictions¶

At observed inputs

Fitted-value covariance is $\sigma^2H$ . Small singular values do not by themselves make these variances diverge.

At a new input

Prediction can be unstable when the new input points along a weakly observed design direction. Extrapolation makes this distinction especially important.

Although coefficients contain $1/s_j$ , the fitted vector has covariance

\operatorname{Cov}(\widehat Y\mid X)=\sigma^2U_rU_r^\top.

(7)

Each training fitted value has variance $\sigma^2h_{ii}\leq\sigma^2$ . Its variance does not diverge merely because a nonzero singular value becomes small.

At a new input $x_0$ , however, full-rank OLS gives

\boxed{\operatorname{Var}(x_0^\top\widehat\beta\mid X) =\sigma^2\sum_{j=1}^p\frac{(x_0^\top v_j)^2}{s_j^2}.}

(8)

A new input with a substantial component in a weakly observed direction can have high prediction variance. This is especially relevant when extrapolating beyond the predictor relationships present in training data.

Exact Dependence & Estimability¶

If $v\in\operatorname{null}(X)$ , then $X(\beta+cv)=X\beta$ for any $c$ . The training data cannot distinguish these parameter vectors. A new mean $x_0^\top\beta$ is identifiable from the linear mean model only if $x_0$ is orthogonal to the null space, equivalently $x_0\in\operatorname{row}(X)$ .

A pseudoinverse chooses a coefficient convention; it does not create information about unidentifiable directions.

Spectral Motivation for Ridge¶

OLS keeps every observed response-space direction with weight one:

\widehat y_{\mathrm{OLS}}=\sum_{j=1}^r u_j u_j^\top y.

(9)

Ridge instead uses

\widehat y_\lambda=\sum_{j=1}^r \underbrace{\frac{s_j^2}{s_j^2+\lambda}}_{w_j(\lambda)}u_j u_j^\top y.

(10)

Directions with small $s_j^2$ relative to $\lambda$ are strongly suppressed. These are rotated design directions, not necessarily individual original predictors.

Worked Shrinkage Comparison

Suppose $s_1=10$ , $s_2=0.1$ , and $\lambda=1$ .

Direction	$s_j^2$	OLS coefficient variance divided by $\sigma^2$	Ridge fitted-direction weight
Strong	100	0.01	$100/101\approx0.9901$
Weak	0.01	100	$0.01/1.01\approx0.0099$

Ridge nearly preserves the strong direction and largely removes the weak one. It reduces uncertainty at the cost of bias if the true signal has a component in the suppressed direction.