Ridge Regression - The Statistics Notebook

Ridge adds a quadratic penalty to least squares. It produces a unique coefficient vector even with redundant columns when all coefficients are penalized and $\lambda>0$ . Its main benefit is a controlled tradeoff between bias and variance.

Penalized Least Squares¶

For the all-coefficients-penalized convention,

\widehat\beta_\lambda=\arg\min_\beta \{\lVert y-X\beta\rVert^2+\lambda\lVert\beta\rVert^2\},\qquad\lambda>0.

(1)

Differentiation gives

-2X^\top(y-X\widehat\beta_\lambda)+2\lambda\widehat\beta_\lambda=0,

(2)

hence

\boxed{\widehat\beta_\lambda=(X^\top X+\lambda I_p)^{-1}X^\top y.}

(3)

For any nonzero $a$ ,

a^\top(X^\top X+\lambda I)a=\lVert Xa\rVert^2+\lambda\lVert a\rVert^2>0.

(4)

Thus the penalized system is positive definite regardless of the rank of $X$ , including when $p>n$ . In computation, solve the system rather than form its inverse.

Constrained Form¶

The related constrained problem is

\min_\beta\ \lVert y-X\beta\rVert^2 \quad\text{subject to}\quad\lVert\beta\rVert^2\leq t.

(5)

Its Lagrangian is

\mathcal L(\beta,\lambda)=\lVert y-X\beta\rVert^2+lambda(\lVert\beta\rVert^2-t).

(6)

For $t>0$ , convexity and a strictly feasible point give the usual Lagrange-multiplier characterization. When the constraint is active, an appropriate multiplier yields a penalized solution. If the constraint is loose enough to contain an OLS minimizer, the multiplier may be zero. The relation between $t$ and $\lambda$ depends on the data; they are not the same numerical parameter.

Geometrically, squared-error contours are ellipsoids and the constraint is a ball. The constrained optimum occurs where the smallest attainable contour meets the ball, unless an unconstrained minimizer already lies inside.

Spectral Shrinkage¶

Using the compact SVD $X=U_rD_rV_r^\top$ ,

\widehat\beta_\lambda =V_r\operatorname{diag}\!\left(\frac{s_j}{s_j^2+\lambda}\right)U_r^\top y,

(7)

\widehat y_\lambda=H_\lambda y,\qquad H_\lambda=U_r\operatorname{diag}\!\left(\frac{s_j^2}{s_j^2+\lambda}\right)U_r^\top.

(8)

For positive singular values, the ridge coefficient component equals its minimum-norm OLS counterpart times

w_j(\lambda)=\frac{s_j^2}{s_j^2+\lambda}\in(0,1).

(9)

Limit	Coefficients	Fitted values
$\lambda\downarrow0$	Minimum-norm least-squares coefficients.	OLS projection $U_rU_r^\top y$ .
$\lambda\to\infty$	Zero, if all coefficients are penalized.	Zero under the same convention.

$H_\lambda$ is symmetric but generally not idempotent. Ridge shrinks a vector already in the fitted subspace, so it is a smoother rather than an orthogonal projection. Its residuals satisfy $X^\top e_\lambda=\lambda\widehat\beta_\lambda$ , not the OLS orthogonality condition.

Bias, Variance & Sampling Distribution¶

Assume $Y=X\beta+\varepsilon$ with $E[\varepsilon\mid X]=0$ and covariance $\sigma^2I$ . Let $B_\lambda=(X^\top X+\lambda I)^{-1}$ . Then

E[\widehat\beta_\lambda\mid X]=B_\lambda X^\top X\beta,

(10)

\boxed{\operatorname{Bias}(\widehat\beta_\lambda\mid X)=-\lambda B_\lambda\beta,}

(11)

\boxed{\operatorname{Cov}(\widehat\beta_\lambda\mid X) =\sigma^2B_\lambda X^\top XB_\lambda.}

(12)

With Gaussian errors, the estimator is multivariate normal with this mean and covariance, possibly degenerate in null-space directions. These formulas assume fixed $\lambda$ ; selecting it using $Y$ makes the overall procedure adaptive.

Mean Squared Error in a Singular Direction¶

For full column rank, let $\theta_j=v_j^\top\beta$ . Then

\operatorname{Bias}(v_j^\top\widehat\beta_\lambda) =-\frac{\lambda\theta_j}{s_j^2+\lambda},\qquad \operatorname{Var}(v_j^\top\widehat\beta_\lambda) =\frac{\sigma^2s_j^2}{(s_j^2+\lambda)^2},

(15)

\boxed{\operatorname{MSE}(v_j^\top\widehat\beta_\lambda) =\frac{\lambda^2\theta_j^2+\sigma^2s_j^2}{(s_j^2+\lambda)^2}.}

(16)

The variance is no larger than OLS’s $\sigma^2/s_j^2$ , but the bias is generally nonzero. The best tradeoff depends on unknown signal strength, so prediction-based tuning is needed.

For example, with $s_j=0.1$ , $\sigma^2=1$ , $\theta_j=1$ , and $\lambda=1$ , OLS MSE is 100. Ridge MSE is $1/1.01\approx0.9901$ . This is an illustrative weak-signal direction, not a guarantee of improvement for every true coefficient or penalty.

Intercepts & Scaling¶

Penalize all coefficients

Leave the intercept free

Use $X^\top X+\lambda I$ . As $\lambda\to\infty$ , all coefficients and fitted values tend to zero.

In practice, fit

\min_{\beta_0,b}\ \lVert y-\beta_0\mathbf1-Zb\rVert^2+\lambda\lVert b\rVert^2,

(17)

leaving the intercept unpenalized. Let $Z_c=Z-\mathbf1\bar z^\top$ and $y_c=y-\bar y\mathbf1$ . Then

\widehat b_\lambda=(Z_c^\top Z_c+\lambda I)^{-1}Z_c^\top y_c,\qquad \widehat\beta_0=\bar y-\bar z^\top\widehat b_\lambda.

(18)

As $\lambda\to\infty$ , slopes approach zero and predictions approach the training mean $\bar y$ , not zero.

Ridge depends on predictor units. Rescaling a column changes the coefficient norm required to represent the same prediction. Standardize predictors when equal treatment across scales is intended, and estimate centering/scaling using only the training portion of each validation split. Remove or otherwise handle zero-variance columns before standardization.

If the objective is $n^{-1}\operatorname{RSS}+\alpha\lVert b\rVert^2$ , multiplying by $n$ shows that the sum-scale penalty here is $\lambda=n\alpha$ .

Redundant Predictors: Stable Does Not Mean Invariant¶

Ridge remains solvable if a predictor is duplicated, but duplicating it generally changes the fit at fixed $\lambda$ .