Ridge Tuning & GCV - The Statistics Notebook

A ridge fit is easy to calculate for a fixed penalty. The practical question is how much shrinkage improves prediction. Cross-validation and generalized cross-validation compare candidate penalties while accounting for the model’s flexibility.

Effective Degrees of Freedom¶

For a fixed response-independent design and penalty,

\widehat y_\lambda=H_\lambda y,\qquad H_\lambda=X(X^\top X+\lambda I)^{-1}X^\top.

(1)

Under common error variance, the covariance definition of model complexity gives

\boxed{\operatorname{df}(\lambda)=\operatorname{tr}(H_\lambda) =\sum_{j=1}^r\frac{s_j^2}{s_j^2+\lambda}.}

(2)

Each observed singular direction contributes between zero and one effective parameter. With all coefficients penalized, df decreases from $r$ as $\lambda\downarrow0$ to zero as $\lambda\to\infty$ .

With an unpenalized intercept and centered predictor matrix $Z_c$ , the full smoother is

H_\lambda=J+U_r\operatorname{diag}(w_j)U_r^\top,\qquad J=\mathbf1\mathbf1^\top/n,

(3)

so

\operatorname{df}(\lambda)=1+\sum_{j=1}^r w_j.

(4)

The extra one counts the fitted intercept. Here df tends to one as slopes are completely shrunk.

Exact LOO for a Fixed Ridge Procedure¶

For sum-scale ridge with the same numerical $\lambda$ in every deleted fit,

\widehat{\operatorname{Err}}_{\mathrm{LOO}}(\lambda) =\frac1n\sum_i\left(\frac{y_i-\widehat y_{\lambda,i}}{1-h_{\lambda,ii}}\right)^2.

(6)

The identity also holds with a fixed penalty matrix that leaves the intercept unpenalized, provided the relevant deleted systems are invertible.

If software uses mean loss and keeps $\alpha$ fixed, the sum-scale penalty changes from $n\alpha$ to $(n-1)\alpha$ on deletion. The displayed full-fit shortcut assumes a fixed sum-scale penalty. Re-estimating predictor scales, selecting features, or retuning $\lambda$ inside each deletion also changes the fitting procedure; use explicit folds to evaluate such a pipeline.

Generalized Cross-Validation¶

Replace individual leverages by their average:

\boxed{\operatorname{GCV}(\lambda) =\frac{\lVert y-\widehat y_\lambda\rVert^2/n} {[1-\operatorname{df}(\lambda)/n]^2}.}

(8)

Choose a candidate penalty with the smallest score and refit with that penalty. GCV approximates LOO by using a common leverage correction; it is not an additional exact deleted-data formula.

As $\lambda$ increases	Typical consequence
Coefficient norm decreases	Less sensitivity to response noise.
Training RSS increases or stays constant	The fit moves away from its least-squares optimum.
Effective df decreases	The GCV complexity correction becomes smaller.
Bias can increase	Excessive shrinkage can lose predictive signal.

The GCV score itself need not be monotone. Search a sufficiently broad grid, often spaced logarithmically. If the minimum lies at an edge, examine whether extending the range is appropriate. Include the OLS limit only when the denominator is nonzero and the numerical calculation is stable.

The minimum GCV score has been used for selection. It should not be treated as an unbiased final assessment of the chosen model. Use a held-out test set or outer CV when a separate assessment is needed.

Efficient Evaluation Across Penalties¶

For the all-penalized convention, precompute a compact SVD and $c=U_r^\top y$ . Write $y_\perp=(I-U_rU_r^\top)y$ . Then

\widehat y_\lambda=U_r\operatorname{diag}(w_j)c,

(9)

\boxed{\operatorname{RSS}(\lambda) =\lVert y_\perp\rVert^2+\sum_{j=1}^r(1-w_j)^2c_j^2.}

(10)

Orthogonality makes the two residual components add without a cross term. After the SVD, both RSS and df require only sums over $r$ directions for each candidate penalty. For an unpenalized intercept, apply the same decomposition to $y_c$ and $Z_c$ , then add one to df.

Residual Variance Is a Different Quantity¶

For a fixed smoother and true mean vector $\mu$ ,

E[\lVert(I-H)Y\rVert^2\mid X] =\lVert(I-H)\mu\rVert^2 +\sigma^2\{n-2\operatorname{tr}(H)+\operatorname{tr}(H^\top H)\}.

(11)

For correctly specified OLS, the bias term is zero and $H^2=H$ , giving $\sigma^2(n-p)$ . For ridge, the bias term generally remains and $H^2\ne H$ . Therefore $\operatorname{RSS}/[n-\operatorname{tr}(H)]$ is not generally an unbiased variance estimator. GCV does not require a separate estimate of $\sigma^2$ .

Worked Spectral Comparison¶

Consider $n=5$ , two orthogonal design directions with $s^2=(9,1)$ , projected response coordinates $c=(3,1)$ , and $\lVert y_\perp\rVert^2=2$ . All coefficients are penalized in this example.

$\lambda$	Shrinkage weights	df	RSS	GCV
0	$(1,1)$	2.0	2.00	1.1111
1	$(0.9,0.5)$	1.4	2.34	0.9028
9	$(0.5,0.1)$	0.6	5.06	1.3068

Among these three candidates, $\lambda=1$ has the lowest score. It allows a modest RSS increase while substantially reducing effective complexity. This discrete comparison does not claim that 1 minimizes GCV over every positive penalty.