Spline Tuning and Uncertainty - The Statistics Notebook

The smoothing parameter determines how much curvature the estimate can use. Selecting it and describing the uncertainty of the resulting curve are separate tasks. A visually smooth fit can still be biased, and a narrow interval need not account for that bias.

The smoother matrix¶

For a fixed basis and penalty parameter, write

A_\lambda=B(B^\top B+\lambda S)^{−1}B^\top,\qquad\widehat y=A_\lambda y.

(1)

The effective degrees of freedom are $\operatorname{df}(\lambda)=\operatorname{tr}(A_\lambda)$ . This measures how responsive the fitted values are to the observed responses. It need not equal the number of basis coefficients.

At small penalties, df approaches the rank of the unpenalized design. At large penalties, it approaches the dimension of the identifiable penalty nullspace. For the usual curvature penalty with an identifiable intercept and slope, that limit is two.

Validation criteria¶

Using the same fixed basis and the same numerical penalty in each omitted fit, the linear smoother identity gives

\operatorname{CV}(\lambda)=\frac1n\sum_i\left\{\frac{y_i−\widehat y_i}{1−a_{ii}}\right\}^2.

(2)

The denominators must be nonzero and the omitted fitting problems must remain well defined. If preprocessing or basis construction is learned again in each fold, evaluate that complete procedure directly rather than assuming this shortcut still applies.

Generalized cross validation replaces individual leverages by their average:

\operatorname{GCV}(\lambda)=\frac{\lVert y−A_\lambda y\rVert^2/n}{\{1−\operatorname{tr}(A_\lambda)/n\}^2}.

(3)

Search over a range of penalties. If the best score lies at the edge, examine whether the range was adequate. A low score obtained after searching is a selection result; independent assessment still requires a reserved test sample or an outer validation loop.

Sampling variation of the curve¶

Assume the true mean is $B\beta$ and set $M=B^\top B+\lambda S$ . Then

E(\widehat\beta\mid B)=M^{−1}B^\top B\beta,

(4)

\operatorname{Var}(\widehat\beta\mid B)=\sigma^2M^{−1}B^\top BM^{−1}.

(5)

The covariance is not simply $\sigma^2M^{−1}$ . The middle $B^\top B$ appears because the random part enters through $B^\top y$ .

At a new input $x_*$ ,

\widehat f(x_*)=b(x_*)^\top\widehat\beta,

(6)

\operatorname{Var}\{\widehat f(x_*)\mid B\}=\sigma^2b(x_*)^\top M^{−1}B^\top BM^{−1}b(x_*).

(7)

For normal errors and fixed $\lambda$ , this is the exact sampling variance around the expectation of the smoothed estimate. The expectation can differ from the true curve because of shrinkage or an insufficient basis.

What an interval includes¶

A displayed band of the form $\widehat f(x)\pm1.96\operatorname{se}\{\widehat f(x)\}$ is a pointwise approximation for the mean curve when smoothing bias is negligible. It is not automatically a simultaneous 95 percent band across every $x$ .

Predicting a new response adds its noise variance. With independent future errors, the variance around the expected prediction error includes the estimation variance plus $\sigma^2$ . Neither calculation automatically includes uncertainty from selecting $\lambda$ on the same sample.

Estimating the noise variance¶

The lecture uses residual error with an effective degrees of freedom adjustment. For a fixed smoother and true mean $\mu$ , the exact expectation is

E\lVert(I−A)y\rVert^2=\lVert(I−A)\mu\rVert^2+\sigma^2\{n−2\operatorname{tr}(A)+\operatorname{tr}(A^\top A)\}.

(8)

For ordinary least squares with the correct mean space, the bias term vanishes and $A^2=A$ , recovering the familiar residual degrees of freedom. For a penalized spline neither simplification generally holds. Dividing RSS by $n−\operatorname{tr}(A)$ is therefore a working adjustment, not a general proof of unbiasedness.