Prediction Error & Bias-Variance - The Statistics Notebook

Distinguish the error of one fitted model from the average performance of a learning procedure. Then separate prediction error into noise, bias, and variance.

Which Error Are We Measuring?¶

Let $\widehat f_T=A(T)$ and let $(X_0,Y_0)$ be an independent new pair.

Quantity	Definition	What is averaged?
Training error	$\operatorname{err}_{\mathrm{train}}(T)=n^{-1}\sum_i L(Y_i,\widehat f_T(X_i))$	Losses on observations used to fit.
Conditional generalization error	$\operatorname{Err}(T)=E[L(Y_0,\widehat f_T(X_0))\mid T]$	New inputs and outputs, keeping the fitted rule fixed.
Average generalization error	$\operatorname{Err}_n=E_T[\operatorname{Err}(T)]$	New observations and repeated training sets of size $n$ .

Model assessment estimates predictive performance. Model selection compares algorithms, model classes, or tuning choices. Reusing an assessment set for selection makes its reported error optimistic for the selected choice.

Train-Test Splitting¶

Split observations into disjoint training and test sets. Fit only on $T_{\mathrm{train}}$ and compute

\widehat{\operatorname{Err}}_{\mathrm{test}}= \frac{1}{n_{\mathrm{test}}}\sum_{i\in\mathrm{test}}L(Y_i,\widehat f_{T_{\mathrm{train}}}(X_i)).

(1)

For an independent, representative test set and a fitting procedure chosen without inspecting it,

E[\widehat{\operatorname{Err}}_{\mathrm{test}}\mid T_{\mathrm{train}}]=\operatorname{Err}(T_{\mathrm{train}}).

(2)

Increasing the test size improves precision for this fixed fitted model. The estimate does not directly assess a different model refitted on all the data.

Data role	Permitted use
Training	Estimate coefficients and all learned preprocessing steps.
Validation or inner CV	Choose the algorithm, features, and tuning parameters.
Final test or outer CV	Assess the completed selection procedure.

A split spends data on both fitting and evaluation. With limited data, cross-validation reuses observations across different fitting/evaluation roles while keeping each scored prediction held out.

Why Training Error Can Mislead¶

An interpolating rule can satisfy $\widehat f_T(X_i)=Y_i$ for every training observation, giving zero training error. Its behaviour between those points can still be poor. Zero residuals impose no general guarantee about loss at new inputs.

A training score is not always numerically below a realized test score. Optimism is an expectation statement under specified assumptions, not a pointwise inequality for every data split. The exact fixed-input version is developed on the next page.

Bias-Variance Decomposition¶

Bias: systematic error

Compare the average fitted prediction across training sets with the true regression function. Square that difference before adding it to prediction error.

Variance: sensitivity

Hold the input fixed and change the training set. Prediction variance measures how much the fitted prediction moves.

Write

Y_0=m(x_0)+\varepsilon_0,\qquad E[\varepsilon_0\mid X_0=x_0]=0,\qquad \operatorname{Var}(\varepsilon_0\mid X_0=x_0)=\sigma^2(x_0).

(3)

The new error is independent of the training set. Define

\overline f(x_0)=E_T[\widehat f_T(x_0)],\qquad \operatorname{Bias}(x_0)=\overline f(x_0)-m(x_0).

(4)

Then

\boxed{ E_{T,Y_0\mid x_0}[(Y_0-\widehat f_T(x_0))^2] =\sigma^2(x_0)+\operatorname{Bias}(x_0)^2+ \operatorname{Var}_T(\widehat f_T(x_0)).}

(5)

Component	Interpretation	Can fitting reduce it?
Irreducible noise	Different responses at the same input.	Not using these inputs alone.
Squared bias	Distance between the average fitted prediction and the truth.	A richer suitable model may help.
Variance	Sensitivity of the prediction to the training sample.	More data, averaging, or constraints may help.

Normality is unnecessary for this identity. Finite second moments and the independent new observation are the key requirements.

Complexity & Prediction¶

Increasing flexibility often reduces approximation bias but increases sensitivity to the training data. This motivates a balance; it does not prove that every algorithm has a U-shaped test-error curve or monotone bias and variance.

Training squared error cannot increase when least squares is minimized exactly over nested model classes. This fact alone says nothing about their ordering on new data.

Worked Example: A Biased Rule Can Predict Better

At an input $x_0$ , suppose the noise variance is 4.

Procedure	Bias	Prediction variance	Expected squared prediction error
A	0	9	$4+0+9=13$
B	2	1	$4+4+1=9$

Procedure B predicts better despite being biased: its variance reduction is larger than its increase in squared bias. Bias must be squared before adding it to the variance.

Connection to classical regression: under the Gauss-Markov assumptions, least squares is best among linear unbiased coefficient estimators. That restricted variance comparison does not establish that it has the smallest prediction error among biased or nonlinear alternatives.