Reading Model Results - The Statistics Notebook

The practical notes show how LASSO and tree calculations appear in fitted model output. The useful part to retain is the connection between each number and the prediction question. Code is omitted here.

A LASSO coefficient vector¶

In the supplied car example, horsepower is predicted using fuel consumption, weight, rear axle ratio, and quarter mile time. One selected fit reports an intercept near 486.014 and slopes near $−2.9165$ , 21.9896, 0, and $−19.6920$ respectively.

The printed dot for the axle ratio represents an exact zero. It does not mean a missing estimate. For this particular fit, that predictor contributes nothing once the other terms and the penalty are accounted for.

For a new row $(24,2.5,3.5,18.5)$ in the same predictor order, the fitted formula gives

486.014−2.9165(24)+21.9896(2.5)+0(3.5)−19.6920(18.5)\approx106.69.

(1)

These are the supplied example’s fitted coefficients, not a new analysis. The result depends on the sample, preprocessing, fold allocation, and selected penalty. The reported penalty value should not be transferred to another dataset or another objective scale.

A slope describes the change in this fitted conditional prediction when one predictor increases while the others are held fixed. It is not automatically a causal effect, and the shrinkage used to select the model changes its interpretation relative to an unpenalized estimate.

Training fit versus prediction¶

The example reports a training $R^2$ near 0.805, calculated as

R^2=1−\frac{\sum_i(y_i−\widehat y_i)^2}{\sum_i(y_i−\bar y)^2}.

(2)

This summarizes fit to those observations. It does not say that 80.5 percent of future variation will be explained. On a separate test set, an analogous score can be negative if predictions are worse than its reference mean predictor.

Use a validation curve to select the penalty and a separate assessment to report performance. A visually flat part of the curve suggests that several candidate complexities perform similarly at the resolution of the available data.

Leakage in the hockey example¶

The supplied hockey example predicts total stoppages. Counts for individual periods are components of that total. Feeding those counts into a model intended to predict the total before the game would reveal parts of the answer that are not yet available.

The correct decision depends on the prediction time. A value available only after the outcome is known cannot be used as an ordinary input for an earlier forecast. Identifiers also need scrutiny because they can encode grouping or time rather than a transferable predictive relationship.

A roughly symmetric marginal response histogram is not a requirement for squared error regression. Examine the conditional mean, residual behavior, prediction range, and whether the chosen loss matches the question. Counts may motivate a different response model even when a histogram looks smooth.

A tree complexity table¶

A complexity table compares candidate tree sizes. Its number of splits describes structural complexity; for a binary tree, the number of leaves is one greater. Relative training error divides the candidate’s training error by the root node error. It decreases as useful training splits are added.

The validation error column estimates prediction performance using omitted folds. Its standard error column describes uncertainty in that estimate. Choose a size using validation error or a stated simplicity rule, rather than treating the smallest training error as the best tree.

Software may normalize its complexity parameter by the root node error. Such a parameter is related to the theoretical $\alpha$ in $\operatorname{RSS}(T)+\alpha|T|$ , but the raw numerical values are not automatically interchangeable.

Forest diagnostics in the companion material¶

A random forest aggregates many trees fitted with randomized observations and candidate features. Bootstrap sampling means sampling with replacement, so some original observations are absent from each tree’s training sample.

For each observation, an out of bag prediction uses only trees that did not train on that observation. This provides a useful internal assessment. If it is used repeatedly to choose settings, the selected score is no longer an untouched final evaluation.

The supplied classification example reports 61 correct absent cases, 3 false alarms, 12 missed present cases, and 5 correct present cases. Overall error is $15/81\approx18.52\%$ , but the missed fraction among present cases is $12/17\approx70.59\%$ . A single overall error rate hides that difference.

These categorical diagnostics are included to interpret the companion example. For numerical regression, use prediction loss such as mean squared error rather than a confusion matrix. The existing forest notes develop the ensemble construction further.

Prediction time in the game example¶

Army size, workers, and resources describe aspects of the state of a game. Ratios express relative advantage; differences express absolute advantage. A ratio also needs a defined rule when its denominator is zero.

Features must come from the time at which the prediction would actually be made. Keep observations from the same game together when splitting data, otherwise snapshots of a single outcome can appear in both training and validation. Later snapshots may contain more signal, but they answer a later forecasting question.

A single tree does not have an out of bag error unless a resampling procedure has explicitly created such predictions. Label each reported score by how its predictions were obtained, rather than calling every error estimate out of bag.