Basic

Metrics¶

Expected MSE¶

For a continuous outcome, prediction error comes from systematic error, sensitivity to the training sample, and noise in the new outcome.

We want both low bias and low variance. Irreducible noise sets a lower bound on prediction error.

Accuracy¶

• Used for Categorical problems.

• Refers to the percentage of correctly identified observations.

Sensitivity and Specificity¶

We want high sensitivity and high specificity, but changing the classification threshold usually trades one against the other.

Multi-Class Classification¶

• Precision/recall must be evaluated for each class when there are multiple classes.

• Macroaveraging:

  $$\frac{1}{K}\sum^K_{k=1}\text{Precision}_k$$

  (6)

  • Each classes get equal weights during evaluation. By ignoring class size, gives rare categories the same weight as frequent categories.

  • E.g., Computing $F$-values for each 2,2 classification table and averaging the $F$-values afterwards.

• Microaveraging:

  • Aggregate all predictions first, then compute the metric.

  • Sum all TP, FP, FN across classes.

  • Gives more weight to large classes.

  • Useful when we care about the overall accuracy, and when the data is imbalanced.

  • Compute the metric after pooling TP, FP, and FN; averaging class-specific scores instead would be macroaveraging.

  • E.g., Pooling all the 2,2 tables into a single 2,2 table and computing $F$-values based on the pooled tbable.

ROC¶

A receiver operating characteristic (ROC) curve shows how the true-positive and false-positive rates change as the classification threshold moves.

Points near the upper-left corner combine high sensitivity with a low false-positive rate.

Constructing the curve¶

F-Measure¶

F₁ combines precision and recall. It does not reward correct negative predictions, so a large number of true negatives cannot inflate the score. Values range from 0 to 1, with higher values indicating a better precision–recall balance.

With balanced classes and symmetric error costs, accuracy may also be useful. RMSE applies to numerical prediction, rather than class labels. Emphasize precision to limit false positives, and recall to limit missed positives.

Baseline¶

Bayes Classifier¶

• A baseline model with an assumption that test errors are minimized when each new observation is assigned to the most likely class.

• For a new observation, choose the class with the largest probability.

• From the graph above, there are two noticeable boundaries where can lead to a change of membership to a new observation, one at 0 and the other at 8.

• Error rate (For a given observation $x$): $1-\text{max}_j\Pr(Y=j|x)$

• Error rate (Overall Bayes error rate): $1-E(\text{max}_j\Pr(Y=j|x))$

  • The Bayes error rate is the minimal error rate.

  • However, in practice, Bayes error rate is never known, because the underlying functions are not known.

Overfitting and Underfitting¶

Overfitting¶

• Overfitting happens when the model fits random noise of a sample rather than the generalizable relationship.

• When does it happen?

  • When the model has too many parameters relative to the number of observations.

  • This is why advanced machine learning algorithms must be watchful towards overfitting; they are designed to have a lot of parameters.

  • When the model is too flexible; when a model can fit a wide variety of shapes and patterns in the data.

• As can be seen on the graph above, overfitted model can twist and bend very closely on data.

Underfitting and validation¶

An underfitted model misses a pattern that is present in the data. Both training and validation errors can remain high. An overfitted model often has low training error but noticeably higher validation error. The gap is useful evidence, although different training and validation populations can also cause a gap.

Use the training set to estimate parameters and a validation set, or cross-validation within the training data, to choose model settings. Keep the test set for evaluating the chosen procedure. Trying many settings on the test set gradually makes it part of model selection.

In $K$-fold cross-validation, each fold is held out once while the other $K-1$ folds are used for fitting. Preprocessing must also be fitted again within each training fold. For repeated measurements, keep observations from the same subject together. For future prediction, use splits that preserve time order.

Tips¶

Log Transformation¶

• When $y$ or any variable spans multiple magintude, it is easier to take a log of it. (E.g., $log(y)$)

  • Log transformation reduces the scale of very large values more than small ones.

  • E.g.:

  • $y = 10$ → $log(y) = 2.3$

  • $y = 1,000,000$ → $log(y) = 13.8$

  • Hence, log transformations can shrink gaps dramatically, and prevent large values from dominating the model.

  • Can reduce right skewness of a positive variable and make some relationships easier to model; improvement is not guaranteed. A logarithm needs positive inputs, and back-transforming a prediction of log response does not generally give the mean response.

Nonlinear models¶

• A model is nonlinear if parameters ($\beta$) appear nonlinearly, not the predictors ($x_i$). For example,

  $$y = \frac{\beta_1}{1+e^{\beta_2x}}$$

  (14)

  ... is nonlinear, because you cannot seperate $\beta_1$ and $\beta_2$ into a clean linear combination. For example, neural networks have their weights interacting across layers and are an example of nonlinear model.

Linear Model¶

• This model above is called as ordinary linear regression (OLR).

• Least squares gives the best linear unbiased estimator under the Gauss–Markov assumptions: a correctly specified linear conditional mean, full column rank, and errors with zero conditional mean, constant variance, and no correlation.

  • AKA, has the lowest variance.

  • Refers to a specific way to esimate the coefficients $\beta$.

  • Chooses $\hat{\beta}$ by minimizing: $\text{RSS} = \sum(y_i-\hat{y_i})^2$.

Hyperparameter and Parameter¶