LASSO

LASSO stands for least absolute shrinkage and selection operator. It estimates a linear predictor while penalizing the sum of the absolute coefficient values. Unlike the quadratic ridge penalty, the absolute value penalty has a corner at zero, which can make an exactly zero coefficient optimal.

The estimation problem¶

With centered training data, solve

The intercept is restored on the original scale using $\widehat\beta_0=\bar y−\sum_j\bar x_j\widehat\beta_j$. If predictors were also scaled, first convert slopes back to their original units.

Penalty comparisons depend on predictor scale. A variable measured in thousands may need a much smaller coefficient than the same variable measured in units. Standardizing slopes puts their penalties on a comparable footing. Learn the scaling from the training sample, including separately within each validation fold.

Constrained geometry¶

An equivalent constrained problem minimizes residual error subject to $\lVert\beta\rVert_1\leq t$, with a corresponding choice of $t$ for a penalized solution. Larger $\lambda$ imposes stronger shrinkage; larger $t$ allows a larger coefficient budget.

In two dimensions the LASSO constraint is a diamond with corners on the coordinate axes. Elliptical residual contours can first touch a corner, setting one coefficient to zero. The ridge constraint is smooth, so it has no comparable corner favoring exact zeros.

This is a tendency rather than a guarantee that ridge coefficients can never equal zero. Symmetry or particular data can make a ridge coefficient zero. Ridge simply does not usually create sparsity by a threshold mechanism.

Optimality at a corner¶

For a scalar $b$, the subgradient of $|b|$ is 1 if $b>0$, $−1$ if $b<0$, and any value in $[−1,1]$ if $b=0$. A subgradient supplies the supporting slopes where an ordinary derivative does not exist.

Let $r=y−X\widehat\beta$. Convex optimality requires

There are two cases:

1. If $\widehat\beta_j\ne0$, then $x_j^\top r=\lambda\operatorname{sign}(\widehat\beta_j)$.

2. If $\widehat\beta_j=0$, then $|x_j^\top r|\leq\lambda$.

A zero coefficient is therefore compatible with a whole interval of residual correlations. It need not have exactly zero correlation with the response.

An orthogonal calculation¶

When $X^\top X=I$, the problem separates into independent coordinates. Write $z_j=x_j^\top y$. Then

For $z=(3,0.8,−2)$ and $\lambda=1$, the fitted coefficients are $(2,0,−1)$. The middle coordinate is removed, while the other two move toward zero by one unit. This calculation is illustrative, rather than an output from the supplied datasets.

For correlated predictors, the relevant quantity changes as the other coefficients change. Applying this formula independently to the raw correlations is not the general solution; coordinate descent handles that dependence.

Correlation and uniqueness¶

The objective is convex, so its optimality conditions characterize global minima. Coefficients need not be unique when the design is rank deficient. For positive $\lambda$, all minimizers nevertheless have the same fitted vector $X\widehat\beta$, because squared loss is strictly convex in that vector.

Strongly correlated predictors may substitute for one another. A small sample change can alter which is retained while leaving predictions similar. Sparse prediction and reliable scientific variable identification are different objectives.