Spline Bases - The Statistics Notebook

A global polynomial uses the same coefficients over the entire input range. Raising its degree adds flexibility everywhere, including places where the data give little information. Splines distribute that flexibility over intervals while joining the pieces smoothly.

Polynomial pieces and knots¶

A knot is a location where neighboring polynomial pieces meet. Without continuity constraints, a separate polynomial in each interval can jump at the knots. Such jumps are often implausible when estimating a smooth mean response.

Use degree $d$ and order $q=d+1$ . A cubic has degree 3 and order 4. For simple interior knots, an order $q$ spline has continuous derivatives through order $q−2$ . Thus a cubic spline has continuous values, first derivatives, and second derivatives, while its third derivative may jump.

With $k$ interior knots, there are $k+1$ polynomial pieces. Initially they require $q(k+1)$ coefficients. Matching $q−1$ quantities at each knot imposes $k(q−1)$ constraints. The remaining dimension is

q(k+1)−k(q−1)=q+k.

(1)

For a cubic spline with three interior knots, this gives $4+3=7$ basis coefficients before additional boundary restrictions.

The truncated power basis¶

A cubic spline can be written as

f(x)=\beta_0+\beta_1x+\beta_2x^2+\beta_3x^3+\sum_{r=1}^{k}\theta_r(x−\tau_r)_+^3.

(2)

Each added term is zero to the left of its knot. To the right, it is cubic. Its value and first two derivatives agree across the knot, so continuity is built into the basis rather than imposed after fitting.

This representation is useful for understanding the spline space. Numerically, powers of $x$ and shifted powers can have very different scales and strong correlations. Another basis can represent the same curves more conveniently.

The B spline recursion¶

Let $t_i$ be a nondecreasing knot sequence. Start with interval indicators,

B_{i,1}(x)=\mathbf1\{t_i\leq x<t_{i+1}\}.

(3)

Build higher orders recursively:

B_{i,q}(x)=\frac{x−t_i}{t_{i+q−1}−t_i}B_{i,q−1}(x)+\frac{t_{i+q}−x}{t_{i+q}−t_{i+1}}B_{i+1,q−1}(x).

(4)

A term with a zero denominator contributes zero, as occurs with repeated boundary knots. Endpoint evaluation also follows the boundary convention of the chosen basis.

The recursion blends adjacent pieces. An order $q$ basis function is supported only on $[t_i,t_{i+q})$ . At most $q$ basis functions are nonzero in an ordinary interior knot interval. Changing one coefficient therefore changes the represented curve only locally, although refitting all coefficients after changing data can still affect the whole fit.

The basis functions are nonnegative. With the complete basis and the usual boundary construction, they sum to one over the basic interval. These properties make the coefficients useful as local weights, although an individual coefficient is not generally the curve’s value at a knot.

Regression splines and natural boundaries¶

A regression spline fixes a knot sequence and estimates its coefficients from the observations. With design matrix $B$ , ordinary least squares solves $\min_\beta\lVert y−B\beta\rVert^2$ . If $B$ has full column rank, the coefficient estimate is $(B^\top B)^{−1}B^\top y$ .

Too few knots can exclude important curvature. Many knots can fit noise unless flexibility is controlled. Knot placement and the smoothing penalty answer different questions: knots define the available space, while the penalty determines how much of it the estimate uses.

A natural cubic spline is linear outside its boundary knots. This restricts the behavior near the edges, where high degree polynomials can be unstable. Linear extrapolation is still an assumption; natural boundaries do not establish what the true relationship does beyond the observed range.

Flexibility between observations¶

Different curves can pass through exactly the same observations yet behave very differently between them. A polynomial of high degree may oscillate, while independently fitted polynomial pieces may jump at their boundaries. A spline joins its pieces smoothly, but smooth joins alone do not prevent it from fitting noise.

Adding knots enlarges the space of possible curves. It does not determine how much of that flexibility the fitted curve should use. Choosing the number of knots or applying a roughness penalty helps control this flexibility; validation on held out observations helps assess whether it improves prediction.

The important comparison is not which curve passes closest to every training point. It is which restrictions are plausible for the underlying mean and which choice predicts well on new observations.