Contrasts & Multiple Comparisons - The Statistics Notebook

A global ANOVA asks whether all means are equal. Contrasts answer focused questions, and multiple-comparison procedures control error across a family of claims.

The worked examples continue the one-way ANOVA example: A, B, and C have means 6, 8, and 11, with four observations each, $MSE=2/3$ , and 9 residual degrees of freedom.

Contrasts: Focused Questions About Means¶

A contrast is a linear combination

C=\sum_{i=1}^m c_i\mu_i,\qquad\sum_i c_i=0.

(1)

The zero-sum condition removes the overall level, leaving a comparison.

Question for A, B, C	Coefficients $(c_A,c_B,c_C)$
A minus B	$(1,-1,0)$
C minus the average of A and B	$(-1/2,-1/2,1)$
Linear trend across three equally spaced levels	$(-1,0,1)$
Curvature across three equally spaced levels	$(1,-2,1)$

Under the independent common-variance one-way model,

\widehat C=\sum_i c_i\bar Y_{i\cdot},\qquad \operatorname{SE}(\widehat C)=\sqrt{MSE\sum_i\frac{c_i^2}{n_i}}.

(2)

T_C=\frac{\widehat C-C_0}{\operatorname{SE}(\widehat C)}\sim t_{N-m} \quad\text{under }H_0:C=C_0.

(3)

An individual two-sided confidence interval is

\widehat C\pm t_{1-\alpha/2,N-m}\operatorname{SE}(\widehat C).

(4)

For testing $C=0$ , define

SS_C=\frac{\widehat C^2}{\sum_i c_i^2/n_i},\qquad F_C=\frac{SS_C}{MSE}=T_C^2\sim F_{1,N-m}.

(5)

Here $SS_C$ denotes a contrast sum of squares, not the total condition sum of squares.

Worked Contrast: C vs. the Average of A and B

For the ANOVA example, use $c=(-1/2,-1/2,1)$ :

\widehat C=11-\frac{6+8}{2}=4.

(6)

\operatorname{SE}(\widehat C) =\sqrt{\frac23\left(\frac{1/4}{4}+\frac{1/4}{4}+\frac14\right)}=0.5.

(7)

Thus $t=8$ on 9 df. The individual 95% confidence interval is

4\pm2.2622(0.5)=(2.8689,5.1311).

(8)

The contrast SS is $4^2/(3/8)=42.6667$ , giving $F=64=t^2$ .

For the contrast A minus B, $SS_C=(-2)^2/(1/4+1/4)=8$ . These two contrasts are orthogonal and their sums of squares add to $42.6667+8=50.6667$ , the treatment SS.

Multiplying all coefficients by a nonzero constant scales the estimate and SE; the two-sided test conclusion stays the same. A negative multiplier reverses the interpretation of a one-sided alternative.

Orthogonal & Polynomial Contrasts¶

Two contrast estimates have covariance

\operatorname{Cov}(\widehat C,\widehat D)=\sigma^2\sum_i\frac{c_i d_i}{n_i}.

(9)

They are orthogonal when $\sum_i c_i d_i/n_i=0$ . With equal group sizes this reduces to $\sum_i c_i d_i=0$ . There are at most $m-1$ mutually orthogonal nonzero contrasts. A full orthogonal set partitions treatment SS.

For equally spaced numeric levels and equal replication:

Number of levels	Linear	Quadratic	Cubic
3	$(-1,0,1)$	$(1,-2,1)$
4	$(-3,-1,1,3)$	$(1,-1,-1,1)$	$(-1,3,-3,1)$

Polynomial contrasts describe trends for quantitative levels. They do not imply a meaningful trend among arbitrary category names. Unequal spacing or replication requires coefficients and orthogonality checks appropriate to that design.

A prespecified contrast can be tested directly without first rejecting the global ANOVA. Several planned contrasts still form a testing family when joint error control is intended; planning alone does not eliminate multiplicity.

Multiple Comparisons¶

With $m$ means, all pairwise comparisons form a family of

K=\binom m2=\frac{m(m-1)}2

(10)

tests. The family-wise error rate (FWER) is the probability of at least one false rejection in that family.

If $K$ null tests were independent and each used level $\alpha$ ,

\operatorname{FWER}=1-(1-\alpha)^K.

(11)

With five means, $K=10$ and $1-0.95^{10}=0.4013$ . This is an illustration under independence, not the exact FWER for overlapping pairwise tests, which are dependent.

The one-way pooled pairwise statistic is

t_{ij}=\frac{\bar Y_{i\cdot}-\bar Y_{j\cdot}} {\sqrt{MSE(1/n_i+1/n_j)}},\qquad df=N-m.

(12)

Procedure	Rule	Main qualification
Fisher’s protected LSD	After global rejection, use ordinary pairwise $t$ tests.	Generally lacks strong FWER control across many groups.
Bonferroni	Test each of $K$ hypotheses at $\alpha/K$ ; adjusted $p=\min(1,Kp)$ .	Controls FWER without requiring independence; can be conservative.
Holm	Sort $p$ -values; compare $p_{(j)}$ to $\alpha/(K-j+1)$ , stopping at the first failure.	Strong FWER control without independence; at least as powerful as Bonferroni.
Šidák	Use $\alpha^*=1-(1-\alpha)^{1/K}$ .	Exact family calibration for independent tests; not a universal rule for dependence.
Tukey HSD / Tukey-Kramer	Use the studentized range distribution for all pairs.	Suited to all pairwise mean comparisons under the normal common-variance model.

For Bonferroni two-sided simultaneous intervals,

\widehat C_\ell\pm t_{1-\alpha/(2K),\nu}\operatorname{SE}(\widehat C_\ell).

(13)

For Tukey-Kramer pairwise intervals,

\bar Y_{i\cdot}-\bar Y_{j\cdot}\ \pm\ q_{1-\alpha;m,\nu}\sqrt{\frac{MSE}{2}\left(\frac1{n_i}+\frac1{n_j}\right)},\qquad \nu=N-m.

(14)

With equal group size $n$ , the margin simplifies to $q\sqrt{MSE/n}$ . Tukey HSD has exact simultaneous coverage under its balanced normal model; Tukey-Kramer extends the procedure to unequal sizes, generally conservatively.

Holm Example: Know When to Stop

For four hypotheses, suppose sorted $p$ -values are $0.004,0.013,0.030,0.200$ and $\alpha=0.05$ .

Step	$p_{(j)}$	Threshold	Decision
1	0.004	0.0125	Reject.
2	0.013	0.0167	Reject.
3	0.030	0.0250	Stop; do not reject.
4	0.200	0.0500	Do not reject after stopping.

Bonferroni would reject only the first. The Holm adjusted $p$ -values are $0.016,0.039,0.060,0.200$ .

Define the family before seeing which comparisons look promising. A significant global ANOVA is not permission for unlimited unadjusted follow-up tests.

The cell includes the required ANOVA summaries so it runs independently of the previous page.