Paired & Randomization Tests - The Statistics Notebook

When observations are matched, the comparison must preserve the pairs. This page explains paired tests, design-based randomization, and planning sample size and power.

At a Glance¶

Paired analysis

Randomization analysis

Reduce each independent pair to one difference. The sample size is the number of independent pairs.

Paired Comparisons¶

Use pairing when the same unit receives two conditions, or when two units are meaningfully matched. Examples include two measurements on the same participant or matched specimens from the same batch.

Define a consistent subtraction order:

D_j=Y_{1j}-Y_{2j},\qquad j=1,\ldots,n.

(1)

Here $n$ counts pairs, not the $2n$ individual measurements. The comparison becomes a one-sample problem for the differences:

\bar D=\frac1n\sum_jD_j,\qquad s_D^2=\frac{\sum_j(D_j-\bar D)^2}{n-1}.

(2)

To test $H_0:\mu_D=\Delta_0$ ,

T=\frac{\bar D-\Delta_0}{s_D/\sqrt n}\sim t_{n-1}\quad\text{under }H_0.

(3)

The exact result assumes independent, normally distributed differences. The two sets of raw measurements need not individually satisfy an independent-sample model. A two-sided confidence interval is

\bar D\pm t_{1-\alpha/2,n-1}\frac{s_D}{\sqrt n}.

(4)

Why Pairing Can Improve Precision¶

\operatorname{Var}(Y_1-Y_2)=\sigma_1^2+\sigma_2^2-2\operatorname{Cov}(Y_1,Y_2).

(5)

When responses within pairs are positively correlated, subtracting removes shared unit-to-unit variation. With equal marginal variance $\sigma^2$ and correlation $\rho$ ,

\operatorname{Var}(\bar D)=\frac{2\sigma^2(1-\rho)}{n}.

(6)

Strong positive correlation can give much better precision than an independent comparison of the same size.
Arbitrary matching does not guarantee a gain. Negative within-pair correlation can increase variance.
For crossover experiments, randomize or counterbalance treatment order. Time trends, learning, fatigue, and carryover can otherwise mimic treatment effects.
If a treatment permanently changes the unit, applying both conditions may not be appropriate.

Randomization & Permutation Tests¶

A randomization test asks: How unusual is the observed statistic among assignments allowed by the design, if treatment has no effect?

Choose a statistic, such as $\bar Y_1-\bar Y_2$ .
Compute its observed value $T_{\mathrm{obs}}$ .
Generate treatment assignments consistent with the original randomization.
Recompute the statistic for each assignment.
Count results at least as extreme as observed.

For a complete enumeration of equally likely assignments $\mathcal A$ , a two-sided absolute-statistic test has

p=\frac{\#\{a\in\mathcal A:|T_a|\geq|T_{\mathrm{obs}}|\}}{|\mathcal A|}.

(9)

For $B$ independently sampled random assignments, a standard Monte Carlo estimate is

\widehat p=\frac{1+\sum_{b=1}^B I(|T_b|\geq|T_{\mathrm{obs}}|)}{B+1}.

(10)

The added 1 includes the observed arrangement and avoids reporting a zero estimated $p$ -value.

Design	Valid rearrangement
Completely randomized independent groups	Reassign labels while retaining the original group sizes.
Randomized matched pairs	Swap treatment labels within each pair, equivalent to sign-flipping each difference.
Randomized blocks	Reassign labels within blocks, preserving allocation counts.
Cluster-randomized experiment	Reassign whole clusters, not individual measurements.

A bootstrap resamples observations to approximate sampling uncertainty. A permutation test rearranges labels to construct a null distribution. These are different procedures.

Planning Sample Size & Power¶

Choose the smallest effect worth detecting, $\delta$ , before data collection. Specify a significance level $\alpha$ , desired power $1-\beta$ , and a plausible variance from earlier work or a pilot study.

For a balanced, two-sided independent comparison with common variance $\sigma^2$ , a normal approximation gives

n\approx\frac{2\sigma^2(z_{1-\alpha/2}+z_{1-\beta})^2}{\delta^2} \quad\text{units per group}.

(12)

For paired differences with variance $\sigma_D^2$ ,

n\approx\frac{\sigma_D^2(z_{1-\alpha/2}+z_{1-\beta})^2}{\delta^2} \quad\text{pairs}.

(13)

Here $z_q$ is the $q$ quantile of the standard normal distribution. Round up. These are planning approximations; small-sample $t$ procedures, attrition, clustering, and a different design may require different calculations.

Change, holding other inputs fixed	Required sample size
Higher desired power	Increases.
Larger response variance	Increases.
Smaller significance level	Increases.
Smaller target effect	Increases; halving $\delta$ roughly quadruples $n$ .
More effective pairing	Decreases when it reduces $\sigma_D^2$ .

Increasing the number of correlated measurements is not the same as increasing the number of independent experimental units. Large samples do not fix confounding.

These calculations compare the paired normal-model test with exact within-pair sign randomization for the same illustrative differences.