Two-Group Comparisons

Compare two independent treatment groups by connecting hypotheses, effect estimates, standard errors, and uncertainty. Start with Experimental Design to check what counts as an independent experimental unit.

At a Glance¶

Pooled test

Welch test

Use the common-variance model when its assumptions are justified. Pool the two variance estimates and use the corresponding residual degrees of freedom.

Simple Comparative Experiments¶

A simple comparative experiment has one design factor with two levels: treatment vs. control, list vs. tile layout, or low vs. high ad frequency.

For the independent-group quantitative-response model,

Y_{ij}\overset{\mathrm{ind}}{\sim}N(\mu_i,\sigma_i^2),\qquad i=1,2,\quad j=1,\ldots,n_i.

(1)

Symbol	Meaning
$Y_{ij}$	Response of unit $j$ in treatment group $i$ .
$n_i$	Number of independent units in group $i$ .
$\mu_i$	Population mean response under treatment $i$ .
$\sigma_i^2$	Population response variance under treatment $i$ .
$\Delta=\mu_1-\mu_2$	Mean difference of interest.
$\widehat\Delta=\bar Y_1-\bar Y_2$	Estimated mean difference.

The basic model assumes independent observations within and between groups. It does not account for pairing, blocks, or clusters. Those designs require an analysis that preserves their dependence structure.

Hypotheses & Evidence¶

Let $\Delta_0$ be the hypothesized difference, usually 0.

Scientific question	Null hypothesis	Alternative hypothesis
Is there a difference?	$H_0:\Delta=\Delta_0$	$H_A:\Delta\ne\Delta_0$
Is group 1 larger?	$H_0:\Delta\leq\Delta_0$	$H_A:\Delta>\Delta_0$
Is group 1 smaller?	$H_0:\Delta\geq\Delta_0$	$H_A:\Delta<\Delta_0$

Choose the direction before looking at outcomes. One-sided tests are calibrated at the null boundary $\Delta=\Delta_0$ .

Test Statistic¶

T=\frac{\text{estimated effect}-\text{null effect}}{\text{estimated standard error}}.

(2)

The numerator measures the observed departure from the null. The denominator measures ordinary sampling variability. A difference of 2 units can be weak or strong evidence depending on its standard error.

Under $H_0$ , we need a known reference distribution, exact or approximate, that lets us evaluate the statistic without unknown nuisance parameters.

What a $p$ -Value Means¶

The $p$ -value is the probability, under the null model, of a test statistic at least as extreme as observed, in the direction(s) specified by $H_A$ .

Small $p$ : the observed result is unusual under $H_0$ .
It is not $P(H_0\mid\text{data})$ and is not the probability that the result is “due to chance.”
At a preselected significance level $\alpha$ , reject when $p\leq\alpha$ .
Failure to reject means insufficient evidence against $H_0$ ; it does not establish equality.

Type I Error, Type II Error & Power¶

Decision	$H_0$ true	$H_0$ false
Reject $H_0$	Type I error	Correct detection
Do not reject $H_0$	Correct non-rejection	Type II error

\alpha=P(\text{reject }H_0\mid H_0\text{ true}),\qquad \beta(\Delta)=P(\text{do not reject }H_0\mid\Delta\text{ is the specified alternative}).

(3)

\operatorname{Power}(\Delta)=1-\beta(\Delta).

(4)

For composite nulls, $\alpha$ is the controlled upper bound on rejection probability. Power depends on the true effect, variance, sample size, and test. Increasing sample size generally increases power; lowering $\alpha$ at fixed sample size generally reduces it.

Statistical significance asks about evidence. Practical significance asks whether the effect is large enough to matter. Report the estimated difference and its uncertainty alongside the test.

Comparing Two Independent Means¶

Sample Summaries¶

\bar Y_i=\frac{1}{n_i}\sum_{j=1}^{n_i}Y_{ij},\qquad s_i^2=\frac{1}{n_i-1}\sum_{j=1}^{n_i}(Y_{ij}-\bar Y_i)^2.

(5)

Estimating the group mean leaves $n_i-1$ degrees of freedom for its variance.

Independence gives

\operatorname{Var}(\bar Y_1-\bar Y_2)=\frac{\sigma_1^2}{n_1}+\frac{\sigma_2^2}{n_2}.

(6)

This identity explains both versions of the two-sample $t$ test.

Student’s Pooled Two-Sample $t$ Test¶

Use a pooled estimate when a common population variance, $\sigma_1^2=\sigma_2^2=\sigma^2$ , is justified.

s_p^2=\frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}.

(7)

This is a degrees-of-freedom weighted average of the two sample variances, not generally their simple average.

\operatorname{SE}_{p}=s_p\sqrt{\frac1{n_1}+\frac1{n_2}},\qquad T_p=\frac{(\bar Y_1-\bar Y_2)-\Delta_0}{\operatorname{SE}_{p}}.

(8)

Under independent normal observations with a common variance,

T_p\mid H_0\sim t_{n_1+n_2-2}.

(9)

Welch’s Two-Sample $t$ Test¶

Welch’s method estimates each variance separately:

\operatorname{SE}_{W}=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}},\qquad T_W=\frac{(\bar Y_1-\bar Y_2)-\Delta_0}{\operatorname{SE}_{W}}.

(12)

Its reference distribution is approximately $t_\nu$ , with Welch-Satterthwaite degrees of freedom

\nu= \frac{\left(s_1^2/n_1+s_2^2/n_2\right)^2} {\dfrac{(s_1^2/n_1)^2}{n_1-1}+\dfrac{(s_2^2/n_2)^2}{n_2-1}}.

(13)

$\nu$ can be non-integer; use it directly in software.
It is not generally equal to $\min(n_1,n_2)-1$ .
Welch does not require equal variances. It still requires an appropriate independent-group design and a reasonable sampling approximation.

Which Test Fits?¶

Situation	Analysis
Independent groups; common variance supported by the model/design	Pooled two-sample $t$ .
Independent groups; unequal or uncertain variances	Welch two-sample $t$ .
Same units measured twice or matched pairs	Analyze within-pair differences; do not use an independent-group standard error.
Blocks or clusters	Include that structure in the model.
Binary individual outcomes	Use a binary-response/proportions analysis.

Check independence from the design, and examine outliers, skewness, and group spreads from the data. Normality makes the pooled test exact; large-sample approximations can help with non-normal data, but do not fix dependence or severe data problems.

$t$ -Test Tails & Confidence Intervals¶

Let $t_{\mathrm{obs}}$ be the observed statistic and $G_\nu$ the CDF of the chosen $t$ reference distribution.

Alternative	$p$ -value
$\Delta\ne\Delta_0$	$2[1-G_\nu(\lvert t_{\mathrm{obs}}\rvert)]$
$\Delta>\Delta_0$	$1-G_\nu(t_{\mathrm{obs}})$
$\Delta<\Delta_0$	$G_\nu(t_{\mathrm{obs}})$

A one-sided $p$ -value is half the two-sided value only when the observed effect points in the chosen alternative’s direction. An effect strongly in the opposite direction gives a one-sided $p$ -value near 1.

A two-sided $100(1-\alpha)\%$ confidence interval for $\Delta$ is

\widehat\Delta\ \pm\ t_{1-\alpha/2,\nu}\operatorname{SE}(\widehat\Delta).

(14)

Use the matching standard error and degrees of freedom: pooled or Welch. Here $t_{q,\nu}$ is the $q$ quantile of $t_\nu$ .

An interval excluding $\Delta_0$ corresponds to rejection by the matching two-sided test.
A 95% procedure covers the fixed true difference in 95% of repeated samples under its assumptions.
Including 0 does not establish that effects are negligible; inspect the entire interval.

Worked Example: Pooled vs. Welch Calculations

Illustrative summary data, constructed for practice:

Group	$n_i$	$\bar y_i$	$s_i$	$s_i^2$
1	10	8	2	4
2	12	6	3	9

Test $H_0:\mu_1-\mu_2=0$ against a two-sided alternative at $\alpha=0.05$ .

Pooled calculation, conditional on a common-variance assumption:

s_p^2=\frac{9(4)+11(9)}{20}=6.75,

(15)

\operatorname{SE}_p=\sqrt{6.75}\sqrt{\frac1{10}+\frac1{12}}=1.1124,

(16)

t_p=\frac{8-6}{1.1124}=1.7979,\qquad \nu=20.

(17)

The two-sided $p$ -value is 0.0873, and the 95% interval is

2\pm2.0860(1.1124)=(-0.3205,\ 4.3205).

(18)

Welch calculation:

\operatorname{SE}_W=\sqrt{\frac4{10}+\frac9{12}}=1.0724,

(19)

t_W=1.8650,\qquad\nu\approx19.1905.

(20)

The two-sided $p$ -value is 0.0775, and the 95% interval is $(-0.2430,\ 4.2430)$ .

Both methods fail to reject at 5%. The estimated difference is 2 response units, but the intervals still allow no difference and fairly large positive differences. This does not show that the treatments are equivalent. Choose the variance model on substantive grounds, not by whichever result has the smaller $p$ -value.

This cell uses the same illustrative summaries. sf evaluates the upper tail directly; std expects standard deviations, not variances.

Comparing Two Variances: The $F$ Test¶

This test asks whether population variances are equal. It does not test equality of means.

H_0:\sigma_1^2=\sigma_2^2,\qquad H_A:\sigma_1^2\ne\sigma_2^2.

(21)

For independent normal samples, keeping the group labels fixed,

F=\frac{s_1^2}{s_2^2},\qquad F\mid H_0\sim F_{n_1-1,n_2-1}.

(22)

The first degrees of freedom belong to the numerator; the second belong to the denominator. If the ratio is inverted, swap the degrees of freedom too.

Two-Sided $p$ -Value¶

Let $H_{d_1,d_2}$ be the $F$ CDF. The usual equal-tail two-sided calculation is

p=\min\left\{1,\ 2\min\left[H_{d_1,d_2}(f_{\mathrm{obs}}),\ 1-H_{d_1,d_2}(f_{\mathrm{obs}})\right]\right\}.

(23)

Equivalently, reject if $f_{\mathrm{obs}}<F_{\alpha/2,d_1,d_2}$ or $f_{\mathrm{obs}}>F_{1-\alpha/2,d_1,d_2}$ .

For a preselected alternative $\sigma_1^2>\sigma_2^2$ , use the upper-tail probability with fixed group labels. For $\sigma_1^2<\sigma_2^2$ , use the lower tail.

The classical variance-ratio test is sensitive to non-normality. A nonsignificant result should not automatically select a pooled mean test. Welch avoids requiring a common variance when equality is uncertain.

Example: Ad Frequency & Visit Duration¶

An ad-frequency experiment compares one ad every eight posts with one ad every five posts, using visit duration as the response.

Item	Approximate reported summary
Group 1	Lower ad frequency; mean duration approximately 4.9.
Group 2	Higher ad frequency; mean duration approximately 3.0.
Sample sizes	500 observations per condition.
Variance ratio	Approximately 0.9376.
Two-sided variance-test $p$	Approximately 0.472.
Pooled mean-test statistic	Approximately 30.
Two-sided mean-test result	Extremely small $p$ -value, reported below 10^-16.

These are rounded reported results, not a recomputation from raw data. The available summaries do not specify the time unit or whether repeated visits from a user were treated as independent; the model depends on the assignment and measurement design.

With $\Delta=\mu_{\text{lower ads}}-\mu_{\text{higher ads}}$ , the observed difference is positive. Therefore:

The two-sided comparison provides strong evidence of a mean difference under the stated model.
“Higher ad frequency increases duration” corresponds to $H_A:\Delta<0$ .
A large positive statistic is in the wrong direction for that alternative, so its one-sided $p$ -value is near 1.

The reported data point toward shorter visits with more frequent ads. Causal interpretation additionally requires valid randomized assignment and an analysis that respects the experimental units.

At a Glance¶

Simple Comparative Experiments¶

Hypotheses & Evidence¶

Test Statistic¶

What a ppp-Value Means¶

Type I Error, Type II Error & Power¶

Comparing Two Independent Means¶

Sample Summaries¶

Student’s Pooled Two-Sample ttt Test¶

Welch’s Two-Sample ttt Test¶

Which Test Fits?¶

ttt-Test Tails & Confidence Intervals¶

Comparing Two Variances: The FFF Test¶

Two-Sided ppp-Value¶

Example: Ad Frequency & Visit Duration¶

What a $p$ -Value Means¶

Student’s Pooled Two-Sample $t$ Test¶

Welch’s Two-Sample $t$ Test¶

$t$ -Test Tails & Confidence Intervals¶

Comparing Two Variances: The $F$ Test¶

Two-Sided $p$ -Value¶