Experimental Design - The Statistics Notebook

A useful experiment starts with a clear question, a measurable response, and a fair assignment of treatments. This page covers planning, experimental units, and the principles that protect a comparison.

Planning: QPDAC¶

QPDAC = Question → Plan → Data → Analysis → Conclusion. PPDAC uses Problem in place of Question.

Stage	What to decide
Question / Problem	State a specific question, a target population, and a measurable outcome.
Plan	Choose the response, factors, units, treatment assignment, and sample size.
Data	Follow the plan; record outcomes, assignment, and any departures from the protocol.
Analysis	Estimate treatment effects and quantify uncertainty using a model that respects the design.
Conclusion	Answer the original question in context, with effect size, uncertainty, and limitations.

A good question is clear, understandable to a general audience, and tied to a specific metric.
“Are our job posts good?” is vague. “Does a casual rather than formal tone increase the proportion of viewers who click?” names the comparison and response.
Most design decisions happen before data collection. A small $p$ -value cannot repair a confounded comparison.

Response, Factors & Experimental Units¶

Term	Meaning	Example: job-post experiment
Response	Outcome recorded for each observational unit.	Whether a viewer clicks; summarize as click-through rate.
Design factor	Variable deliberately varied because its effect is of interest.	Tone of the post.
Level	A particular setting of a factor.	Formal or casual.
Treatment	A specific combination of design-factor levels.	Casual tone + short format, if both tone and format vary.
Experimental unit	Smallest entity independently assigned a treatment.	Viewer, if assignment is by viewer.
Observational unit	Entity on which a measurement is recorded.	A viewing event; there may be several per viewer.
Replicate	An additional independent experimental unit receiving a treatment.	Another independently assigned viewer.

Four Factor Roles¶

Role	What we do	Why
Held constant	Keep the variable at one level.	Prevent it from explaining treatment differences.
Design factor	Deliberately vary its levels.	Estimate the effect of interest.
Nuisance factor	Account for a variable that affects the response but is not the focus.	Reduce unwanted variation or confounding.
Allowed to vary	Let it differ between units.	Practical when it cannot be controlled; randomization helps protect the comparison.

The role depends on the question and assignment mechanism. Time of day can be held constant, used as a block, or studied as a design factor. “Nuisance” describes scientific relevance; “allowed to vary” describes handling, so an uncontrolled nuisance variable can fall into both descriptions.

Worked Design Example: Job-Post Click Rate

Question: Does post tone change click-through rate for a particular job?

Component	Plan
Response	Click/no click for each viewer within a fixed observation window.
Experimental unit	Viewer, randomly assigned to one version.
Design factor	Tone: formal vs. casual.
Held constant	Job, salary range, format, and observation window.
Nuisance variables	Time of exposure, device, and applicant status.
Possible blocking	Randomize tone within time windows or applicant-status groups.
Allowed to vary	Other viewer characteristics, protected by random assignment.

The treatment-level response summary is

\widehat p_i=\frac{\text{viewers who click in group }i}{\text{viewers assigned to group }i}.

(1)

Changing both tone and salary would make the comparison ambiguous: either could explain a difference. Repeated impressions from one viewer are not independent viewer-level replicates.

Analysis boundary: individual click outcomes are binary. A proportions or binary-response model is appropriate; the normal-response $t$ formulas below are introduced for quantitative outcomes.

Other Examples: Baseball & Chalk

Component	Baseball	Chalk breaks
Possible response	Runs scored, run differential, or win/loss; choose one.	Breaks per writing session of fixed duration.
Possible design factor	A strategy deliberately assigned to games.	Caffeine vs. no caffeine, if assigned.
Experimental unit	Game, if strategy is assigned by game.	Writer-session, if caffeine is assigned by session.
Held constant	Team, for a study about one team.	Chalk brand and board type.
Nuisance variables	Opponent strength, fatigue, weather.	Writer, room, time of session.

“Number of runs” is an outcome, not an experimental unit. Home/away differences may be observed rather than randomized. Similarly, merely observing caffeine habits makes the chalk comparison observational.

Sessions involving the same writer may be correlated. Treating every chalk mark or every session as an independent replicate without accounting for this can understate uncertainty.

Randomization, Replication & Blocking¶

Principle	What it does	What it does not guarantee
Randomization	Assigns treatments by chance; prevents systematic assignment based on known or unknown unit characteristics.	Exact balance in every realized sample.
Replication	Applies treatments to multiple independent units; estimates error and improves precision.	Removal of bias from a flawed assignment process.
Blocking	Groups similar units and randomizes treatments within groups; allows within-block comparisons.	Removal of every source of variation.

Random assignment supports causal comparisons within the study. Random sampling supports generalization to a target population. They solve different problems.

Confounding: treatment differences cannot be separated from another variable. If all formal posts run in the morning and all casual posts run at night, tone is confounded with time.

Replication vs. Repeated Measurement¶

Suppose two batches receive different recipes, and 20 cookies are measured per batch.

The batches receive treatments, so there is only one experimental unit per recipe.
The 20 cookies are subsamples; they do not create 20 independent recipe assignments.
Replicating the recipe comparison requires additional independently assigned batches.

Repeated measurement can improve measurement precision, but it does not create independent treatment replication.