The default claim of no difference or no effect, e.g. 'the new checkout does not change conversion'.
Example
Common mistakes
Saying "we accepted H0" or "the test proved there's no effect". A non-significant result means the evidence was not strong enough, often because the sample was too small.
Writing the hypotheses after seeing the data. Choosing one-sided H1 or a new metric once you know which way the numbers went inflates false positives.
Treating "reject H0" as "H1 is big and important". With huge samples, trivial differences become significant; check the effect size and confidence interval.
Stating H0 about the sample instead of the population. The sample means are what they are; hypotheses are about the true conversion rates or averages behind them.
Testing a vague H1. "The redesign improves things" needs a metric, a direction and, ideally, a minimum effect worth detecting before the test starts.
Learn it in the course
p-value and statistical significance · What a p-value really says, what it doesn't, how the 0.05 significance level turns it into a decision, and statistical versus practical significance.
Two-sample t-test · The two-sample t-test (Student's t-test): comparing the means of two groups and deciding whether the gap is bigger than chance.
Power analysis and sample size · A power analysis tells you how many users an experiment needs to detect the effect you care about.
Every statistical test starts with two competing claims. The null hypothesisH0 is the boring default: there is no difference, no effect, nothing going on. The alternative hypothesisH1 is the claim you would like to support: there is a difference. In an A/B test on conversion:
H0:pB=pAH1:pB=pA
Why start from "nothing happened"? Because it is the only claim precise enough to calculate with. Assuming H0 is true, we can work out how much the results would wobble from sampling noise alone, and then ask how surprising our actual data would be in that world. That surprise is the p-value. If p<α (usually 0.05), the data are hard to square with H0, and we reject it in favour of H1. If not, we fail to rejectH0.
The wording matters. Failing to reject H0 does not prove there is no effect; it means this test, with this sample, did not find convincing evidence. Think of a court: "not guilty" is not the same as "innocent". Likewise, rejecting H0 says an effect is probably not zero, not that it is large or worth shipping; that is a question of effect size and practical significance.
The alternative can be two-sided ("B differs from A", the safe default) or one-sided ("B is better than A"). Choose it, along with α and the primary metric, before you look at the results. Picking the direction after seeing the data quietly doubles your chance of a false positive, the type I error.
CatChow tests a checkout redesign: 1,000 users see the old checkout (A), 1,000 see the new one (B).
Hypotheses, written before launch:
H0: the redesign doesn't change conversion, pB=pA
H1: it does change conversion, pB=pA (two-sided, α=0.05)
Results: A converts 52 of 1,000 (5.2%), B converts 68 of 1,000 (6.8%). The difference is 1.6 percentage points.
Test under H0. If there were truly no difference, both groups would share one conversion rate, estimated by pooling: 120/2,000=0.06. The standard error of the difference is
SE=0.06×0.94×(10001+10001)=0.0001128≈0.0106
so z=0.016/0.0106≈1.51. The two-tailed p-value is about 0.13.
Decision.0.13>0.05, so we fail to reject H0. The correct summary is not "the redesign doesn't work" but "a 1.6 pp gap this size shows up about 13% of the time from noise alone, so 1,000 users per group can't tell us yet". The 95% confidence interval for the difference, about −0.5 to +3.7 pp, still includes zero and also includes a meaningful win. The next step is a larger test, not a verdict.
