Rejecting a true H0: declaring an effect that does not exist. Its probability is alpha.
Example
Common mistakes
Reading α as "the chance our winner is fake". α is the false-positive rate among tests where nothing is going on; the share of your declared winners that are fake can be much higher, especially when most ideas don't work.
Ignoring type II errors. An underpowered test that "found nothing" often gets an idea killed; check the power before calling a result negative.
Peeking and stopping early. Checking daily and stopping at the first p < 0.05 pushes the real type I error rate well above 5%.
Testing many metrics and reporting the one that moved. With 20 independent metrics and no real effects, the chance of at least one false positive is about 64%.
Lowering α without adding data. Fewer false wins, but more missed real effects; only a bigger sample or less noisy metric improves both.
Learn it in the course
Two-sample t-test · The two-sample t-test (Student's t-test): comparing the means of two groups and deciding whether the gap is bigger than chance.
A hypothesis test ends with a decision, and a decision can be wrong in two different ways. Cross the truth with the decision and you get four outcomes:
H0 true (no real effect)
H0 false (real effect)
Reject H0 ("it works!")
type I error, false positive
correct: power, 1−β
Don't reject H0 ("no evidence")
correct
type II error, false negative
A type I error is declaring a win that isn't there: you ship a checkout change that actually does nothing. Its probability, when H0 is true, is the significance level α you chose, usually 5%. A type II error is missing an effect that is real: a good idea gets shelved because the test "showed nothing". Its probability is β, and 1−β is the test's statistical power. A common target is 80% power, meaning β=20%.
The two errors pull against each other. Lowering α from 5% to 1% makes false wins rarer but, with the same sample, makes misses more common. The main way to reduce both at once is more data (or less noisy metrics); that is what sample size calculations are for.
Which error is worse depends on the business. A false positive costs engineering time and can quietly hurt a metric you weren't watching. A false negative costs the revenue of an idea you never launched. Two situations inflate type I errors far above α: checking many metrics or variants (the multiple comparisons problem), and peeking at a running test and stopping the moment it looks significant.
CatChow's product team runs checkout tests with 1,000 users per group, two-sided, α=0.05.
Type I error: the copy tweaks that did nothing. Over a quarter the team tests 20 small button-copy changes. Suppose none of them has any real effect, so H0 is true every time. Each test still has a 5% chance of a false win:
expected false positives: 20×0.05=1
chance of at least one: 1−0.9520≈1−0.358=0.64
So there is about a 64% chance the quarterly report celebrates at least one "winning" copy change that is pure noise.
Type II error: the redesign that really works. Now suppose a redesign truly lifts conversion from 5% to 6%. With 1,000 users per group the standard error of the difference is about 0.05×0.95/1000+0.06×0.94/1000≈0.0102, so the real 1 pp lift is only about 0.01/0.0102≈0.98 standard errors. The test declares a win only when ∣z∣>1.96, so the power is roughly
P(Z>1.96−0.98)=P(Z>0.98)≈0.16
That gives β≈0.84: the test misses this real lift about 5 times out of 6. For 80% power the team would need about 8,150 users per group.
Same team, same α: in the first case too many false wins, in the second a near-certain miss. The fix for the second is sample size; the fix for the first is fewer, pre-registered tests or a multiple-comparison correction.
