Peeking in A/B testing turned a 5% false-positive rate into 27.6% in 100,000 simulated tests. When to stop an A/B test, and how sequential testing helps.
What does a p-value mean? Here is the p-value meaning in one sentence: it is the probability of seeing a difference at least as large as the one you observed if, in reality, there were no difference at all. A small p-value (conventionally below 0.05) means your result would be surprising in a world where the change did nothing, so you have some evidence that it did something; that is what "statistically significant" means. It does not tell you the probability that your variant is better, how big the effect is, or whether the result matters for the business. Most bad decisions around A/B tests come from reading more into a p-value than it can carry.
How to interpret a p-value: a quick example
CatChow, an online pet-food shop, tests a "free shipping over 30 dollars" banner on the product page. After two weeks:
Group
Visitors
Orders
Conversion
Control (no banner)
10,000
400
4.0%
Variant (banner)
10,000
460
4.6%
The variant converts 0.6 percentage points better, a 15% relative lift. Is that real, or could random noise produce a gap like this?
A two-proportion z-test gives z≈2.09 and p≈0.036.
Translated: if the banner truly had zero effect, you would see a gap of 0.6 points or more (in either direction) in about 3.6% of experiments this size, just from the luck of which visitors landed in which group. That is fairly unlikely, so the team has reasonable evidence that the banner helps.
How a p-value is calculated
Every p-value comes from the same three steps:
Assume the boring world. The null hypothesis H0 says the change does nothing: both groups have the same true conversion rate.
Measure how far the data is from that world. Turn the observed difference into a test statistic, here z=SEp^2−p^1, where SE is the standard error of the difference. The bigger ∣z∣, the further the data sits from "no effect".
Ask how often the boring world produces something this extreme. That share is the p-value.
Then you compare it with a threshold chosen before the test, the significance level α (usually 0.05). If p<α, you call the result statistically significant.
What a p-value does not mean
It is not the probability that the null hypothesis is true
"p=0.036, so there is a 3.6% chance the banner does nothing" is the most common misreading, and it is wrong. The p-value is calculated assumingH0 is true; it cannot also tell you how likely H0 is.
Here is why that matters. Suppose only 1 in 10 ideas your team tests actually works, your tests have 80% power and you use α=0.05. Out of 1,000 tests:
100 ideas really work; the tests catch 80 of them.
900 ideas do nothing; 5% of those, 45 tests, still come out "significant" by chance.
So 125 tests are "winners", and 45 of them are false: 36% of your significant results are noise, even though every one of them had p<0.05. The p-value alone can't tell you which ones.
Of 1,000 tests where 1 in 10 ideas works, 80 real effects and 45 false alarms reach p < 0.05, so 36% of the 125 significant results are noise
It is not the probability that the variant is better
"97% chance the banner wins" is the same mistake in reverse. Statements like that need a Bayesian analysis with an explicit prior. Some A/B testing tools report them; a classic p-value does not.
It is not the size of the effect
A tiny, useless effect can have a very small p-value if the sample is large enough. With a million visitors per group, a change from 4.00% to 4.08% conversion gives p≈0.004. Highly significant, and possibly not worth the engineering time to maintain.
The reverse is also true. Take the banner test again, but with only 1,000 visitors per group and the same 4.0% vs 4.6% rates: now p≈0.51. The effect is exactly the same size; the test is just too small to separate it from noise.
A large p-value is not proof of no effect
p=0.51 does not mean "the banner doesn't work". It means "this test couldn't tell". Absence of evidence is not evidence of absence, especially in an underpowered test. Before you kill an idea, check whether the test had enough users to detect the effect you care about.
Statistical vs practical significance
Two different questions:
Statistical significance: is the effect distinguishable from noise?
Practical significance: is it big enough to matter?
You need both. The fix is to always look at the effect size and its confidence interval next to the p-value. For the banner test, the 95% confidence interval for the difference is roughly +0.04 to +1.16 percentage points. That range says two things at once: the effect is probably positive (the interval doesn't include zero, which matches p<0.05), and it could be anything from negligible to large. That is far more useful for a decision than "p=0.036".
Traps that make p-values lie
Peeking. If you check the results every day and stop the moment p drops below 0.05, the real false-positive rate climbs well above 5% (see why peeking at A/B tests breaks them). P-values assume you analyse once, at the sample size you planned.
Many metrics, one winner. Test 10 independent metrics with no real effect at α=0.05, and the chance that at least one comes out "significant" is 1−0.9510≈40%. With 20 metrics it is about 64%. Pick one primary metric before launch, or correct for multiple comparisons.
Slicing after the fact. "Not significant overall, but significant for iOS users in Kyiv on weekends" is almost always a false positive. Segments you didn't plan in advance are ideas for the next test, not conclusions.
Treating 0.049 and 0.051 as opposites. The 0.05 line is a convention, not a law of nature. A result at 0.051 is almost exactly as strong as one at 0.049.
How to report results to stakeholders
Stakeholders don't need the definition of a p-value. They need to know what happened, how sure you are, and what you recommend. A template that works:
The banner increased conversion from 4.0% to 4.6% (+0.6 pp, +15%). The 95% confidence interval for the lift is +0.04 to +1.16 pp, so the effect is very likely positive but could be small. The result is statistically significant (p=0.036). With about 43,000 product-page visitors a month, the point estimate means roughly 250 extra orders a month; at the low end of the range the gain would be marginal. Recommendation: ship it and keep monitoring conversion and average order value for four weeks.
Notice what it does: effect first, uncertainty as a range, p-value as supporting detail, and a decision tied to business impact. Avoid "the test proved", "97% confidence the banner wins" and "no effect" for a non-significant result; say "we couldn't detect an effect larger than X" instead.
Key takeaways
The p-value is the probability of data at least this extreme if there were no real effect.
It is not the probability that the null hypothesis is true or that the variant wins.
A small p-value doesn't mean a big effect; a large one doesn't mean no effect.
Always report the effect size and a confidence interval alongside the p-value.
Decide α, the primary metric and the sample size before launch, and don't peek.
FAQ
What does a p-value less than 0.05 mean?
It means that if the change had no real effect, results as extreme as yours would appear less than 5% of the time. By convention you then call the result statistically significant. It does not mean there is a 95% chance your hypothesis is right.
Is a p-value of 0.06 "almost significant"?
It is weak evidence, about as weak as 0.04 is. Rather than arguing about the threshold, look at the confidence interval and the business stakes. If the plausible range includes effects that would matter, consider a longer or larger test instead of declaring victory or defeat.
Can a p-value be zero?
Not exactly. Tools may display "p = 0.000" or "p < 0.001" when the value is very small. Report it as "p < 0.001", and remember that with huge samples even trivial effects reach tiny p-values.
Why do A/B testing tools show "probability to beat control" instead?
Those tools use Bayesian methods, which answer a different question: given the data and a prior assumption, how likely is the variant to be better. It can be easier to explain, but it is not the same number as a p-value, and it also depends on choices you should understand before trusting it.
Learn it hands-on
The free lesson p-value and significance builds the p-value from scratch on a CatChow experiment, with quizzes that target exactly the misreadings above and a practice task checked by AI. It is part of the free course Statistics for Product Managers and Marketers.
To check your own numbers, the free statistical significance calculator takes the counts of two groups and returns the p-value with a one- or two-sided test, a plain-language reading and a picture of where your result falls on the curve.
