Peeking in A/B testing turned a 5% false-positive rate into 27.6% in 100,000 simulated tests. When to stop an A/B test, and how sequential testing helps.
By Sergey BruhPublished 10 min read
A free course for PMs and marketers: short lessons, real product data and exercises with instant feedback.
Start the free course10 min read
The Bonferroni correction on a four-variant A/B/n test: why multiple comparisons create false winners, how Holm compares, and what it costs in traffic.
9 min read
How long to run an A/B test: divide the required sample by eligible daily traffic, round up to whole weeks, and plan for novelty effects and sales.
9 min read
What minimum detectable effect (MDE) means, relative vs absolute, and how to set it from business value and your traffic. With a worked example.
Sequential testing is the honest way to look at an A/B test before it ends: you plan the looks in advance and give each one a stricter threshold than 0.05, so the overall false-positive rate stays at 5%. Its opposite is peeking in A/B testing: checking the results while the test is still running and stopping as soon as the dashboard shows . It is a problem because every extra look gives random noise one more chance to cross the significance line, so you ship far more false winners than the 5% you agreed to. We simulated 100,000 tests in which both versions were identical: one look at the end called 5.0% of them "significant", while a look every day for four weeks called 27.6% "significant" at some point. Opening the dashboard is not the mistake. Making the decision on the first green number is.
CatChow, an online pet-food shop, tests a "Subscribe and save" block on the product page. Emily, the product manager, planned the test properly: the baseline conversion is 4%, the smallest lift worth shipping is 10% relative, and that takes about 39,500 visitors per group, or four weeks at 3,000 visitors a day. (The calculation is in our article on A/B test sample size.)
Then the dashboard does this:
On day 9 the variant is 14.4% ahead and the result is significant. Victor, the head of marketing, writes: "It's already green, just ship it." Emily waits. By day 28 the gap has shrunk to 0.9% and .
Here is the catch: this trajectory is one of the 100,000 tests from the simulation below, chosen because it shows the pattern clearly. In every one of them both groups had exactly the same true conversion rate. The block did nothing. Day 9 was noise that looked like a win.
An A/A test compares a version with itself, so every "significant" difference is a false positive by construction. The settings:
At every look we calculated the p-value on all the data collected so far and asked one question: was this test "significant" at least once?
One planned look keeps the promise: 5.0%. A harmless-sounding weekly check already gives 12.7%. With daily checks more than one test in four turns green at some point.
Three more facts about the 27,550 false alarms from daily checking:
About half of the false alarms favoured B. A team that ships whenever B turns green would have released a do-nothing change in 13.8% of these tests, against 2.5% with a single look.
A p-value is not a progress bar that fills up steadily. While data comes in, the gap between the groups wanders up and down, and the p-value wanders with it, most of all early on, when the sample is small.
The 5% guarantee covers one pre-planned moment: if there is no real effect, the chance that the result sits beyond the line at that moment is 5%. "Stop as soon as it's green" asks a different question: will the result cross the line at any moment? Compare "will it rain at noon next Saturday?" with "will it rain at any time this month?". The second is far more likely, even though the weather is the same.
The rule is also one-sided. A green result ends the test; a grey one gets another try tomorrow. Noise only has to get lucky once.
This is a relative of the multiple comparisons problem, where testing many metrics produces a "winner" by chance. But looks overlap (day 10 contains all of day 9's data), so 28 looks give 27.6% and not the of 28 independent attempts.
Not looking at all is a bad idea too: a broken variant costs real money.
What you should not do is use the p-value or the lift of the primary metric, the one the decision is based on, as a trigger to stop and ship. For guardrails, agree on the stop conditions before launch ("we stop if checkout errors double"), so that they don't become a second way to peek.
"We'll ship only if it stays significant two days in a row." In the simulation this rule still flagged 17.2% of A/A tests. Better than 27.6%, but more than three times the nominal rate.
"Not significant yet, let's give it another week." Extending a test until it turns green is the same peeking, just at the other end.
Trusting the lift from a test that was stopped early. Even when the effect is real, the first crossing happens when noise is pushing in your favour, so the lift that goes into the forecast is too optimistic.
Option 1: fix the sample size and the duration in advance. Calculate the sample with the A/B test sample size calculator, round up to whole weeks and write four things into the test brief: the primary metric, the sample size, the end date and the decision rule. Analyse the primary metric once, at the end, and report the lift together with its confidence interval.
Option 2: use sequential testing. If an early decision is really valuable, use a method designed for repeated looks. The idea is simple: you set the looks in advance and spread the 5% error budget across them, so each look has a stricter threshold than 0.05. In our simulation four weekly looks with at each of them flagged 5.0% of A/A tests, the same as one look at 0.05.
The price is sensitivity. With a real lift from 4.0% to 4.4%, the single look on day 28 found the winner in 82.5% of simulated tests and the four-look scheme in 74.7%, but its tests took 19.7 days on average instead of 28. Many experimentation platforms have a sequential mode built in; check which statistics are behind the green badge in yours.
Victor doesn't need a lecture on stopping rules. He needs a reason, a date and a promise:
"Green on day 9 is not the result yet. In tests like ours where nothing really changed, more than one in four turns green at some point if you check every day, and four out of five of those are no longer green at the end. We agreed on four weeks because that is what it takes to see a 10% lift. On day 28 I'll bring the final number with its range, and if it's a win, we ship the next morning."
Name the costs too: waiting takes 19 more days, while a false winner means a feature that does nothing and a forecast built on a lift that was never there.
Yes, as long as the look can't change the decision. Check that the test runs correctly, that the split is even and that guardrail metrics are healthy. The primary metric is analysed at the planned end, or at the looks your sequential method defines.
Only under a rule set in advance. A far stricter threshold does protect you: daily looks with flagged 0.9% of A/A tests in our simulation. But fixing that threshold before the test is exactly what a sequential method does. Deciding after you have seen the data that the p-value is "small enough" is still peeking.
It is a way to analyse an A/B test several times while it runs without inflating false positives. You fix the number of looks and their thresholds before launch, so each look uses a stricter cut-off than 0.05 and the whole test keeps a 5% false-positive rate. The trade-off: a little less sensitivity in exchange for stopping early when the effect is clear. In our simulation four weekly looks at kept the A/A false-positive rate at 5.0%.
Stop and report what you have: the observed lift and its confidence interval, which shows how large an effect could still be hiding. Don't extend the test just to chase significance. If the question is still important, plan a new test with a larger sample.
The free lesson Pitfalls: peeking, multiple testing, novelty effect lets you practise on CatChow's banner test that was "already significant" after two days, with quizzes and a practice task checked by AI. It is part of the free course Statistics for Product Managers and Marketers.
