# Pitfalls: peeking, multiple testing, novelty effect

> The most common ways real A/B tests go wrong, and how to avoid them.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: A/B testing in practice
- Lesson 12/24, ~37 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/ab-testing-pitfalls

## Checking in on the dashboard, again

**Alex Tutor:** Another CatChow test is running, and as the product team's numbers person you've been watching it closely.

**Sofia:** The banner test has been live for 2 days and it's already significant! p = 0.03. Ship it?

**Alex Tutor:** How many times have you checked that dashboard since it launched?

**Sofia:** ...maybe six? I check every few hours.

**Alex Tutor:** That's **peeking** - checking results repeatedly and stopping the moment you like what you see. It quietly breaks the 5% error rate you're relying on.

![A cat opens an oven door to peek at a tray of fish-shaped treats that are still pale and half-baked, with the timer far from done.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/ab-testing-pitfalls-illustration-2.webp)

*Every peek is a new chance to pull the tray out too early.*

> **Peeking (optional stopping)**
>
> Checking a test's results repeatedly and stopping as soon as they cross a significance threshold. Each look is a fresh chance for a false positive, so the true error rate climbs well past the nominal 5%.

**Alex Tutor:** With a fixed 5% threshold checked repeatedly, the *true* chance of a false alarm somewhere along the way is much higher than 5%.

> **Exercise:** Stopping a test as soon as it first reaches p < 0.05, then declaring victory, keeps your actual false-positive rate at 5%.

**Alex Tutor:** There's a cousin problem: testing many metrics at once, not just peeking at one over time.

**Sofia:** We do track a lot of metrics on every test - conversion, AOV, retention, NPS, CTR...

**Alex Tutor:** Right - and if you test 5 independent metrics at α = 0.05 each, assuming none of them truly changed, at least one will likely come up 'significant' by chance.

**Alex Tutor:** This is the **multiple comparisons problem**: each test has a 5% false-positive chance, but run enough of them and the *combined* chance climbs fast.

![A blindfolded cat throws many darts at a row of five targets; most miss, but one lands right in an orange bullseye.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/ab-testing-pitfalls-illustration-3.webp)

> **Exercise:** You test 5 independent guardrail metrics, each at α = 0.05, and none of them truly changed. What is the probability that at least one shows a false positive, as a percentage?

**Alex Tutor:** One standard fix: the **Bonferroni correction** - divide your significance threshold by the number of tests, so the combined risk stays near 5%.

$$
\alpha_{\text{per test}} = \frac{\alpha_{\text{family}}}{m}
$$

*Bonferroni correction: divide the desired overall (family-wise) significance level by the number of comparisons, m.*

**Alex Tutor:** For those same 5 metrics at a family-wise 0.05, each metric now needs its own p below 0.05 / 5.

**Sofia:** That makes each individual test stricter to pass.

**Alex Tutor:** Exactly - it's conservative on purpose. Fewer false alarms, at the cost of needing a clearer signal to call anything 'significant.'

> **Exercise:** Using the Bonferroni correction for 5 comparisons at a family-wise α = 0.05, what per-test significance threshold should you use?

**Alex Tutor:** Let’s check that it actually works. With the stricter bar, what’s the combined risk now?

> **Exercise:** After the Bonferroni correction, each of the 5 independent metrics is tested at α = 0.01. If none of them truly changed, what is the chance that at least one still shows a false positive, as a percentage?

**Alex Tutor:** And now read one result against that bar. This is where people slip most often.

> **Exercise:** A test tracks 10 metrics with a Bonferroni correction at a family-wise α = 0.05. One metric comes out at p = 0.008. It counts as statistically significant.

## Novelty and other traps

**Alex Tutor:** A different problem now: sometimes an effect is real on day one, then quietly disappears.

**Sofia:** Like people just noticing the new banner because it's new?

**Alex Tutor:** Exactly - the **novelty effect**. Regular users react to *change itself*, not necessarily to the change being better. That reaction fades.

![On the left, a cat leaps excitedly at a shiny new food bowl; on the right, the same cat lies bored next to the very same bowl.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/ab-testing-pitfalls-illustration-4.webp)

*Some reactions are to the change itself, and they fade once it stops being new.*

**Alex Tutor:** It's why day-one results can flatter a losing idea, and why you should watch a metric's trend over the whole test, not just its opening spike.

**Alex Tutor:** Here’s a banner test that ran three weeks: +3 points in week 1, +1 in week 2, +0.2 in week 3. Two ways to read it.

> **Exercise:** A banner test ran three weeks. The lift in conversion was +3 points in week 1, +1 in week 2 and +0.2 in week 3, about +1.4 points averaged over the test. Which reading is right?

**Alex Tutor:** (Side note: these pitfalls apply to any metric - an arithmetic average, a proportion, whatever you're tracking - not just conversion rate.)

> **Exercise:** What's the safest way to handle wanting to check results before your pre-computed sample size is reached?

## Matching the pitfalls

**Alex Tutor:** Let's make sure the vocabulary from this whole lesson is solid - match each pitfall to its description.

**Sofia:** There's a lot of near-synonyms here, ngl.

**Alex Tutor:** Fair - that's exactly why this matters. Getting them mixed up in a real retro is an expensive mistake.

**Sofia:** Alright, let's match them.

> **Exercise:** Match each pitfall to its description.

**Alex Tutor:** Good. Now let's separate the real causes of a false 'winner' from the things that don't actually cause one.

**Sofia:** I have a feeling one of the options is a trap.

**Alex Tutor:** You'd be right. Read carefully.

**Sofia:** Okay, deep breath.

> **Exercise:** Which of these are classic causes of a false 'winning' variant? Select all that apply.

## Writing it up

**Alex Tutor:** Let's put this into a message you could actually send a teammate.

**Sofia:** The 'already significant after 2 days' situation from the start?

**Alex Tutor:** That one exactly.

**Sofia:** Okay, let me write it.

> **Exercise:** CatChow's dashboard shows the banner test 'already significant' after 2 days, but the pre-registered sample size needs 2 weeks. In 2-3 sentences, explain to a teammate why they shouldn't ship yet.

**Alex Tutor:** Solid. One more scenario - it stacks a few of today's pitfalls together.

**Sofia:** Great, more than one at once.

**Alex Tutor:** CatChow ran a 'gift-wrap' upsell test for 3 weeks, tracking 12 metrics.

**Sofia:** That's already two red flags from today, isn't it?

> **Exercise:** Write a one-paragraph recommendation for the team.

**Alex Tutor:** That's the module, really: question the headline number, check if it's actually significant, and check *how* you got there.

**Sofia:** I'm going to look at every test result differently now.

**Alex Tutor:** That's the goal. Go forth and stop peeking at dashboards every hour.

- Previous lesson: [Conversion rates and the two-proportion z-test](https://productovi.com/courses/statistics-for-pm-and-marketers/conversion-rate-tests.md)
- Next lesson: [One-way ANOVA: comparing many groups](https://productovi.com/courses/statistics-for-pm-and-marketers/anova.md)

---
Productovi Learning. Free, no sign-up needed to read.