# Many tests at once

> The complement rule, multiplying independent chances, the chance of at least one false alarm, counting pairs and the Bonferroni split.

- Course: [Math for Statistics and Unit Economics](https://productovi.com/courses/math-basics.md)
- Module: Probability and distributions
- Lesson 28/36, ~26 min
- Source: https://productovi.com/courses/math-basics/many-tests-at-once

## Five metrics, one test

**Alex Tutor:** Still on Klasna's product team. The growth team tested a new price page and checked five metrics: sign-ups, trial bookings, package purchases, Club conversion and average package size. One came out "significant" at α = 0.05.

**Sofia:** One out of five sounds like a real effect.

**Alex Tutor:** Let’s check. Suppose the page changed nothing at all. Each check then has a 0.95 chance of staying quiet. What’s the chance that all five stay quiet?

**Alex Tutor:** For independent events, "this and that" means multiplying: 0.95 × 0.95 = 0.9025 for two, and 0.95⁵ = 0.774 for five. The complement, at least one false alarm, is 1 − 0.774 = **0.226**, almost a quarter.

**Alex Tutor:** Before the formula, try the multiplying rule on two everyday events.

> **Exercise:** At Klasna, a tutor confirms a lesson the day before with probability 0.7, and the learner shows up with probability 0.9. If the two are independent, what share of lessons are both confirmed and attended?

$$
P(\text{at least one false alarm}) = 1 - (1 - \alpha)^m
$$

*α: the false-alarm rate of one check; m: the number of independent checks.*

**Sofia:** So "at least one" is easiest through "none".

**Alex Tutor:** Always. Counting every way to get one, two or more alarms is long; "none" is one product, and 1 minus it gives the rest.

**Sofia:** But our five metrics aren’t independent. Sign-ups and trial bookings move together.

**Alex Tutor:** True, and related metrics give fewer separate chances to be fooled, so the real figure sits somewhere between 5% and 22.6%. The formula gives the top of that range, which is the safe number to keep in mind.

> **Exercise:** A dashboard compares 10 metrics before and after a release, each at α = 0.05. If nothing really changed, what is the chance of at least one false alarm, in percent, to one decimal?

> **Exercise:** Three independent metrics each have a 5% chance of a false alarm, so the chance of at least one false alarm is exactly 15%.

**Alex Tutor:** And the objection about related metrics, as a quick check.

> **Exercise:** Klasna’s five price-page metrics are closely related (sign-ups and trial bookings move together). So the chance of at least one false alarm is higher than the 22.6% that 1 − 0.95⁵ gives.

## How many false alarms to expect

**Alex Tutor:** A second way to see it: the expected number of false alarms is m × α. Five metrics give 5 × 0.05 = 0.25. Twenty give 20 × 0.05 = 1: on average, one fake winner per twenty checks.

![At α = 0.05, the chance of at least one false alarm is 5% for 1 check, 22.6% for 5, 40.1% for 10 and 64.2% for 20; expected false alarms m × 0.05.](https://productovi.com/courses/math-basics/diagrams/m7-l4-false-alarm-growth.png)

*Each extra check is another ticket in a lottery nobody wanted to play.*

**Sofia:** So with twenty metrics, one "winner" is exactly what pure noise looks like.

**Alex Tutor:** Yes. That’s why slicing a test into many segments after the fact almost always "finds" something. The finding may be real, but one lucky slice isn’t evidence yet.

![A row of calm cats on a bench, each with a small bell; one cat rings its orange bell excitedly at an empty spot.](https://productovi.com/courses/math-basics/inline/m7-l4-i1.webp)

*With enough watchers, someone always rings at nothing.*

> **Exercise:** An analyst splits a null result into 40 segments (cities, devices, ages) and tests each at α = 0.05. How many "significant" segments should we expect even if the change did nothing?

## Counting pairs

**Alex Tutor:** Tests with several variants multiply checks too. The price page had four versions, A, B, C and D. Comparing every version with every other: each meets three others, 4 × 3 = 12, and halving removes A–B counted again as B–A.

$$
C(k, 2) = \frac{k(k-1)}{2}
$$

*The number of pairs among k items: 4 variants give 6, 5 give 10, 10 give 45.*

> **Exercise:** A test has five variants, and the team wants to compare every variant with every other. How many comparisons is that?

## The Bonferroni split

**Alex Tutor:** The simplest fix is to share the 5% between the checks: test each one at α ÷ m. With six pairs, 0.05 ÷ 6 ≈ 0.0083 per pair. Then the chance of any false alarm stays at about 5% or below.

![Four variants give 4 × 3 ÷ 2 = 6 pairs; with a Bonferroni split each pair is tested at 0.05 ÷ 6 ≈ 0.0083, or its p is multiplied by 6.](https://productovi.com/courses/math-basics/diagrams/m7-l4-pairs.png)

*Count the comparisons first, then share α between them.*

**Sofia:** And the software shows "adjusted p" instead. Same thing?

**Alex Tutor:** The same rule turned around: adjusted p = min(1, m × p), compared with the usual 0.05. A pair with p = 0.012 among six gives 6 × 0.012 = 0.072: not significant any more. The min caps it at 1, because a probability can’t exceed 1.

> **Exercise:** Among six pairwise comparisons, the pair B vs D has p = 0.004. What is its Bonferroni-adjusted p?

**Alex Tutor:** Bonferroni is strict: it protects against false alarms and makes real effects harder to see. The better defense is cheaper: pick one primary metric before the test and treat the rest as hints for the next test.

**Alex Tutor:** Last question before your turn: which plan would you sign off?

> **Exercise:** Klasna plans a price-page test with four variants and 20 metrics. Which plan keeps the false-alarm risk near 5% without making real effects very hard to see?

## Your turn

**Alex Tutor:** A marketer writes in the team chat: "The new ad worked nowhere overall, but in Lviv it’s significant, p = 0.03! We checked 20 cities. Let’s roll it out in Lviv."

> **Exercise:** A marketer writes: "The new ad worked nowhere overall, but in Lviv it’s significant, p = 0.03! We checked 20 cities. Let’s roll it out in Lviv." In two or three sentences, explain why the Lviv result is weak evidence and what you would do instead. Use numbers.

## Where you will meet this

> **In short**
>
> - Independent events: "and" multiplies, 0.95 × 0.95 × … = 0.95^m.
> - **At least one** = 1 − none: P(false alarm somewhere) = 1 − (1 − α)^m. Five checks: 22.6%; twenty: 64.2%.
> - Expected false alarms = **m × α**.
> - Pairs among k variants: **k(k − 1)/2**.
> - **Bonferroni:** test each at α ÷ m, or adjusted p = min(1, m × p). Best defense: one primary metric chosen in advance.

> **These lessons use it**
>
> - [Pitfalls: peeking, multiple testing, novelty effect](/courses/statistics-for-pm-and-marketers/ab-testing-pitfalls): 1 − (1 − α)^m and the Bonferroni correction.
> - [One-way ANOVA: comparing many groups](/courses/statistics-for-pm-and-marketers/anova): why many pairwise t-tests mislead.
> - [A/B test plan: hypothesis, metrics, guardrails](/courses/product-analytics/test-design): one primary metric chosen before the test.
> - [A/B/n test results calculator](/tools/abn-test-results): pairwise comparisons with the correction built in.

**Alex Tutor:** Next time: the normal curve, where 1.96 comes from and why area means probability.

- Previous lesson: [Expected value and scenarios](https://productovi.com/courses/math-basics/expected-value.md)
- Next lesson: [The normal curve](https://productovi.com/courses/math-basics/the-normal-curve.md)

---
Productovi Learning. Free, no sign-up needed to read.