# Bonferroni correction in A/B/n testing: compare several variants without a false winner

> The Bonferroni correction on a four-variant A/B/n test: why multiple comparisons create false winners, how Holm compares, and what it costs in traffic.

- Source: https://productovi.com/blog/abn-test-multiple-comparisons
- By: Sergey Bruh
- Published: 2026-10-02
- Topics: Multiple testing, Post-hoc comparisons, Tests for proportions

The Bonferroni correction divides the 5% significance level by the number of comparisons (or, equivalently, multiplies each p-value by it), so that the chance of any false positive across the whole test stays at 5%. You need it in A/B/n testing, where you compare a control with two or more alternatives at once, and **every extra variant adds comparisons, each of which is one more chance for a false positive, so the variant with the lowest p-value cannot simply be declared the winner**. With four variants there are six possible pairs. Test each pair at the usual 5% level and the chance of at least one false positive is about 20–26% instead of 5%. The fix is to decide in advance which comparisons you will make and to apply a correction (Bonferroni or its step-by-step version, Holm) that keeps the risk for the whole test at 5%. It works, and it costs traffic.

## Four buttons and one tempting winner

CatChow, an online pet-food shop, tests the label on its checkout button. Emily, the product manager, had three new ideas, so all three went into the test next to the current label. It ran for four weeks at about 3,000 checkout visitors a day, split equally:

| Variant     | Button label      | Visitors | Orders | Conversion |
| ----------- | ----------------- | -------- | ------ | ---------- |
| A (control) | Place order       | 21,000   | 840    | 4.00%      |
| B           | Buy now           | 21,000   | 861    | 4.10%      |
| C           | Complete my order | 21,000   | 948    | 4.51%      |
| D           | Get it delivered  | 21,000   | 1,006  | 4.79%      |

Emily runs a [two-proportion z-test](/courses/statistics-for-pm-and-marketers/conversion-rate-tests/theory) for every pair and gets four "significant" results out of six. D has the highest conversion and the lowest p-value against the control, so the slide almost writes itself: "D wins, +19.8%". Two questions come first. How many of those four results could be noise? And is D really better than C?

## The multiple comparisons problem: why the risk grows with every comparison

One comparison at $\alpha = 0.05$ has a 5% chance of a false positive when there is no real difference. With $m$ independent comparisons, the chance that at least one of them fires is:

$$
P(\text{at least one false positive}) = 1 - (1 - \alpha)^m
$$

| Variants | Comparisons with control only | Risk  | All pairs | Risk  |
| -------- | ----------------------------- | ----- | --------- | ----- |
| 2        | 1                             | 5.0%  | 1         | 5.0%  |
| 3        | 2                             | 9.8%  | 3         | 14.3% |
| 4        | 3                             | 14.3% | 6         | 26.5% |
| 5        | 4                             | 18.5% | 10        | 40.1% |
| 6        | 5                             | 22.6% | 15        | 53.7% |

The formula assumes independent comparisons. Inside one test they share data (every pair with A uses the same control group), so the real risk is somewhat lower: a simulation gives about 12% for all pairs of three variants and about 20% for all pairs of four. Either way it is several times the 5% you agreed to. This is the **multiple comparisons problem**: the lowest of six [p-values](/blog/p-value-explained) means much less than the same p-value in a plain A/B test.

## The two-step approach

**Step 1: an overall test.** Is there any difference between the variants at all? For conversion rates this is a chi-square test on the table of orders and non-orders for all four variants. CatChow's data gives $\chi^2 = 20.49$ with 3 degrees of freedom and $p \approx 0.0001$. The variants are not all the same, but the test does not say which ones differ. For averages such as order value, the same logic is [ANOVA followed by post hoc tests](/courses/statistics-for-pm-and-marketers/anova/theory).

**Step 2: pairwise comparisons with a multiple testing correction.** Compare the pairs, but hold each p-value to a stricter bar, so that the risk of at least one false positive across the whole family of comparisons stays at 5%. The protection comes from this step: the correction works with or without an overall test.

## How the Bonferroni correction works, and the Holm alternative

**Bonferroni** splits the 5% evenly. With six comparisons each one has to pass $0.05 / 6 \approx 0.0083$. Equivalently, multiply every p-value by 6 (capping the result at 1) and compare it with 0.05.

**Holm** spends the same 5% step by step. Sort the p-values from smallest to largest, multiply the smallest by 6, the next by 5, then by 4 and so on. Walk down the list and stop at the first adjusted value of 0.05 or more: that comparison and everything after it are not significant.

| Pair   | Difference, pp | Raw p    | Bonferroni p | Holm p |
| ------ | -------------- | -------- | ------------ | ------ |
| D vs A | 0.79           | < 0.0001 | 0.0005       | 0.0005 |
| D vs B | 0.69           | 0.0006   | 0.0036       | 0.0030 |
| C vs A | 0.51           | 0.0090   | 0.0543       | 0.0362 |
| C vs B | 0.41           | 0.0365   | 0.2192       | 0.1096 |
| D vs C | 0.28           | 0.1790   | 1            | 0.3581 |
| B vs A | 0.10           | 0.6032   | 1            | 0.6032 |

Adjusted values are calculated from unrounded p-values. What the table says:

- Without a correction, four pairs pass 0.05. After Bonferroni two remain: D beats A and D beats B.
- Holm also confirms C vs A (0.0362), which Bonferroni narrowly rejects (0.0543). Both methods hold the family risk at 5%, but Holm is never stricter than Bonferroni, so it misses fewer real effects.
- D has the highest conversion, but D vs C is far from significant under either method. This test cannot tell C and D apart.

![Of six pairwise comparisons, four pass p < 0.05 without a correction and only two survive Bonferroni: D beats A and B, but the test cannot tell D from C](/blog/abn-test-multiple-comparisons/diagram.png)

The honest summary is "D beats the current button and B; there is no evidence that D is better than C", not "D wins".

To check your own numbers, use the free [A/B/n test results calculator](/tools/abn-test-results). It takes three to six variants, compares every pair with a two-sided z-test, applies the Bonferroni correction and adds an adjusted interval for each difference. For CatChow's counts it returns the Bonferroni values from the table and the verdict that no single variant outperformed all the others. It has no overall test, Holm correction or control-only mode.

## Control only or all pairs?

If the only question is "does any new label beat the current one?", three comparisons are enough: B, C and D against A. Bonferroni then multiplies by 3, and C vs A gets an adjusted p of 0.0271, which is significant.

The price is the conclusion you are allowed to draw. You may say "C and D each beat the control", but nothing about D against C: naming the single best variant needs all pairs. The family has to be fixed before launch. Choosing the smaller one after seeing the results is a quieter way of picking the lowest p-value.

## What it costs in traffic

Two things get more expensive at once: the threshold is stricter, and traffic is split more ways. For a 4% baseline, a 10% relative lift (4.0% to 4.4%) and 80% power for each comparison:

| Design                   | Threshold per comparison | Users per variant | Total users | Days at 3,000 a day |
| ------------------------ | ------------------------ | ----------------- | ----------- | ------------------- |
| A/B                      | 0.05                     | 39,475            | 78,950      | 27                  |
| 3 variants, all pairs    | 0.0167                   | 52,654            | 157,962     | 53                  |
| 4 variants, control only | 0.0167                   | 52,654            | 210,616     | 71                  |
| 4 variants, all pairs    | 0.0083                   | 60,904            | 243,616     | 82                  |
| 5 variants, all pairs    | 0.005                    | 66,955            | 334,775     | 112                 |

Compared with a plain A/B test, four variants with all pairs need 54% more users per variant and twice as many groups: about 3.1 times the traffic, or 12 full weeks instead of 4.

So what did CatChow's four weeks buy? With 21,000 visitors per variant and a threshold of 0.0083, the test had 80% power only for lifts of about 17% or more. The same four weeks as a plain A/B test, with 42,000 visitors per group, would have caught a lift of about 10%.

The [A/B test sample size calculator](/tools/ab-test-sample-size) plans a two-variant test. For a rough A/B/n estimate, set its significance level to 1% (close to the corrected 0.83%, so slightly optimistic) and multiply the users per variant by the number of variants.

## The same problem in metrics and segments

Variants are only one way to multiply comparisons. Check one A/B test on 10 metrics and the chance of at least one false positive is about 40%. Slice it by 8 segments (device, channel, new or returning customers) and it is about 34% for the segments alone. "The new button works for returning Android users" is usually the multiple comparisons problem in disguise.

Choose one primary metric in advance and treat segment findings as hypotheses for the next test. [Peeking](/blog/ab-test-peeking) at the results every day multiplies comparisons too. A correction for variants covers none of this.

## Common mistakes

**Declaring the highest rate the winner.** The highest observed conversion is a fact about this sample. A winner also needs a significant difference from the other variants after the correction.

**Running each pair through a separate A/B calculator.** Each calculation knows nothing about the others, so nothing corrects for their number.

**Dropping variants after the fact.** Removing B and C because they "obviously lost" and re-running D vs A as a plain A/B test is choosing comparisons after seeing the data.

**Reading "not significant" as "equal".** The adjusted interval for D minus C runs from −0.27 to 0.82 percentage points: D may be clearly better, or slightly worse.

## What to do in practice

1. Before launch, write down the variants, the primary metric, the family of comparisons (control only or all pairs) and the correction.
2. Plan the sample with the corrected threshold and the real traffic split, then round up to full weeks.
3. Analyse once, at the end, and report adjusted p-values with intervals.
4. Prefer fewer, bolder variants. Button labels that differ by a word or two will differ from each other by a fraction of a percentage point, and telling them apart takes months. Variants that differ in substance have a chance of an effect you can detect. If traffic is limited, pick the strongest idea by judgement or customer research and run a plain A/B test.

## Key takeaways

- In A/B/n testing the risk of at least one false positive grows with the number of comparisons: about 1 in 4 for six independent comparisons at 5%.
- Fix the family of comparisons and the correction before launch.
- Bonferroni multiplies every p-value by the number of comparisons. Holm does it step by step and misses fewer real effects.
- The highest conversion is not a winner until it differs significantly from the other variants.
- Four variants with all pairs need about three times the traffic of an A/B test.

## FAQ

### What is A/B/n testing, and how is it different from multivariate testing?

A test where traffic is split between a control and two or more alternatives of the same element, for example four button labels. The "n" stands for any number of variants. A multivariate test is different: it varies several elements at once and compares their combinations.

### Bonferroni or Holm: which correction should I use?

Both keep the risk for the whole family of comparisons at the chosen level. Holm is never stricter, so it is the better default for p-values. Bonferroni is simpler to explain and comes with matching adjusted intervals. Choose before you see the results.

### Do I need a correction if I only compare variants with the control?

Yes. Three comparisons with the control at 5% each still give up to a 14% risk of at least one false positive. The correction is milder than for all pairs, because the family is smaller: three comparisons instead of six.

## Learn it hands-on

Multiple comparisons, peeking and the novelty effect break tests that look perfectly sound. The free lesson [Pitfalls: peeking, multiple testing, novelty effect](/courses/statistics-for-pm-and-marketers/ab-testing-pitfalls/theory) walks through them on CatChow's tests, with quizzes and a practice task checked by AI. It is part of the free course _Statistics for Product Managers and Marketers_.

## Related lessons

- [Pitfalls: peeking, multiple testing, novelty effect](https://productovi.com/courses/statistics-for-pm-and-marketers/ab-testing-pitfalls.md)
- [One-way ANOVA: comparing many groups](https://productovi.com/courses/statistics-for-pm-and-marketers/anova.md)
- [Conversion rates and the two-proportion z-test](https://productovi.com/courses/statistics-for-pm-and-marketers/conversion-rate-tests.md)
- [Power analysis and sample size](https://productovi.com/courses/statistics-for-pm-and-marketers/power-and-sample-size.md)
- [p-value and statistical significance](https://productovi.com/courses/statistics-for-pm-and-marketers/p-value.md)