# One-way ANOVA: comparing many groups

> Testing three or more variants at once with a one-way ANOVA, without inflating false positives.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: Many groups and before/after
- Lesson 13/24, ~40 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/anova

## Three variants: ANOVA vs t-tests

**Alex Tutor:** Still at CatChow, still on the product team, and here's a test to read. Marketing wrote three versions of the welcome email for new users and finally tested them: each newcomer got the Classic, the Playful or the Discount-first one.

![Three envelopes fly from a laptop to a cat's mailbox: a plain one, one covered in paw prints and one with an orange tag on it.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/anova-illustration-2.webp)

**Sofia:** And Discount-first clearly wins, right? People love discounts.

**Alex Tutor:** Maybe. We measured first-month spend for 5 new users per variant. Before comparing means, we need a test built for three groups at once, not two.

**Sofia:** Why not just run a t-test for each pair: Classic vs Playful, Playful vs Discount, Classic vs Discount?

**Alex Tutor:** Each t-test risks a 5% false alarm on its own. Run three, and the chance that at least one screams "significant!" by pure luck climbs well above 5%.

> **Exercise:** CatChow tested three onboarding emails on 5 new users each and wants to compare average first-month spend across all three. Why is running three separate t-tests (one per pair) a bad idea?

## How one-way ANOVA splits the variance

**Alex Tutor:** A one-way analysis of variance (one-way ANOVA) runs one test for all three groups; "one-way" means one factor, here the email variant. It asks: is the spread *between* group means big compared with the spread *within* each group?

$$
F = \dfrac{MS_{between}}{MS_{within}}
$$

*The F-statistic: between-group mean square divided by within-group mean square.*

**Alex Tutor:** Between-group variance measures how far the three variant means sit from the overall average. Big differences between variants push it up.

**Alex Tutor:** Within-group variance is the noise: how much users *inside* the same variant differ from their own group's mean. You'd see it even with no real effect at all.

![Three small groups of cats, each huddled closely around its own food bowl, with wide empty spaces between the groups.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/anova-illustration-3.webp)

*ANOVA asks whether the gaps between groups are large compared with the spread inside each group.*

**Sofia:** So a big F just means the variants are different somehow?

**Alex Tutor:** Right idea. ANOVA's null hypothesis (H0) says all three population means are equal. A large F is evidence against that, but it doesn't yet say which variant is different.

> **Exercise:** The null hypothesis of a one-way ANOVA is that at least one group mean differs from the others.

## One-way ANOVA example: computing F

**Alex Tutor:** Here's first-month spend, in USD, for the 5 users in each variant: Classic 18, 19, 20, 21, 22; Playful 23, 24, 25, 26, 27; Discount-first 28, 29, 30, 31, 32.

**Average first-month spend by variant**

| Email variant | Mean spend |
| --- | --- |
| Classic | 20 |
| Playful | 25 |
| Discount-first | 30 |

*The three variant means are evenly spaced 5 USD apart, while users inside each variant vary by only a couple of dollars.*

**Alex Tutor:** Grand mean across all 15 users: (20 + 25 + 30) / 3 = 25 dollars. Each variant's mean sits exactly 5 dollars from it.

$$
SS_{between} = n\sum_{j=1}^{k}(\bar{x}_j - \bar{x})^2
$$

*Between-group sum of squares: n users per group times the squared distance of each group mean from the grand mean.*

**Alex Tutor:** With n = 5 per group: SS_between = 5 × ((20−25)² + (25−25)² + (30−25)²) = 5 × (25 + 0 + 25) = 250.

**Alex Tutor:** Within-group SS adds each user's squared distance from their own group mean: 10 + 10 + 10 = 30. Three groups leave 15 − 3 = 12 degrees of freedom for it.

> **Exercise:** Using SS_between = 250 (df = 2) and SS_within = 30 (df = 12) from the CatChow email test, compute the F-statistic.

**Alex Tutor:** F = 50 is enormous. For df(2, 12), a result this extreme would happen by chance far less than 1 in 1000 times if the variants truly performed the same.

**Alex Tutor:** Try the whole calculation once more on smaller numbers, so the steps stick: three checkout layouts, 4 users each.

> **Exercise:** Another CatChow test: three checkout layouts, 4 users each, mean order values \$40, \$44 and \$48. SS_within = 54. Compute the F-statistic (two decimals).

**Sofia:** So it's decided: Discount-first is the winner, we ship it everywhere?

**Alex Tutor:** Before I answer, pick the summary you’d actually send to marketing.

> **Exercise:** CatChow tested three welcome emails, Classic, Playful and Discount-first (mean first-month spend \$20, \$25 and \$30). The ANOVA gave F = 50, p < 0.001. Which summary would you send to marketing?

**Alex Tutor:** Not yet. ANOVA only tells us the three means aren't all equal. It doesn't say whether Discount-first beats Playful, or just barely beats Classic.

**Alex Tutor:** Comparing all pairs safely needs a Bonferroni correction: divide alpha by the number of comparisons to keep the overall risk near 5%.

> **Exercise:** You plan to compare all 3 pairs of variants (Classic–Playful, Playful–Discount, Classic–Discount) after the significant ANOVA, using a Bonferroni correction to keep the family-wise error rate at 0.05. What significance threshold (alpha) should each pairwise test use?

## Post hoc tests: which groups differ? (Tukey's HSD)

**Alex Tutor:** This safe pairwise comparison after a significant ANOVA is called a post hoc test. The common choice for equal-sized groups is Tukey's HSD test.

**Sofia:** HSD... sounds dramatic. What does it actually do?

**Alex Tutor:** It compares every pair of group means using one shared threshold, built from the within-group variance, so the overall false-positive rate across all pairs stays near 5%.

![A cat uses one measuring stick to compare three food bowls pair by pair, with arcs linking every pair of bowls.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/anova-illustration-4.webp)

**Alex Tutor:** Run it here and Discount-first beats both Classic and Playful, and Playful also beats Classic. All three differ, matching how far apart their means already looked.

> **Post hoc test**
>
> A pairwise comparison run after a significant ANOVA with built-in control of multiple comparisons (Tukey's HSD, Bonferroni, Games–Howell).

> **Exercise:** Match each ANOVA term to what it describes.

## ANOVA assumptions and the Kruskal-Wallis test

**Alex Tutor:** ANOVA assumes roughly normal data and similar variances in each group, the same assumptions as the t-test.

**Alex Tutor:** When those are badly broken (tiny samples, wild outliers), use the Kruskal-Wallis H test instead: Mann-Whitney's idea, extended to three or more groups.

**Sofia:** Ranking again, like we did for two groups?

**Alex Tutor:** Exactly the same spirit: convert every value across all groups to its rank, then compare rank sums instead of means. It trades a little power for safety against outliers.

**Alex Tutor:** Here’s a messier dataset from the loyalty team. Which test would you reach for?

> **Exercise:** CatChow compares first-month spend across three loyalty offers, 6 users each. One group contains two bulk buyers who spent over \$400, while everyone else spent \$15–\$40. Which test fits best?

> **Exercise:** Which statements about one-way ANOVA and its F-statistic are true? Select all that apply.

## Wrap-up

**Alex Tutor:** One ANOVA beats three t-tests: one honest false-positive rate, one F-statistic, and within-group noise as the baseline to compare against.

**Sofia:** So next time marketing says "the third variant obviously won", I get to ask "obviously, or F-statistically"?

**Alex Tutor:** Almost. Also look at the actual group means and the chart, not just F. A huge F with a tiny, meaningless dollar difference isn't a business win.

**Alex Tutor:** Two tasks left: explain the post hoc step in your own words, then advise on a slightly trickier CatChow test.

> **Exercise:** In 2–4 sentences, explain why a significant one-way ANOVA result on its own doesn't tell CatChow which onboarding email variant to ship.

**Alex Tutor:** Nice reasoning. Let's stress-test it with a trickier, real CatChow case.

**Alex Tutor:** This one won't hand you a tidy F = 50; the result sits right on the fence.

**Sofia:** Great, so I actually have to think instead of just reading the F-statistic?

**Alex Tutor:** Exactly. That's the whole point of statistics literacy for PMs.

> **Exercise:** What would you tell marketing, and what would you do next?

- Previous lesson: [Pitfalls: peeking, multiple testing, novelty effect](https://productovi.com/courses/statistics-for-pm-and-marketers/ab-testing-pitfalls.md)
- Next lesson: [Two-way ANOVA and interactions](https://productovi.com/courses/statistics-for-pm-and-marketers/multifactor-anova.md)

---
Productovi Learning. Free, no sign-up needed to read.