# Repeated measures ANOVA

> Tracking the same users across several points in time with a repeated measures ANOVA or the Friedman test.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: Many groups and before/after
- Lesson 16/24, ~41 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/repeated-measures

## Same users, three checkpoints: a within-subjects design

**Alex Tutor:** Same loyalty program, same CatChow product team: we're tracking monthly orders for the same 5 loyalty subscribers at three checkpoints: signup week, week 4, and week 8.

**Sofia:** So it's like the before/after test, just with three time points instead of two?

**Alex Tutor:** Exactly the same spirit, extended. Measuring the same subjects more than twice makes it a repeated-measures (within-subjects) design.

![The same cat at three stops along a path, with a stack of food parcels beside it that grows taller at each stop.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/repeated-measures-illustration-2.webp)

**Sofia:** Could I just run three paired t-tests: Week0–Week4, Week4–Week8, Week0–Week8?

**Alex Tutor:** You could, but you'd hit the same multiple-comparisons problem as running three t-tests instead of a one-way ANOVA.

**Alex Tutor:** Repeated-measures ANOVA does for many time points what the paired t-test did for two: one test, one honest error rate.

> **Exercise:** Why not just run three separate paired t-tests (Week0–Week4, Week4–Week8, Week0–Week8) instead of one repeated-measures ANOVA?

## Same trick as before: remove the subject

**Alex Tutor:** Monthly orders per subscriber: U1 2, 6, 10; U2 6, 7, 11; U3 7, 11, 12; U4 8, 13, 15; U5 12, 13, 17 (Week 0, Week 4, Week 8).

**Monthly orders per subscriber over time**

| Checkpoint | U1 | U2 | U3 | U4 | U5 |
| --- | --- | --- | --- | --- | --- |
| Week 0 | 2 | 6 | 7 | 8 | 12 |
| Week 4 | 6 | 7 | 11 | 13 | 13 |
| Week 8 | 10 | 11 | 12 | 15 | 17 |

*Each subscriber starts at a different level, but all five rise by roughly the same amount from checkpoint to checkpoint.*

**Alex Tutor:** Every subscriber's baseline differs by design (6, 8, 10, 12, 14 on average), yet each one climbs by a similar amount over time.

**Alex Tutor:** Repeated-measures ANOVA splits total variability into three pieces, not two: Subjects, Time, and Error, instead of one-way ANOVA's Between/Within.

$$
SS_{total} = SS_{subjects} + SS_{time} + SS_{error}
$$

*Total sum of squares splits into subject variability, the time effect, and whatever's left over.*

**Sofia:** So it's one-way ANOVA, plus an extra 'Subjects' slice carved out of the error?

![A round cake cut into three slices, one large, one medium and one thin, while a cat gets ready to carry the large slice away.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/repeated-measures-illustration-3.webp)

*Pull the differences between subscribers out into their own slice, and the leftover noise gets much smaller.*

> **Exercise:** A repeated-measures ANOVA splits total variability into three parts: between-subject variability, the time effect, and a leftover error term, instead of one-way ANOVA's two-part Between/Within split.

## Repeated measures ANOVA example: the numbers

**Alex Tutor:** Grand mean is 10. Subject means are 6, 8, 10, 12, 14. Time means (Week 0, 4, 8) are 7, 10, 13.

**Alex Tutor:** Summing squared deviations from 10 across all 15 numbers gives SS_total = 220.

**Alex Tutor:** From the subject means: SS_subjects = 120. From the time means: SS_time = 90.

> **Exercise:** CatChow tracks monthly orders for the same 5 loyalty subscribers at 3 checkpoints. For this repeated-measures design, SS_total = 220, SS_subjects = 120, and SS_time = 90. Compute SS_error (the residual).

**Alex Tutor:** Degrees of freedom: subjects 5 − 1 = 4, time 3 − 1 = 2, error (5 − 1)(3 − 1) = 8. MS_time = 90 / 2 = 45. MS_error = 10 / 8 = 1.25.

**Alex Tutor:** Compare that error's 8 degrees of freedom with the 15 − 3 = 12 you'd get by mistakenly treating all 15 numbers as one independent-groups design by week alone.

**Sofia:** More degrees of freedom sounds like it should be better, though?

**Alex Tutor:** More df doesn't help if the error term itself is inflated by all the between-subject spread we just pulled out into its own slice.

> **Exercise:** With SS_time = 90 (df = 2) and SS_error = 10 (df = 8), compute the F-statistic for the time effect.

## The payoff: much more power

**Alex Tutor:** To see the payoff, first do it the wrong way: pretend the three weeks had different people.

> **Exercise:** CatChow tracked monthly orders for the same 5 loyalty subscribers at 3 checkpoints: 15 numbers with SS_total = 220. Ignore who is who and treat the 15 numbers as three independent groups by week: SS_time = 90 (df = 2), and the error is SS_total − SS_time = 220 − 90 = 130 on 12 df. Compute this naive F (two decimals).

**Alex Tutor:** Ignoring who's who, error SS would be SS_total − SS_time = 220 − 90 = 130 over 12 df: MS_error ≈ 10.83, giving F ≈ 4.15.

**Alex Tutor:** Properly paired by subject, the exact same data gives F = 36, an enormous gain in power from identical 15 numbers.

![A cat looks at the same kibble through two lenses: through the foggy one nothing stands out, through the clear one an orange upward trail is plain.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/repeated-measures-illustration-4.webp)

**Sofia:** Same data, wildly different F. That's a little unsettling.

**Alex Tutor:** It's the same edge the paired t-test had over the independent-samples t-test, just extended to three or more time points.

> **Exercise:** Match each term to what it describes.

## Not normal enough? The Friedman test

**Alex Tutor:** Repeated-measures ANOVA assumes roughly normal, similarly spread differences between time points.

**Alex Tutor:** When that's shaky, or the sample is tiny, the Friedman test is the fallback: it ranks each subject's own measurements and compares rank sums across time points.

**Sofia:** Ranks again? Feels like this whole course is just "when in doubt, rank it."

**Alex Tutor:** Not far off. Ranks trade a little power for a lot of safety against outliers and skewed data.

**Alex Tutor:** Let’s run it on our five subscribers. The ranking part is easy here, because everyone rises steadily.

> **Exercise:** Friedman test on CatChow's monthly orders for 5 loyalty subscribers at 3 checkpoints: within each subscriber, rank the three checkpoints 1–3. Every subscriber rises steadily, so the rank sums are 5, 10 and 15. Compute the Friedman statistic: Q = 12 / (n·k·(k + 1)) × ΣR² − 3n(k + 1), with n = 5 subscribers and k = 3 checkpoints.

> **Exercise:** Which statements about repeated-measures designs are true? Select all that apply.

## Wrap-up

**Alex Tutor:** Same subjects, several checkpoints: pull the subject variability into its own slice, and the real time effect stands out far more clearly.

**Alex Tutor:** Even with F = 36, still check the actual size of the change per subscriber, not just the F-statistic, before calling it a business win.

**Alex Tutor:** So how would you write it up for the team? Two drafts.

> **Exercise:** Repeated-measures ANOVA on CatChow's loyalty cohort (the same 5 subscribers at signup, week 4 and week 8): monthly orders 7 → 10 → 13 on average, F(2, 8) = 36, p ≈ 0.0001. Which summary is right?

**Sofia:** Got it: same subjects, one 'Subjects' slice removed, then judge the trend on its own merits.

**Alex Tutor:** Two tasks left: explain that F jump in your own words, then a case with a tiny, lopsided repeated-measures sample.

> **Exercise:** CatChow tracked monthly orders for the same 5 loyalty subscribers at 3 checkpoints. In 2–4 sentences, explain why these 15 measurements gave F ≈ 4.15 when treated as an independent-groups design by week, but F = 36 as a proper repeated-measures analysis.

**Alex Tutor:** Now the case: NPS tracked at three checkpoints for the same 5 power users after a new feature.

**Alex Tutor:** Four users trend nicely upward, but one user's score crashes at the last checkpoint after an unrelated billing error.

**Sofia:** Ready.

**Alex Tutor:** Go for it.

> **Exercise:** What test would you use, and how would you handle the outlier?

- Previous lesson: [Paired t-test: before and after](https://productovi.com/courses/statistics-for-pm-and-marketers/paired-tests.md)
- Next lesson: [Correlation coefficient: how two metrics move together](https://productovi.com/courses/statistics-for-pm-and-marketers/correlation.md)

---
Productovi Learning. Free, no sign-up needed to read.