# Two-way ANOVA and interactions

> A two-way ANOVA for when two factors act together, e.g. a promo that works on mobile but not on desktop.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: Many groups and before/after
- Lesson 14/24, ~39 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/multifactor-anova

## One banner, two devices: a two-way ANOVA

**Alex Tutor:** One more test from the CatChow product team's backlog, and a new one: a discount banner on the product page. This time we split the results by device: Desktop vs Mobile, Control vs Discount banner.

**Sofia:** So it's basically a regular A/B test, just with an extra column?

**Alex Tutor:** More than that. We're testing two factors at once: the banner and the device. Each explanatory variable is called a factor, and together they form a two-way ANOVA.

![A two-by-two grid: a desktop monitor and a phone, each shown once with a plain screen and once with an orange banner on screen.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/multifactor-anova-illustration-2.webp)

**Sofia:** If the banner adds 7 dollars to average revenue overall, it adds 7 dollars everywhere, right?

**Alex Tutor:** That's exactly the assumption we need to check. Let's look at the four groups this design creates before trusting one overall number.

> **Exercise:** Why might looking only at the discount banner's overall average revenue uplift (ignoring device) be misleading?

## A 2×2 factorial design: four cells

**Alex Tutor:** Session revenue, USD, 3 users per cell: Desktop-Control 38, 40, 42; Desktop-Discount 40, 42, 44; Mobile-Control 28, 30, 32; Mobile-Discount 40, 42, 44.

**Mean session revenue by banner and device**

| Banner | Desktop | Mobile |
| --- | --- | --- |
| Control | 40 | 30 |
| Discount | 42 | 42 |

*The desktop and mobile lines are not parallel: the discount banner barely moves desktop revenue but lifts mobile revenue much more.*

**Alex Tutor:** The four cell means: Desktop-Control 40, Desktop-Discount 42, Mobile-Control 30, Mobile-Discount 42.

**Alex Tutor:** Notice the lines aren't parallel. If the banner simply added a fixed amount everywhere, both lines would rise by the same height.

**Sofia:** The mobile line jumps way more steeply. Is that the interaction thing?

> **Exercise:** An interaction effect means that the effect of one factor is the same at every level of the other factor.

## Main effects: averaging over the other factor

**Alex Tutor:** A main effect is the average effect of one factor across all levels of the other. Average the banner's effect over both devices.

**Alex Tutor:** Averaging desktop and mobile together: Control mean is 35, Discount mean is 42. That's the banner's main effect, ignoring device.

**Alex Tutor:** So the banner's main effect is 42 − 35 = 7 dollars, averaged across both devices.

**Sofia:** And the device factor, does it have a main effect too?

**Alex Tutor:** Desktop's mean is 41, mobile's is 36, averaged over both banner states. So device's main effect is 5 dollars: desktop earns more regardless of the banner.

> **Exercise:** Cell means: Desktop-Control $40, Desktop-Discount $42, Mobile-Control $30, Mobile-Discount $42. Compute the banner's main effect: the discount mean minus the control mean, averaged over both devices.

## Interaction effect: when the whole isn't the sum of its parts

**Alex Tutor:** An interaction effect is when one factor's effect depends on the level of the other. It shows up exactly as those non-parallel lines.

**Alex Tutor:** On desktop the banner adds 42 − 40 = 2 dollars. On mobile it adds 42 − 30 = 12 dollars. Same banner, very different effect.

![The same orange banner on two screens: a cat at a desktop computer barely reacts, while a cat holding a phone jumps with joy.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/multifactor-anova-illustration-3.webp)

*An interaction: how much the banner helps depends on the device.*

**Sofia:** So mobile users just love discounts more. Let's ship the banner only there, done?

**Alex Tutor:** That's a fair reading of this data, but check it's not just desktop already sitting near a ceiling before generalising too far.

> **Exercise:** Cell means: Desktop-Control \$40, Desktop-Discount \$42, Mobile-Control \$30, Mobile-Discount \$42. Compute the interaction effect size: the banner's uplift on mobile (Mobile-Discount − Mobile-Control) minus its uplift on desktop (Desktop-Discount − Desktop-Control).

**Alex Tutor:** For contrast, here’s a test where the segments differ but the change doesn’t. Is that an interaction?

> **Exercise:** A different CatChow test, an email layout (A or B) by device. Mean order values: Desktop-A \$30, Desktop-B \$36, Mobile-A \$22, Mobile-B \$28. These cell means show an interaction between layout and device.

## Reading the two-way ANOVA table

$$
F_{banner} = \dfrac{MS_{banner}}{MS_{error}}, \quad F_{device} = \dfrac{MS_{device}}{MS_{error}}, \quad F_{banner \times device} = \dfrac{MS_{banner \times device}}{MS_{error}}
$$

*A two-way ANOVA computes three separate F-statistics: one per main effect, plus one for their interaction.*

**Alex Tutor:** For this test: F_banner ≈ 36.75, F_device ≈ 18.75, F_interaction ≈ 18.75, all on df(1, 8). All three clear the critical value of about 5.32.

**Alex Tutor:** Each F uses the same error term. Check one yourself from the table: SS_device = 75 on 1 df, SS_error = 32 on 8 df.

> **Exercise:** From the banner × device ANOVA table: SS_device = 75 with 1 degree of freedom, SS_error = 32 with 8 degrees of freedom. Compute F_device.

**Sofia:** All three significant? So everything matters equally?

**Alex Tutor:** They matter, but the interaction is the headline: it changes how you should even read the two main effects.

> **Exercise:** Match each term from a multifactor ANOVA to what it describes.

## What this means for the rollout

**Alex Tutor:** With a significant interaction, one 'roll out everywhere or nowhere' decision is the wrong shape for the question.

**Alex Tutor:** Treat a significant interaction as a flag to segment the decision: it's plausible to ship on mobile while holding or testing further on desktop.

**Alex Tutor:** So which recommendation would you put in front of the team?

> **Exercise:** Banner uplift: +\$2 per session on desktop, +\$12 on mobile, +\$7 overall; the interaction is significant. Which recommendation fits?

**Sofia:** Feels like Simpson's paradox's cousin.

**Alex Tutor:** Close: an unnoticed interaction is exactly how a reassuring overall average can hide a segment where the change isn't helping much at all.

![A smiling cat sits on a wide rug while another cat lifts a corner and finds a smaller, unimpressed cat hidden underneath.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/multifactor-anova-illustration-4.webp)

> **Simple effects**
>
> After a significant interaction, examine each factor's effect separately at each level of the other (its "simple effects"), the same discipline as a post hoc test, one level deeper.

**Sofia:** So even more testing after the testing?

**Alex Tutor:** Exactly the same discipline as ANOVA's post hoc step, just applied one layer deeper, to combinations instead of single groups.

> **Exercise:** Which statements about factorial designs and interactions are true? Select all that apply.

## Wrap-up

**Alex Tutor:** One banner, two very different stories: +2 dollars on desktop, +12 on mobile. A single 'it works' verdict erases that gap.

**Alex Tutor:** Whenever a chart of group means shows crossing or clearly non-parallel lines, that's your cue to look for an interaction before reporting one overall number.

**Sofia:** Got it: check the lines before trusting the average.

**Alex Tutor:** One task to put that into your own words, then a trickier case with three channels instead of two devices.

> **Exercise:** In 2–4 sentences, explain why CatChow shouldn't just look at the discount banner's overall average uplift (+7 dollars) when deciding whether to roll it out on both desktop and mobile.

**Alex Tutor:** Nice. One more, mixing both main effects and an interaction.

**Alex Tutor:** This one has no chart, just a results table. Read it like an analyst would, one channel at a time.

**Sofia:** No pressure.

**Alex Tutor:** You'll be fine, that's exactly what this course is for.

> **Exercise:** What would you recommend, channel by channel?

- Previous lesson: [One-way ANOVA: comparing many groups](https://productovi.com/courses/statistics-for-pm-and-marketers/anova.md)
- Next lesson: [Paired t-test: before and after](https://productovi.com/courses/statistics-for-pm-and-marketers/paired-tests.md)

---
Productovi Learning. Free, no sign-up needed to read.