Two-way ANOVA and interactions

A one-way ANOVA compares groups that differ by one factor: a categorical variable such as the email variant. Real tests often have two: the banner (control or discount) and the device (desktop or mobile).

Crossing them gives a factorial design. With two levels each, that's a 2×2 grid of four cells: desktop-control, desktop-discount, mobile-control, mobile-discount. A two-way ANOVA analyses all four at once.

Why not just compare discount against control overall? Because the overall number assumes the banner does the same thing everywhere. In the CatChow data the overall uplift is +$7, but that's an average of +$2 on desktop and +$12 on mobile. Neither device actually sees +$7.

A factorial design answers three questions in one go:

• does the banner matter, on average? (its main effect);
• does the device matter, on average? (its main effect);
• does the banner's effect depend on the device? (the interaction).

The third question is the one a single overall average can never answer.

An interaction effect means the effect of one factor depends on the level of the other. The discount banner adds $2 on desktop and $12 on mobile: the banner's effect depends on the device.

The quickest way to see it is a line chart of the cell means: banner on the horizontal axis, one line per device.

• Parallel lines: no interaction. The banner lifts both devices by the same amount; the device gap stays the same with or without the banner.
• Non-parallel lines: a possible interaction. Here desktop goes 40 → 42 and mobile goes 30 → 42, so the mobile line climbs much more steeply.
• Crossing lines: a strong interaction, where a change can help one segment and hurt another.

A chart can only suggest an interaction; small samples make lines wobble. The two-way ANOVA tests it formally with its own F-statistic.

The practical meaning: if there's an interaction, "what does the banner do?" has no single answer. The honest answer is "it depends on the device", followed by the numbers for each.

A main effect is the average effect of one factor across all levels of the other. You get it from marginal means: average the cells along a row or a column.

CatChow's discount-banner test has four cell means of session revenue: desktop-control $40, desktop-discount $42, mobile-control $30, mobile-discount $42. For the banner, average over both devices:

• control: (40 + 30) / 2 = 35;
• discount: (42 + 42) / 2 = 42;
• banner main effect: 42 − 35 = $7.

For the device, average over both banner states:

• desktop: (40 + 42) / 2 = 41;
• mobile: (30 + 42) / 2 = 36;
• device main effect: 41 − 36 = $5.

(With equal cell sizes, these marginal means match the averages over all users in the row or column.)

Main effects are useful summaries, but they are averages over the other factor. When there's no interaction, the banner's $7 is true on every device. When there is one, the $7 is a blend that no single segment actually experiences.

The common slip is to compare just one pair of cells, for example 42 − 40 = 2 on desktop, and call that the banner's main effect. That's a simple effect for one device, not the average over both.

In a 2×2 design, the size of the interaction is how much the effect of one factor changes between the two levels of the other. Compute the effect in each segment, then subtract. In CatChow's discount-banner test, session revenue went from $40 to $42 on desktop and from $30 to $42 on mobile:

• banner uplift on mobile: 42 − 30 = $12;
• banner uplift on desktop: 42 − 40 = $2;
• interaction size: 12 − 2 = $10.

You get the same $10 looking the other way round. The device gap is 40 − 30 = $10 without the banner and 42 − 42 = $0 with it, a change of $10. An interaction is symmetric: if the banner's effect depends on the device, the device's effect depends on the banner.

An interaction of 0 means parallel lines: the banner adds the same amount on both devices. The bigger the number, the more misleading the overall main effect becomes.

Keep the order consistent (mobile minus desktop, or the reverse) and say which you used; the sign tells you which segment gains more. And remember that a large difference in sample means still needs the interaction F-test to show it's not just noise.

Not every gap between segments is an interaction. Take an email layout test by device: Desktop-A $30, Desktop-B $36, Mobile-A $22, Mobile-B $28.

• Layout effect on desktop: 36 − 30 = $6.
• Layout effect on mobile: 28 − 22 = $6.
• Interaction size: 6 − 6 = 0.

On a line chart the two lines are parallel. Desktop sits $8 above mobile with either layout: that's a main effect of device, not an interaction. Layout B adds $6 everywhere: a main effect of layout. The effects simply add up, which is why this pattern is called additive.

In this situation the overall summary is honest: "layout B adds about $6 per order, on any device". One rollout decision fits all segments.

The trap is to see that mobile is lower and conclude the layout "works differently" there. Interaction is about whether the effect differs, not whether the levels differ. Ask: does the change add the same amount in each segment? If yes, the lines are parallel, whatever their height.

Real data are never perfectly parallel, which is why the interaction gets a formal test.

A two-way ANOVA table has one row per effect plus one for the error (the within-cell noise). For the banner × device test with 3 users per cell:

• banner: SS = 147, df = 1, MS = 147, F = 147 / 4 = 36.75;
• device: SS = 75, df = 1, MS = 75, F = 75 / 4 = 18.75;
• banner × device: SS = 75, df = 1, MS = 75, F = 18.75;
• error: SS = 32, df = 8, MS = 32 / 8 = 4.

The degrees of freedom follow simple rules: (levels − 1) for each factor, their product for the interaction (1 × 1 = 1), and N minus the number of cells for the error (12 − 4 = 8).

Every F shares the same denominator, MS_error. Each one is then compared with the critical value for its own df: about 5.32 for df(1, 8) at α = 0.05. All three clear it here.

The classic slip is dividing SS by SS: 75 / 32 ≈ 2.34 looks non-significant, but it skips the degrees of freedom. Always convert sums of squares to mean squares first.

Four terms, one design:

• factor: a categorical explanatory variable whose levels define the groups. Banner (control, discount) and device (desktop, mobile) are factors;
• main effect: the average effect of one factor across all levels of the other, read from the marginal means ($7 for the banner, $5 for the device);
• interaction effect: when one factor's effect depends on the level of the other ($2 on desktop, $12 on mobile);
• two-way (multifactor) ANOVA: one analysis with two or more factors that tests each main effect and their interaction, each with its own F-statistic.

After a significant interaction comes one more step: simple effects. You look at the effect of one factor separately at each level of the other: the banner on desktop alone, the banner on mobile alone. It's the same discipline as a post hoc test after one-way ANOVA, one layer deeper.

With three factors there are three main effects, three two-way interactions and one three-way interaction, which is why most product tests stop at two factors: the interpretation multiplies quickly.

A significant interaction changes the shape of the decision. "Roll out everywhere or nowhere" assumes one effect; the data say there are two.

A practical sequence:

1. Read the simple effects: +$12 on mobile, +$2 on desktop. Test each one if needed; this is the post hoc step for factorial designs.
2. Decide per segment. Mobile's gain is large and clear: ship there. Desktop's +$2 may not cover the cost of the discount; hold, or run a focused desktop test.
3. Ask why. Maybe mobile shoppers respond more to discounts. Or maybe desktop revenue already sits near a ceiling, so there's little room to move. The explanation affects what you try next.
4. Report the segments, not the blend. "+$7 overall" is true arithmetic but describes no real user.

This is the same lesson as Simpson's paradox from a different angle: an overall average can look reassuring while hiding a segment where the change barely helps, or even hurts.

When the lines actually cross, segmenting stops being optional: a global rollout would make one group worse off.

The order in which you read a two-way ANOVA matters. Start with the interaction, then look at the main effects.

• Interaction not significant: the lines are roughly parallel, and each main effect is a fair summary. "The banner adds about $7, on any device" is a reasonable sentence.
• Interaction significant: the main effects become averages of different stories. The banner's $7 is a blend of $2 and $12; the device's $5 is a blend of $10 and $0. Report them with caution, and lead with the simple effects instead.

Visually, non-parallel lines in a chart of cell means are the cue to check. Formally, the interaction has its own F-statistic, separate from the two main effects.

For decisions this has a direct consequence: with a significant interaction, one global "ship or don't" choice may be right for some segments and wrong for others. Decide segment by segment.

A note on wording: "significant" here still means "unlikely to be noise", not "large". Always pair it with the dollar sizes.