# Who is in the test: triggers and cluster randomization

> Who belongs in an A/B test: triggered enrollment, dilution, and cluster randomization of whole groups when users affect each other.

- Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md)
- Module: Did the change work?
- Lesson 19/32, ~29 min
- Source: https://productovi.com/courses/product-analytics/who-is-in-the-test

## Dilution: why a test found nothing

**Alex Tutor:** Maya has an old Halves test result that disappointed the team, and she wants you to look at it again: who should have been in that A/B test at all?

**Maya:** In spring we tested a “Make it recurring?” suggestion. When someone adds a bill that repeats last month’s, same name and amount, Halves offers to repeat it automatically. Everyone who opened the app was split 50/50. Result: a fraction of a point, interval through zero.

**Alex Tutor:** Look at who was in it. Of about 34,000 weekly active users, only around 5,100 a week add a bill that repeats an earlier one. That’s 15%. The other 85% were in the test but could never see the suggestion.

**Sofia:** But that 85% was split evenly as well. Doesn’t it just cancel out?

**Alex Tutor:** It cancels in the difference, not in the noise. That’s **dilution**: most of the sample carries variance and zero effect, so a real change is spread thin until the interval swallows it.

![One drop of orange dye almost vanishes in a big tub of water but clearly colors a small glass beside it.](https://productovi.com/courses/product-analytics/inline/who-is-in-the-test-illustration-2.webp)

*Dilution: users who never meet the change bury its effect in noise.*

**Alex Tutor:** Try it. Among people who add a repeat bill, 24% set up a recurring one within a week, and the suggestion adds 3 points. What is the effect across the whole test?

> **Exercise:** In spring Halves tested a “Make it recurring?” suggestion, which offers to repeat a bill automatically when a user adds one that repeats last month's. The test split all **34,000** weekly active users. **5,100** of them added a repeat bill and could see the suggestion; among them it raised recurring set-ups by **+3 pp**. Nobody else was affected. What is the effect across **all** users in the test, in percentage points?

**Maya:** We wrote that test up as “the suggestion doesn't work” and moved on. Was that wrong?

> **Exercise:** In spring Halves tested a “Make it recurring?” suggestion, shown only when a user adds a bill that repeats last month's. The test split all 34,000 weekly active users, of whom about 5,100 added such a bill. The result was a fraction of a point, with an interval through zero. Which sentence describes the result **correctly**?

## Triggered enrollment: start counting where the change starts

**Alex Tutor:** The fix is **triggered enrollment**: a user enters the analysis at the first moment the two versions could differ, here adding a repeat bill. Log that moment in control too, where no suggestion appears. You may assign the variant earlier; the count starts at the trigger.

**Theo:** What did splitting everyone actually cost?

**Alex Tutor:** Use last lesson’s formula. At the trigger, 24% → 27% needs **3,306** users per variant: two weeks of repeat bills. Across everyone the rates become 5.3% → 5.75%, and it needs **40,414** per variant, about twelve times as many.

![Only 15% of users can see the suggestion; counting from the trigger needs 3,306 users per variant instead of 40,414 when everyone is split.](https://productovi.com/courses/product-analytics/diagrams/m5-l3-triggered-enrollment.png)

**Sofia:** Could the trigger be setting up the recurring bill? Those are the people the feature is for.

**Alex Tutor:** No: the suggestion changes **who** does that. A trigger must happen the same way in both versions, before the change can act. Our paywall test follows the same rule: users enter at their first `paywall_viewed`, not at `trial_started`.

> **Exercise:** Halves's “Make it recurring?” suggestion appears when a user adds a bill that repeats one from the month before, and offers to repeat it automatically. Which enrollment trigger is right for a rerun of its test?

## Network effects in A/B tests: when users share a screen

**Maya:** Next quarter we want to test a **group plan**: one member pays, and everyone in the group gets Plus. Same 50/50 split by user?

**Alex Tutor:** That breaks. Picture a flat of three split across variants. One flatmate in the new variant buys the group plan, and the two in control get Plus for free. Control is no longer control.

**Sofia:** And they talk. “Why does your paywall offer a group plan and mine doesn't?”

**Alex Tutor:** That's a **network effect**: one user's variant leaks into another's experience. The fix is **cluster randomization**: flip the coin for whole groups, so every member of a group gets the same variant.

![Split by user, one flatmate's group plan gives Plus to flatmates in control; split by group, every member of a flat gets the same variant.](https://productovi.com/courses/product-analytics/diagrams/m5-l3-cluster-randomization.png)

![Three flatmates on one sofa peer at each other's phones; one screen shows an orange offer the other two don't have.](https://productovi.com/courses/product-analytics/inline/who-is-in-the-test-illustration-3.webp)

*When a group shares one balance, split the test by group, not by person.*

> **Exercise:** Which Halves changes should be randomized by **whole group** rather than by user? Select all that apply.

## What cluster randomization costs: the design effect

**Alex Tutor:** There's a price. Members of one group behave alike: they travel together, pay rent together, leave together. Three flatmates carry less information than three strangers.

![Three flatmates walk in step down a street, carrying matching shopping bags and matching receipts.](https://productovi.com/courses/product-analytics/inline/who-is-in-the-test-illustration-4.webp)

*Members of one group act alike, so a test by group needs more users.*

**Alex Tutor:** The **design effect** says how many more users you need. m is the average group size in the test; ICC is how alike members are, from 0 (strangers) to 1 (identical).

$$
\text{DE} = 1 + (m - 1)\times \text{ICC}
$$

*Design effect: multiply the users a user-level test would need by DE.*

**Sofia:** For Halves, say groups of 3 and an ICC of 0.15, from past group data?

**Alex Tutor:** Then work it out, starting from the 6,735 users per variant a user-level test would need.

> **Exercise:** A user-level test would need **6,735** users per variant. Halves tests a group plan (one member pays, everyone in the group gets Plus), randomized by group: average size **m = 3**, **ICC = 0.15**. How many **users** per variant does it need? Round up.

**Alex Tutor:** Group size matters as much as ICC. Try a market where groups are a little bigger.

> **Exercise:** Halves tests a group plan (one member pays, everyone gets Plus), randomized by group, in a market where groups average **m = 4** members. ICC is still **0.15**, and a user-level test would need **6,735** per variant. How many **users** per variant? Round up.

## Analyze at the randomization unit

**Alex Tutor:** One more rule. If the coin was flipped per group, the analysis must treat the group as the unit. The plain per-user interval would come out too narrow: too confident about an effect that's really shakier.

**Maya:** And the trigger for the group plan?

**Alex Tutor:** Same rules as before, applied to a group. Think it through first.

> **Exercise:** Halves tests a group plan, offered on the paywall: one member pays, everyone in the group gets Plus. The test is randomized by group. When should a **group** enter the test?

**Alex Tutor:** The same idea at group level: a group enters when any member first sees the paywall. From then on, the whole group is in.

**Sofia:** So before every test: who can be affected, when they enter, and what we flip the coin for: the randomization unit.

> **Exercise:** In a test randomized by group, you can analyze users as if each was randomized alone: 8,756 users in groups of 3 (ICC 0.15) give the same precision either way.

## Your turn

**Theo:** My idea for the summer: when a **trip group** is created abroad, new expenses default to the local currency instead of the user's home one.

**Alex Tutor:** A good one to plan. Every member adds expenses into the same group, and they all see the same balances.

**Sofia:** So the three questions again: who, when and what unit.

> **Exercise:** Theo, the Halves growth marketer, wants to test this for the summer: when a **trip group** is created abroad, new expenses default to the local currency instead of the user's home one. Every member adds expenses into the same group and sees the same balances. Plan who is in Theo's test in 3–5 sentences: the enrollment trigger, the unit of randomization and why, and what that choice means for the size of the test and its analysis.

- Previous lesson: [How big and how long](https://productovi.com/courses/product-analytics/test-size.md)
- Next lesson: [Case: reading a finished test](https://productovi.com/courses/product-analytics/reading-a-test.md)

---
Productovi Learning. Free, no sign-up needed to read.