Case: the version or the traffic?

Before arguing with a before/after number, check that it's built fairly and reproduce it. Halves shipped version 5.2.0 with a new onboarding on 31 August. Week-2 retention of new users went from 28.9% in August to 39.8% in September, and Maya wants to put "+11 points" in the board update.

A good analysis starts by checking the comparison, not by doubting it. The three fairness rules from Pre/post analysis: comparing versions fairly all pass here: new users grouped by sign-up week, a fixed window (active on at least one of days 8–14), and complete cohorts only.

Then reproduce the number from the data, pooling rows before dividing. August (5.1.0): 705 active of 2,441 new users, 705 ÷ 2,441 = 28.88%. September (5.2.0): 807 ÷ 2,028 = 39.79%. The raw gap, +10.9 pp, is computed correctly.

Reproducing first confirms you're reading the same data as the person who made the claim, so the conversation can move on to the real question. Here that question is what the 10.9 points are a number of.

The release wasn't the only thing that changed at the end of August. Theo paused the paid social campaign on 1 September, and September is moving season: new flatmates, new rent to split, while summer trips wind down. Each of those changes who signs up.

The shares of new users show how much:

• flats: 29.95% in August → 1,265 of 2,028 = 62.38% in September;
• trips: 44.53% → 9.91%;
• paid social: 34.08% → 12.03%.

And these segments retain very differently. In August flats kept 41.45% on days 8–14 and couples 38.04%, while trips kept only 15.18% and paid-social users 23.32%. September swapped many of the worst-retaining users for the best-retaining ones.

That alone can lift the average by many points without any user behaving differently: a mix shift, as described in Volume, quality and Simpson's paradox. When several things change in the same week, the raw before/after number carries all of them at once, and the release gets credit for whatever moved.

So after reproducing the raw number, the next question is always the same: who came, before and after, and how do those groups usually retain?

If a change reaches every new user, a real effect of several points should show as a consistent rise in every segment, not as small moves in both directions. Inside each group type, the September rates move like this:

• couples: 209 ÷ 562 = 37.19%, against 38.04% in August: slightly down;
• flats: 44.58% against 41.45%: up about three points;
• trips: 16.92% against 15.18%: up, on only 201 September users.

Up, down, up: no consistent rise. Everyone who signed up in September went through the new onboarding, so a strong effect would have lifted couples too.

Two cautions keep this reading honest. Small segments are noisy: 201 trip users or 562 couples give rates that can easily move a point or two by chance. And "no consistent pattern" is not the same as "no effect": a small genuine improvement could be hidden in that noise.

So the segment view supports "no sign of a large onboarding effect", which already rules out crediting the 11 points to it. It can't measure a small one either way.

A segment is itself a mix of smaller segments. Of the three group types, flats rose most from August to September: from 41.45% to 44.58%.

Maya will point at that, so look one level down. Flats arrive through different channels, and channels retain differently: invite flats keep around 47–50%, paid-social flats around 30–34%. In August 279 of 731 flats came by invite (38.2%); in September 640 of 1,265 (50.6%).

So the September flats are a different blend of flats, with many more coming through the best channel. Part of their "improvement" is the same mix shift again, just hidden inside one segment.

This is why one split is rarely enough: a shift at any level can move a rate that looks like a clean comparison. The practical answer is to split by all the variables you know matter at once, here channel × group type, and compare cell by cell, or re-weight the cells to one fixed mix.

When two things shift at once, hold the mix fixed on their cells, not on one variable. Here the cells are channel × group type, nine in all, and as in Traffic mix and seasonality you can adjust from both sides.

September's rates at August's mix: weight each September cell rate by that cell's share of August's users. Result: 28.82%, against August's 28.88%, a difference of −0.06 points. If September had August's users, retention would be the same.

August's rates at September's mix: weight each August cell rate by that cell's share of September's users. Result: 40.14%, against September's actual 39.79%. If August had September's users, it would already have been at about 40%.

Both say the same thing from opposite sides: the mix explains the whole jump from 28.9% to 39.8%, and at a fixed mix the two months are almost identical.

Why cells and not just group types? Re-weighting by group type alone gives September about 30.4% at August's mix: better, but still 1.5 points of channel shift left inside it. Splitting by both variables at once removes both shifts, cell by cell.

With the evidence in, each candidate explanation for the jump from 28.9% to 39.8% can be checked:

• The mix changed (more flats and invites, fewer trips and paid users). Supported: at a fixed mix the two months are 28.82% and 28.88%, and August's rates at September's mix give 40.14%.
• The onboarding added about 11 points. Not supported: like for like, retention barely moved, and segment rates go both up and down.
• Pausing paid social made the remaining users more loyal. Not supported: the pause removed a low-retaining group, which lifts the average without anyone retaining better.
• Incomplete cohorts inflate the after number. Not the issue: every cohort here has its full days 8–14. (In other cases this is the first thing to check.)

The pattern to remember: a before/after jump that disappears at a fixed mix was produced by who was counted, not by how they behaved. That's a complete explanation of the raw number. It's not a measurement of the onboarding's own effect, which could still be small and positive or small and negative.

Theo's version of the story: "pausing paid social raised retention by eleven points". It's the same trap with a different hero. The pause cut paid social from 34.08% to 12.03% of new users, and paid users retain worst (23.32% in August). Fewer of them in the average lifts the average. Nobody who stayed retained better because of it.

The trap repeats one level down. Paid social's own rate went from 23.32% to 28.69%, which looks like the paid users improved. But within paid social the mix changed too: 370 of 832 August paid users were trip users (44%), against 24 of 244 in September (10%). Fewer trip users inside the channel lifts the channel's rate in exactly the same way.

For a channel, the useful question was never its effect on the average. It's what its users bring in against what they cost: the paid users who did retain were real users, and the pause also means fewer users overall. Judge the campaign on value and cost, and keep it out of the retention story.

A board update has to be short and survive the question "are you sure?". For this release that means four moves:

1. The raw number, honestly: week-2 retention of new users rose from 28.9% to 39.8%, built fairly.
2. Why: September's sign-ups changed: flats 30% → 62% of new users, trips 45% → 10%, paid social 34% → 12%.
3. Like for like: at August's mix, September is 28.8% against 28.9%. The jump is the mix.
4. Next: keep a randomized holdout group on the old onboarding for a month, and report retention by segment and at a fixed mix.

Leave out both "the new onboarding lifted retention by 11 points" and "the onboarding did nothing". Neither is supported.

A holdout group is a random slice of users kept on the old version. It lives through the same September, with the same flats and the same paused campaign, so the difference between the two groups is the onboarding.

How it fits together: reproduce, find who changed, read segments and cells, adjust both ways, name what's supported, propose a holdout. Module 6 answered what we can learn without a test; the next module asks whether Halves makes money.