# Case: the version or the traffic?

> Did the new Halves version help, or did the users change?

- Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md)
- Module: What can we learn without a test?
- Lesson 24/32, ~42 min
- Source: https://productovi.com/courses/product-analytics/did-the-version-help

## The best release ever?

**Alex Tutor:** Good news at Halves for once, and Maya wants it in the board update. First it comes to you.

**Maya:** 5.2.0 shipped the new onboarding on 31 August. Week-2 retention of new users went from **28.9% in August to 39.8% in September**. Eleven points! That's going in the board update.

**Theo:** Also worth knowing: I paused the paid social campaign on 1 September. The summer budget ran out.

**Alex Tutor:** And September is moving season: new flatmates, new rent to split, while summer trips wind down. Three things changed in the same week. So we check in order: is the number fair, who came, and what like for like shows.

![On one street, travelers put suitcases away, new flatmates carry moving boxes in, and a billboard is being covered with a blank sheet.](https://productovi.com/courses/product-analytics/inline/did-the-version-help-illustration-2.webp)

*Three things changed at once: the release, the campaign and the season.*

**Dataset (CSV):** [Weekly sign-up cohorts, August and September](https://productovi.com/datasets/product-analytics/did-the-version-help/halves-autumn-cohorts.csv)

New Halves users, weeks of 3 August to 21 September 2026. Version 5.2.0 with the new onboarding was released on 31 August. Every cohort has its full days 8–14.

- `signup_week`: Monday of the sign-up week
- `app_version`: Version installed: 5.1.0 in August, 5.2.0 from 31 August
- `channel`: invite, app_store_search or paid_social
- `first_group_type`: Type of the first group the user joined: flat, couple or trip
- `new_users`: New users in this cell
- `active_days_8_14`: Of them, opened the app on at least one of days 8–14

**Sofia:** The fairness rules pass: new users by sign-up week, a fixed days 8–14, every cohort complete.

**Alex Tutor:** Good. Then reproduce Maya's before number first, so we know we're reading the same data.

> **Exercise:** What share of **5.1.0** (August) new users were active in days 8–14? Percent, two decimals.

## Who came in September

**Maya:** So my number is right.

**Alex Tutor:** The number is right; the question is what it's a number of. Who are these people? In August, 45% of new users joined through a trip group and 34% came from paid social. Both groups retain worst.

**Sofia:** And in September trips shrink and paid stops. What about flats?

**Alex Tutor:** Check.

> **Exercise:** What share of **5.2.0** (September) new users had `flat` as their first group type? Percent, two decimals. (August: 29.95%.)

## Like with like

**Alex Tutor:** So September is full of the users who always retain best. Now compare inside each group type.

**Maya:** Flats went from 41.5% to 44.6%. That's still up!

> **Exercise:** What share of **5.2.0 `couple`** users was active in days 8–14? Percent, two decimals. (5.1.0 couples: 38.04%.)

**Alex Tutor:** Look one level down. September's flats came more through invites, the best channel: invite flats retain around 47–50%, paid flats around 30–34%. Group type alone still hides a channel shift.

![A magnifying glass over a group of new flatmates reveals that most of them arrived holding invitation envelopes.](https://productovi.com/courses/product-analytics/inline/did-the-version-help-illustration-3.webp)

*One level down, September's flats came more often through invites.*

**Alex Tutor:** Put a number on it before Maya argues: how many of September's flats came by invite?

> **Exercise:** What share of **September (5.2.0) `flat`** users came through `invite`? Percent, one decimal. (August flats: 38.2%.)

**Alex Tutor:** One summary number, as before: September's rate in each channel × group type cell, weighted by August's mix. It comes to **28.82%**, against 28.88% in August: **−0.06 points**.

**Theo:** How can flats go up and the whole thing stay flat?

**Alex Tutor:** Because inside flats the channel mix moved too: more invites. Re-weighting by channel and group type together fixes both at once, cell by cell.

**Alex Tutor:** Check it from the other side too: give August's users September's mix.

> **Exercise:** Halves's week-2 retention of new users was 28.88% in August (5.1.0) and 39.79% in September (5.2.0). Take **August's** rate in each channel × group type cell and weight it by that cell's share of **September's** new users. What retention do you get? Percent, one decimal.

**Sofia:** So the whole eleven points is who signed up.

> **Exercise:** Halves's week-2 retention of new users jumped from 28.9% in August to 39.8% in September. Version 5.2.0 shipped a new onboarding on 31 August; the same week the paid social campaign was paused and moving season began. What best explains the jump?

## Theo's turn

**Theo:** Then my pause did it! Cutting paid social raised retention by eleven points.

**Alex Tutor:** It raised the **average**. Nobody retained better because of it. And the question for paid social was never its retention alone; it's what its users are worth against what they cost.

**Sofia:** Same trap, different hero.

![Someone lifts the smallest apples out of a basket; what remains looks finer, though no apple has grown.](https://productovi.com/courses/product-analytics/inline/did-the-version-help-illustration-4.webp)

*Removing a group lifts the average without anyone staying longer.*

**Theo:** But my paid users did retain better in September: 28.69% against 23.32%.

> **Exercise:** **True or false:** Halves paused its paid social campaign on 1 September. Paid-social users' week-2 retention rose from 23.32% in August to 28.69% in September, so the users the campaign still brought in became more loyal.

**Alex Tutor:** Before Theo's reply, pick the one line for Maya's board update: the line the board will remember.

> **Exercise:** Week-2 retention of Halves new users rose from 28.9% in August to 39.8% in September, after version 5.2.0 shipped a new onboarding; re-weighted to August's mix of channel × group type, September is 28.8%. Which line should go into the board update?

> **Exercise:** Halves's week-2 retention of new users rose from 28.9% in August to 39.8% in September. Theo, the growth marketer who paused the paid social campaign on 1 September, says: “Pausing paid social raised retention.” Reply to him in 2–4 sentences.

## Case: the board update

**Maya:** So what do I tell the board? I still think the onboarding is better.

**Sofia:** Could a holdout have answered it?

**Alex Tutor:** Yes. A random slice of new users kept on the old onboarding lives through the same September, with the same flats and the same paused campaign. Then the difference is the onboarding.

**Alex Tutor:** It might be, by a little. This data can't show a small effect either way. Write what it does show, what it can't, and how we'd find out properly next time.

> **Exercise:** Write the board update: did 5.2.0 help, or did the traffic change?

**Alex Tutor:** That's the whole module in one paragraph: build the number fairly, ask who came, compare like with like, and hand the causal question to a test.

- Previous lesson: [Selection bias: users who do X retain better](https://productovi.com/courses/product-analytics/correlation-and-selection.md)
- Next lesson: [How to calculate ARPU and ARPPU](https://productovi.com/courses/product-analytics/revenue-metrics.md)

---
Productovi Learning. Free, no sign-up needed to read.