# Pre/post analysis: comparing versions fairly > Pre/post analysis: the rules that make a before-and-after comparison of app versions worth reading. - Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md) - Module: What can we learn without a test? - Lesson 21/32, ~30 min - Source: https://productovi.com/courses/product-analytics/version-comparison ## “Did we break retention?” Why a version split misleads **Alex Tutor:** A worrying dashboard at Halves: Maya thinks a release broke something, and you're the one to check it with a pre/post analysis: the app before and after the release. **Maya:** v4.9.0 went out on Monday, 11 May, with the new first-expense flow. My dashboard says week-2 retention fell from **29% to 15%**. Did we break something? **Sofia:** Can't we just split today's users by version? Everyone on 4.9.0 against everyone still on 4.8.0: same week, same weather. **Alex Tutor:** That's the tempting one, and it's tilted before you start. Nobody drew lots for the update. People who open Halves every few days get 4.9.0 within a day or two, often automatically; people who drifted away sit on 4.8.0 for weeks. The 4.9.0 side is packed with your most loyal users. ![Eager regulars crowd forward to install an update on their phones first, while occasional users stroll up much later.](https://productovi.com/courses/product-analytics/inline/version-comparison-illustration-2.webp) *Nobody drew lots for the update: the keenest users move first.* **Sofia:** So the new version would “win” even if it changed nothing. **Alex Tutor:** Yes: the groups picked themselves. This is an **observational comparison**, not a coin flip, so fairness has to be built in. Take **new users grouped by sign-up date**: a newcomer installs whatever is current, so the date decides the version, not their enthusiasm. Then two more rules: **the same fixed window** for every cohort, and **only cohorts that have lived all of it**. > **Exercise:** Halves shipped v4.9.0, with a new first-expense flow, on Monday 11 May; before that everyone was on v4.8.0. Which comparison of the two versions is fair enough to read? ## Which cohorts have finished? **Alex Tutor:** Here are daily sign-up cohorts from 13 April to 31 May, pulled on the morning of 1 June. Retention means opening the app on at least one of **days 8–14** after sign-up; the sign-up day is day 0. **Dataset (CSV):** [Daily sign-up cohorts around v4.9.0](https://productovi.com/datasets/product-analytics/version-comparison/halves-version-cohorts.csv) New Halves users from 13 April to 31 May 2026, one row per sign-up day. v4.9.0 was released on 11 May. Data pulled on the morning of 1 June. - `signup_date`: Sign-up day (day 0) - `app_version`: Version new users installed that day - `days_observed`: Days after the sign-up day seen by 1 June - `new_users`: New users that day - `returned_any_time`: Opened the app on any later day up to 31 May - `active_days_8_14`: Opened the app on at least one of days 8–14 (so far) **Sofia:** A cohort from 25 May has `days_observed` = 6. Its day 8 hasn't even happened, so its zero means nothing. **Alex Tutor:** Exactly. And a cohort with 10 days observed has seen only part of its window. It needs `days_observed` of at least 14. ![A baker takes only the golden loaves out of an oven, leaving the pale ones that are still baking.](https://productovi.com/courses/product-analytics/inline/version-comparison-illustration-3.webp) *Count only cohorts that have lived through the whole period.* > **Exercise:** Halves's daily sign-up cohorts (v4.9.0 from 11 May) are compared on activity in days 8–14 after sign-up; the data, pulled on the morning of 1 June, runs through 31 May. How many **v4.9.0** cohorts have lived their full days 8–14 by the morning of 1 June? **Alex Tutor:** The cutoff isn't fixed. Every new pull of the data moves it, so make sure you can recompute it. > **Exercise:** Halves compares daily sign-up cohorts on activity in days 8–14 after sign-up (the sign-up day is day 0). The data runs through **7 June**. What is the **latest sign-up date** whose cohort has lived its full days 8–14? ## Before and after analysis: pool the cohorts **Alex Tutor:** Start with v4.8.0: all 28 of its cohorts are complete. Sum `active_days_8_14` and `new_users`, then divide. Sum first; averaging the daily rates would give small days the same weight as big ones. **Sofia:** So a quiet Tuesday and a busy Saturday count by their users, not as equals. **Alex Tutor:** Right. Pool the users, then divide once. > **Exercise:** What share of **v4.8.0** new users were active in days 8–14? Percent, two decimals. **Maya:** And my 15% for v4.9.0? **Alex Tutor:** That's all 21 v4.9.0 cohorts, 14 of which couldn't be fully counted yet. Most of your drop is calendar, not code. Take only the seven complete cohorts. ![On 1 June all 28 v4.8.0 cohorts are complete, but only the 7 v4.9.0 cohorts from 11–17 May have had their full days 8–14; compare only those.](https://productovi.com/courses/product-analytics/diagrams/m6-l1-complete-cohorts.png) **Sofia:** 11 to 17 May, then. Same sums, same division. > **Exercise:** Take only the **complete** v4.9.0 cohorts (11–17 May). What share was active in days 8–14? Percent, two decimals. ## A metric without a fixed horizon **Theo:** My report uses “returned at any time”: 66.8% for v4.8.0 against 47.1% for v4.9.0. That's a drop too. **Alex Tutor:** A cohort from mid-April had six weeks to come back once; a cohort from late May had days. That column measures age, not the version. **Sofia:** So a metric without a fixed horizon always favors older cohorts. > **Exercise:** “Returned at any time since sign-up” is a fair metric for comparing v4.8.0 and v4.9.0 cohorts, because it uses all the data we have. **Theo:** What if I keep “any time” but use only the complete cohorts? That should fix it. > **Exercise:** Take only the **complete** v4.9.0 cohorts (11–17 May). What share **returned at any time**? Percent, two decimals. ## What a before/after comparison can and can't prove **Maya:** So v4.9.0 is actually better? +2.33 points? **Alex Tutor:** Careful twice. First, chance: seven days, 551 users. The Wilson-based interval for the difference runs from about **−1.9 to +6.7 pp**. It could be a small loss or a decent gain. **Alex Tutor:** Second, even a narrow interval covers only chance. Anything else that changed in May, like school holidays, a campaign or the weather, sits inside the same number. Before/after can show a difference; it can't prove the release caused it. **Maya:** Then give me one line I can send to the team without someone correcting me tomorrow. > **Exercise:** Complete sign-up cohorts show 31.22% of v4.9.0 new users active in days 8–14 against 28.88% for v4.8.0: +2.33 pp, 95% interval about −1.9 to +6.7 pp. Which line can Maya, the Halves product lead, send to the team about v4.9.0 and week-2 retention? ![A gardener admires a taller plant after adding fertilizer, while the sun and a rain cloud overhead also did their part.](https://productovi.com/courses/product-analytics/inline/version-comparison-illustration-4.webp) *Before and after can't separate the release from everything else that changed.* **Sofia:** Then the honest answer is: we didn't break it, and we can't claim we improved it yet. > **Exercise:** After v4.9.0 shipped on 11 May, the dashboard of Maya, the Halves product lead, showed week-2 retention falling from 29% to 15%. Answer her question “Did v4.9.0 break week-2 retention?” in 3–5 sentences. Use the numbers from the dataset, say why her 15% is misleading, and what the comparison can and can't support. **Alex Tutor:** Put a date in the calendar: in two weeks, 21 v4.9.0 cohorts will be complete instead of seven. Same rules, a sharper number. - Previous lesson: [Case: reading a finished test](https://productovi.com/courses/product-analytics/reading-a-test.md) - Next lesson: [Traffic mix and seasonality](https://productovi.com/courses/product-analytics/traffic-and-seasons.md) --- Productovi Learning. Free, no sign-up needed to read.