# Case: reading a finished test

> Effect, interval, guardrails and the decision.

- Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md)
- Module: Did the change work?
- Lesson 20/32, ~44 min
- Source: https://productovi.com/courses/product-analytics/reading-a-test

## Six weeks later

**Alex Tutor:** Six weeks on, the Halves paywall test is over, and the product team is waiting for your reading.

**Maya:** The paywall test is done! I'm putting **“Trials +74%”** in the all-hands deck.

**Theo:** And iOS alone is up 2 points. Can we ship there first?

**Alex Tutor:** Let's read it in the order the plan fixed six weeks ago: check the split, then the primary metric with its interval, then the guardrail, then the rule. Trials and platforms come last.

**Dataset (CSV):** [Halves paywall test: control vs trial_first](https://productovi.com/datasets/product-analytics/reading-a-test/halves-paywall-test.csv)

Users enrolled at their first paywall view, 3–30 August 2026, each followed for 14 days. One row per enrollment week, platform and variant.

- `enroll_week`: Monday of the week the users entered the test
- `platform`: ios or android
- `variant`: `control` (current paywall) or `trial_first` (leads with a 7-day trial)
- `users`: Users enrolled
- `trials_started`: Users who started a trial within 14 days (secondary)
- `paid_14d`: Users with a first paid charge within 14 days (primary)
- `active_days_8_14`: Users who added an expense in days 8–14 (guardrail)

**Alex Tutor:** The split first. 6,897 users in control, 6,792 in trial_first. With a 50/50 assignment that gap is normal; a chi-square check gives 0.81, well under 3.84. A big imbalance would mean broken assignment, and then nothing else is worth reading.

**Sofia:** What would a broken split look like?

**Alex Tutor:** Something like 7,400 against 6,300. It's usually a bug: one variant crashes on older phones, or logs some users twice. You fix it and rerun; you don't interpret it.

**Alex Tutor:** Make sure you'd spot one. A 50/50 test, about 13,700 users in total.

> **Exercise:** A 50/50 test enrolled about 13,700 users. Which split should make you **stop and look for a bug** before reading any metric?

![Two scales: one holds two nearly equal piles of person tokens and stays level, the other tilts sharply to one side.](https://productovi.com/courses/product-analytics/inline/reading-a-test-illustration-2.webp)

*Check the split first: a lopsided one usually means a bug.*

**Sofia:** Then the primary: sum `paid_14d` and `users` for each variant and divide.

> **Exercise:** Use the dataset of the Halves paywall test (control vs trial_first). What share of **control** users paid within 14 days? Answer in percent, to two decimals.

## The effect and its interval

**Alex Tutor:** trial_first, same sums: 345 of 6,792, so 5.08%.

**Sofia:** So is the effect 5.08 − 3.77, or 5.08 ÷ 3.77?

**Alex Tutor:** The effect is always **treatment − control**, in percentage points. The ratio is a relative lift, and it must be labelled as one.

**Theo:** Points sound smaller than percent.

**Alex Tutor:** Which is exactly why a readout says which one it means.

> **Exercise:** In the Halves paywall test, 260 of 6,897 control users and 345 of 6,792 trial_first users paid within 14 days (the primary metric). What is the effect, **treatment − control**, in percentage points? Two decimals.

**Alex Tutor:** Now the interval. For each rate we use the **Wilson** interval. It behaves well when rates are small, like 4%, where the simple ± formula can mislead.

$$
\frac{p + \frac{z^2}{2n} \pm z\sqrt{\frac{p(1-p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}
$$

*Wilson 95% interval for a rate p = x / n, with z = 1.96. Control: 3.35% to 4.25%. trial_first: 4.58% to 5.63%.*

**Alex Tutor:** For the difference, combine the two Wilson intervals. Each bound of the effect borrows the matching distances from both rates. Our [A/B test results calculator](/tools/abn-test-results) uses the same method.

$$
L = d - \sqrt{(p_T - L_T)^2 + (U_C - p_C)^2}, \quad U = d + \sqrt{(U_T - p_T)^2 + (p_C - L_C)^2}
$$

*Interval for d = p_T − p_C (treatment − control) from the Wilson intervals [L_T, U_T] and [L_C, U_C].*

**Sofia:** So for the lower bound: how far trial_first could fall, and how far control could rise.

> **Exercise:** In the paywall test, control paid at p_C = 3.77% (Wilson 95% interval 3.35% to 4.25%) and trial_first at p_T = 5.08% (Wilson 95% interval 4.58% to 5.63%), so the effect is d = 1.31 pp. The two intervals combine into a lower bound L = d − √((p_T − L_T)² + (U_C − p_C)²). Compute the **lower bound** of the 95% interval for treatment − control, in percentage points. Two decimals.

**Alex Tutor:** Now put it in one line for the deck, the way it should be quoted.

> **Exercise:** In the Halves paywall test, the share who paid within 14 days (the primary metric) went from 3.77% in control to 5.08% with the trial-first paywall. Which headline reports the effect **correctly**?

## Power, guardrail, rule

**Alex Tutor:** Primary: **+1.31 pp, 95% interval +0.62 to +2.00 pp**. The test was sized for 6,735 users per variant to catch +1 pp and got more in both. It had the power it was built for.

![Two clusters of arrows in a target, each spread over a small area, sitting clearly apart from each other.](https://productovi.com/courses/product-analytics/inline/reading-a-test-illustration-3.webp)

*An effect comes with a range; here the whole range sits above zero.*

**Alex Tutor:** Guardrail, `active_days_8_14`: 60.53% in control, 60.60% in trial_first, **+0.07 pp, interval −1.57 to +1.70 pp**. The margin was −3 pp.

![The primary effect, +1.31 pp (+0.62 to +2.00), clears 0 and the guardrail stays above −3 pp, so the paywall ships; Android alone is inconclusive.](https://productovi.com/courses/product-analytics/diagrams/m5-l4-test-readout.png)

**Sofia:** So that's not “no effect on usage”. It's “any drop is smaller than we agreed to accept”.

> **Exercise:** The Halves trial-first paywall test's results for all users: primary (paid within 14 days) **+1.31 pp, 95% interval +0.62 to +2.00 pp**; guardrail (`active_days_8_14`) **+0.07 pp, interval −1.57 to +1.70 pp**. By platform, the primary is +2.00 pp on iOS and +0.29 pp on Android (interval −0.69 to +1.28). Apply the decision rule from the plan: ship if the primary's interval is entirely above 0 and the guardrail's interval is above −3 pp. What is the call?

## The tempting numbers

**Maya:** Fine, it ships. But can I still say “trials +74%”? It sounds much better.

**Alex Tutor:** It's a secondary metric and a relative lift. The claim is the primary: +1.31 pp, from 3.77% to 5.08%, interval +0.62 to +2.00. Trials explain how it worked, not whether.

**Alex Tutor:** Compute the trials effect the same way as the primary, in points, so you can see what “+74%” actually is.

> **Exercise:** Use the dataset of the Halves paywall test. What is the effect on **trials started**, treatment − control, in percentage points? Two decimals.

**Theo:** And the platforms? iOS is +2.00 pp, Android +0.29 pp with an interval from −0.69 to +1.28.

**Alex Tutor:** The test wasn't sized for platforms, and the rule is for all users. Android's interval includes 0 **and** +1 pp: that's inconclusive, not “no effect”. Worth a follow-up, not a separate decision.

![Someone holds up one shiny pebble picked from a pile of plain ones, while a colleague points back at the whole pile.](https://productovi.com/courses/product-analytics/inline/reading-a-test-illustration-4.webp)

*A tempting slice isn't the result; the planned main number is.*

**Alex Tutor:** Before you answer Theo, check his iOS slice against the whole plan, guardrail included: on iOS it's −1.53 pp, from −3.66 to +0.60.

> **Exercise:** **True or false:** in the Halves paywall test, on iOS alone the interval for the primary metric (paid within 14 days) is +1.05 to +2.96 pp, so shipping trial_first on iOS only would be the safest call.

> **Exercise:** Theo, the Halves growth marketer, proposes shipping trial_first on iOS only, “because Android shows nothing”. By platform, the primary is +2.00 pp on iOS (95% interval +1.05 to +2.96, guardrail interval −3.66 to +0.60) and +0.29 pp on Android (interval −0.69 to +1.28); the plan's rule was written for all users. Reply to him in 2–4 sentences.

## Case: the readout

**Maya:** I'll forward the readout to the whole team. Short, with the numbers, and something I can defend when someone asks “are you sure?”

**Sofia:** And the numbers we haven't computed, like the interval for trials?

**Alex Tutor:** Same method, same dataset, if you want them. They won't change the decision, though.

**Alex Tutor:** Then write it the way the plan reads: primary with its interval, power, guardrail, the tempting numbers in their place, the decision, and what to watch next.

> **Exercise:** Write the readout of the paywall test for Maya.

**Alex Tutor:** Notice what made this easy: every question had its answer written down before the data arrived. Reading the test was just checking the numbers against the plan.

- Previous lesson: [Who is in the test: triggers and cluster randomization](https://productovi.com/courses/product-analytics/who-is-in-the-test.md)
- Next lesson: [Pre/post analysis: comparing versions fairly](https://productovi.com/courses/product-analytics/version-comparison.md)

---
Productovi Learning. Free, no sign-up needed to read.