# Case: a beta cohort panic-sold > Judge an A/B test of a "pause before selling" screen with a small sample and decide what to ship and measure next. - Course: [Long-Term Investing: Index Funds, Risk and Your Own Behavior](https://productovi.com/courses/long-term-investing.md) - Module: The users are the risk (and so are we) - Lesson 28/32, ~39 min - Source: https://productovi.com/courses/long-term-investing/case-panic-cohort ## The test from the spring drop **Alex Tutor:** End of September at Zapas, the last day of the behavior month. As the PM you now have the results of a test that ran during the spring drop, and a launch in six weeks. **Grace:** For two weeks in month 6, near the bottom of the drop, everyone who tapped "Sell all" was split at random. Control got the usual confirmation. The other half got a pause screen: their own plan, what the sale would lock in, and a "Sell anyway" button. **Ethan:** And it worked! 35% sold in control, 23% with the pause. A third less panic selling. I want that headline on launch day. **Grace:** Sixty users in each group. Before any headline, let's see if that's more than chance. **Dataset (CSV):** ["Pause before selling" A/B test, spring drop (synthetic)](https://productovi.com/datasets/long-term-investing/case-panic-cohort/ab_pause_screen.csv) 120 users who tapped "Sell all" during the two test weeks in month 6: the arm they saw (control or pause) and whether they went on to sell. **Share who completed the sale, by arm (synthetic)** | Arm | Completed the sale, % | | --- | --- | | Control | 35 | | Pause screen | 23.3 | *Users who tapped "Sell all" during the two test weeks. Control: 21 of 60. Pause screen: 14 of 60.* > **Exercise:** In control **21 of 60** users sold; with the pause screen **14 of 60** did. What is the difference between the two selling rates, in percentage points? Round to one decimal. **Sofia:** 11.7 points on 60 people. My gut says that's big. **Grace:** With 60 people one user is worth 1.7 points. Seven users switching their minds is the whole gap. Gut feelings about small samples are usually too confident. ## More than chance? **Sofia:** How do we check that? **Grace:** A two-proportion z-test. Pool both arms to estimate the noise, then ask how many "noise units" the gap is. You've seen it in the statistics course. $$ z = \frac{p_1 - p_2}{\sqrt{\bar p (1 - \bar p)\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}, \quad \bar p = \frac{x_1 + x_2}{n_1 + n_2} $$ *Two-proportion z-test: p₁, p₂ are the selling rates, x₁, x₂ the sellers, n₁, n₂ the arm sizes.* > **Exercise:** Control: 21 of 60 sold; pause screen: 14 of 60. Compute the z-statistic of the two-proportion test (pooled rate = 35 ÷ 120). Round to two decimals. **Ethan:** So it didn't work? **Grace:** We don't know yet. "Not significant" means "this test can't tell". The screen may cut selling a lot, a little or not at all. Sixty users per arm is too few to say. > **Exercise:** **True or false:** with z = 1.41 and p = 0.16, the test shows that the pause screen has no effect on selling. ## How big a test? **Sofia:** So how many users would we need to tell for sure? **Grace:** Never for sure, but well enough. If the true drop were 35% to 23.3%, catching it 80% of the time at the 5% level takes about 235 users per arm. **Ethan:** 470 people tapping "Sell all"? The whole beta is 2,400 users, and the spring test got only 120. **Grace:** After launch we'll have far more users, but they tap "Sell all" mostly when markets fall. In a calm year the test fills slowly. So we decide the size now and let it run until the next drop fills it. $$ n \approx \frac{(z_{\alpha/2} + z_{\beta})^2 \,[p_1(1-p_1) + p_2(1-p_2)]}{(p_1 - p_2)^2} $$ *Users needed per arm. For 5% two-sided and 80% power: z_α/2 = 1.96, z_β = 0.84.* > **Exercise:** How many users per arm would Zapas need to detect a drop from **35.0%** to **23.3%** (exact rates 21/60 and 14/60) with 80% power at the 5% two-sided level? Use z_α/2 = 1.959964 and z_β = 0.841621, and round **up** to a whole user. ![35.0% vs 23.3% on 60 users each gives z 1.41, p 0.16: don't announce a win, keep the 50/50 split to about 235 per arm; selling stays two taps away.](https://productovi.com/courses/long-term-investing/diagrams/m7-l4-test-decision.png) ## What to say, what to ship **Ethan:** I still need a line for the launch post. **Grace:** You get an honest one. And one rule I won't bend: the pause screen informs, it never blocks. The money is theirs; selling stays two taps away. > **Exercise:** Which line can Zapas publish about the pause-screen test? **Sofia:** What exactly may the pause screen do, then? **Grace:** Show the user's own plan and the numbers. Not hide the button, not count down, not guilt-trip. And we measure complaints as carefully as sales. **Ethan:** A countdown would cut selling even more. **Grace:** Probably, and it would also trap people who have a real reason to sell: a medical bill, a lost job. We don't win tests by locking doors. > **Exercise:** Which features are acceptable on Zapas's pause-before-selling screen? Select all that apply. ## Your turn: the result and the decision **Ethan:** Fine. First give me the line for the team chat, then the full decision. **Alex Tutor:** The chat line is two or three sentences. The decision is the case: what ships at launch, what you measure, and when you'll decide. > **Exercise:** Write the 2–3 sentence update for the team chat on the pause-screen test: the two rates with group sizes, what the significance check says, and what happens next. > **Exercise:** Decide what Zapas ships at launch after the pause-screen test, and how it will know whether the screen works. > **Key ideas** > > - Report an A/B result as both rates with group sizes and the absolute gap: 35.0% vs 23.3% (n = 60 each), 11.7 points. > - z = 1.41, p = 0.16: inconclusive, not "no effect". Detecting a 35% → 23.3% drop needs about 235 users per arm. > - Fix the stopping rule in advance; sell taps come mostly in falls, so the sample may need the next drop. > - A behavior feature informs and never blocks: the user's plan, the numbers, and "Sell anyway" one tap away; measure complaints too. > - Never turn an inconclusive test into a launch headline. > **Educational, not advice** > > This lesson explains how investing works; it doesn't tell you what to buy. Investments can fall, and you can get back less than you put in. All Zapas and Tessaro data is synthetic. Check the current rules where you live or ask a licensed professional. **Alex Tutor:** Next month: launching for Ukraine. First question: in which currency should a user keep money for a goal three years away? - Previous lesson: [Rules made in calm weather: a one-page investment policy](https://productovi.com/courses/long-term-investing/investment-policy.md) - Next lesson: [The currency of your goals and the safe part at home](https://productovi.com/courses/long-term-investing/currency-of-goals.md) --- Productovi Learning. Free, no sign-up needed to read.