Case: a beta cohort panic-sold

The test: for two weeks of the spring drop, every beta user who tapped "Sell all" was randomly assigned to one of two arms.

• Control (60 users): the usual confirmation. 21 sold, 35.0%.
• Pause screen (60 users): their own plan, what the sale would lock in, a "Sell anyway" button. 14 sold, 23.3%.

There are two ways to state the difference:

• absolute: 35.0 − 23.3 = 11.7 percentage points;
• relative: 11.7 ÷ 35.0 = 33.3% fewer sellers.

Both are honest if labelled. The relative one sounds bigger, which is why headlines prefer it. Report the absolute difference next to the two rates and the group sizes.

Random assignment is what makes the comparison fair: the two arms differ only in the screen. But with 60 users per arm, chance alone can produce sizeable differences. That's the next check.

To see whether two rates differ by more than chance, compare the gap with the noise you'd expect if the screen did nothing.

1. Pooled rate, as if both arms were one group: (21 + 14) ÷ 120 = 29.2%.
2. Standard error of the difference: √(0.292 × 0.708 × (1/60 + 1/60)) ≈ 0.083, about 8.3 points.
3. z = gap ÷ standard error = 0.117 ÷ 0.083 ≈ 1.41.

A z of 1.41 means the gap is about 1.4 times the typical noise. For a two-sided test at 5%, you'd need about 1.96. The matching p-value is 0.16: if the pause screen had no effect at all, a gap this large or larger would show up about one time in six.

The method and its limits are taught step by step in the statistics course, in Conversion rates and the two-proportion z-test.

A test can end in three ways, and "not significant" is easy to misread.

• Significant: the gap is unlikely to be chance alone (at 5%, p below 0.05). You can say there is an effect, with its uncertainty.
• Not significant, small sample: the test couldn't tell. A real effect may be there; the data just can't separate it from noise. This is Zapas's case: p = 0.16 on 60 users per arm.
• Not significant, large sample: a big test that still finds nothing makes a large effect unlikely, though a small one may remain.

So two wrong conclusions are equally tempting: Ethan's "a third less panic" (treating noise as proof) and "it didn't work" (treating no proof as proof of nothing).

The useful question is "how big a test would we need?". That depends on the effect you hope to detect, the error rates you accept and the base rate. More in Power analysis and sample size.

Before running a test again, decide how big it must be to answer the question.

You need four inputs:

• the base rate: 35% of control users sold;
• the smallest effect worth detecting: say the drop we saw, to 23.3%;
• the significance level: 5% two-sided (z = 1.96);
• the power: an 80% chance of detecting the effect if it's real (z = 0.84).

Plugging them in gives about 235 users per arm, roughly 470 people who tap "Sell all", four times the spring test. A smaller true effect needs far more: the sample grows with 1 ÷ (gap)².

For Zapas there's a twist: users tap "Sell all" mostly during falls. Calm months add few test users, so reaching 470 may take until the next big drop. Plan for that: keep the split running from launch and fix the stopping rule (235 per arm) in advance, so nobody stops early on a lucky week.

An honest report of a small test has four parts:

1. What was tested, in one line: a pause screen for users tapping "Sell all" during a fall.
2. Both rates and the sizes: 35.0% (21 of 60) vs 23.3% (14 of 60).
3. The uncertainty, in plain words: "too few users to be sure" (z = 1.41, p = 0.16 for the team).
4. What happens next: the test continues to a set size.

What turns it dishonest: relative numbers without the base ("a third less"), dropped group sizes, and verbs like "cuts", "stops" or "proves". In an investing product, a claim about protecting users from their own panic is a promise about their money, and it must be backed as carefully as a return figure.

Internally the same rule holds: the team chat gets the rates, the sizes and the p-value, not "the pause screen works".

A feature that tries to change what users do with their money needs limits that no test result can override.

Allowed: information and reflection.

• the user's own plan, especially their "in a fall" line;
• the numbers: what the sale would lock in, the change since they started, their horizon;
• one extra, clearly labelled step ("Sell anyway").

Not allowed: obstruction and pressure.

• countdowns or waiting periods before a user can sell;
• hidden or greyed-out sell buttons, confusing wording;
• guilt, shame or fear messages.

Measured alongside the main metric: complaints and support tickets, time to complete a sale, account closures. If the screen lowers selling but raises complaints, it's a warning, not a win.

The reason is simple: the money belongs to the user. A product earns the right to nudge only by being transparent about it.

The team chat is where test results turn into folklore, so the first message matters. Three sentences cover it:

1. The numbers with their sizes: 35.0% (21 of 60) in control vs 23.3% (14 of 60) with the pause screen, 11.7 points lower.
2. What the check says: z = 1.41, p = 0.16, inconclusive. It isn't proof of an effect, and it isn't proof there is none.
3. What happens next: the split continues to a size fixed in advance, about 235 users per arm, and nobody announces a result before then.

Avoid both shortcuts: "it works, a third less panic" and "it failed". Each will be repeated long after the details are forgotten.

The pause-screen case has no clean answer from the data, which is exactly what makes it a product decision.

A strong decision separates five things:

1. The evidence: 35.0% vs 23.3% on 60 users per arm, z = 1.41, p = 0.16: inconclusive.
2. The launch choice: keep a randomized split (or ship widely with a held-out control) and a stopping rule fixed now, about 235 per arm; expect the sample to fill mostly during the next fall.
3. What is said publicly: nothing that claims an effect.
4. Metrics: the share of "Sell all" taps that end in a sale, then re-buys and the users' money-weighted return; guardrails on complaints, time to sell and closures.
5. Design limits and the plan: inform, never block; lead with the user's own "In a fall" line.

Uncertainty is not a reason to do nothing; it's a reason to decide how you'll learn.