Power analysis and sample size

Every test can go wrong in two ways. It can "find" an effect that isn't there (a type I error, controlled by α), or it can miss an effect that is there (a type II error, with probability β).

Statistical power is the other side of β:

power = 1 − β

It's the probability that the test comes out significant when the effect you care about is real. The usual target is 80%: if the redesign truly lifts conversion by the amount you planned for, the test will detect it four times out of five, and miss it once.

Why a product team should care: an underpowered test usually ends with "no significant difference". People then read that as "the idea doesn't work" and drop it, even though the test never had a fair chance to see the effect.

Power isn't something you read off after the test. You set it beforehand, by choosing the sample size. That calculation is a power analysis, and it's what the rest of this lesson builds.

The sample-size formula for comparing two conversion rates has three ingredients: the critical values, the difference you want to detect, and how noisy the outcome is. For a yes/no outcome, the noise is p(1 − p).

For a 5% baseline and a 6% target:

• 0.05 × 0.95 = 0.0475;
• 0.06 × 0.94 = 0.0564;
• sum = 0.1039.

That sum goes on top of the formula. p(1 − p) is largest at p = 0.5 (0.25) and shrinks toward 0 and 1, so a metric around 50% is noisier per user than one around 5%. Small conversion rates are still expensive to test, though, because the differences between them are tiny.

Keep p as a decimal. Writing 5 × 95 + 6 × 94 = 1,039 looks similar but is 10,000 times bigger, and if the difference below is still in decimals, the sample size comes out absurd. Pick one unit and use it everywhere; decimals are the safer habit.

The minimum detectable effect (MDE) is the smallest true difference you want the test to be able to catch. It's a business decision: what lift would be worth shipping?

In the sample-size formula the MDE sits in the denominator squared:

n ≈ (z_α/2 + z_β)² × [p1(1 − p1) + p2(1 − p2)] / (p1 − p2)²

So the required sample moves with 1 / MDE². Halve the MDE and you need about 4 times the users; cut it to a third and you need about 9 times.

With a 5% baseline at α = 0.05 and 80% power:

• detecting 5% → 7% (2 points) needs about 2,207 users per variant;
• detecting 5% → 6% (1 point) needs about 8,146.

The ratio isn't exactly 4 because the variance term changes a little with p2, but "half the MDE, four times the users" is the right instinct.

This is why chasing tiny lifts is expensive. Before asking for a 0.5-point MDE, check whether a lift that small would even change the decision.

Sizing a conversion test takes four small calculations. Take a 10% baseline and a 12% target at α = 0.05 (two-sided) and 80% power:

1. Critical values: (1.96 + 0.84)² = 2.8² = 7.84. These two numbers only change if you change α or power.
2. Variance term: 0.10 × 0.90 + 0.12 × 0.88 = 0.09 + 0.1056 = 0.1956.
3. Difference squared: (0.12 − 0.10)² = 0.02² = 0.0004.
4. Divide: n ≈ 7.84 × 0.1956 / 0.0004 ≈ 3,834 users per variant.

Then double it for the total (about 7,668 for two variants) and round up: a sample-size answer is a minimum, so 3,833.76 becomes 3,834, never 3,833.

The step that goes wrong most often is the denominator. The difference is 0.02, but the formula needs its square, 0.0004. Divide by 0.02 instead and you get about 77 users per variant, a test that could only detect enormous effects.

This formula is an approximation for two proportions. Online calculators may differ by a few percent; that's fine for planning.

A power analysis often returns an uncomfortable number: 8,146 users per variant for a page that sees 1,500 a week. There are honest ways out and dishonest ones.

Honest moves change the plan:

• accept a bigger MDE. Commit only to catching a larger lift. Because n follows 1 / MDE², this helps a lot;
• run longer until the sample is reached, and say up front how long that is;
• test where the traffic is: a busier page or step of the funnel;
• reduce noise in the metric, for example by excluding bot traffic.

Dishonest moves hide the problem:

• running for a fixed short window and reporting whatever comes out;
• loosening α just to make the calculator happy. At α = 0.10 the sample drops by about a fifth (to roughly 6,416), but the false-positive rate doubles;
• adding metrics so that "something" turns significant.

If no honest option fits, the honest answer is to not run this test now, or to run it and call it directional, not a decision.

Stakeholders don't think in users per variant; they think in weeks. The conversion is simple:

duration = (users per variant × number of variants) / eligible users per week

For CatChow's newsletter sign-up test, which needs 3,834 users per variant: 3,834 × 2 = 7,668 users. At 1,200 visitors a week that's 7,668 / 1,200 ≈ 6.39 weeks.

Two rounding rules:

• round up, because the sample size is a minimum;
• round to full weeks. Behaviour on Mondays isn't behaviour on Saturdays, so a test that stops mid-week over-represents some days. 6.39 becomes 7 weeks.

Count only the traffic that actually reaches the tested element: if the change sits on the payment step, page views of the home page don't help.

The classic slip is forgetting the second group: 3,834 / 1,200 ≈ 3.2 weeks looks twice as fast, but each variant gets only half the traffic. And if the honest answer comes out at many months, go back to the MDE rather than stopping early.

Planning a test means deciding how often you're willing to be wrong in each direction. Four terms do that work:

• type I error (α): you reject a true null hypothesis. A false positive: you ship a banner that does nothing. α = 0.05 means you accept this 5% of the time when there's no effect;
• type II error (β): you fail to reject a false null hypothesis. A false negative: you drop a change that really works;
• statistical power (1 − β): the chance of correctly detecting a real effect of the size you planned for. 80% is the usual target;
• minimum detectable effect (MDE): the smallest true difference the test is designed to catch.

α and power are probabilities; the MDE is a size. The three are tied together by the sample size: once you fix any two of them, the data you collect decides how good the third can be.

A memory hook: type I is crying wolf when there's no wolf; type II is missing the wolf that's really there.

Each choice in a power analysis has a price in users. Starting from 8,146 per variant (5% → 6%, α = 0.05, 80% power):

• smaller MDE: the biggest effect, because it's squared. 5% → 5.5% needs roughly four times as many users;
• higher power: going from 80% to 95% swaps z_β = 0.84 for 1.645, so (1.96 + 1.645)² ≈ 13.0 instead of 7.84, about 13,503 per variant;
• stricter α: α = 0.01 uses 2.576 instead of 1.96, so (2.576 + 0.84)² ≈ 11.67, about 12,124 per variant;
• noisier outcome: a baseline nearer 50% has a bigger p(1 − p) per user.

The reverse moves (a larger MDE, lower power, looser α) all cut the sample, and each one has a cost: you miss smaller effects, miss real effects more often, or get more false positives.

So the conversation with stakeholders is a trade-off, not a calculator setting. Decide which risk the business can live with, then let the sample size follow.

A p-value above 0.05 says one thing: this data didn't give strong evidence of a difference. Whether that's informative depends on power.

• In a well-powered test (the planned sample was reached), a non-significant result is useful: if there were an effect as big as your MDE, you'd probably have seen it. You can say "any effect is likely smaller than the MDE".
• In an underpowered test, the same p-value says almost nothing. With about 1,500 users per variant instead of 8,146, a real 1-point lift would usually come out non-significant. Missing it was the expected outcome.

So before writing "the redesign doesn't work", check what the test could have detected. A confidence interval for the difference helps: if it runs from −1 to +2 points, a 1-point gain is still entirely plausible.

The practical rule: report non-significant results from small tests as inconclusive, show the interval, and decide whether a properly sized test is worth running. Dropping an idea on an underpowered "no" is how good changes quietly die.