# Power analysis and sample size

> A power analysis tells you how many users an experiment needs to detect the effect you care about.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: A/B testing in practice
- Lesson 10/24, ~38 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/power-and-sample-size

## Why 'just run it and see' fails

**Alex Tutor:** You're still answering CatChow's data questions for the product team, and the checkout redesign is next in line.

**Sofia:** Let's launch the checkout redesign test to 200 users each and check back Friday.

**Alex Tutor:** Before we do - what effect are we trying to catch, and can 200 users even show it if it's real?

**Sofia:** I... hadn't thought about that part.

**Alex Tutor:** That's **statistical power**: the probability your test detects a real effect, if one truly exists. Low power means real wins get missed. Checking it before launch is called a **power analysis**.

![In a dark room a cat shines a small, weak flashlight, and its narrow beam misses an orange toy mouse sitting just outside the light.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/power-and-sample-size-illustration-2.webp)

*Low power is a weak flashlight: the effect can be right there and still go unseen.*

> **Statistical power (1 - β)**
>
> The probability of correctly detecting a real effect (rejecting a false null hypothesis) - the complement of a type II error (β).

**Alex Tutor:** Miss a real effect and you either ship nothing when you should have, or worse - you decide the idea 'doesn't work' and drop it for good.

> **Exercise:** Higher statistical power means a lower chance of a type II error (failing to detect a real effect that exists).

**Alex Tutor:** Four things drive power: your sample size, the effect size you care about, your significance level (α), and the data's natural variability.

**Sofia:** Effect size... you mean like a percentage lift?

**Alex Tutor:** In conversion tests, we usually call it the **minimum detectable effect (MDE)** - the smallest true difference you actually want your test to be able to catch.

**Alex Tutor:** Smaller MDE = harder to catch = you need way more users. It's the single biggest lever on required sample size.

![A cat holds a huge magnifying glass over a heap of blue kibble, and only through the lens can a tiny orange piece be seen.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/power-and-sample-size-illustration-3.webp)

**Alex Tutor:** (Side note: unlike Module 1's AOV example, this lesson works with proportions, not an arithmetic average - the formulas differ slightly.)

## Power analysis: sizing a conversion test

**Alex Tutor:** Let's size a real one. CatChow's checkout page converts at a baseline p1 = 5.0%. Marketing wants to detect a rise to p2 = 6.0% - that's the MDE.

$$
n \approx \frac{(z_{\alpha/2}+z_{\beta})^2 \left[p_1(1-p_1)+p_2(1-p_2)\right]}{(p_1-p_2)^2}
$$

*Approximate sample size per variant for a two-proportion test at significance level α (two-sided) and power 1 - β.*

**Alex Tutor:** For the usual α = 0.05 (two-sided) and 80% power, the critical values are z = 1.96 and z_β = 0.84.

> **Exercise:** Compute p1(1-p1) + p2(1-p2) for p1 = 5.0% and p2 = 6.0% (as a decimal proportion, not a percentage).

**Alex Tutor:** Plugging in: n ≈ (1.96+0.84)² × 0.1039 / (0.05-0.06)² = 7.84 × 0.1039 / 0.0001 ≈ 8,146 users **per variant**.

**Sofia:** 8,146 *per variant*? That's over 16,000 total for a 1-point lift?!

**Alex Tutor:** Yes - small baseline rates and small absolute lifts are expensive to detect. That's the real cost of chasing tiny effects.

**Alex Tutor:** And it's sharply non-linear: shrink the MDE and the required sample explodes, because it's the *squared* difference in the denominator.

> **Exercise:** If you halve the MDE you're targeting (keeping α, power and baseline variability fixed), the required sample size is multiplied by about how much?

**Alex Tutor:** Now size one on your own. The newsletter sign-up form is a different page with a different baseline, but the same four steps.

> **Exercise:** CatChow’s newsletter sign-up form converts at 10%. Marketing wants a test able to detect a rise to 12% at α = 0.05 (two-sided) and 80% power (z values 1.96 and 0.84). About how many users does each variant need?

**Alex Tutor:** So what do you do when the sample-size calculator says 8,146 but your test page only gets 3,000 visits total in two weeks?

**Sofia:** Just run it for two weeks and check whatever p-value comes out?

**Alex Tutor:** That test would be badly underpowered - likely to miss a real 1-point lift even if it's there. You have real options instead.

**Alex Tutor:** Accept a bigger MDE (only detect a 2-point jump, not 1), run longer to accumulate the traffic, or test on a higher-traffic page instead.

> **Exercise:** The calculator says you need 8,146 users per variant; you only get 1,500/week. What's the honest move?

**Alex Tutor:** Whichever option you pick, someone will ask "so how long?". Turn the sign-up test into weeks.

> **Exercise:** CatChow's newsletter sign-up test needs 3,834 users per variant (two variants). The form gets about 1,200 visitors a week, split evenly between the variants. How many **full weeks** should the test run?

## Power, type I and type II errors

**Alex Tutor:** Let's pin down the vocabulary - power ties directly to the two error types from the significance-testing lessons.

**Sofia:** Type I, type II... I always mix those up.

**Alex Tutor:** Fair - most people do. Match each term below and it should click.

**Sofia:** Okay, throw it at me.

> **Exercise:** Match each term to its definition.

**Alex Tutor:** Nailed it. Now let's check which levers actually change how many users you need.

**Sofia:** I'll guess: smaller MDE needs more people.

**Alex Tutor:** Right instinct. There are three more levers just like that one.

**Sofia:** Bring on the multi-select.

> **Exercise:** Which of these increase the required sample size, holding everything else fixed? Select all that apply.

**Alex Tutor:** Now imagine someone skipped all this and ran the checkout test with whatever traffic they had. Here is what came back.

> **Exercise:** CatChow's checkout redesign test (5.0% baseline conversion) needed 8,146 users per variant to detect a 1-point lift. Someone ran it anyway with about 1,500 users per variant. The result: p = 0.41. Which summary is right?

## Power analysis before the test, not after peeking

**Alex Tutor:** One more habit to build: doing the power analysis and fixing the sample size *before* the test starts, not after seeing early results.

**Sofia:** Actually... our team's rule is every test runs exactly one week, no exceptions. 'That's our process.'

**Alex Tutor:** That's a trap disguised as consistency. A week's worth of traffic isn't the same as a week's worth of *power*.

![Two identical jars fill under one clock: a thick stream of kibble fills one high, while a thin trickle leaves the other nearly empty.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/power-and-sample-size-illustration-4.webp)

*A week of a busy page and a week of a quiet page are not the same amount of evidence.*

**Sofia:** Because traffic isn't constant...

> **Exercise:** Your team's rule is that every A/B test runs exactly one week, no exceptions. In 2-3 sentences, explain to a teammate why fixing the test duration at 'always one week' is worse than fixing the required sample size.

**Alex Tutor:** Good - now a messier, real situation to reason through.

**Sofia:** Bring it.

**Alex Tutor:** CatChow's checkout page gets about 500 sessions/week. Marketing wants to test a new 'subscribe & save' banner, targeting a lift from 5.0% to 6.0% conversion.

**Sofia:** That's the exact example from before - 8,146 per variant, right?

> **Exercise:** Should CatChow launch this test on schedule?

**Alex Tutor:** That's the real skill: catching an underpowered plan *before* it wastes a month proving nothing.

**Sofia:** Next: actually running the conversion-rate test properly, right?

**Alex Tutor:** Exactly - proportions, the z-test, and reading uplift without fooling yourself.

- Previous lesson: [95% confidence intervals](https://productovi.com/courses/statistics-for-pm-and-marketers/confidence-intervals.md)
- Next lesson: [Conversion rates and the two-proportion z-test](https://productovi.com/courses/statistics-for-pm-and-marketers/conversion-rate-tests.md)

---
Productovi Learning. Free, no sign-up needed to read.