# Conversion rates and the two-proportion z-test

> Comparing conversion rates between variants with a two-proportion z-test and reading uplift correctly.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: A/B testing in practice
- Lesson 11/24, ~44 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/conversion-rate-tests

## Is the A/B test result significant?

**Alex Tutor:** The checkout test at CatChow has finished, and the product team wants your read on it first.

**Sofia:** Results are in! Control converts at 5.2%, treatment at 6.8%. That's a 31% lift - let's ship it!

**Alex Tutor:** Hold on - 31% *relative* lift sounds huge, but that's computed from only 1,000 users per variant. Let's actually test whether it's real.

**Sofia:** Wait, isn't 6.8% just... bigger than 5.2%? What's there to test?

**Alex Tutor:** Bigger in *this* sample, yes. But both numbers came from random users - a **two-proportion z-test** tells us if the gap is bigger than sampling noise would explain.

![A cat squints through a light haze at two almost identical food bowls, one blue and one orange; the orange bowl is only slightly fuller.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/conversion-rate-tests-illustration-2.webp)

**Dataset (CSV):** [CatChow checkout redesign A/B test](https://productovi.com/datasets/statistics-for-pm-and-marketers/conversion-rate-tests/catchow-checkout-redesign.csv)

One row per user: which variant they saw (control or treatment) and whether they converted (1) or not (0). Synthetic data made for this course.

- `user_id`: User identifier
- `variant`: Which variant the user saw: control or treatment
- `converted`: 1 if the user completed checkout, 0 otherwise

**Alex Tutor:** Here's the raw data, one row per user: which variant they saw, and whether they converted.

> **Exercise:** Using the dataset **CatChow checkout redesign A/B test**, what is the relative lift of treatment (6.8%) over control (5.2%), as a percentage?

## Two-proportion z-test, step by step

**Alex Tutor:** 30.8% relative lift is the headline number. But for the test, we first need the **pooled conversion rate** - as if both groups were one.

$$
\bar{p} = \frac{x_1+x_2}{n_1+n_2}
$$

*Pooled proportion: total conversions divided by total users, both variants combined.*

![Cat paws tip a blue bowl and an orange bowl so their kibble pours together into one large shared bowl in the middle.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/conversion-rate-tests-illustration-3.webp)

*The pooled rate pretends, for a moment, that both variants are one big group.*

**Alex Tutor:** Control had 52 conversions from 1,000; treatment had 68 from 1,000. Combined: 120 conversions from 2,000 users.

**Sofia:** So pooled is just 120 over 2,000?

**Alex Tutor:** Exactly. It's used only to compute the standard error under the assumption that both variants truly convert at the same rate.

> **Exercise:** What is the pooled conversion rate p̄, as a percentage, given 52 conversions from 1,000 (control) and 68 from 1,000 (treatment)?

**Alex Tutor:** Before we use it, one question about why the pooled rate is there at all.

> **Exercise:** Why does the z-test use the pooled rate (6%) to compute the standard error, instead of each group’s own rate?

**Alex Tutor:** (Side note: proportions use their own SE formula here, not the one for an arithmetic average - the logic rhymes with Module 1, but the formula differs.)

**Alex Tutor:** Now the standard error, using the pooled rate: SE = sqrt[p̄(1-p̄) x (1/n1 + 1/n2)].

$$
SE = \sqrt{\bar{p}(1-\bar{p})\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}
$$

**Alex Tutor:** Plugging in: sqrt[0.06 x 0.94 x (1/1000 + 1/1000)] = sqrt[0.0564 x 0.002] = sqrt(0.0001128) ~ 0.0106.

**Sofia:** And then z is just the gap divided by that?

**Alex Tutor:** Exactly: z = (p2 - p1) / SE = 0.016 / 0.0106 ~ 1.51.

> **Exercise:** A z-statistic of about 1.51 corresponds to a two-tailed p-value of roughly 0.13, which is below the usual 0.05 significance threshold - so this result is statistically significant.

**Alex Tutor:** So with only 1,000 users per variant, that 30.8% relative lift isn't distinguishable from noise at the usual bar.

**Sofia:** But treatment really did convert better *in this sample*. Doesn't that count for something?

**Alex Tutor:** It's a real observation - just not strong enough evidence yet. The 95% CI for the *difference* makes that concrete.

**Alex Tutor:** That interval is about -0.5 to +3.7 percentage points. It includes zero - 'no real difference' is still a plausible explanation.

> **Exercise:** CatChow's checkout redesign converted at 6.8% against 5.2% for control, with 1,000 users per variant: a 30.8% relative lift. Sofia, a product manager, wants to ship it immediately. What should she check first?

## Z-test or chi-square test for conversion?

**Sofia:** An analyst on the data team runs a chi-square test on tables like this. Would that give a different answer?

**Alex Tutor:** Same question, same answer. For a 2×2 table (variant × converted or not) the **chi-square test** compares the observed counts with the counts you'd expect if both variants converted at the pooled 6%: 60 conversions and 940 non-conversions each.

**Alex Tutor:** Add up (observed − expected)² / expected over the four cells: 64/60 + 64/60 + 64/940 + 64/940 ≈ 2.27. That's exactly z² (1.51² ≈ 2.27), and the p-value is the same, about 0.13.

**Alex Tutor:** So for two variants, pick either. The z-test also gives you the direction and a CI for the difference; chi-square is the one that scales to three or more variants or outcomes.

**Alex Tutor:** Now run the whole test yourself on a different experiment: two email subject lines, 1,500 recipients each.

> **Exercise:** A separate email test: subject line A gets 90 orders from 1,500 recipients, subject line B gets 120 from 1,500. Compute the two-proportion z-statistic (B minus A, pooled SE). Round to two decimals.

**Alex Tutor:** Let's lock in the whole procedure - the steps, in order, from raw counts to a decision.

**Sofia:** I keep mixing up when the pooled rate comes in versus the individual rates.

**Alex Tutor:** This ordering exercise should settle that once and for all.

**Sofia:** Let's go.

> **Exercise:** Put the steps of a two-proportion z-test in the right order.

**Alex Tutor:** One last check - can you spot which of these follow-up claims actually hold up?

**Sofia:** Multiple correct answers again? Fine, I'm ready.

**Alex Tutor:** This one ties together everything we just covered, including a callback to last lesson's sample-size math.

**Sofia:** No pressure, then.

> **Exercise:** The checkout test: control 52 orders from 1,000 users (5.2%), treatment 68 from 1,000 (6.8%), z ≈ 1.51, p ≈ 0.13. Which of these statements about this test are correct? Select all that apply.

## Explaining the headline-lift trap

**Alex Tutor:** First, a quick one. Two possible lines for the weekly report. Which one goes in?

> **Exercise:** Which line belongs in the weekly report about the checkout test?

**Alex Tutor:** This is the pattern for every conversion-rate test you'll run at CatChow. Let's put it in your own words.

![A cat holds a magnifying glass over one small orange kibble piece; inside the lens it looks huge, while on the table it stays tiny.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/conversion-rate-tests-illustration-4.webp)

*A big relative lift can sit on top of a tiny absolute gap.*

**Sofia:** You want me to explain the headline-lift trap?

**Alex Tutor:** Exactly that - for someone in marketing who's never seen a z-test.

**Sofia:** Got it, give me a second.

> **Exercise:** In 2-3 sentences, explain to a marketing colleague why 'treatment converts 31% better than control' is a misleading headline for this specific test: 5.2% vs 6.8% conversion, 1,000 users per variant, p ≈ 0.13.

**Alex Tutor:** Nice framing. Now a full memo to write, using everything from this lesson.

**Sofia:** A whole memo?

**Alex Tutor:** Just one solid paragraph. Marketing is waiting on your recommendation before the Friday planning call.

**Sofia:** Okay - walk me through the ask.

> **Exercise:** Write your recommendation for the Friday planning call.

**Alex Tutor:** That balance - not overclaiming a win, not dismissing a real signal - is the whole job here.

**Sofia:** Next lesson: all the ways this can still go wrong even when the math is right?

**Alex Tutor:** Exactly - peeking, multiple metrics, novelty effects. The pitfalls that break tests that look perfectly sound.

- Previous lesson: [Power analysis and sample size](https://productovi.com/courses/statistics-for-pm-and-marketers/power-and-sample-size.md)
- Next lesson: [Pitfalls: peeking, multiple testing, novelty effect](https://productovi.com/courses/statistics-for-pm-and-marketers/ab-testing-pitfalls.md)

---
Productovi Learning. Free, no sign-up needed to read.