Conversion rates and the two-proportion z-test

When one variant converts better than another, there are two honest ways to state the gap, and they sound very different.

• Absolute lift is the plain difference, in percentage points (pp): 6.8% − 5.2% = 1.6 pp.
• Relative lift is that difference as a share of the control rate: (6.8 − 5.2) / 5.2 = 0.016 / 0.052 ≈ 30.8%.

Both are correct, but they answer different questions. Relative lift tells you how big the change is compared with where you started. Absolute lift tells you how many extra people convert: 1.6 pp means about 16 extra orders per 1,000 visitors.

With small baseline rates, relative lifts look dramatic. Going from 1% to 1.5% is a 50% lift and only half a point in absolute terms. So a headline that only says "+31%" invites people to imagine a much bigger business effect than 16 orders per 1,000.

Always divide by the control rate, and always say which kind of lift you mean: "pp" for absolute, "%" for relative.

The two-proportion z-test starts from a "what if": suppose the redesign makes no difference at all (the null hypothesis). Then both groups share one true conversion rate, and the best estimate of it uses all the data at once. That's the pooled conversion rate:

p̄ = (x1 + x2) / (n1 + n2)

where x is the number of conversions and n the number of users in each group. For the checkout test: (52 + 68) / (1,000 + 1,000) = 120 / 2,000 = 0.06, or 6.0%.

With equal group sizes this happens to equal the plain average of the two rates. With unequal groups it doesn't, and the shortcut gives the wrong number. Say control had 40 conversions from 800 users (5%) and treatment 90 from 1,200 (7.5%). Pooled: 130 / 2,000 = 6.5%. The average of the two rates, 6.25%, gives the smaller group too much weight.

So pool the counts, not the percentages: add up conversions, add up users, then divide.

The two-proportion z-test and the confidence interval for the difference use slightly different standard errors, and the reason is what each one assumes.

The test asks: if the redesign made no difference, how unusual would this gap be? So it works inside the null hypothesis, where both groups share one rate. The best estimate of that rate is the pooled p̄, and the SE is

√(p̄(1 − p̄)(1/n1 + 1/n2))

The interval asks a different question: how big could the real difference be? It doesn't assume the rates are equal, so each group keeps its own:

√(p1(1 − p1)/n1 + p2(1 − p2)/n2)

In the checkout test both come out at about 0.0106, so the choice barely matters. It matters more when the groups differ a lot in size or rate. Pooling isn't a way to make results significant; it's the honest SE for the specific question the test asks.

If you remember one line: pooled when testing "no difference", separate when estimating "how much".

The z-statistic measures the gap between the two rates in standard errors:

z = (p2 − p1) / SE

For CatChow's checkout redesign test (5.2% vs 6.8%, a gap of 0.016, with an SE of about 0.0106): 0.016 / 0.0106 ≈ 1.51. The gap is about one and a half standard errors wide.

To decide, compare it with the critical value or read the p-value. For a two-sided test at α = 0.05, the critical value is 1.96: a |z| above it is significant. z = 1.51 falls short, and the matching p-value is about 0.13.

The p-value is the probability of seeing a gap at least this big if there were truly no difference. Here that's about 13%: noise alone produces a gap like this fairly often. Since 0.13 is above 0.05, the result is not statistically significant.

Two misreadings to avoid. A p-value of 0.13 is not "87% sure the redesign works". And "not significant" is not "no effect": it means this sample can't tell the two apart yet.

The z-test gives a yes/no answer. The confidence interval for the difference shows the whole picture: how big the true gap could plausibly be.

(p2 − p1) ± 1.96 × √(p1(1 − p1) / n1 + p2(1 − p2) / n2)

Note that here each group keeps its own rate: the pooled rate belongs only to the test, which assumes no difference.

For the checkout test: √(0.052 × 0.948 / 1,000 + 0.068 × 0.932 / 1,000) ≈ 0.0106, so the margin is 1.96 × 0.0106 ≈ 2.1 pp. The interval is 1.6 ± 2.1, roughly −0.5 to +3.7 percentage points.

Read it in two ways:

• it includes zero, so "no real difference" is still plausible. That matches p > 0.05;
• it reaches +3.7 pp, so a large real win is plausible too.

That's why the right call is "we don't know yet", not "ship it" and not "it failed". The interval tells stakeholders exactly how much uncertainty is left.

Here is the full test on a second example, so you can check each step. Subject line A: 90 orders from 1,500 emails. Subject line B: 120 from 1,500.

• rates: p1 = 0.06, p2 = 0.08, a gap of 0.02 (2 pp, a 33% relative lift);
• pooled rate: (90 + 120) / 3,000 = 0.07;
• SE: √(0.07 × 0.93 × (1/1,500 + 1/1,500)) = √(0.0651 × 0.001333) ≈ 0.00932;
• z = 0.02 / 0.00932 ≈ 2.15;
• |z| > 1.96, p ≈ 0.03: significant at α = 0.05.

The 95% CI for the difference, with separate SEs, runs from about +0.2 to +3.8 pp. It doesn't include zero, which matches the significant test.

Compare with the checkout test: a similar relative lift (33% vs 31%), but 1,500 users per group instead of 1,000 and a bigger absolute gap. The evidence crosses the line here and didn't there. Significance depends on the gap and the sample, never on the headline percentage alone.

Watch the parentheses: the 1/n1 + 1/n2 term sits inside the square root.

The whole procedure fits on a sticky note:

1. Each group's rate: p1 = 52 / 1,000 = 5.2%, p2 = 68 / 1,000 = 6.8%.
2. Pooled rate: p̄ = 120 / 2,000 = 6%. It describes the world where the null hypothesis is true.
3. Standard error from the pooled rate: √(p̄(1 − p̄)(1/n1 + 1/n2)) = √(0.06 × 0.94 × 0.002) ≈ 0.0106.
4. z = (p2 − p1) / SE = 0.016 / 0.0106 ≈ 1.51.
5. Compare with 1.96, or read the p-value (≈ 0.13).
6. Decide: not significant at α = 0.05.

The individual rates come first because you need them for the difference; the pooled rate only enters at step 3.

A data team may run a chi-square test on the 2×2 table instead. For two variants it's the same test in another form: the chi-square statistic equals z² (1.51² ≈ 2.27) and the p-value is identical. The z-test also gives the direction of the gap; chi-square extends to three or more variants.

Everything in this test points the same way, and it's worth seeing how the pieces fit:

• p ≈ 0.13 > 0.05 and the 95% CI for the difference includes zero are the same fact told twice. At the 5% level, an interval that contains zero always goes with a non-significant two-sided test, and vice versa (exactly so when both use the same standard error; with the pooled and unpooled versions, almost always).
• The test was underpowered. Using the sample-size formula for two proportions, (z_α/2 + z_β)² × [p1(1 − p1) + p2(1 − p2)] / (p1 − p2)², for 5.2% → 6.8% at α = 0.05 and 80% power: 7.84 × (0.052 × 0.948 + 0.068 × 0.932) / 0.016² ≈ 3,450 users per variant. The test had 1,000.
• So the outcome was predictable. Even if the redesign really adds 1.6 points, a test this small would usually come out non-significant.

The honest conclusion is neither "it works" nor "it doesn't". It's "we haven't collected enough data to tell", with a concrete next step: run to the planned sample size.

A conversion test result should survive being forwarded without you in the room. A line that does that has four parts:

1. both rates: 5.2% → 6.8%, so readers see the baseline;
2. the absolute difference in percentage points: +1.6 pp;
3. the uncertainty: 95% CI −0.5 to +3.7 pp;
4. the status: significant or not, and what happens next.

The relative lift (+31%) can go in brackets as context, but never alone. On its own it hides the baseline, and with small rates it makes modest changes look huge.

The classic trap is the headline lift: a big relative number from a small test, reported as a result. It gets repeated in meetings, ends up in a roadmap deck, and nobody remembers that the interval included zero.

If a stakeholder only wants one sentence, give them the status first: "Not decided yet: +1.6 points so far, could be anywhere from slightly negative to +3.7." That's short, honest and hard to misquote.