← GlossaryPearson's chi-square test
Tests association between categorical variables by summing (observed - expected)^2 / expected over all cells.
Common mistakes
- Feeding in percentages instead of counts. The statistic depends on sample size; "5.2% vs 6.8%" without the counts behind it can't be tested.
- Counting the same user more than once. A table of sessions or page views where one customer appears many times breaks the independence assumption and makes the result look far more certain than it is.
- Ignoring small expected counts. If any expected cell count falls below about 5 (common in niche segments), switch to Fisher's exact test.
- Reading a significant 3-variant result as "C wins". Chi-square only says the rates are not all equal. Follow up with pairwise comparisons and a multiple-comparisons correction.
- Treating a big χ² as a strong effect. χ² grows with the sample; report the conversion rates and the lift as the effect size.
Pearson's chi-square test checks whether two categorical variables are related, for example variant (A / B / C) and outcome (ordered / didn't order), or acquisition channel and chosen plan. The data go into a contingency table of counts: one row per category of the first variable, one column per category of the second.
The idea is to compare what you observed with what you'd expect if the variables were independent. The expected frequency of each cell is
E=grand totalrow total×column total
and the test statistic adds up the squared gaps, scaled by the expected counts:
χ2=∑E(O−E)2
It has (rows−1)(columns−1) degrees of freedom. For a 2×2 table (df = 1) the 5% critical value is 3.84; software gives the exact p-value.
Chi-square vs t-test. Student's t-test compares the means of a quantitative variable (such as average order value) between two groups. Chi-square works with counts in categories. For a classic A/B test on conversion, a 2×2 chi-square test (without continuity correction) is equivalent to the two-proportion z-test: χ2=z2 and the p-values match. Chi-square earns its keep when you have three or more variants or outcome categories and want a single overall test instead of a pile of pairwise comparisons.
Check the assumptions: each user is counted once, you use raw counts rather than percentages, and every expected count is at least about 5 (otherwise use Fisher's exact test). A significant result only says that some association exists. To see where it comes from and how strong it is, look at the per-cell gaps and the conversion rates themselves.
CatChow's checkout redesign test as a contingency table:
Expected counts if the variant made no difference:
- ordered: 20001000×120=60 in each group;
- didn't order: 20001000×1880=940 in each group.
Chi-square statistic:
χ2=60(52−60)2+60(68−60)2+940(948−940)2+940(932−940)2
=1.067+1.067+0.068+0.068≈2.27
With df = (2 − 1)(2 − 1) = 1, the critical value is 3.84. Since 2.27 < 3.84 the result is not significant (p ≈ 0.13).
Cross-check: the two-proportion z-test on the same data gave z≈1.51, and 1.512≈2.27. Same test, two costumes. If CatChow had added a third checkout variant, the table would be 3×2 with df = 2, and one chi-square test would check all three conversion rates at once.
