effect size

How big a difference or relationship is, independent of sample size, e.g. absolute lift, relative lift or Cohen's d.

Effect size answers "how much?" rather than "is there anything at all?". It is the magnitude of a difference or relationship, expressed so that it does not grow just because you collected more data. A p-value blends two things, the size of the effect and the size of the sample; effect size isolates the first.

In product and marketing work you'll meet a few common forms:

absolute lift: $p_B - p_A$ , in percentage points;
relative lift: $\frac{p_B - p_A}{p_A}$ , in per cent;
difference in means in natural units: dollars of average order value, minutes per session;
standardised difference, most often Cohen's d: the difference in means divided by the pooled standard deviation. Useful for comparing effects across metrics with different units or for power calculations;
strength of a relationship: a correlation $r$ , $R^2$ or an odds ratio.

Why significance without effect size misleads: "significant" only tells you the effect is probably not zero. With a million users, a 0.05-point lift is significant; with a thousand users, a 1.6-point lift may not be. A report that says only "p < 0.05" hides which of those worlds you're in.

So always report the effect size together with its confidence interval, and plan tests around the smallest effect worth detecting, the minimum detectable effect (MDE), which in turn sets the sample size you need for adequate power. For business decisions, natural units (orders, dollars, percentage points) are usually the most useful; standardised measures help when units are arbitrary, such as survey scores.

Example

Two CatChow tests from the same quarter:

Test	Users per group	Control	Variant	Absolute lift	Relative lift	p
Checkout redesign	1,000	5.2%	6.8%	+1.6 pp	+30.8%	≈ 0.13
Free-delivery banner	600,000	4.00%	4.10%	+0.10 pp	+2.5%	≈ 0.005

Relative lifts: $\frac{6.8 - 5.2}{5.2} = \frac{1.6}{5.2} \approx 30.8\%$ and $\frac{4.10 - 4.00}{4.00} = 2.5\%$ .

Only the banner is statistically significant, yet the redesign's observed effect is 16 times larger ( $1.6 / 0.10$ ).

Translate both into money. Say checkout sees 100,000 visitors a month and the average order is $50:

redesign: $100{,}000 \times 0.016 = 1{,}600$ extra orders, about $80,000 a month;
banner: $100{,}000 \times 0.001 = 100$ extra orders, about $5,000 a month.

The right reading: the banner's effect is real but small. The redesign's effect could be large but is still uncertain (its 95% confidence interval runs from about −0.5 to +3.7 pp), so the next step is a longer test, not a verdict.

Common mistakes

Reporting only the verdict. "Significant winner, p = 0.01" without the lift and its interval leaves leadership unable to judge whether it matters.
Mixing percentage points and per cent. 5.0% → 5.5% is +0.5 pp or +10% relative. Say which one you mean, every time.
Being dazzled by relative lift on a tiny baseline. 0.1% → 0.2% is "+100%", yet it may be a handful of orders.
Trusting the effect from a small or early-stopped test. Tests that barely cross significance tend to overstate the true effect (the "winner's curse"). Look at the full confidence interval and expect some shrinkage after launch.
Judging business value with generic labels. Calling d = 0.2 "small" says nothing about whether it's worth money; translate the effect into orders or revenue.

Learn it in the course

Power analysis and sample size · A power analysis tells you how many users an experiment needs to detect the effect you care about.
p-value and statistical significance · What a p-value really says, what it doesn't, how the 0.05 significance level turns it into a decision, and statistical versus practical significance.