p-value and statistical significance

Every test in this module, t or U, ends in one number: the p-value. Its definition is roundabout on purpose:

Assuming the null hypothesis is true (there's no real difference), the p-value is the probability of getting a test statistic at least as extreme as the one you observed, from sampling noise alone.

Read it as a thought experiment. Pretend CatChow's new upsell widget at checkout does nothing. Rerun the A/B test thousands of times with fresh customers. How often would chance alone produce a gap as large as the one you saw, or larger? That share is the p-value.

• A small p-value means data like yours would be rare in a no-difference world. That's evidence against H0.
• A large p-value means data like yours is ordinary under H0. The test found nothing surprising.

Notice what the condition does: the whole calculation lives inside the "no difference" world. That's why a p-value can't tell you how likely H0 is, how big the effect is, or whether it will replicate.

A p-value of 0.02 invites three tempting sentences, all wrong.

1. "There's a 2% chance H0 is true." The p-value was computed by assuming H0 is true. It's the probability of the data given H0, not of H0 given the data. Those are different numbers, just as "most cats have four legs" differs from "most four-legged animals are cats".
2. "There's a 98% chance we'd get the same result again." Replication depends on the true effect and the size of the next sample. A result with p = 0.02 can easily fail to reach significance the second time.
3. "p = 0.001 proves a huge, important effect." The p-value mixes effect size with sample size. A tiny effect on a huge sample gets a tiny p-value too.

The correct sentence is less exciting: "If there were no real difference, data this extreme would show up about 2% of the time by chance." Then add the effect size and the sample size, so the reader can judge importance separately.

Most p-value mistakes happen in the sentence, not in the calculation. A safe report line has three parts.

1. The condition first. Start from the no-difference world: "If the badge made no difference…". That keeps the direction of the probability right.
2. The data, not the hypothesis. "…a gap this large would appear by chance about 3% of the time." The probability describes data, never H0 or H1.
3. The effect and the sample. "The badge group ordered 4% more often (12,000 sessions per group)." Without these, readers will treat the p-value as a measure of importance.

Short versions are fine as long as they don't flip the condition: "significant at 5% (p = 0.03)" is safe. "97% sure it works" is not, and neither is "3% chance it's a fluke": both quietly turn a statement about data into a statement about the hypothesis.

If the audience isn't technical, lead with the effect size and its uncertainty, and keep the p-value as a supporting detail.

When p comes back above α, the honest conclusion is: we didn't find enough evidence to reject H0. That's different from "H0 is true" or "the feature doesn't work".

Absence of evidence isn't evidence of absence. A test can miss a real effect for several reasons:

• the sample was small, so the standard error was large and only big effects could clear the bar;
• the metric was noisy, with wide natural variation;
• the effect is real but modest, smaller than the test could detect.

Missing a real effect is a type II error. With small samples it's common, not exotic.

So a non-significant result calls for a careful next step rather than a verdict. Ask how big an effect the test could realistically have detected. Look at the observed difference: is it near zero, or a promising gap that just isn't certain? If it's promising and matters for the business, run longer or with more traffic before deciding. And in the report, write "no significant difference found", never "no difference".

To turn a p-value into a decision, you need a threshold chosen before you look at the data: the significance level, α. The rule is one line: reject H0 if p < α.

Almost everyone uses α = 0.05. It's a convention from early statistics, not a law of nature: nothing special happens between p = 0.049 and p = 0.051. What α really sets is your tolerance for false alarms: with α = 0.05, about 1 in 20 tests of a change that does nothing will still come out "significant".

For tests based on the normal distribution there's a handy landmark. In a standard normal curve, about 95% of the area lies within ±1.96 of the centre, so 5% lies in the two tails together. A z-score of about 1.96 (in either direction) therefore sits exactly on the p = 0.05 line for a two-tailed test. It's the precise version of the "95% within 2 standard deviations" rule.

A statistic beyond ±1.96 means p < 0.05; beyond about ±2.58 means p < 0.01.

Four terms make up the decision framework.

• p-value: assuming H0 is true, the probability of data at least this extreme by chance.
• Significance level (α): the threshold you fix in advance, usually 0.05.
• Statistical significance: p < α. The data would be surprising in a no-difference world, so you reject H0. It answers "is the effect probably not zero?"
• Practical significance: is the effect big enough to matter for the business? It answers "should we act on it?"

The last two are independent. A result can be:

• statistically significant but practically trivial: +0.02% conversion on millions of sessions;
• practically promising but not statistically significant: +12% order value on 60 orders per group;
• both, which is what you want before shipping;
• neither.

No formula gives practical significance. You set it from the business: what lift pays for the build, the risk and the maintenance? Ideally decide that minimum before the test too, so the result can't talk you into lowering it.

A change from 5.0% to 5.5% conversion can be described in two honest ways, and mixing them up is a classic reporting slip.

• Absolute lift: the plain difference, 5.5% − 5.0% = 0.5 percentage points (pp).
• Relative lift: the difference as a share of the baseline, (5.5 − 5.0) / 5.0 × 100 = 10%.

Both are correct; they answer different questions. Relative lift is handy for comparing tests across metrics. Absolute lift, multiplied by volume, gives the business impact: 0.5 pp of 500,000 customers is 2,500 extra conversions.

The danger is in the words. "Conversion went up by 0.5%" is ambiguous: it could mean 0.5 pp (a 10% relative lift) or a 0.5% relative lift (from 5.0% to 5.025%), a twentyfold difference. Always say "percentage points" for absolute changes of a rate.

The example also shows why significance isn't the end of the story. p < 0.001 here comes largely from the huge sample. Whether 2,500 extra conversions justify the change is a business judgement.

A tiny p-value tells you an effect is very likely real. It doesn't tell you whether the effect is worth anything. To judge practical significance, translate the lift into units the business cares about.

1. Absolute lift: 2.1% − 2.0% = 0.1 percentage points.
2. Per unit of volume: 0.1 pp of 100,000 sessions = 100 extra orders.
3. Into money: if each order brings, say, $4 of contribution after costs, that's about $400 per 100,000 sessions.
4. Compare with the cost: engineering time, maintenance, any risk to other metrics.

With a million sessions per group, the test can detect differences this small, so p < 0.001 is no surprise. The same 0.1 pp lift on 10,000 sessions per group would not be significant at all. The effect hasn't changed; only the test's ability to see it has.

A useful habit: before the test, write down the smallest lift that would change your decision. After the test, check both: is it significant, and is it above that line?

A test statistic is roughly effect divided by standard error, and the standard error shrinks with the square root of the sample size. So the same effect produces very different p-values depending on how much data you have.

Take a 3% relative lift in conversion, 5.00% to 5.15%:

• with 2,000 sessions per group the standard error of the difference is about 0.69 points, z ≈ 0.22, p ≈ 0.83;
• with 200,000 per group it's about 0.07 points, z ≈ 2.2, p ≈ 0.03;
• with 1,000,000 per group it's about 0.03 points, z ≈ 4.8, p < 0.001.

The effect never changed. Only the test's ability to see it did.

Two lessons follow. First, a non-significant small test is weak evidence either way: it may simply be underpowered. Second, a significant large test isn't automatically important: with enough data, even trivial effects pass. That's why you report the effect size next to the p-value, and why the sample size is planned in advance, a topic later in the course.