# p-value and statistical significance

> What a p-value really says, what it doesn't, how the 0.05 significance level turns it into a decision, and statistical versus practical significance.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: Comparing two groups
- Lesson 8/24, ~39 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/p-value

## What the p in p-value actually says

**Alex Tutor:** Back to CatChow's product team and our widget test: t ≈ 6.97, and I told you it was 'statistically significant'. Time to unpack what statistical significance actually means.

**Sofia:** I assumed 'significant' just meant 'big and important'.

**Alex Tutor:** Common trap, and a big part of why this lesson exists. Every test we've run boils down to one number: the **p-value**. Its actual definition is oddly roundabout.

**Alex Tutor:** The p-value is: *assuming the null hypothesis is true* (the groups really don't differ), the probability of getting a test statistic — like our t or U — at least as extreme as the one we calculated, just from sampling noise.

**Sofia:** That's a mouthful. Can you say it again, slower?

**Alex Tutor:** Slowly: pretend there's truly no difference between control and variant. The p-value asks 'if that pretense were true, how often would random sampling alone hand us a result this extreme or more?' Small p-value = that pretense looks implausible.

![A cat flips a coin that keeps landing on the same side, a tower of identical coins grows beside it, and the cat raises an eyebrow.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/p-value-illustration-2.webp)

*If nothing were going on, how often would chance alone look like this?*

**Alex Tutor:** Let's make sure that definition sticks before we go further.

> **Exercise:** Which is the closest correct definition of a p-value?

## What it does NOT say

**Alex Tutor:** Now the important part: three ways people misread that number, constantly, in real product meetings.

**Sofia:** Hit me. I've probably said at least one of them out loud.

> **Three things a p-value is not**
>
> It is not the probability that H0 is true. It is not the probability the result would replicate. And a *small* p-value does not measure how big or important the effect is — it only measures how surprising the data would be if there were truly no effect.

**Sofia:** Ouch, I've definitely said 'there's a 2% chance the null hypothesis is true' before.

**Alex Tutor:** You're in very good company — that exact mistake shows up in published papers. Let's stress-test your new understanding.

> **Exercise:** Which of these are common misinterpretations of a p-value that you should avoid saying out loud? Select all that apply.

**Alex Tutor:** Now pick the right sentence yourself. This is the exact line that ends up in reports.

> **Exercise:** A new free-delivery badge test gives p = 0.03. Which sentence reads the p-value **correctly**?

**Sofia:** Okay, so what CAN we say when p comes back above 0.05? That the effect definitely isn't there?

**Alex Tutor:** Careful, that's another classic trap — and a big enough one that it deserves its own question.

**Alex Tutor:** It ties back to type II errors from the t-test lesson: absence of evidence isn't evidence of absence, especially with a small sample.

![A cat searches a dark room with a narrow flashlight beam while a toy mouse sits just outside the lit circle.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/p-value-illustration-3.webp)

*A small sample is a narrow beam: missing something doesn't mean it isn't there.*

**Alex Tutor:** One more common one, framed as true or false this time.

> **Exercise:** Getting a p-value above 0.05 (failing to reject H0) proves that the null hypothesis is true.

## Is it statistically significant? The 0.05 significance level

**Alex Tutor:** So how small is 'small enough'? Before looking at data, we pick a threshold — the **significance level, α** — almost always 0.05 by convention. If p < α, we reject H0.

$$
\text{reject } H_0 \iff p < \alpha
$$

*The whole decision rule in one line: compare the p-value to the significance level chosen in advance.*

**Sofia:** Why 0.05 specifically? Did someone prove that's the magic number?

**Alex Tutor:** Not really — it's a convention from early 20th-century statistics that stuck. Nothing sacred happens at p = 0.0499 versus p = 0.0501. Treat 0.05 as a useful line in the sand, not a law of nature.

**Alex Tutor:** Quick reference point: for a two-tailed test, a z-score of about 1.96 lands exactly on the classic 0.05 line.

> **Exercise:** For a two-tailed z-test, a z-score of about 1.96 corresponds to what classic p-value threshold?

**Sofia:** So for our widget test, t ≈ 6.97 was way past that 1.96-ish territory. No wonder you called it significant.

**Alex Tutor:** Exactly — and if you ran it through software, it would report an actual p-value far below 0.001, not just 'less than 0.05'.

**Sofia:** And the Mann-Whitney page test? U = 2 against a critical value of 5?

**Alex Tutor:** Also comfortably significant at 0.05 — software would give you an exact p there too, just computed a different way under the hood. Same logic, different engine.

**Alex Tutor:** Let's match the whole decision framework together — the pieces from this lesson and the last one.

> **Exercise:** Match each term to its definition.

## Statistical significance is not practical significance

**Alex Tutor:** Here's the twist that catches PMs specifically: p-value depends heavily on sample size. With enough data, even a tiny, commercially useless difference becomes 'statistically significant'.

**Sofia:** So a huge company could get p < 0.001 on a change that changes nothing that matters?

**Alex Tutor:** Precisely. That's the gap between **statistical significance** (is the effect probably not zero?) and **practical significance** (is the effect big enough to act on?). CatChow needs both before shipping anything.

![A huge magnifying glass makes a tiny kibble crumb look enormous, while a cat holds the real crumb on its paw and shrugs.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/p-value-illustration-4.webp)

*Significant is not the same as big enough to matter.*

**Alex Tutor:** Concrete numbers, your turn to compute.

> **Exercise:** CatChow tested a tiny checkout copy tweak on 500,000 customers per group: control converts at 5.0%, variant at 5.5% — statistically significant at p < 0.001. What is the relative lift, (variant - control) / control × 100?

**Alex Tutor:** A percentage alone hides the money. Here's a fresh test: put its lift into orders.

> **Exercise:** A checkout test on 1,000,000 sessions per group: control converts at 2.0%, variant at 2.1%, p < 0.001. How many **extra orders per 100,000 sessions** does the variant bring?

**Alex Tutor:** And the flip side of the same coin: one effect, two very different p-values.

> **Exercise:** Two CatChow tests show the same 3% relative lift in conversion (5.00% → 5.15%). The test with 2,000 sessions per group gives p ≈ 0.83; the one with 1,000,000 per group gives p < 0.001. What explains the difference?

## Wrap-up

**Sofia:** Honestly, the 'p = 0.049 is significant but p = 0.051 isn't' thing has always bothered me a little.

**Alex Tutor:** You've basically rediscovered a genuine, ongoing debate among statisticians. Good instinct.

**Sofia:** Wait, statisticians actually argue about this?

**Alex Tutor:** Constantly — some journals have even proposed dropping the word 'significant' altogether in favour of just reporting the actual p-value and effect size. Put that discomfort into words for me — why is treating 0.05 as a hard line, rather than a rough guide, a bit misleading?

> **Exercise:** In your own words, why is treating p = 0.05 as a hard pass/fail line (0.049 'works', 0.051 'doesn't') a bit misleading, even though it's a useful convention?

**Alex Tutor:** Nicely reasoned. Time to put the whole module to work: t-test, Mann-Whitney, and now p-values, all in one messy real report.

**Sofia:** Let me guess, someone got overexcited about a tiny number again.

**Alex Tutor:** You're getting good at this. This one even has your favourite ingredient: a huge sample size making a tiny number look impressive.

**Sofia:** My favourite ingredient. Great, I have a type now.

> **Exercise:** Review the report before it reaches the leadership deck.

- Previous lesson: [Mann–Whitney: when data isn't normal](https://productovi.com/courses/statistics-for-pm-and-marketers/mann-whitney.md)
- Next lesson: [95% confidence intervals](https://productovi.com/courses/statistics-for-pm-and-marketers/confidence-intervals.md)

---
Productovi Learning. Free, no sign-up needed to read.