counterfactual baseline

Also called: expected baseline, incremental, holdout baseline

What a metric would have been without the change, estimated from a holdout: baseline × the holdout’s growth. Incremental result = actual − expected.

A campaign’s raw "before vs after" mixes two things: what the campaign did and what would have happened anyway (seasons, a market trend, a holiday). A counterfactual baseline estimates the second part from a holdout, a comparable group or region that didn’t get the campaign.

If the holdout grew by g, the test group would likely have grown by the same g without the campaign:

$\text{expected} = \text{before} \times (1 + g), \quad \text{incremental} = \text{actual} - \text{expected}$

Costs are then divided by the incremental result, not by everything that happened: cost per incremental referral, incremental CAC. Dividing by all results counts customers who would have come anyway and makes a campaign look far cheaper than it is.

Example

Referrals in a test region went from 400 to 560; a holdout region grew from 300 to 330 (+10%).

Expected: 400 × 1.10 = 440.
Incremental: 560 − 440 = 120.
At a cost of 36,000 UAH: 300 UAH per incremental referral, not 36,000 ÷ 560 = 64.

Common mistakes

Dividing cost by all results. Divide by the incremental ones.
Subtracting the holdout’s count instead of applying its growth rate.
A holdout that isn’t comparable (different size, season or audience) gives a biased baseline.

Learn it in the course

What do we divide by? Denominator traps · ARPU vs ARPPU, LTV per cohort user vs per survivor, win rate with or without "no decision", gross vs net margin, concentration with the item in the total, and incremental results against a control group.
Flows: customers and cash · One rule for every stock, end = start + in − out: MRR, churn, NRR and GRR; profit vs cash, working capital, DSO and runway; CAC payback and incrementality against a holdout.