What would have happened anyway

A campaign's result is not the total that showed up after it. It's the difference from what would have happened anyway. That difference is called incrementality: the customers the marketing added.

To see "what would have happened anyway", you need a baseline. The strongest one is a holdout: similar places or people deliberately left without the campaign. Whatever happens there is what the campaign cities would have done without the ads.

Klasna's September 2026 restart campaign ran in nine Ukrainian cities from 20 August to 30 September, for 150,000 UAH. Four similar cities were left out on purpose. Hanna chose them before launch, by size and by how many new families each had last September, so they looked like the campaign cities. Klasna ran no restart campaign in September 2025, so that month is a clean second baseline.

The rule that makes a holdout trustworthy: choose it before you see any results. Picking the comparison cities afterwards, even "similar" ones, lets you pick whichever makes the campaign look good, and the test bends without anyone meaning to.

Before judging a campaign's claim, split it into the groups that make it up. Each group has its own "would have happened anyway".

The restart campaign "brought 410 families" in the campaign cities. Split them by what they were before September:

• new families, whose first lesson fell in September 2026: 275;
• returning spring families, who studied in February–May 2026 and came back: 135.

Share of returners: 135 ÷ 410 = 32.93%. About a third of the "families the campaign brought" were already Klasna families.

That matters because returners and new families have very different baselines. Every September about 30% of spring families come back to Klasna, ads or no ads. New families depend much more on whether anyone has heard of Klasna.

So the next two steps take each group in turn: what the campaign added to returners, and what it added to new families. Only the sum of those two is the campaign's result.

The slip: dividing the cost by the whole 410 (150,000 ÷ 410 = 365.85 per family) and comparing that with what a new family cost last year. It mixes customers you already had with ones you won.

Customers who come back every season don't prove a campaign worked. Compare their return rate where the campaign ran and where it didn't.

Every September about 30% of Klasna's spring families come back. In September 2026:

• campaign cities: 135 of 450 spring families returned, 135 ÷ 450 = 30%;
• holdout cities: 45 of 150, 45 ÷ 150 = 30%.

The same rate with and without the ads. Without the campaign, 450 × 30% = 135 were expected to return in the campaign cities: exactly what happened. So the campaign added about none of the returners. They'd have come back anyway.

Counting all 135 as campaign results is the trap; so is counting the holdout's 45, which is simply a different group of cities. And you don't need to ask each family: the comparison of rates answers the question better than memories would.

What follows for next September: paid ads aimed at families who return anyway are mostly wasted. The reminder they need costs nothing in a message to families who agreed to hear from Klasna.

For new customers, last year is a baseline, but the market moves too. Take off the growth the holdout shows before you credit the campaign.

Uliana's point: new families in the campaign cities grew from 230 last September to 275 this one, 45 more. But in the holdout cities new families also grew, from 80 to 88, with no campaign at all:

• holdout growth: 88 ÷ 80 = 1.10 (+10%);
• expected in the campaign cities without the campaign: 230 × 1.10 = 253;
• actual: 275;
• added by the campaign: 275 − 253 = 22 families.

That's 5.37% of the claimed 410. The other 23 of Uliana's 45 would have come anyway, through word of mouth or a growing market.

Three baselines, three answers for the same campaign: no baseline gives 410, last year alone gives 45, last year plus the holdout gives 22. The more of "what would have happened anyway" you take off, the closer you get to what the ads really did.

The slip: subtracting the holdout's 8 extra families (275 − 230 − 8 = 37). Growth is applied as a rate, because the cities are different sizes.

Once you know how many customers the campaign added, put them in money, and use contribution, not revenue: only what Klasna keeps after tutors, video and payment fees.

Return on contribution = (incremental contribution − cost) ÷ cost.

A new family leaves about 8,170 UAH of contribution in its first twelve months, on Klasna's planning curve. For the 22 families the campaign added:

• incremental contribution: 22 × 8,170 = 179,740 UAH;
• after the cost: 179,740 − 150,000 = 29,740;
• return: 29,740 ÷ 150,000 = 19.83%.

The campaign paid back within twelve months, by a small margin. And the cost per incremental family is 150,000 ÷ 22 = 6,818.18 UAH, not the 365.85 in the first report.

Same campaign, three answers again: no baseline, 410 families; last year alone, 45 families and 145.1%; last year plus the holdout, 22 families and 19.83%.

Two slips: computing the return on revenue (it ignores tutor pay and fees), and on every family who showed up in the period (it credits the campaign with customers it didn't add).

"Cost per family" depends entirely on which families you divide by. Divide by the customers the campaign added, and compare like with like.

For the restart campaign:

• per claimed family: 150,000 ÷ 410 = 365.85 UAH;
• per incremental family: 150,000 ÷ 22 = 6,818.18 UAH.

The first number made Leo say "ten times cheaper than last October, let's double it". Last October, ads brought new families at about 4,000 UAH each. On the honest number, the September campaign was more expensive per added family, not cheaper.

It still paid back, because a new family leaves about 8,170 of contribution in twelve months, more than 6,818. But the margin is thin, so "double it" isn't supported. Spending more usually reaches people who are less and less likely to buy, so each extra family tends to cost more, not the same.

The habit: whenever someone quotes a cost per customer, ask "per which customers?" If the answer includes people who would have come anyway, the number flatters the campaign.

A holdout gives an estimate, not an exact count. Say how uncertain it is, and report a range next to the central number.

The 22 added families come from the holdout cities' average growth of 10%. But the four holdout cities didn't grow equally: each grew between 9.09% and 11.11%. Apply each end to the campaign cities' 230:

• at 11.11% growth, about 19 families were added;
• at 9.09%, about 24.

So the honest statement is: about 22 new families (roughly 19–24), a return on contribution of about 6% to 31%. Small, positive, uncertain.

That's still useful. It says the campaign probably paid back, that the effect is far smaller than 410 or 45, and that nothing here justifies doubling the budget.

Two opposite slips. Reporting "22 families, 19.83%" as if exact hides how fragile it is. And "the range is too wide, so we can't say anything" throws away a clear finding: every end of the range is far below the claim, and above zero.

"How did you hear about us?" feels like direct proof. It isn't. Attribution answers are clues; the holdout and the baselines are the evidence.

In the campaign cities, 168 September families answered "an ad". Against 22 added families, that's far too many. And 14 families in the holdout cities, where the campaign never ran, gave the same answer. People remember the last thing they saw, and returners saw the ads too. So 168 "ad" answers don't prove 168 sign-ups caused by the campaign.

What the September data does support:

• the campaign added about 22 new families, not 410;
• it paid back on contribution within twelve months, by a small and uncertain margin;
• ads aimed at spring families who return anyway are mostly wasted.

What it doesn't: that doubling the budget would double the families. Nothing here says the effect grows in step with spend.

How it fits together: split the claim, take off returners and the holdout's growth, price the added customers on contribution, and give the range. The next lesson turns measuring into a monthly habit of looking and acting.