Reading a price test

The first number everyone reads from a price test is conversion: buyers ÷ visitors in each arm. It answers one question: how many people said yes at this price.

Klasna's test ran eight weeks, 1 February to 28 March 2027, for new visitors in Ukraine buying the English 16-lesson package. Arm A (620 a lesson) had 232 buyers from 4,000 visitors: 5.80%. Arm B (670) had 221 from 4,000: 5.53%. Arm C (720), added in week 5, had 84 from 1,600: 5.25%.

Leo's reading, "620 converts best, keep it", stops too early. Conversion says how many said yes, not how much each yes was worth. A higher price almost always converts a little worse, and it can still earn more. So conversion is where the reading starts, never where it ends. To count it, sum buyers and visitors over all weeks of an arm, then divide; don't average weekly rates.

To compare prices fairly, measure money per visitor, not buyers. Revenue per visitor (RPV) = sales ÷ visitors = conversion × package price. It multiplies how many said yes by what each yes paid, so it puts both in one number.

It also fits how a test works: every visitor costs the same to attract, whichever price they see. The arm that earns more per visitor earns more from the same traffic.

In Klasna's test: A sold 2,301,440 UAH from 4,000 visitors, 575.36 per visitor. B sold 2,369,120 from 4,000, 592.28 per visitor. B converted a little worse and still brought in 2.94% more per visitor, because each B package costs 10,720 instead of 9,920.

RPV is the first money view. The business keeps only part of each sale, so the next step is the same calculation for contribution.

Revenue per visitor counts the whole price. But a higher price changes contribution far more than revenue, because the costs per lesson stay the same: the extra price is almost all extra contribution.

For a 16-pack: contribution per package = 16 × (price × 0.975 − 390 payout − 30 video and AI). At 620 a package leaves 2,952 UAH; at 670 it leaves 3,732. The extra 800 of package price brings 780 of contribution; only the 2.5% fee comes off.

Then contribution per visitor = buyers × contribution per package ÷ visitors. A: 232 × 2,952 / 4,000 = 171.22. B: 221 × 3,732 / 4,000 = 206.19, or +20.42%.

That's the gap that matters for the business. A 2.94% lead in revenue per visitor becomes a 20.42% lead in what Klasna keeps from each visitor.

A higher price will lose some buyers; the question is how many it can lose before it stops paying. Work out that threshold, then compare it with what the test shows.

For revenue per visitor, B pays as long as its conversion is above A's × 620 / 670. So B could convert up to 1 − 620 / 670 = 7.46% worse than A before its revenue per visitor fell to A's level. For contribution the cushion is much wider: 1 − 2,952 / 3,732 = 20.9%.

The gap the test observed is B's 5.53% against A's 5.80%: 4.66% lower, and within noise. Even a true loss somewhat larger than that would leave B ahead on revenue, and a loss up to 20.9% would leave it ahead on contribution.

This turns a vague worry ("B converts worse") into a clear test: is the loss bigger than the price can afford? Here it isn't close.

A price that wins the first sale can still lose later if its buyers don't come back. So a price test also checks retention: here, the share of buyers who bought again within 60 days.

In Klasna's test, 95 of A's 232 buyers came back: 40.95%. For B, 90 of 221: 40.72%. The difference is a fraction of a point, well within noise: families who paid 670 stayed just as well as those who paid 620.

This matters because a higher price can attract a different kind of buyer, or disappoint more of them after the first package. If B's buyers had returned much less often, its first-purchase lead could shrink or vanish over a year. They didn't, so the money-per-visitor lead holds.

Count retention the same way as conversion: sum repeat buyers over all weeks of an arm and divide by all its buyers.

Any two arms will differ a little by chance, even if the price made no difference at all. Before calling a gap a loss, ask how often a gap this large would appear by chance. That's what a p-value says.

B converted at 5.53% against A's 5.80%, with 4,000 visitors in each arm. If price made no difference, a gap at least this large would turn up about 59% of the time: p ≈ 0.59. So the test can't say B loses buyers. It also can't rule out a small loss, which is why you compare with the loss that would matter (7.46% for revenue, 20.9% for contribution).

Two common misreadings. p ≈ 0.59 is not "a 59% chance that B is better"; it's about how surprising the data would be if there were no difference. And a gap within noise doesn't mean the test failed: it answered its question, that B earns more per visitor without a clear loss of buyers.

A test compares prices fairly only if the arms ran at the same time, under the same conditions. Demand moves with the calendar; an arm that ran only in busy weeks looks better for reasons that have nothing to do with its price.

Nazar added arm C (720) in week 5, because B looked fine. So A and B ran all eight weeks, and C only the last four, in March, when the exam season starts and every arm sells better. Over the whole test C earned 604.80 per visitor, the highest headline number.

Compare like with like: weeks 5–8 only. A: 112 buyers of 1,600 visitors, sales 1,111,040, 694.40 per visitor. B: 710.20. C: 604.80, the lowest. Within the same weeks the picture flips.

That's why arms aren't added midway. If a new price is worth testing, start a new test with all arms from the same day.

A headline number can win for the wrong reason. Before choosing a price, ask whether its arm faced the same conditions as the others.

Arm C's 604.80 per visitor is higher than A's and B's eight-week figures only because C ran in the exam peak while their totals include the quieter February. In the same weeks C is the lowest: 604.80 against 694.40 (A) and 710.20 (B). Its conversion was 5.25% against A's 7.00% in those weeks, a gap unlikely to be chance (p ≈ 0.04). And only a third of its buyers came back within 60 days (33.33%). So the test doesn't support 720; it never ran outside the peak at all.

How it fits together: read conversion, then money per visitor in revenue and contribution, check retention and noise against the loss that would matter, and compare arms over the same weeks. The next lesson puts this test into the yearly pricing review, next to realized prices, costs and competitors.