Fair and lawful price tests

A price test shows different prices to comparable groups of buyers and compares what they do. It's the most direct way to learn how buyers respond to a price before changing it for everyone.

What makes a test fair is how the price is assigned: by chance (new visitors split at random) or by time (one price for everyone new in February, another in March, stated openly on the page). Each visitor keeps the same price on every visit.

What a test must not do is vary the price by who the person seems to be: their phone, their browsing, how often they come back. That's no longer a test but a personalized price, set from someone's data. As of October 2026, EU rules require telling consumers when a price was personalized by automated decision-making. And a family that finds out feels cheated, whatever the law says.

A test needs a clear population: the buyers who may see a test price. Choose it so the result is useful and nobody is treated unfairly.

Klasna's February test covers only new visitors in Ukraine, paying in hryvnia. Three groups stay out:

• Existing families keep the list price. A family that has paid 620 for two years shouldn't suddenly see 670; that would be a price change without notice, not a test.
• Families abroad stay out: their EUR prices are another question, and EU consumer rules apply to them directly.
• Families with a social discount are never part of a test: the discount exists for a reason and isn't a lever to experiment with.

A narrow population is a feature. It answers one question cleanly, how new families in Ukraine respond to the English 16-pack price, instead of mixing several.

A test is only as good as the question it was set up to answer. Before the first visitor, write down three things: the main measure, the length and the sample. Deciding after you've seen the data lets you pick whatever number looks best.

For a price test, conversion alone is the wrong main measure. A higher price almost always converts a little worse; what matters is whether it earns more. So Klasna's main measure is revenue and contribution per visitor, plus repeat purchases, to check that buyers at the higher price stay as well. Complaints and total sales of one arm say little: arms of different size or length can't be compared on totals.

The length covers full weeks and avoids ending on a lucky day; the sample has to be big enough to tell a real gap from noise. How to size a test is covered in the course Product Analytics: From Data to Decisions.

In a test, some buyers pay more than others for the same thing, and they didn't choose which group they were in. A fair test protects them.

Klasna's rule: if the test ends at the lower price, everyone who bought at the higher one gets the difference back, as lessons. A family that paid 670 in February while the test settled on 620 is made whole, without having to ask.

This does three things. It keeps the test fair to the people in it. It makes the test easy to defend if a family compares prices with a neighbour. And it lets the business test boldly, because a test price that turns out too high costs only the difference, not trust.

The safety net is written down before the test starts, together with the other house rules: the same price on every visit, existing customers at the list price, social discounts never part of a test.

A split test needs many buyers in each group. With a handful of enquiries, splitting them in half tells you nothing: each group is too small for any difference to mean something.

Dana gets two or three enquiries a month. So she tests in sequence: one price for everyone new for a period, then compare with the period before. From 15 March to 30 April 2027 she quoted 750 to every new enquiry, and her price post said 750 too. Her current learners and her waitlist kept 600, as promised.

A sequential test is honest because everyone new in the same period sees the same price, openly. Its weakness is that the two periods can differ in more than price: a season, a busy month, a post that happened to travel. So it's read with care, and best with a rule written before it starts.

With small numbers, a share can look precise and mean very little. Always check how much one answer moves it.

In Dana's test four of six new enquiries said yes to 750: 4 / 6 = 66.67%. Before the test, 11 of 14 had started at 600: 78.57%. The gap looks like twelve points lost to the higher price.

But with six enquiries, one more yes would make it 83.33% and one more no 50%: each answer moves the share by about 17 points. And the test ran in spring, her busiest season, so the two periods differ in more than price. The two who said no gave different reasons: one the price, one the schedule.

So 66.67% is a fact about six people, not a measured acceptance rate. To compare rates you need many more answers than a one-person practice gets in six weeks. What a small test can do is the subject of the next card.

A small test can't measure a rate, but it can answer a simpler question: is there a wall at this price? A wall means buyers stop saying yes almost entirely.

To use a small test this way, write the decision rule before you start. Dana's rule: keep 750 if at least half of new enquiries accept it. Four of six did, so the rule is met: there's no wall at 750. She keeps it for new learners from 1 May and keeps watching what new enquiries say.

What she can't conclude: that 750 lowered her acceptance rate (six answers can't show that), or that she could charge 960 just as easily (she never quoted it). Going back to 600 would throw away the one thing the test did show.

The rule written in advance is what makes the result honest. Without it, any result could be read as a win or a loss after the fact.

Put together, a fair price test follows a fixed order, and the order is what protects it from wishful reading:

1. Choose what to vary and who takes part: the price of one product, for new buyers only.
2. Write down the main measure, the length and the decision rule: revenue and contribution per visitor and repeat purchases; full weeks; what result means what.
3. Set the safeguards: the same price on every visit, existing customers at the list price, no social discounts, the difference refunded if the test ends lower.
4. Run it without changing it midway: no new arms, no early stop because one week looks good.
5. Read the result against the rule and decide.

The rule comes before the data, the safeguards before the first visitor, the decision after the planned end.

How it fits together: a fair test varies price by chance or time, for a clear group, with a measure, a rule and a safety net decided in advance; small businesses test in sequence and look for walls. The next lesson reads Klasna's February–March test, where one arm was added midway.