Backtest basics and traps

A backtest runs a written rule on past prices as if you had traded it then: every entry, every exit, every cost. It is cheap, it needs only a price table, and it is the first honest look at a rule that has passed the checkability test from A one-page trading plan.

Two habits make it honest. Run the rule on all the prices you have, not on the stretch you like. And include costs on every trade, because a rule that wins a little per trade can lose once costs are in.

Nora's rule on Bluefin Energy shows both. Her 9 trades from February to April made +1,233 after costs. All 20 trades from January to September made −57: before costs only +23, and 20 round trips at 4.00 each cost 80.

Every backtest result is then followed by one question: what could be wrong with this test? A good number is where the checking starts, not where it ends.

The trades outside Nora's test window tell the rest of the story. Trade 1 (+160) and trade 11 (+62) sit just before and just after the climb. Trades 12 to 20, from May to September, all lose, from −50 to −259. Together the 11 trades outside the window make −1,290.

Nothing about the rule changed in May. The market did. After April Bluefin stops trending and moves sideways: it pokes to a new 10-day high, then gives it back, like the false breakouts in Support, resistance and breakouts in module 5. A rule that buys every new high buys each poke at its top.

Notice the count too. The whole series has 20 trades. Under 30 is too few to trust a result either way, good or bad, so even −57 is not a verdict. It only says that the window's +1,233 came from the trend, not from the rule.

The first trap is testing a rule on one stretch that happens to suit it. From 3 February to 18 April Bluefin climbed from about 40 to 56. In a steady rise almost any "buy" rule wins: buy on a new high, buy on a dip, buy on Mondays. The test then measures the trend, not the rule.

A trend-following rule and a range are a bad pair, and every market has both. So a fair test covers several kinds of market: a rise, a fall and a sideways stretch, as in Trend and range in module 5. If the rule makes money only in one of them, write that down as its limit; it is useful knowledge, not a failure.

A quick check on any test you see: what was the price doing during the test window? If the answer is "going up the whole time", the result says little about a buy rule.

Hindsight means choosing how to test after you have already seen the answer. Nora picked February to April "because that's where the chart looked nice". The chart looked nice because the price was rising, so the test was decided by the result it was meant to check.

The cure is a little discipline before you look:

• choose the period in advance, for example "all of 2025", and use all of it, sideways months included;
• split it: build and adjust the rule on the first part only;
• keep the last part untouched and run the finished rule on it once, as the final check.

The untouched part works only if it stays untouched. Peek at it, adjust, and run again, and it becomes one more part you fitted.

Costs, fills and trade counts are separate checks; none of them repairs a period chosen with hindsight.

Overfitting means adjusting a rule until it fits one particular past, noise included. The usual route: try many versions on the same prices and keep the best. Nora tried every look-back from 5 to 20 days; the 5-day version made +1,907, while her original 10-day version made −57.

The best of many is partly luck. With enough versions, one will always look great on any price series, because it happens to match that series' random wiggles. Those wiggles won't repeat.

Signs to watch for:

• many versions tried, only the winner shown;
• very exact settings ("5 days, a 2.3% stop, only on Tuesdays");
• most of the profit from one short stretch.

Simple rules with few settings overfit less. And the honest step after any search is the one from the hindsight card: run the chosen version once on prices nobody has looked at. If it falls apart there, it was fitted, not found.

When someone shows you the best of many versions, ask for the rest of the list. Nora's table has 16 look-backs on the same Bluefin prices and the same costs. The best, 5 days, made +1,907. The average of all 16 is 6,023 ÷ 16 ≈ 376, and the middle of the list sits around +148.

So the winner is about five times a typical version. That gap is the part of +1,907 you should expect to lose on new prices: it comes from fitting this year's wiggles, not from the idea.

The shape of the table matters too. The results fall smoothly from 5 days to 10, then stay flat. That says the shorter look-backs mainly caught more of the one February–April climb, the trap from the first card of this lesson in a new form.

A fair summary of a search reports the typical result and how widely the versions differ, not the top line.

The last trap hides in other people's tests. A forum post lists "ten rules that beat the market for five years". Which rules could make the list? Only the ones still running. The rules that blew up were quietly dropped, so the list shows survivors and calls them typical.

That is survivorship bias: you see only what survived. Lists of "the best funds of the decade" work the same way, because funds that closed simply vanish from the table, so the average looks better than anyone's real result. Often the person sharing the list doesn't even know the failures are missing.

The question that fixes it: how many started, not just how many are left? Ten winners out of ten is impressive; ten out of five hundred is what luck alone would produce.

Together with one lucky period, hindsight, overfitting and too few trades, it makes five ways a test can look better than the rule really is.

In a backtest every order fills exactly at the price in the table. Real orders don't: market and stop orders slip by a little, and a morning gap can open straight past a stop, as the slippage and gaps from modules 1 and 3 showed. So a real result usually comes out a bit worse than the backtest, so experienced traders such as Marc trim every result before trusting it.

That completes the questions that follow every backtest result:

• costs in? whole period, chosen in advance?
• how many versions tried? one run on unseen prices?
• at least 30 trades? any failures missing?
• fills you could really get?

A rule that passes all of them is not proven, only not yet disproven. The past doesn't promise the future.

How it fits together: a backtest is cheap evidence, and each trap makes it look better than the rule is. A rule that survives the checklist earns the next step: trading it small on a practice account and recording every trade in a journal.