# Backtest basics and traps

> Test a rule on past prices and see the traps: one lucky period, hindsight, too many tweaks and missing failures.

- Course: [Trading Basics: How It Works, How Not to Lose Everything](https://productovi.com/courses/trading.md)
- Module: How do I turn an idea into a plan I can test?
- Lesson 26/32, ~29 min
- Source: https://productovi.com/courses/trading/backtest-traps

## Nine trades, eight winners

**Alex Tutor:** Back on your Pebble practice account, Nora has done something sensible for once: she tested a rule on past prices before trading it. Let's hear it.

**Nora:** When Bluefin Energy closes above every close of the last 10 days, buy 100 shares at the next open and sell at the close 5 days later. February to April: 9 trades, 8 winners, +1,233 after costs!

> **Nora's rule**
>
> **Entry:** when Bluefin Energy closes above every close of the last 10 days (the 10 days are the rule's **look-back**), buy 100 shares at the next open.
> **Exit:** sell at the close 5 trading days later.
> **Costs:** 4.00 per round trip.
> **Nora's test:** 3 February to 18 April, 9 trades, 8 winners, +1,233 after costs.

**Alex Tutor:** Running a rule on past prices as if you'd traded then is a **backtest**. It's cheap and useful, and also very easy to fool yourself with. Here are the prices and every trade the rule made over nine months.

**Dataset (CSV):** [Bluefin Energy daily prices, January to September 2025](https://productovi.com/datasets/trading/backtest-traps/m7_bfe_prices.csv)

Made-up prices from the Pebble practice market.

**Dataset (CSV):** [Every trade of Nora's rule](https://productovi.com/datasets/trading/backtest-traps/m7_bt_trades.csv)

100 shares per trade, 4.00 of costs per round trip; result is after costs. in_test_window = yes marks the trades Nora tested.

**Alex Tutor:** Her test window ran from 3 February to 18 April. In those weeks the price climbed from about 40 to 56. In a rise like that, almost any "buy" rule wins.

![Bluefin weekly candles, January to September. The test window, 3 Feb to 18 Apr, is the climb from about 40 to 56. After April the price goes sideways and trades 12–20 all lose.](https://productovi.com/courses/trading/diagrams/m7-l2-test-window.png)

*Made-up prices from the Pebble practice market.*

**Sofia:** So what happens on the rest?

> **Exercise:** Add up the result column for all 20 trades. What does the rule make over the whole series, after costs, in dollars?

## Trap 1: one lucky period

**Nora:** Minus 57? The same rule?

**Alex Tutor:** Same rule. After April the price stops trending and moves sideways: it jumps to a new 10-day high, then gives it back. The rule buys every jump at the top.

**Sofia:** How much did those later trades lose?

> **Exercise:** How much do the 11 trades marked in_test_window = no make together, after costs, in dollars?

**Alex Tutor:** So the window showed the trend, not the rule. And note: the whole series has only 20 trades. Under 30 is too few to trust a result either way, good or bad.

![The 9 trades in the window made +1,233 after costs, the 11 outside it lost 1,290, so all 20 trades made −57. And 20 trades is too few to trust either way.](https://productovi.com/courses/trading/diagrams/m7-l2-window-vs-all.png)

**Nora:** I picked February to April because that's where the chart looked nice.

**Alex Tutor:** That's **hindsight**: choosing the test after you've seen the answer. Pick the period before you look, use all of it, ranges included, and keep the last part untouched for one final check.

![Nora cheers at the rising stretch of a price line framed by a curtain, while Alex pulls the curtain back to show the sideways months after it and Marc studies them.](https://productovi.com/courses/trading/diagrams/m7-l2-scene.png)

> **Exercise:** Why did the test on February–April make the rule look so good?

**Alex Tutor:** Quick check on how Nora chose her window.

> **Exercise:** Choosing the test period after looking at the chart is fine, as long as costs are included.

## Trap 2: too many tweaks

**Nora:** Fine, I fixed it. I tried every look-back from 5 to 20 days on the whole series. 5 days makes +1,907!

**Dataset (CSV):** [The same rule with 16 look-backs](https://productovi.com/datasets/trading/backtest-traps/m7_bt_tweaks.csv)

Same prices, same costs; net_result is after costs.

**Alex Tutor:** 16 versions on the same prices, keep the best: that's **overfitting**. The rule gets bent to fit this exact past. The 5-day version wins mostly because it caught more of the one uptrend.

![The same rule with look-backs of 5 to 20 days on the same prices: the best, 5 days, makes +1,907 after costs, Nora's 10 days −57. The best of many is partly luck.](https://productovi.com/courses/trading/diagrams/m7-l2-lookbacks.png)

**Marc:** The honest way: choose the version on the first part of the data, then run it once on a part you've never looked at. If it falls apart there, it was fitted, not found.

> **Exercise:** Which are signs that a backtest result may be overfitted? Pick all that apply.

**Alex Tutor:** Before we move on, look at the whole table of 16 versions, not just the top line.

> **Exercise:** In the table of 16 look-backs, what is the average net result of all 16 versions, after costs, in dollars? Round to a whole dollar.

## Trap 3: the missing failures

**Alex Tutor:** The last trap hides in other people's tests. A forum lists "ten rules that beat the market for five years". Who made the list? Only rules still running. The ones that blew up were deleted.

**Marc:** Same with "the best funds of the decade": funds that closed simply vanish from the table, so the average looks better than anyone's real result.

**Alex Tutor:** That's **survivorship bias**: you only see what survived. Ask how many started, not just how many are left.

**Nora:** So it's like the shop window in an ad: only the good bread on display.

**Alex Tutor:** Exactly, except here the seller may not even know the failures are missing.

> **Exercise:** Match each trap with what it looks like.

## What could be wrong with this test?

**Marc:** One more thing I learned the hard way. In a backtest every order fills at the price in the table. Real orders slip, and a morning gap can jump straight past a stop. So I always knock a bit off a backtest result.

**Alex Tutor:** So every backtest result ends with one question: what could be wrong with this test? Costs in? Whole period? How many versions tried? At least 30 trades? Any failures missing? Fills you could really get?

**Nora:** And if a rule passes all of that?

**Alex Tutor:** Then it's worth testing on a practice account. Not proven, just not yet disproven. The past doesn't promise the future.

**Alex Tutor:** One more look at Marc's habit of trimming every backtest result.

> **Exercise:** Marc, the experienced trader in this course, always knocks a little off a backtest result before he trusts it. Why?

> **Exercise:** Nora's new post: "My 5-day breakout rule made +1,907 after costs on Bluefin this year. Proven!" In 2–4 sentences, say what could be wrong with this test.

> **Key ideas**
>
> - A **backtest** runs a rule on past prices. Costs are always in, and the result is always followed by "what could be wrong with this test?"
> - **One lucky period:** 9 trades in an uptrend made +1,233; all 20 trades made −57 after costs.
> - **Hindsight:** choose the test period before you look at the chart, and use all of it.
> - **Overfitting:** the best of many versions on the same prices is partly luck; check it once on unseen prices.
> - **Survivorship bias:** lists of winners hide the ones that failed. And under 30 trades is too few to tell.

> **Educational, not advice**
>
> This course explains how trading works; it never tells you what to buy or sell. A good backtest is not a promise: most people who trade lose money, and you can lose all of it. For taxes and rules, ask a licensed professional where you live.

- Previous lesson: [A one-page trading plan](https://productovi.com/courses/trading/one-page-plan.md)
- Next lesson: [Practice account, starting small and the journal](https://productovi.com/courses/trading/practice-account-and-journal.md)

---
Productovi Learning. Free, no sign-up needed to read.