Broker API HubMenu

Technique explained as code

How a backtest lies

A backtest is easy to write and easier to fool yourself with. Six common mistakes, each one demonstrated in code on invented prices, so you can see the size of the error for yourself.

Independent and unofficialEducational onlyInvented data onlyCode run

Risk warning: CFDs and spread bets are complex instruments and come with a high risk of losing money rapidly due to leverage. You can lose money on these products, and each provider publishes its own loss figure under FCA rules. You should consider whether you understand how they work and whether you can afford to take the high risk of losing your money. Automated and API trading can amplify losses through bugs, connectivity failures and runaway orders.

Teaching material only. This page explains how a technique works as code and how to test an idea honestly. It is not investment advice, not a recommendation and not an invitation to trade. Every price on it is invented, so no result here says anything about any real market, and nothing here suggests that a technique makes money.

What it is

A backtest runs a rule over past prices and adds up what would have happened. It is the cheapest test there is. It is also the easiest to get wrong, because almost every mistake you can make moves the answer in the flattering direction.

This page makes six of those mistakes on purpose. Each script is short, and each prints the honest answer next to the flattering one.

The idea behind it

Every demonstration below runs on invented prices that are a random walk. That matters. The script that made them gives tomorrow no memory of today, so no rule can have a real advantage on this file. Whenever a result below looks good, you know for certain that the test is at fault and not that a rule has been found.

That is the habit this page is meant to build. When a test of your own looks good, the first question is not how to use it. It is which of these mistakes you have made.

Look ahead: using a price you could not have known

Look ahead means a decision uses information from after the moment it was supposedly made. It is nearly always an accident: a slice that runs one position too far, or a column that was filled in at the end of the day and read at the start.

The rule here buys at the open and sells at the close of a day, after a close above the highest close of the five days before. The honest version takes its signal from yesterday. The other takes it from today, whose close is not known at the open. The two differ by one index.

Data the script reads: invented-prices.csv. Invented values, never a real market. Save it in one folder and run the script from that folder.

The file that was run: lie_1_look_ahead.py

# How a backtest lies, part 1: look ahead.
# The same rule is run twice. The honest run decides with prices up to yesterday. The other
# run lets today's close leak into the decision, which no one could have known at the open.
# Prices come from invented-prices.csv: a random walk made by a script, never a real market.
import csv

opens, closes = [], []
with open("invented-prices.csv", newline="") as f:
    for row in csv.DictReader(f):
        opens.append(float(row["open"]))
        closes.append(float(row["close"]))

DAYS = 5             # example value only
COST = 0.10          # points paid on every completed trade


def run(peek):
    """The rule: after a close above the highest close of the DAYS days before it,
    buy at the open and sell at the close of the same day."""
    total = 0.0
    trades = wins = 0
    for day in range(DAYS + 1, len(closes)):
        # The whole mistake is this one line. The signal should come from yesterday.
        # With peek on it comes from today, whose close is not known when the day opens.
        signal = day if peek else day - 1
        if closes[signal] > max(closes[signal - DAYS:signal]):
            result = closes[day] - opens[day] - COST
            total += result
            trades += 1
            if result > 0:
                wins += 1
    return trades, wins, total


for label, peek in [("Deciding with prices up to yesterday:", False), ("Deciding with today's close already known:", True)]:
    trades, wins, total = run(peek)
    print(label)
    print("  Trades:", trades, "of which above zero:", wins)
    print("  Total points after costs:", round(total, 2))
print("Simply holding over the same days:", round(closes[-1] - opens[DAYS + 1] - COST, 2))

What the run printed

Deciding with prices up to yesterday:
  Trades: 169 of which above zero: 76
  Total points after costs: -16.54
Deciding with today's close already known:
  Trades: 169 of which above zero: 164
  Total points after costs: 149.67
Simply holding over the same days: -7.07

Output of the run on , copied as printed. Every input to this run was invented, so the result says nothing about any real market.

How to read it

Same rule, same prices, same number of trades. Deciding from yesterday, fewer than half the trades ended above zero and the total was below zero. With today's close leaked in, 164 of 169 trades ended above zero and the total is enormous.

No rule does that. A record where nearly every trade gains is the clearest sign of look ahead there is, and the right response is to hunt for the leak, not to celebrate.

The guard is to be strict about time in every line. For each value a decision uses, ask at what moment it became known. In this site's scripts the habit is visible in the slices, which always stop before today.

Overfitting: trying many settings and keeping the top one

Every rule has settings, and it is natural to try a few. The mistake is to try many, keep the one that scored highest, and report its score as if it were the only one you ran.

This script runs a moving average crossover with 343 different pairs of lengths over the first half of the prices, takes the top scoring pair, and runs it on the second half.

Data the script reads: invented-prices.csv. Invented values, never a real market. Save it in one folder and run the script from that folder.

The file that was run: lie_2_many_settings.py

# How a backtest lies, part 2: trying many settings and keeping the one that scored highest.
# A moving average crossover is run with every pair of lengths in a grid over the first half
# of the prices. The top scoring pair is then run on the second half, which it has never seen.
# Prices come from invented-prices.csv: a random walk made by a script, never a real market.
import csv

closes = []
with open("invented-prices.csv", newline="") as f:
    for row in csv.DictReader(f):
        closes.append(float(row["close"]))


def average(values):
    return sum(values) / len(values)


COST = 0.10          # points paid on every completed trade
HALF = len(closes) // 2


def run(fast_days, slow_days, first, last):
    """Total points after costs for one pair of lengths over days first..last-1."""
    holding = False
    entry = total = 0.0
    for day in range(max(first, slow_days), last):
        fast = average(closes[day - fast_days:day])      # up to yesterday only
        slow = average(closes[day - slow_days:day])
        if not holding and fast > slow:
            holding, entry = True, closes[day]
        elif holding and fast < slow:
            total += closes[day] - entry - COST
            holding = False
    if holding:
        total += closes[last - 1] - entry - COST
    return total


# Every pair of lengths in the grid, scored on the first half only.
scores = {}
for fast_days in range(2, 21):
    for slow_days in range(10, 101, 5):
        if fast_days < slow_days:
            scores[(fast_days, slow_days)] = run(fast_days, slow_days, 0, HALF)

ranked = sorted(scores, key=scores.get, reverse=True)
top = ranked[0]
middle = ranked[len(ranked) // 2]

print("Settings tried:", len(scores))
print("Settings that ended above zero on the first half:", len([s for s in scores.values() if s > 0]))
print("Average of all settings on the first half:", round(average(list(scores.values())), 2))
print("Simply holding over the first half:", round(closes[HALF - 1] - closes[0] - COST, 2))
print("Top scoring setting on the first half:", top, "with", round(scores[top], 2), "points")
print("Middle of the table on the first half:", middle, "with", round(scores[middle], 2), "points")
print("The top scoring setting on the second half:", round(run(top[0], top[1], HALF, len(closes)), 2), "points")
print("The middle setting on the second half:", round(run(middle[0], middle[1], HALF, len(closes)), 2), "points")
print("Simply holding over the second half:", round(closes[-1] - closes[HALF] - COST, 2))

# How well did the first half predict the second? Count the settings that ended above zero in both.
both = 0
for fast_days, slow_days in ranked[:20]:
    if run(fast_days, slow_days, HALF, len(closes)) > 0:
        both += 1
print("Of the 20 highest on the first half, above zero on the second half:", both)

What the run printed

Settings tried: 343
Settings that ended above zero on the first half: 290
Average of all settings on the first half: 4.24
Simply holding over the first half: 1.02
Top scoring setting on the first half: (13, 20) with 12.6 points
Middle of the table on the first half: (11, 90) with 5.18 points
The top scoring setting on the second half: -14.07 points
The middle setting on the second half: -14.35 points
Simply holding over the second half: -8.55
Of the 20 highest on the first half, above zero on the second half: 0

Output of the run on , copied as printed. Every input to this run was invented, so the result says nothing about any real market.

How to read it

On the first half 290 of the 343 settings ended above zero, and the top one scored many times what holding did. Anyone shown that table alone would think they had found something, and would think the long list of other settings that also gained was confirmation.

On the second half the top setting lost, by about as much as a setting taken from the middle of the table. Of the twenty highest on the first half, none ended above zero on the second. The first half score carried no information about the second.

The settings were fitted to the noise in one stretch of prices, and noise does not repeat. The guard is to hold back data that plays no part in choosing, to test the chosen setting on it once, and to report how many settings were tried.

Many random rules: one of them always looks good

This is the same mistake with the disguise taken off. Each rule here is a list of coin flips made before any price is read: in for this week, out for that week. The rules know nothing about the prices and cannot.

The script scores 1,000 of them on the first half, keeps the top one, and runs it on the second half. It does that five times, with a fresh 1,000 rules each time.

Data the script reads: invented-prices.csv. Invented values, never a real market. Save it in one folder and run the script from that folder.

The file that was run: lie_3_random_rules.py

# How a backtest lies, part 3: try enough rules and one of them will look good.
# Each "rule" here is a list of coin flips made before any price is read: in for this week,
# out for that week. It knows nothing. 1,000 such rules are scored on the first half of the
# prices, the top one is kept, and it is then run on the second half. This is done five times.
# Prices come from invented-prices.csv: a random walk made by a script, never a real market.
import csv
import random

closes = []
with open("invented-prices.csv", newline="") as f:
    for row in csv.DictReader(f):
        closes.append(float(row["close"]))

COST = 0.10          # points paid on every completed trade
HALF = len(closes) // 2
WEEKS = len(closes) // 5 + 1
RULES = 1000


def run(decisions, first, last):
    """Follow a list of weekly in or out decisions over days first..last-1."""
    holding = False
    entry = total = 0.0
    for day in range(first, last):
        want_in = decisions[day // 5]
        if want_in and not holding:
            holding, entry = True, closes[day]
        elif holding and not want_in:
            total += closes[day] - entry - COST
            holding = False
    if holding:
        total += closes[last - 1] - entry - COST
    return total


hold_first = closes[HALF - 1] - closes[0] - COST
hold_second = closes[-1] - closes[HALF] - COST
print("Simply holding: first half", round(hold_first, 2), " second half", round(hold_second, 2))
print("batch  above holding  top rule, first half  same rule, second half  all rules, second half")

for batch in range(1, 6):
    rng = random.Random(batch)                   # a different set of 1,000 rules each time
    rules = [[rng.random() < 0.5 for _ in range(WEEKS)] for _ in range(RULES)]
    first_half = [run(rule, 0, HALF) for rule in rules]
    top = max(range(RULES), key=lambda i: first_half[i])
    above = len([s for s in first_half if s > hold_first])
    second = run(rules[top], HALF, len(closes))
    everyone = sum(run(rule, HALF, len(closes)) for rule in rules) / RULES
    print(f"{batch:5d}  {above:13d}  {first_half[top]:20.2f}  {second:22.2f}  {everyone:22.2f}")

What the run printed

Simply holding: first half 1.02  second half -8.55
batch  above holding  top rule, first half  same rule, second half  all rules, second half
    1            400                 33.11                    7.85                   -5.97
    2            387                 24.00                   -2.24                   -6.06
    3            425                 26.08                    6.27                   -6.39
    4            400                 30.68                    6.84                   -6.11
    5            404                 27.74                   -8.30                   -6.32

Output of the run on , copied as printed. Every input to this run was invented, so the result says nothing about any real market.

How to read it

In all five batches around 400 of the 1,000 coin flip rules ended above holding on the first half, and the top rule scored between 24 and 33 points while holding made about one. It happened every time. Search 1,000 rules of any kind and the top one will look like a discovery.

The fourth column is the same rule on the second half. The scores are scattered on both sides of zero, from about minus eight to plus eight, and look nothing like the first half. The last column gives the average of all 1,000 rules on the second half as the yardstick for a rule that knows nothing.

Some of the top rules landed above that average and some below, which is what chance looks like. Their second half decisions are fresh coin flips with no connection to the first half. Whenever you hear that a rule was found by scanning thousands of candidates, this is the picture to keep in mind.

Ignoring costs: the rule that only works for free

Every trade costs something: the gap between the buying and selling price, commission, and the difference between the price you wanted and the price you got. A test that leaves this out measures a market that does not exist.

The script runs the rate of change rule from the momentum page five times. Only the cost per trade changes.

Data the script reads: invented-prices.csv. Invented values, never a real market. Save it in one folder and run the script from that folder.

The file that was run: lie_4_no_costs.py

# How a backtest lies, part 4: leaving out costs.
# One rule, the rate of change rule from the momentum page, is run again and again with a
# different cost per trade each time. Nothing else changes.
# Prices come from invented-prices.csv: a random walk made by a script, never a real market.
import csv

closes = []
with open("invented-prices.csv", newline="") as f:
    for row in csv.DictReader(f):
        closes.append(float(row["close"]))

DAYS = 10            # example value only
START = DAYS + 1


def run(cost):
    """In when price is above where it was DAYS days ago, out when below. Returns trades and total."""
    holding = False
    entry = total = 0.0
    trades = 0
    for day in range(START, len(closes)):
        change = closes[day - 1] / closes[day - 1 - DAYS] - 1      # up to yesterday only
        if not holding and change > 0:
            holding, entry = True, closes[day]
        elif holding and change < 0:
            total += closes[day] - entry - cost
            trades += 1
            holding = False
    if holding:
        total += closes[-1] - entry - cost
        trades += 1
    return trades, total


trades, free = run(0.0)
print("Trades:", trades)
print("cost per trade   total points")
for cost in [0.0, 0.05, 0.10, 0.20, 0.50]:
    print(f"{cost:14.2f}   {run(cost)[1]:12.2f}")
print("Cost per trade at which the total reaches zero:", round(free / trades, 3))
print("Simply holding, which pays the cost once:", round(closes[-1] - closes[START] - 0.10, 2))

What the run printed

Trades: 45
cost per trade   total points
          0.00           8.56
          0.05           6.31
          0.10           4.06
          0.20          -0.44
          0.50         -13.94
Cost per trade at which the total reaches zero: 0.19
Simply holding, which pays the cost once: -10.11

Output of the run on , copied as printed. Every input to this run was invented, so the result says nothing about any real market.

How to read it

With no cost the total is above zero. Each step up in cost takes a fixed amount off for each of the 45 trades, and between a cost of 0.10 and 0.20 the total goes below zero.

The line under the table is the useful number: the cost per trade at which the total reaches zero. If that figure is close to any realistic cost, the result is too fragile to mean anything, because nobody knows their future costs to that precision.

The guard is to put a cost in from the first run, to make it pessimistic, and to print the break even cost beside every result. The more often a rule trades, the more this matters.

Survivorship: only counting what is still here

A list of instruments that exist today leaves out everything that failed along the way. Test on today's list and you have quietly removed the worst outcomes before you start.

The invented universe file holds 30 instruments that each follow a random walk with no drift. Any that closes below 60 is removed from the list, as a failed company is removed from an index. The script compares the average change of the ones still listed at the end with the average of all 30.

Data the script reads: invented-universe.csv. Invented values, never a real market. Save it in one folder and run the script from that folder.

The file that was run: lie_5_survivorship.py

# How a backtest lies, part 5: survivorship.
# invented-universe.csv holds 30 invented instruments. Each starts at 100 and follows its own
# random walk with no drift. One that closes below 60 is removed, and its cells are empty from
# then on. The question: what does "buy all of them on day one and hold" look like if you only
# count the ones still on the list at the end? No trading rule is involved, so there is no
# cost per trade to apply: this page is about which instruments get counted at all.
import csv

first_price = {}
last_price = {}
still_listed = {}

with open("invented-universe.csv", newline="") as f:
    for row in csv.DictReader(f):
        for name, cell in row.items():
            if name == "date":
                continue
            still_listed[name] = cell != ""
            if cell != "":
                first_price.setdefault(name, float(cell))
                last_price[name] = float(cell)          # ends up holding the last price it ever had


def percent_change(name):
    return (last_price[name] / first_price[name] - 1) * 100


def average(values):
    return sum(values) / len(values)


everyone = sorted(first_price)
survivors = [n for n in everyone if still_listed[n]]
removed = [n for n in everyone if not still_listed[n]]

print("Instruments on day one:", len(everyone))
print("Still listed on the last day:", len(survivors))
print("Removed along the way:", len(removed))
print("Average change, survivors only:", round(average([percent_change(n) for n in survivors]), 1), "percent")
print("Average change, removed only:", round(average([percent_change(n) for n in removed]), 1), "percent")
print("Average change, everything that was there on day one:", round(average([percent_change(n) for n in everyone]), 1), "percent")
print("Survivors that ended above where they started:", len([n for n in survivors if percent_change(n) > 0]), "of", len(survivors))
print("All instruments that ended above where they started:", len([n for n in everyone if percent_change(n) > 0]), "of", len(everyone))

What the run printed

Instruments on day one: 30
Still listed on the last day: 17
Removed along the way: 13
Average change, survivors only: 13.3 percent
Average change, removed only: -40.4 percent
Average change, everything that was there on day one: -10.0 percent
Survivors that ended above where they started: 10 of 17
All instruments that ended above where they started: 10 of 30

Output of the run on , copied as printed. Every input to this run was invented, so the result says nothing about any real market.

How to read it

Thirteen of the 30 were removed. The 17 that remain show an average rise, and most of them ended above where they started. Counting all 30, the average is a fall, and only a third ended higher.

Nothing was wrong with any single price. The error is entirely in which instruments were counted. The survivors look good because surviving was the selection.

The guard is to ask where a list came from and as of what date. A test needs the list as it stood at the start of the period, including everything that later disappeared. The same applies to any group you can only see the survivors of.

Too few trades: a record that could easily be luck

A total made of three trades is three events. It cannot tell a rule from a run of luck, however large it is.

The first half of the script runs the crossover from lesson six over its 120 day sample file and prints each trade. The second half leaves prices behind and counts how often 10,000 coin flippers, with no skill at all, end up with a record that looks like skill.

Data the script reads: sample-candles.csv. Invented values, never a real market. Save it in one folder and run the script from that folder.

The file that was run: lie_6_few_trades.py

# How a backtest lies, part 6: too few trades.
# First, the crossover from lesson six on its short sample file, trade by trade. Then a count of
# how often pure coin flipping produces a record that looks impressive, for short and long runs.
# Prices come from sample-candles.csv: invented, never a real market.
import csv
import random

closes = []
with open("sample-candles.csv", newline="") as f:
    for row in csv.DictReader(f):
        closes.append(float(row["close"]))


def average(values):
    return sum(values) / len(values)


FAST, SLOW = 5, 20   # the same example values as lesson six
COST = 0.10          # points paid on every completed trade

holding = False
entry = 0.0
results = []
for day in range(SLOW, len(closes)):
    fast = average(closes[day - FAST:day])           # up to yesterday only
    slow = average(closes[day - SLOW:day])
    if not holding and fast > slow:
        holding, entry = True, closes[day]
    elif holding and fast < slow:
        results.append(closes[day] - entry - COST)
        holding = False
if holding:
    results.append(closes[-1] - entry - COST)

print("Days of prices:", len(closes))
print("Each trade, in points:", [round(r, 2) for r in results])
print("Total points after costs:", round(sum(results), 2))
print("Total without the largest trade:", round(sum(results) - max(results), 2))
print("Simply holding over the same days:", round(closes[-1] - closes[SLOW] - COST, 2))

# Now forget prices. 10,000 pretend traders each flip a fair coin for every trade.
# None of them has any skill. How many end up with a record that looks like skill?
rng = random.Random(11)
PEOPLE = 10_000


def lucky(trades, wins_needed):
    """Count the coin flippers who win at least `wins_needed` of `trades` flips."""
    count = 0
    for _ in range(PEOPLE):
        wins = sum(1 for _ in range(trades) if rng.random() < 0.5)
        if wins >= wins_needed:
            count += 1
    return count


print("Coin flippers out of", f"{PEOPLE:,}", "who won at least:")
print("  5 of 6 trades:", lucky(6, 5))
print("  17 of 20 trades:", lucky(20, 17))
print("  83 of 100 trades:", lucky(100, 83))
print("  60 of 100 trades:", lucky(100, 60))

What the run printed

Days of prices: 120
Each trade, in points: [0.9, 7.59, -1.81]
Total points after costs: 6.68
Total without the largest trade: -0.91
Simply holding over the same days: 7.0
Coin flippers out of 10,000 who won at least:
  5 of 6 trades: 1173
  17 of 20 trades: 17
  83 of 100 trades: 0
  60 of 100 trades: 305

Output of the run on , copied as printed. Every input to this run was invented, so the result says nothing about any real market.

How to read it

The rule made three trades. One of them supplies more than the whole total: take it away and what is left is below zero. That is a single event, and nothing general follows from it.

Among 10,000 coin flippers, well over a thousand won at least five trades out of six. A record of five wins in six is something more than one person in ten gets by flipping a coin. Seventeen of twenty still happened to a handful of them. Eighty three of a hundred happened to none.

The last line is a caution in the other direction. Sixty wins in a hundred sounds modest, and a few hundred of the 10,000 managed it by chance. More trades help, but a large sample of a small effect can still be luck. The guard is to count the trades before reading the total, and to ask how often a coin would have done as well.

More ways a test misleads

What to try changing

These are programming exercises. Each one changes the code and asks you to explain what you see. None of them is a search for settings to use.

How this page was made

Each script on this page was run with Python 3.12 on Windows, standard library only, from a folder holding the data files listed. No network and no account. Each exited with code 0, and what is shown is everything it printed.

The data files whose names begin with the word invented are written by make_invented_data.py from fixed seeds, so anyone can make the same files again and get the same output. This site is not published if the code or a data file has changed since the run.

Sections