Every Backtest Works, Which Is Exactly the Problem
Testing a strategy against historical data is necessary and proves very little. The list of ways a backtest can look excellent while describing nothing real is long.
The Structural Problem
A backtest applies a set of trading rules to historical data and reports what would have happened.
The difficulty is that the researcher has already lived through the period, knows what happened, and will run the test many times with variations. Each variation is informed, consciously or not, by knowledge of the outcome.
The result is a process almost guaranteed to produce a strategy that fits the past. Fitting the past is easy. That is the only thing a backtest can demonstrate.
A backtest cannot distinguish a real effect from a coincidence in the sample, because both look identical inside the sample.
The Failure Modes
| Problem | What happens |
|---|---|
| Look ahead bias | Uses information unavailable at the time |
| Survivorship bias | Failed securities absent from the universe |
| Overfitting | Parameters tuned to sample noise |
| Ignored costs | Spreads, commissions, market impact |
| Capacity assumptions | Assumes fills at prices unavailable in size |
| Regime dependence | Works only in the conditions tested |
Look Ahead Is the Subtle One
Look ahead bias means using information that was not available when the trade would have been made. It is rarely obvious.
Financial statement data is a common source. A company's annual figures for a year ending in December are not published until months later, but many databases attach them to the year end date. A backtest reading that field on 1 January is trading on information nobody had.
Index membership is another. Testing a strategy on today's index constituents applies membership retroactively to a period when those companies had not yet been added, and inclusion tends to follow good performance.
Restated figures are a third. Databases often carry the corrected numbers rather than what was originally reported, and the correction happened afterwards.
Overfitting Is the Unavoidable One
Every parameter choice is an opportunity to fit noise. A moving average length, a rebalancing frequency, a threshold, a holding period. Each is selected by trying values and keeping the best.
With enough parameters, any historical series can be fitted almost perfectly. The fitted model then fails immediately out of sample, because it learned the specific noise of the training data rather than any persistent relationship.
The defences are partial. Use few parameters. Prefer round numbers over precisely optimised ones. Check that performance is stable across nearby parameter values, since a strategy that works at a 20 day window and fails at 19 and 21 has found noise. And reserve data that is never used during development.
Costs Change the Answer
Backtests frequently assume execution at the closing price with no cost. Real trading pays the spread, pays commissions, and moves the price when trading in size.
For a strategy holding positions for months, this reduces returns modestly. For a strategy trading daily, it frequently eliminates the entire edge. Many published high frequency strategies are profitable only in the absence of the costs that would necessarily be incurred to run them.
Market impact is the hardest to model, because it depends on the size being traded and the liquidity at the time, neither of which the historical data reveals.
What Makes a Backtest Worth Anything
An economic rationale stated before the test, so the strategy is testing a hypothesis rather than searching for one. Few parameters, chosen for simplicity. Realistic costs. Performance across multiple regimes and markets rather than one favourable period. Stability under small changes to the rules. And a genuine holdout period that was never examined during development.
Even with all of that, a backtest is weak evidence. Live results with real money remain the only strong evidence, and there is no way to obtain them in advance.
The Bottom Line
Backtests are necessary and nearly worthless as proof, because the researcher already knows the history and will search until something fits. Look ahead bias, survivorship, overfitting, and ignored costs each independently produce results that cannot be repeated. Demand an economic reason stated first, few parameters, honest costs, and an untouched holdout, and still treat the output as a hypothesis rather than a finding.