Every Backtest Works, Which Is Exactly the Problem

The Structural Problem

a backtest applies a set of business rules to historical data and reports what would have happened

The difficulty is that the researcher has already lived through the period knows what happened and will run the test many times with variations. Each variation is informed consciously or not by knowledge of the result

The result is a process that is almost guaranteed to produce a strategy that fits the past. Adapting the past is easy. That's the only thing a backtest can prove

This is not an argument for skipping the exercise. A failing rule in history is dead and should be abandoned cheaply and discovering it costs an afternoon rather than a year of living capital. The mistake is to treat a pass as proof in the same way that a failure is proof. Failure is informative because it is difficult to achieve by accident. Success is not because the entire apparatus is built to produce it

The Failure Modes

These are not variations of an error. Each is independently sufficient to produce a result that cannot be repeated they are usually present together and their effects go in the same direction. Each of them makes the reported performance better than reality rather than making it worse which is why the combined error is never a wash

Look Ahead Is the Subtle One

Lookahead bias means using information that was not available when the operation would have been performed. It is rarely obvious

Data from financial statements is a common source. A company's annual figures for a year ending in December are not published until months later but many databases attach them to the year-end date. A backtest that reads that field on January 1 is operating on information that no one had

Index membership is another. Testing a strategy with current index constituents applies membership retroactively to a period when those companies had not yet been added and inclusion tends to follow good performance

Restated figures are one-third.Databases often contain corrected numbers rather than those originally reported and the correction occurred later

General defense is a point-in-time discipline meaning that each field carries the date on which that value was made public and the test can only read values ​​whose date has already passed. Databases created this way exist and cost more which is the main reason the shortcut persists

What makes the dangerous family member look ahead is how small the leak can be while still dominating the outcome. A few days of prior knowledge of a profit figure is worth more than most genuine advantages so a strategy tainted in this way doesn't look even slightly better than it should. It looks extraordinary and the extraordinary backtesting results should prompt a search for the leak before they prompt anything else

The Universe Inherits Its Own Bias

Survival enters the backtest through the universe and not through the rules. If eligible securities are drawn from a current quote file companies that went bankrupt somewhere in the middle of the sample are simply not present so the strategy never has a chance to buy something that later went to zero

The distortion is not uniform across strategies. It falls most heavily on rules that buy what is cheap or small because those categories contain the highest failure rates and a rule that selects the problem looks like a rule that selects the value once the problem has been removed from the record

Overfitting Is the Unavoidable One

Each parameter choice is an opportunity to adapt to noise. A moving average length a rebalancing frequency a threshold a holding period. Each is selected by testing values and keeping the best

With enough parameters any historical series can be fit almost perfectly. Then the fitted model immediately fails out of sample because it learned specific noise from the training data rather than any persistent relationship

Defenses are partial.Use few parameters.Prefer round numbers to precisely optimized ones.Verify that performance is stable at all nearby parameter values as a strategy that works on a 20-day window and fails on 19 and 21 has encountered noise.And reserve data that is never used during development

The stability check deserves emphasis because it is the only diagnostic that costs nothing and is omitted anyway. Plotting the result over the entire range of a parameter rather than reporting the best value. A true effect produces a wide region where the strategy works acceptably with the best value somewhere within it. A tight artifact produces a single peak surrounded by mediocrity and the peak is the value that is written in the presentation

It also matters that the parameter count is higher than it appears. The rules discarded along the way were also parameters. So was the decision to test this asset class this period this rebalancing convention and this signal definition. The reported model has few parameters. The process that produced it had many and the second number is what determines how much of the result is noise

Costs Change the Answer

Backtests often involve executing at the closing price at no cost. Real trading pays the spread pays commissions and moves the price when trading in size

Arithmetic is unforgiving of anything traded frequently. A strategy that completely replaces your portfolio each month sells 100 percent and buys 100 percent each time so about 24 times the value of the portfolio changes hands in a year. At 5 basis points of the cost of the round trip that's 1.2 percent annually. At 20 basis points which is nothing to write home about for smaller or less liquid names it's4.8 percent annually and a 4 percent gross lead has turned into a 0.8 percent loss before something goes wrong

Compare this to a strategy that turns over 20 percent of the portfolio per year. This is 0.4 times the value of the traded portfolio costing 2 basis points per year on the low estimate and 8 on the high estimate. The same cost assumptions that destroy the fast strategy are a rounding error for the slow strategy

For a strategy that holds positions for months the costs modestly reduce returns. For a day trading strategy they often eliminate all advantage. Many published high-frequency strategies are profitable only in the absence of the costs that would necessarily be incurred to execute them

Capacity Is a Separate Question

The impact on the market is the most difficult to model because it depends on the size being traded and the liquidity at that time something that historical data does not reveal

The historical record shows the price at which the transactions were made and the volume that was traded. It does not show what price would have been required to execute an order that never existed. A backtest that assumes a fill at close assumes a counterparty that was not there

Therefore capacity is a property of the strategy and not a detail of its implementation. A rule concentrated on the smallest least liquid stocks can be genuinely profitable at a size that no professional operation could implement and the same rule applied to the big liquid names where they can absorb capital often stops working because the inefficiency it exploits does not stand up to so much attention

The practical check is to compare the size of the positions involved in the strategy with the volume that was actually traded in those names on those days. If the strategy needs a significant portion of a day's volume the backtest describes prices that would not have been available

Regime Dependence Is the Hardest to Fix

The latter failure mode cannot be engineered out because it is more a failure of history than a failure of method

A strategy tested during a period of falling interest rates has been tested in a rate environment. A strategy tested since the last major crisis has never been tested in a single one. The data may contain thousands of daily observations and still contain only a handful of independent macroeconomic regimes and it is the regimes that determine whether the relationship on which the strategy is based continues to hold

This is why performance across multiple markets and multiple decades carries more weight than a longer series of daily data from a single market. Additional observations from within a regime mostly confirm the regime. Useful evidence comes from asking whether the same rule worked in a different country in a different rate environment or in a period that the researcher was not thinking about when the rule was written

Walking Forward Instead of Looking Back

One method addresses part of the problem honestly which is making the test itself obey the constraint under which the trader operates

A forward test estimates the rule in an early window applies it to the immediately following period without any further adjustment then advances both windows and repeats. The reported performance is the concatenation of those intact future periods. Nothing in the record was produced by a model that had seen the data it was trading

This is more demanding than it seems because every decision must be made within the estimation window including parameter values choosing which values ​​are eligible and thresholds. It is common to see a forward test where the parameters were selected once in the entire sample and then re-estimated as decoration which preserves the appearance of the method and none of its protection

If done correctly it does not eliminate the difficulty. Still the researcher chose to construct this rule rather than another and that choice was based on knowledge of history. It eliminates the most mechanistic form of contamination and the drop in reported performance when a naive test is redone in this way is a useful measure of how much of the original result was in retrospect

What Makes a Backtest Worth Anything

An economic rationale expressed before testing so the strategy is to test a hypothesis rather than pursue one. Few parameters chosen for simplicity. Realistic costs. Performance in multiple regimes and markets rather than one favorable period. Stability in the face of small rule changes. And a genuine resistance period that was never examined during development

Even with all that a backtest is weak evidence. Live results with real money are still the only solid evidence and there is no way to get them in advance

Why Live Results Come In Lower

The gap between the proven result and the negotiated result has a name in trading and is measured rather than assumed. Each element runs in one direction

Orders are not filled at the price that triggered them. Some are not filled at all and the ones that are not filled are disproportionately the trades where the price moved away because other people wanted the same thing meaning the ones missing are the good ones. The positions are limited by risk limits that the backtest never had. Corporate actions borrowing costs on short positions and finance charges appear in a real account and rarely in a simulation

None of these are dramatic on their own. Together they explain why a strategy that shows a solid result on paper routinely delivers a fraction of it and why the sensible expectation for any new rule is that proven performance is substantially reduced rather than proven performance achieved

The Bottom Line

Backtesting is necessary and almost useless as a test because the researcher already knows the story and will search until something fits. Lookahead bias survivorship overfitting and ignored costs independently produce results that cannot be replicated. Require that an economic rationale few parameters honest costs and an intact reserve be stated first and continue to treat the result as a hypothesis rather than a finding. The most useful question you can ask of a solid backtest is not whether the code is correct. It is whatIt would have to have been true in the data for this result to appear even if the strategy was worthless and that question almost always has an answer