Backtest Overfitting
The condition in which a strategy's historical performance reflects fitting to noise in a particular dataset rather than a persistent market effect, so that live performance regresses toward zero or below.
Every historical dataset contains patterns that occurred by chance and will not recur. A backtest cannot distinguish these from durable structure; it can only report how well a configuration fitted the past. Overfitting is what happens when the researcher, iterating over parameters, instruments, date ranges, filters, and entry rules, selects the configuration that fitted the past best — and thereby selects, with high probability, for luck.
The mechanism is selection bias over multiple trials. Run enough variations and some will show excellent historical performance even if no variation has any true edge, for the same reason that some coin flippers in a large enough crowd produce long streaks of heads. The in-sample performance of the chosen configuration is therefore a biased estimator of its future performance, and the bias grows with the number of trials — including the trials the researcher no longer remembers running.
The symptoms are recognisable. Performance that is exquisitely sensitive to parameter values. Equity curves that degrade promptly after the strategy goes live. Rules with no economic rationale that happen to dodge the worst historical drawdowns. A parameter set that works on one instrument and fails on its close substitutes.
The defences are procedural, not statistical cleverness after the fact: fewer researcher degrees of freedom, pre-registration of hypotheses before testing, honest recording of every trial in a kill ledger, walk-forward validation applied once rather than iteratively, and explicit multiple-testing corrections such as the probability of backtest overfitting and the deflated Sharpe ratio. None of these make a backtest trustworthy. They make its untrustworthiness measurable, which is the most that honest research can claim.