Data Snooping
The reuse of one dataset to evaluate many hypotheses — sequentially, collectively, or unconsciously — until something fits, at which point the fit is reported as if it were the only hypothesis ever tried.
Data snooping is multiple testing without the accounting. Its defining feature is that it rarely feels like a violation while it is happening: each step is one reasonable-seeming iteration — adjust the filter, try the other lookback, exclude the weird year — and each step conditions the next hypothesis on the same data that will be used to judge it. By the time something works, the dataset has answered dozens of questions, and the final p-value or Sharpe ratio is reported as though it had answered one.
The individual form is the garden of forking paths: every data-dependent choice is a trial whether or not it was experienced as one. The collective form is larger and stranger — entire research communities mining the same few decades of the same indices means the field’s trial count is enormous before any individual researcher runs their first test. The formal machinery for this is White’s Reality Check (2000), which tests a strategy against the full universe of rules searched rather than in isolation; applied by Sullivan, Timmermann, and White to technical trading rules and to calendar effects, it showed that many celebrated anomalies lose significance once the breadth of the search that found them is priced in. The best rule out of thousands beating a benchmark is not evidence; it is arithmetic.
Snooping cannot be cured downstream. Corrections exist — the probability of backtest overfitting, the deflated Sharpe ratio, false-discovery-rate control — but every one of them is conditioned on an honest count of what was tried, which is precisely the number snooping fails to record. The defences are therefore procedural and boring: pre-registration to fix hypotheses before the data can shape them, and a kill ledger to keep the trial count honest after it starts growing. Backtest overfitting is the damage; data snooping is the process that inflicts it.