# Data Snooping > The reuse of one dataset to evaluate many hypotheses — sequentially, collectively, or unconsciously — until something fits, at which point the fit is reported as if it were the only hypothesis ever tried. - Canonical: https://hadalinstruments.com/glossary/data-snooping/ - Term set: https://hadalinstruments.com/glossary/ --- Data snooping is multiple testing without the accounting. Its defining feature is that it rarely feels like a violation while it is happening: each step is one reasonable-seeming iteration — adjust the filter, try the other lookback, exclude the weird year — and each step conditions the next hypothesis on the same data that will be used to judge it. By the time something works, the dataset has answered dozens of questions, and the final p-value or Sharpe ratio is reported as though it had answered one. The individual form is the garden of forking paths: every data-dependent choice is a trial whether or not it was experienced as one. The collective form is larger and stranger — entire research communities mining the same few decades of the same indices means the *field's* trial count is enormous before any individual researcher runs their first test. The formal machinery for this is White's Reality Check (2000), which tests a strategy against the full universe of rules searched rather than in isolation; Sullivan, Timmermann, and White applied it to calendar effects, which did not survive the correction, and to technical trading rules, where the best rule did survive the check on the historical sample but did not repeat in the period afterwards. The lesson is the same from either end: the best rule out of thousands beating a benchmark is not, on its own, evidence; it is arithmetic until the breadth of the search is priced in. Snooping cannot be cured downstream. Corrections exist — the [probability of backtest overfitting](/glossary/probability-of-backtest-overfitting/), the [deflated Sharpe ratio](/glossary/deflated-sharpe-ratio/), false-discovery-rate control — but every one of them is conditioned on an honest count of what was tried, which is precisely the number snooping fails to record. The defences are therefore procedural and boring: [pre-registration](/glossary/pre-registration/) to fix hypotheses before the data can shape them, and a [kill ledger](/glossary/kill-ledger/) to keep the trial count honest after it starts growing. [Backtest overfitting](/glossary/backtest-overfitting/) is the damage; data snooping is the process that inflicts it. ## Commonly confused with Neighbouring concepts that get used interchangeably, and the distinction that actually separates them. - **[Backtest overfitting](https://hadalinstruments.com/glossary/backtest-overfitting/)** — Data snooping is the process; overfitting is the damage. Naming them separately matters because the remedies differ — you correct overfitting with multiple-testing statistics, and you prevent snooping with procedure, before any statistics exist to apply. - **Exploratory data analysis** — Looking at data to form hypotheses is legitimate and necessary. It becomes snooping when the same data then judges the hypotheses it generated, and the report presents the result as though the hypothesis had arrived from somewhere else. The sin is not the looking; it is the missing accounting. - **p-hacking** — Largely the same practice under a name from a different literature, with a connotation of intent that data snooping does not carry. The forking-paths version needs no intent at all: every data-dependent choice is a trial whether or not it was experienced as one. - **Multiple testing** — Multiple testing is the condition — many hypotheses against one dataset. Snooping is multiple testing without the accounting. The formal corrections all assume you can state how many tests were run, which is precisely the number snooping fails to record. ## How to measure it in your own data A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself. - **Records you need** — A count of every hypothesis the dataset has answered, including the data-dependent choices that did not feel like hypotheses: the filter you adjusted, the lookback you swapped, the year you excluded for being weird. For a field-level view, the count also includes what everyone else has already run against the same few decades of the same indices. - **What you compute** — White's Reality Check tests a strategy against the full universe of rules searched rather than in isolation. Sullivan, Timmermann and White applied it to calendar effects, which did not survive the correction, and to technical trading rules, where the best rule did survive on the historical sample but did not repeat in the period after it. Both outcomes make the same point from opposite ends: the breadth of the search that found a rule is part of the evidence about it. - **What the answer tells you** — The best rule out of thousands beating a benchmark is not evidence; it is arithmetic. If the trial count cannot be stated, no correction can be applied and the honest report is that the result is uncorrectable rather than significant. This is the one failure that genuinely cannot be repaired downstream — every remedy is conditioned on a number that was not recorded. ## Questions and answers ### Is data snooping the same as cheating? No, and treating it that way makes it harder to prevent. Its defining feature is that it rarely feels like a violation while it happens — each step is one reasonable-seeming iteration, and each step quietly conditions the next hypothesis on the data that will judge it. Scrupulous researchers snoop by default; that is why the defence is procedural rather than ethical. ### What is the garden of forking paths? The individual form of snooping, where a researcher makes a series of data-dependent choices that were never experienced as separate tests. No single decision looks like a hypothesis, and yet the dataset has answered every one of them by the time a result is reported. ### Can I correct for data snooping after the fact? Only if you know what was tried. The probability of backtest overfitting, the deflated Sharpe ratio and false-discovery-rate control are all conditioned on an honest trial count. Without it there is nothing to correct against, which is why the count has to be kept while the work happens rather than reconstructed after it. ### Does snooping happen at the level of a whole field? Yes, and it is the larger and stranger version of the problem. When entire research communities mine the same few decades of the same indices, the field's trial count is enormous before any individual researcher runs a first test — so an apparently novel anomaly may be the surviving tail of a search nobody conducted deliberately. ## If this has already cost you If a result was found by searching, the breadth of that search can be priced into it — provided the count is honest. - [Overfit Assay](https://hadalinstruments.com/assays/#overfit-assay) “Is my backtest real, or did I fit it to noise?” Will not establish: Whether the strategy will be profitable. A backtest that survives the battery is a backtest that was not obviously fitted — it is not a forecast, and the report says so on its first page. Intake is not open yet, so none of these can be commissioned today. They are listed here so you know the measurement exists and what it would and would not settle — the [launch list](https://hadalinstruments.com/pricing/#waitlist) hears first. ## Related terms Derived from the links this entry makes and the entries that link back to it. - [Backtest Overfitting](https://hadalinstruments.com/glossary/backtest-overfitting/) The condition in which a strategy's historical performance reflects fitting to noise in a particular dataset rather than a persistent market effect, so that live performance regresses toward zero or below. - [Deflated Sharpe Ratio (DSR)](https://hadalinstruments.com/glossary/deflated-sharpe-ratio/) A test statistic that adjusts an observed Sharpe ratio for the number of trials conducted, the length of the track record, and the non-normality of returns, estimating the probability that the true Sharpe ratio exceeds zero. - [False Discovery Rate (FDR)](https://hadalinstruments.com/glossary/false-discovery-rate/) The expected proportion of rejected null hypotheses that are in fact true — the share of your discoveries that are false, as distinct from the probability of making any false discovery at all. - [Kill Ledger](https://hadalinstruments.com/glossary/kill-ledger/) An append-only research record of every strategy configuration that was tried and abandoned, kept so that multiple-testing corrections can be computed from the true number of trials rather than the remembered one. - [Pre-Registration](https://hadalinstruments.com/glossary/pre-registration/) The practice of recording a hypothesis, its test design, and its success criteria — timestamped — before examining the data that will judge it, so that a confirmation cannot have been shaped by the answer. - [Probability of Backtest Overfitting (PBO)](https://hadalinstruments.com/glossary/probability-of-backtest-overfitting/) A statistic estimating the probability that the strategy configuration selected as best in-sample will underperform the median of its rivals out-of-sample, typically computed via combinatorially symmetric cross-validation. - [Sharpe Ratio](https://hadalinstruments.com/glossary/sharpe-ratio/) The ratio of a return series' mean excess return to its standard deviation — how much return was earned per unit of volatility borne, expressed as a single scalar. - [The Two Lenses](https://hadalinstruments.com/glossary/the-two-lenses/) A framework in quantitative trading where every metric is evaluated through both its theoretical statistical properties and its physical microstructure execution constraints. ## In the research Data Snooping comes up in eight research notes on this site, and this entry lists three of them. - [The edge that was real and worth half a pip](https://hadalinstruments.com/research/the-edge-that-was-real-and-worth-half-a-pip/) My engine's first end-to-end edge hunt found statistically real structure — leans of half to two percent from a coin flip — and none of it is tradable. - [Does COT positioning predict reversals?](https://hadalinstruments.com/research/does-cot-positioning-predict-reversals/) Crowded positioning extremes precede fewer liquidations, not more. The interval excludes zero, p = 0.0066, and the sign is backwards. - [The break that cleared neither bar](https://hadalinstruments.com/research/the-break-that-cleared-neither-bar/) A joint change-point detector flagged one break across twelve market channels, then tested it against two independent bars and cleared neither. ## Cite This Definition APA BibTeX HTML Hadal Instruments. (2026). Data Snooping. Hadal Glossary. https://hadalinstruments.com/glossary/data-snooping/ Version e524a7e, 2026-08-04. @misc{hadal_2026_data-snooping, author = {Hadal Instruments}, title = {Data Snooping}, year = {2026}, url = {https://hadalinstruments.com/glossary/data-snooping/}, howpublished = {Hadal Glossary}, version = {e524a7e}, note = {Pre-launch publication; version dated 2026-08-04} } Source: Hadal Instruments, Data Snooping. Original Research Copy Citation **Version e524a7e** identifies the commit that last changed this page in Hadal's content repository. That repository is not public, so the identifier does not resolve externally — it is published so a citation pins one specific state rather than a moving page. To obtain the exact version cited, use the [press and research route](https://hadalinstruments.com/press/).