Why does my backtest use data that did not exist?

The short answer

Almost always, the answer is not a bug in your loop. It is that the file you loaded was assembled in hindsight — corrected, curated and completed after the dates it describes — and the assembly happened before your code ran, where no indexing discipline can reach it.

Historical datasets are, by default, the present’s account of the past. Economic series get revised. Universes are published as they stand today. Prices are adjusted backwards. Vendors clean ticks years later. Every one of those edits pushes information that did not exist on the simulated date into a row stamped with that date. Your loop then reads it correctly, and the result is a strategy that was traded by someone with a memory of the future. The polite name for that is look-ahead bias — and the point of this article is that it arrives inside the file far more often than it arrives inside your code.

The corrective is to stop asking “is my data accurate?” and start asking a narrower, answerable question: at this simulated instant, what was actually knowable? The set of facts that answers it is what I call the Knowable Set — and the whole method below is a procedure for reconstructing it and then proving your simulation never left it.

Four channels the future arrives through

The four channels are worth separating because each leaves a different trace and each has a different fix. A pipeline can be immune to one and wide open to the next.

1. Revision — the value was restated after the fact

Statistical agencies publish estimates and then improve them. The first print of a release is what participants acted on; the number sitting in a general-purpose database is frequently a later revision, and in some series a much later one. This is not a defect in the agency’s work — the revision is the agency doing its job — but it means the “historical” value for a given month can be a number that nobody possessed in that month.

You can check this without taking anyone’s word for it. The Federal Reserve Bank of St. Louis maintains ALFRED, an archive that preserves the vintages of economic series: what each series looked like at each publication date, rather than only how it reads today. The U.S. Bureau of Labor Statistics documents the same structure for employment data from the other end — preliminary estimates, subsequent monthly revisions, and an annual benchmark revision. The existence of those archives is the fact worth internalising: institutions that care about this problem store the sequence, not the answer, which tells you the sequence is the thing a backtest needs.

The fix is point-in-time data: query the vintage as of the simulated date, not the series as of today.

2. Membership — the universe is today’s survivors

A universe list downloaded now is a list of things that still exist. Test on it and every historical trade has been conditioned on information unavailable at the time: that these particular names would still be here. The delisted, the bankrupt and the absorbed — the positions that would have hurt — are simply absent, and their absence flatters everything computed on what remains.

This is survivorship bias, and its defining property is that it is invisible from inside the dataset. Missing rows do not announce themselves. Detection requires an external record of membership as it stood on each date; in equities, the research-grade databases carry delisting returns as a distinct field precisely because the delisted rows are the ones that carry the information. In FX and CFDs the same problem wears different clothes: symbols are renamed, brokers close and their histories vanish with them, and vendors quietly retire instruments with awkward pasts.

3. Adjustment — the price series was rewritten backwards

Splits, dividends, contract rolls and continuous-futures stitching all rewrite history to make a series continuous. The rewrite is applied with today’s adjustment factors, so the “price” on a date years ago is a number that could not have been quoted then, and a signal keyed to a level, a round number or a ratio is reading a value manufactured by the adjustment convention rather than by the market.

The tell is that the same date carries different prices in files built on different days. If you cannot state which convention produced your series and when it was applied, you cannot say what your level-based rule was actually triggering on.

4. Repair — the vendor cleaned it later

The quietest channel. Ticks classified as bad get filtered; gaps get backfilled from a second source with different latency and different spreads; missing sessions get padded so downstream consumers meet no holes. Each edit is small, defensible, and administered directly into the data pipeline where no downstream statistical check can see it.

Padding leaves the most recognisable signature: the carry-forward artifact, a stretch of repeated values that reads as market calm and actually records absent data. A simulation will happily fill stops and limits against those phantom prices. This is the domain of tick data provenance — knowing what a dataset contains and what has been done to it since capture.

How to test your own data for it

Four tests, in ascending order of effort. The first two are afternoons, and between them they catch most of what matters.

The as-of question

Take one value your strategy depends on and ask the dataset a single question: on what date did this value become knowable, and can the file tell me? Not “is it correct” — when did it arrive.

If the dataset has no field that answers this, you have learned something important and cheap: your pipeline cannot reconstruct the Knowable Set at all, and every downstream conclusion is conditional on the vendor having never edited anything. That is a strong assumption to hold silently.

The two-vintage diff

The most decisive test available to a working researcher, and it costs a wait rather than an analysis.

Take today’s export of a series and put it away untouched. In a month, export the same series over the same historical range, and diff the two files across the overlapping past — rows whose dates are old in both. Any difference is, by definition, an edit applied to history after the fact. There is no other explanation available.

What you are reading is not a magnitude but a structure: which fields move, whether the edits cluster around events or sessions, and whether the earliest history is stable while the recent past churns. A file that changes in its old rows is a file whose old rows were never a record of what was known. Run it on the vendor you actually backtest against, because this is a property of your supplier, not of data in general.

The freeze test

Content-hash the dataset the moment it arrives, before anyone has looked at it, and store the hash beside the file. From then on any modification — a cleaning pass, a resample, a “fix” — produces a different hash and has to be recorded as a lineage step rather than happening silently.

The value shows up later, when a result is challenged. The question “is the file I backtested byte-identical to the file I ingested?” is answerable with a hash and unanswerable without one, and two years is exactly long enough for memory to stop being evidence.

The feature knowability audit

Walk your feature code and, for each input, name the moment it became knowable. Then compare that moment against the bar it is attached to. Three constructions fail this audit routinely and none of them look like a bug:

• Full-sample normalisation. A z-score, a min-max scale or a winsorisation computed over the whole period embeds the period’s own statistics into every historical value. Every row now knows the distribution of its own future.
• Backward-filled joins. Merging a lower-frequency series onto bars, then filling backwards to remove gaps, propagates a value into the interval before it existed. Forward-fill is the honest direction; backward-fill is time travel with a friendly name.
• Labels that reach forward. Target construction almost always looks ahead by design, which is fine — until the label horizon overlaps a training window and the leak arrives through the split rather than through the feature. Purging and an embargo exist for exactly this.

The audit produces a list, and the list is the artifact. It is also the part nobody does, because it is bookkeeping rather than modelling.

The Knowable Set

The single-sentence version: at every simulated instant there is a set of facts that a participant could actually have held, and a backtest is valid only to the extent that it consulted nothing outside it.

The reframe earns its keep by moving the question off the code and onto the data. “Do I have look-ahead bias?” invites a review of your indexing and a false sense of safety when the review comes back clean. “Can I reconstruct the Knowable Set for this simulation?” has a procedural answer — you can or you cannot, and the four tests above tell you which — and it points at the vendor, the ingestion step and the join, which is where the leak usually lives.

It also explains why the discipline feels like a downgrade. Point-in-time work multiplies storage, complicates every query, and produces worse-looking backtests, because the flattering revisions and the buried failures are precisely what it removes. Numbers going down is the evidence that it is working.

What the wrong answers get wrong

“Use a longer history.” More years of a hindsight-assembled file is more hindsight. Length is orthogonal to vintage, and an older sample is often the more edited one, since it has been available for restatement longer.

“Buy premium data.” Price buys coverage, resolution and support. It does not, on its own, buy vintages — a well-known vendor with an immaculate export can still ship you the current version of history and no record of the previous ones. The question to ask a supplier is not “how clean is it?” but “can I request the file as it stood on a past date, and what is your change log?”

“Add a delay to every signal.” Lagging entries by a bar is a sound defence against the crude timing forms and no defence at all against revision, membership or repair. A one-bar delay applied to a revised series is a strategy trading yesterday’s version of the future.

“The out-of-sample period was clean.” Out-of-sample protects against selection, which is a different failure with different tests — treated separately in how to test a backtest for overfitting. A leak in the data pipeline is present in every partition of that data at once, including the held-out one, so partitioning cannot reveal it.

“It reproduces exactly, so it must be right.” Reproducibility says the computation is deterministic. It says nothing about whether the inputs were knowable. A leak reproduces perfectly, every time, which is part of why it survives review.

Where the measured version publishes

The uncomfortable part of this method is not the statistics — there are none. It is that the honest version requires a record kept before you knew you would need it: a hash taken at ingestion, a vintage stored when it was still current, an arrival time written down while it was still trivially available.

That is the part I am building into instrumentation rather than resolutions. Data Forensics is intended to point a fidelity battery at the dataset itself — carry-forward and merge artifacts, per-side staleness, gap structure, and whether bars are point-in-time or quietly revised — and to print “insufficient sample” as a result rather than hiding it. Provenance-Grade Ingestion is the upstream half: content-hashing at the door, registration timestamps, and gap ledgers, so that silent mutation stops being possible rather than being detected late. Both are pre-launch. Neither has published outputs, and this page carries none.