Tick Data Provenance
The documented history of a market dataset: where each tick was captured, when, through what aggregation, and every alteration — cleaning, backfill, merge, adjustment — applied to it since capture.
A historical dataset is a claim about the past, and like any claim it has a chain of custody — usually an undocumented one. Ticks were captured at some venue, through some aggregation, with some latency; they were then filtered, merged, resampled, and “corrected”, often years later, by parties with no record of what they changed. Tick data provenance is the discipline of knowing that history, and its absence is the unexamined assumption underneath most backtests.
The alterations are rarely malicious and never neutral. Vendors remove ticks their filters classify as bad — handing the simulation a smoother past than any live system experienced. Gaps are backfilled from a different source with different latency and different spreads, splicing two microstructures into one series that belonged to neither. Symbols are remapped, instruments with awkward histories are retired, and adjustments are applied retroactively. Each silent edit is a small dose of look-ahead bias or survivorship bias administered directly into the data pipeline, where no validation protocol downstream can detect it. Padding routines add their own contribution: the carry-forward artifact, stability manufactured from missing data.
Provenance discipline is mechanical rather than clever. Content-hash the dataset at ingestion, before anyone has looked at it, so that any later modification produces a new hash and a recorded lineage step. Keep a gap ledger, so missing sessions are visible wounds instead of silent patches. Record when data arrived relative to when hypotheses about it were registered. None of this improves the data; it makes the data’s condition knowable, which is the precondition for point-in-time honesty about what was actually knowable when.
The test of provenance is blunt: can you prove, two years later, that the file you backtested is byte-identical to the file you ingested? The Provenance-Grade Ingestion instrument we are building is intended to make that answer yes by construction.