Definition

91 / 97

Tick Data Provenance

Working definition

The documented history of a market dataset: where each tick was captured, when, through what aggregation, and every alteration — cleaning, backfill, merge, adjustment — applied to it since capture.

A historical dataset is a claim about the past, and like any claim it has a chain of custody — usually an undocumented one. Ticks were captured at some venue, through some aggregation, with some latency; they were then filtered, merged, resampled, and “corrected”, often years later, by parties with no record of what they changed. Tick data provenance is the discipline of knowing that history, and its absence is the unexamined assumption underneath most backtests.

The alterations are rarely malicious and never neutral. Vendors remove ticks their filters classify as bad — handing the simulation a smoother past than any live system experienced. Gaps are backfilled from a different source with different latency and different spreads, splicing two microstructures into one series that belonged to neither. Symbols are remapped, instruments with awkward histories are retired, and adjustments are applied retroactively. Each silent edit is a small dose of look-ahead bias or survivorship bias administered directly into the data pipeline, where no validation protocol downstream can detect it. Padding routines add their own contribution: the carry-forward artifact, stability manufactured from missing data.

Provenance discipline is mechanical rather than clever. Content-hash the dataset at ingestion, before anyone has looked at it, so that any later modification produces a new hash and a recorded lineage step. Keep a gap ledger, so missing sessions are visible wounds instead of silent patches. Record when data arrived relative to when hypotheses about it were registered. None of this improves the data; it makes the data’s condition knowable, which is the precondition for point-in-time honesty about what was actually knowable when.

The test of provenance is blunt: can you prove, two years later, that the file you backtested is byte-identical to the file you ingested? The Provenance-Grade Ingestion instrument I am building is intended to make that answer yes by construction.

Commonly confused with

Neighbouring concepts that get used interchangeably, and the distinction that actually separates them.

Data quality

Quality asks whether the data is good; provenance asks whether you can say what happened to it. A clean dataset with no history is less trustworthy than a messy one with a complete lineage, because the first cannot tell you what the cleaning removed.

Point-in-time data

Point-in-time is about what was knowable on each historical date. Provenance is about what has been done to the record since it was captured. Provenance is the precondition — you cannot make an honest point-in-time claim about a dataset whose alterations are undocumented.

Data cleaning

Cleaning is an alteration, and an undocumented one is indistinguishable from corruption. Vendors remove ticks their filters classify as bad, handing the simulation a smoother past than any live system experienced. The problem is not that they cleaned; it is that nothing recorded what left.

A checksum

A checksum tells you a file changed. Provenance tells you what it was, what changed it, when, and from what source the replacement came. Hashing is one mechanism inside the discipline rather than the discipline itself.

How to measure it in your own data

A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself.

Records you need

Capture venue, capture time, aggregation path and latency for each tick; then every alteration since — filters applied, gaps backfilled and from where, symbols remapped, adjustments applied retroactively. Plus a record of when data arrived relative to when hypotheses about it were registered.

What you compute

Content-hash the dataset at ingestion, before anyone has looked at it, so any later modification produces a new hash and a recorded lineage step. Keep a gap ledger so missing sessions are visible wounds rather than silent patches.

What the answer tells you

The test is blunt: can you prove, two years later, that the file you backtested is byte-identical to the file you ingested. None of this improves the data — it makes the data's condition knowable, which is the precondition for any honest claim about what was knowable when. Watch particularly for gaps backfilled from a different source with different latency and different spreads, which splices two microstructures into one series that belonged to neither.

If this has already cost you

What a dataset actually sampled, and what has been altered since capture, is examinable before another result is built on it.

  • Dataset Forensics Assay“Can I trust the data file my research stands on?”Will not establish: That the data is correct. Structure can prove a file defective; it cannot prove it faithful to the market — only a second independent source can, and the report names that limit.

Intake is not open yet, so none of these can be commissioned today. They are listed here so you know the measurement exists and what it would and would not settle — the launch list hears first.

Questions and answers

Why does it matter what a vendor changed years after capture?

Because every silent edit is a small dose of look-ahead or survivorship bias administered directly into the data pipeline, where no validation protocol downstream can detect it. Purging, embargoing and walk-forward all assume the data at each point is what was available at that point. When the pipeline broke that assumption, every downstream check passes on a compromised result.

Are these alterations malicious?

Rarely, and never neutral. Filters remove ticks classified as bad; gaps get backfilled from whatever source was available; instruments with awkward histories are retired; adjustments are applied retroactively. Each is a defensible operational decision, and collectively they hand you a past that no live system experienced.

What does good provenance discipline actually look like?

Mechanical rather than clever. Content-hash at ingestion before anyone has looked at the data. Keep a gap ledger so a missing session is a visible wound rather than a silent patch. Record when data arrived relative to when hypotheses about it were registered, so the ordering can be checked later.

How would I know if my dataset has provenance problems?

Ask the blunt question first: can you demonstrate that the file you are backtesting is byte-identical to the file you received. If the answer is no, or nobody knows, the dataset's condition is unknown rather than good — and that is a finding you can act on immediately.

Derived from the links this entry makes and the entries that link back to it.

  • Carry-Forward ArtifactA stretch of a market dataset in which missing values were filled by repeating the last known price, manufacturing bars that look like market stability but actually record the absence of data.
  • Epps EffectThe systematic decline of measured correlation between two price series as the sampling interval shrinks; at fine grains the instruments rarely print at the same instant, and the correlation formula reads that asynchrony as independence.
  • Footprint chartA candle opened up: per price level within each bar, the volume that traded — usually split by aggressor side — so the bar shows not just where price went but who paid to move it.
  • Look-Ahead BiasThe use, at a simulated decision point, of any information that would not have been available at that moment in live trading.
  • Order flowThe stream of orders arriving at a market — aggressive orders consuming resting size and passive orders posting it — and, by extension, the discipline of reading that stream from book and tape data.
  • Point-in-Time DataData recorded as it was actually known on each historical date — first-print economic releases, as-of universe membership, unrevised financials — rather than the revised series that exists only in hindsight.
  • Survivorship BiasA distortion introduced when a dataset includes only the entities that survived to the present, so that failures vanish from history and everything measurable looks better than it was.
  • Volume profileA histogram of traded volume by price level over a chosen window — where the trading happened on the price axis, rather than when it happened on the time axis.

Tick Data Provenance comes up in eight research notes on this site, and this entry lists three of them.

Cite This Definition

Hadal Instruments. (2026). Tick Data Provenance. Hadal Glossary. https://hadalinstruments.com/glossary/tick-data-provenance/ Version 426b66a, 2026-08-25.

Version 426b66a identifies the commit that last changed this page in Hadal's content repository. That repository is not public, so the identifier does not resolve externally — it is published so a citation pins one specific state rather than a moving page. To obtain the exact version cited, use the press and research route.