Why do two data feeds give different backtests?
Asked as: why does my backtest give different results with different data feeds
In spot FX there is no single tape: every feed is one venue's filtered, aggregated history. Where feeds diverge, why results move, and how to diff yours.
The short answer
Because there is no single history for either feed to be a copy of. Spot FX has no consolidated tape: every provider’s file is a record of one liquidity pool, sampled one way, cleaned by one set of rules, stamped in one timezone convention. Run the same strategy over two such files and you are not testing one strategy against the market twice — you are testing it against two histories, each partially an artifact of decisions made in someone’s data pipeline.
The disagreement between your backtests is therefore not a nuisance to be averaged away. It is a measurement — of how much of your result depends on the recording rather than the market — and it is one of the cheapest and most informative measurements a strategy developer can take.
WHAT THIS IS — AND WHAT IS NOT PUBLISHED. This article is method: why recordings of the same market legitimately differ and how to locate the disagreement in your own files. No data vendor is measured, named or ranked here, and nothing should be inferred about any. Where a specific file needs auditing, that is a measurement — see the close of this piece. Status of any measured claim: NOT YET PUBLISHED.
Prerequisite Knowledge
You need two exports covering the same instrument and the same window — from two providers, or your platform’s bundled history beside a downloaded alternative — and the strategy result that differs between them. Failing that, one file and this article’s questions still work: most of the checks below interrogate a single file’s internal structure.
Five places the histories fork
1. The pool
Each feed records one aggregation of liquidity providers. Different pools see different quotes at the same instant — legitimately, because in a fragmented market there is no one price to record. This is the floor under everything else: even two perfect recordings of two pools would differ.
2. The sampling
Tick data is not “everything that happened”; it is what survived the provider’s sampling policy — every quote change, or only best-bid-offer changes, or a throttled stream, or bars built from any of those. Two files can describe the same pool and still disagree about how many ticks a fast minute contained, which decides what a backtester believes was tradeable inside it.
3. The cleaning
Providers filter what they classify as bad ticks, and the classification is a judgement call encoded in a pipeline: one file’s removed spike is another file’s retained print. The cleaning is invisible in the product — the file simply arrives without the ticks it lost — and the point-in-time article covers the quietest version, where gaps are backfilled from a second source with different spreads and nobody is told.
4. The clock
Server timezone, daylight-saving convention, and where the weekly boundary falls differ across providers. A mismatched clock convention shifts every session-dependent computation — daily bars, day boundaries for a daily loss limit, the definition of “the news minute” — and produces divergences that look like price disagreements but are actually calendar disagreements.
5. The seams
Weekend handling, holiday sessions, and rollover minutes are where files show their manufacturing marks: padded bars where one provider fills a closed session and another leaves a hole, spread conventions across the Sunday open, contract-roll adjustments on anything future-derived. Seams are rarely where you look and reliably where files disagree.
How to locate the disagreement in ten minutes
Align the two files on a common clock first — otherwise you are diffing calendars, not prices. Then, per session: bar counts, tick counts if you have them, and the spread distribution; then pull the largest price divergences and timestamp them. The pattern of the divergence is the finding. Clustered at news minutes: sampling and cleaning differences, the expected shape — and the reason a perfect news-period backtest is a pipeline claim, not a market claim. Uniform across quiet periods: suspect the clock. Concentrated at weekend seams: session handling. And if your strategy’s verdict flips between the files, weigh the third debunked claim above before shopping for a third feed.
The tick-data provenance entry defines the recording questions precisely; point-in-time data and survivorship bias cover the related ways a history can differ from the past it claims to record.
The Observable Mechanism
Everything here is computable from files you already hold, in a spreadsheet, without any provider’s cooperation. The diff between two histories is one of the few data-quality measurements that requires no reference truth — the disagreement itself is the observable, and its structure tells you which pipeline decisions produced it.
What This Does Not Establish (The Limits)
This article establishes why recordings legitimately differ — not which of your two files is better, whether either is faithful to the venue you trade on, or whether a specific defect explains a specific result. Structure can prove a file defective; it cannot prove it faithful — agreement between two files is only as strong as their independence, and two feeds sharing an upstream source share its defects invisibly. Fidelity to a venue is measurable only against a capture from that venue, which is a different measurement than any diff between downloads.
Where this leads
If the question is what a specific file actually contains — its gaps and their pattern, its seams, its look-ahead violations, with a content-hash manifest pinning the audited bytes to the findings — that is the Data Forensics instrument’s ground, and the Dataset Forensics assay runs it on your file as a one-off, provenance question included. And if the question is what your live feed is doing as it reaches you — the recording you could be making instead of downloading — that capture-side battery belongs to the Broker-Feed Auditor, with its own article on the method.
Claims examined
Claim 01§ claim-fbf6f23f
My broker's historical data is the real market — it came from my broker.
Your broker's history is one venue's filtered view of a fragmented market, further shaped by aggregation, bad-tick cleaning, and session handling before it reached your download. In spot FX no consolidated tape exists for it to be a copy of. That does not make it worthless — for backtesting a strategy you will trade at that venue, the venue's own history is the most relevant one available. It makes it one history among several, with edits you were not shown.
Claim 02§ claim-be93862c
The more expensive data feed is the more accurate one.
Price tracks coverage, depth and convenience — more pairs, more years, nicer formats — none of which is fidelity to the venue you actually trade on. Accuracy is not a property a feed has in isolation; it is a relation between a recording and a venue. A dear institutional feed can be an excellent record of liquidity you will never be quoted, while your own venue's rougher export is the truer description of the prices your orders would have met.
Claim 03§ claim-afd9f76f
Small differences between feeds can't change a backtest's conclusion.
Feed differences are not spread evenly through time — they cluster at news, at rollover, in thin sessions — which is exactly where many strategies concentrate their activity, so a small average difference becomes a large difference on precisely the bars that decide the result. A strategy whose verdict flips between feeds has revealed something important: its edge lives inside the disagreement between recordings, which is to say inside measurement noise, and no feed choice can rescue that.
Each claim above has a permanent address — the § link — whose canonical home is the refutation index, where it carries its variant phrasings and the true proposition stated on its own feet; this article is the evidence behind it. If a claim's text ever changes, it becomes a new claim at a new address, and the old one stops resolving rather than silently meaning something else.
Explore further
Instruments
Research
- How much does the range actually widen at NFP?What 1,820 scheduled-release windows across 28 pairs were measured for — and why the range-expansion factors stay unpublished until their artifact ships.
- Does my backtest have survivorship bias?Asked as:
does my backtest have survivorship bias
- Why does my backtest use data that did not exist?Asked as:
why does my backtest use data that did not exist yet