Does my backtest have survivorship bias?

The short answer

Probably — and the more useful answer is that you cannot find out by looking at the dataset. Survivorship bias is a defect made of absence: the rows that would convict it were never written, so every internal check passes — a check can only inspect the rows that exist. The audit is not of your data. It is of your selections — what was available, what survived to be included, and what left no trace of having been excluded.

That reframing matters because the standard telling of this subject sends a forex trader to look in the wrong place, for a mechanism their backtest mostly does not have, while the doors it actually came through stand open behind them.

The equity story is the wrong map

The way this subject is taught, one mechanism stands in for the whole phenomenon: build a backtest on today’s index constituents and the stocks that delisted are missing, so every long strategy tests better than it would have traded. The story is true, and for an FX trader it is close to useless — currency pairs rarely die the way companies do, so a trader who checks their backtest against the delisting story finds nothing and concludes the bias is absent.

The conclusion does not follow, because the bias is not a fact about delistings. It is any filtering done by attrition, upstream of your analysis, that left no record of what was removed. In a retail FX backtest that selection enters through three doors, and none of them looks like a delisted stock.

The instrument door. Feed histories exist only for venues that survived to serve them. A broker that shut down took its history with it; a vendor that retired an awkward symbol removed its past along with its present; the data you can download today is the data that something chose to keep serving. Which version of history you get is itself venue-shaped — why do two data feeds give different backtests is that subject — but before any two feeds can disagree, the set of feeds that still exist has already been filtered once.

The strategy door — the wide one. By the time a configuration reaches a serious backtest, it has survived a sequence of informal trials: symbols that looked worse, timeframes that were abandoned, parameter sets discarded in a demo and forgotten. Those variants are the strategy pool’s failures, and they are as absent from your final record as a dead company is from today’s index. This door connects survivorship to its downstream twin: backtest overfitting is fitting to noise that is present in the data, survivorship is noise that is absent from it, and the discarded-trial count that how to test a backtest for overfitting asks you to bring is precisely a survivor count from this door.

The record door. Every external record you benchmark against — published strategy results, copy-trading leaderboards, the accounts people link in forums — is a pool whose failures removed themselves. Records that went bad get deleted and restarted; accounts that died stop being linked. What remains visible is the surviving tail of a population whose size nobody wrote down, which is why how do you verify a trading track record treats a record’s existence as the weakest kind of evidence about it.

The Absent Denominator

The three doors share one arithmetic, and naming it is what turns a warning into an audit.

Every statistic computed from a survivor-only record is a numerator whose denominator left no row behind. The backtest return is the numerator; the denominator is every configuration that was tried. The visible leaderboard is the numerator; the denominator is every account that started. The downloadable history is the numerator; the denominator is every feed that once existed. In each case the number in front of you is real — and meaningless in proportion to how much of its denominator is missing, because the missing part did not leave at random. It left by failing.

That is the absent denominator, and it has one property worth engraving: it cannot be recovered from the numerator. No amount of analysis on the surviving data reconstructs the population it survived from. The denominator either was written down at the time, by someone with the discipline to record what was tried before knowing how it would end — or it is gone.

Which is why the one structural defence is to write the denominator down before the outcome exists. That is what pre-registration is, seen from this angle: a denominator fixed in advance, so no later failure can quietly leave it. It is also the reason this site’s own kill ledger publishes its dead hypotheses against the full registry count rather than alone — a kill count without its denominator is a marketing figure, and the register is the denominator, written before the verdicts arrived.

The audit, step by step

Each step is an enumeration, and each can end honestly in either of two ways: a written denominator, or a written admission that the denominator is unrecoverable. Both are findings. Silent optimism is the only wrong result.

1. The strategy door first, because it is yours. Write down every symbol, timeframe and parameter family you tried before the configuration under test, as honestly as memory allows. If the count is a reconstruction rather than a contemporaneous record, mark it as one — a remembered denominator is a floor, not a measurement. From today forward, keep the log the overfitting article calls the Search Receipt; it is this audit’s raw material, kept as it happens instead of excavated afterwards.
2. The instrument door. List the data sources your test depends on and establish, for each, what governs its historical coverage: how far back it reaches, which instruments were retired or renamed, whether gaps are documented or silent. Where the vendor cannot answer, record that the coverage policy is unknown — that sentence is itself a provenance finding, and tick data provenance is the discipline it belongs to.
3. The record door. For any external record your confidence leans on, ask what population it survived from and whether that population’s size was recorded anywhere. If it was not — and for leaderboards and forum records it never is — treat the record as an existence proof, not a performance estimate.
4. Close with the status line. For each door, one sentence: denominator recorded, denominator bounded from memory, or denominator unrecoverable. A backtest with three honest status lines attached is a different object from the same backtest without them, whatever the equity curve looks like.

The step this audit cannot perform is quantification. A recovered count establishes that the bias is present; how large it is depends on how the departed members performed, and their performance departed with them. Where the denominator is gone, the honest status is unquantified, not absent — the same terminus the glossary entry reaches, arrived at here from inside your own workflow.

Where the measured version publishes

Steps two and three run into the same wall: they need to know what a dataset actually contains, how it has been altered since capture, and whether its structure betrays a history its documentation does not mention. That examination — gap patterns, retired-symbol seams, point-in-time violations detectable from structure, a content-hash manifest pinning the audited file to its report — is the Data Forensics instrument. Its companion on the strategy door is the Overfit Auditor, because a trial count and a survivor count are the same number seen from opposite ends. Both are pre-launch, and both carry outputs marked NOT YET PUBLISHED.

The upstream disciplines are point-in-time data and the Knowable Set — what could actually have been known on the date being simulated — and the downstream consequence of ignoring them is the subject of why do backtests fail in live trading.