73 / 97
Probability of Backtest Overfitting (PBO)
Written PBO throughout.
Working definition
A statistic estimating the probability that the strategy configuration selected as best in-sample will underperform the median of its rivals out-of-sample, typically computed via combinatorially symmetric cross-validation.
The probability of backtest overfitting is the probability that the configuration a backtest selected as best in-sample underperforms the median of its rivals out-of-sample. It was introduced by David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu in “The Probability of Backtest Overfitting” — the SSRN working paper, later published in the Journal of Computational Finance (September 2016, doi:10.21314/jcf.2016.322). The statistic attacks the selection problem directly. It does not ask whether a strategy’s backtest is good; it asks whether the act of selecting that strategy from among its tried alternatives is likely to have selected noise.
How it is calculated
The standard estimator, combinatorially symmetric cross-validation (CSCV), works on the full performance history of every configuration tried — not just the winner. The history is partitioned into an even number of blocks. For each way of designating half the blocks as in-sample, the procedure identifies the configuration that performed best on that half, then observes where that same configuration ranks among all configurations on the other half. Repeating this across all combinations yields a distribution of out-of-sample ranks for in-sample winners. PBO is the fraction of combinations in which the in-sample winner falls below the median out-of-sample — the frequency with which “best in backtest” predicted “below average in practice.”
How to read a high or low PBO
A low PBO says selection is finding something persistent across subsamples. A high PBO says the ranking of configurations is unstable: shuffle the data and a different lottery winner emerges, which is the signature of fitting noise. Neither is a certificate. A PBO below one half means the in-sample winner beat the median of its rivals out-of-sample more often than not — and that is all it means. A strategy with an excellent backtest can sit inside a search whose PBO is high, which is the useful case, because the number grades the procedure rather than the survivor.
What the paper measures, and what this page adds
The paper defines the statistic and proposes the estimator, with the selection of a strategy from a set of trials as the object under test.
What this entry adds is the condition the estimator cannot check for itself: the honesty of its input. PBO is conditioned on the recorded set of trials; every configuration that was tried, failed, and quietly deleted biases the estimate downward, making the surviving strategy look more legitimate than it is. This is why PBO is inseparable from the discipline of a kill ledger and pre-registration: the statistic is only as truthful as the trial count behind it. It pairs naturally with the deflated Sharpe ratio, which corrects the winner’s headline statistic for the same multiplicity rather than grading the selection; the two answer different questions and are read together. On this site the estimator runs inside the Overfit Auditor, whose page publishes its reading on this estate’s own research, and the procedure for a backtest you hold is walked through in how to test a backtest for overfitting.
Commonly confused with
Neighbouring concepts that get used interchangeably, and the distinction that actually separates them.
- Deflated Sharpe ratio
Both correct for the same multiplicity and report different things. DSR corrects the winner's headline statistic and returns a probability that its true Sharpe exceeds zero. PBO ignores the headline entirely and asks whether the act of selecting was likely to have selected noise.
- An out-of-sample test
A single out-of-sample result grades one configuration once. PBO uses the full performance history of every configuration tried, across many in-sample and out-of-sample splits, to grade the selection procedure rather than the survivor.
- Overfitting itself
Overfitting is the condition; PBO is an estimate of how likely your selection process is to be producing it. A low PBO is evidence about the procedure, not a certificate for the strategy that came out of it.
- A backtest score
PBO does not ask whether the backtest is good. It asks whether "best in backtest" has been predicting "above average in practice" across subsamples. A configuration with an excellent backtest can sit inside a search with a high PBO, which is the useful case.
How to measure it in your own data
A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself.
- Records you need
The full performance history of every configuration tried — not only the winner. That is the demanding part: the estimator cannot run on a survivor, and most research keeps only survivors.
- What you compute
Combinatorially symmetric cross-validation. Partition the history into an even number of blocks; for each way of designating half as in-sample, find the configuration that performed best on that half and observe where it ranks among all configurations on the other half. PBO is the fraction of combinations in which the in-sample winner falls below the median out-of-sample.
- What the answer tells you
A low value says selection is finding something persistent across subsamples. A high value says the ranking of configurations is unstable — shuffle the data and a different lottery winner emerges, which is the signature of fitting noise. The estimate is biased downward by every configuration that was tried, failed and quietly deleted, so an unrecorded search makes the survivor look more legitimate than it is.
If this has already cost you
The stability of a configuration ranking across subsamples is computable from the full set of variants you tried.
- Overfit Assay“Is my backtest real, or did I fit it to noise?”Will not establish: Whether the strategy will be profitable. A backtest that survives the battery is a backtest that was not obviously fitted — it is not a forecast, and the report says so on its first page.
Intake is not open yet, so none of these can be commissioned today. They are listed here so you know the measurement exists and what it would and would not settle — the launch list hears first.
Work it out yourself
Free calculators that take this concept as an input. Each shows its working, so the number it gives you can be checked rather than taken on trust.
Questions and answers
What question does PBO actually answer?
Not whether your strategy is good. It asks whether the act of selecting that strategy from among the alternatives you tried is likely to have selected noise — how often, across subsamples, the in-sample winner turned out to be below average out of sample. It grades the procedure rather than the survivor.
Why does it need every configuration, not just the winner?
Because the statistic is about ranking stability. Combinatorially symmetric cross-validation needs to know where the in-sample winner sits among all configurations on the held-out half, which is impossible if the rivals were not kept. A PBO computed on a survivor is not the statistic.
What does a high PBO mean in practice?
A high PBO means the ordering of your configurations is unstable across subsamples — resample the data and a different candidate comes first. That instability is what fitting noise looks like from the outside, and it is a stronger warning than any single disappointing out-of-sample result.
Can PBO be gamed?
Not deliberately so much as by omission, which is more common. Every configuration tried, failed and deleted biases the estimate downward. The statistic is only as truthful as the trial count behind it, which is why it is inseparable from keeping a record while the search happens.
Related terms
Derived from the links this entry makes and the entries that link back to it.
Where the term is used
Instrument pages whose published copy uses this term. Each page states what it measures and what it does not establish.
In the research
Probability of Backtest Overfitting (PBO) comes up in two research notes on this site.