Probability of Backtest Overfitting (PBO)
A statistic estimating the probability that the strategy configuration selected as best in-sample will underperform the median of its rivals out-of-sample, typically computed via combinatorially symmetric cross-validation.
Introduced by Bailey, Borwein, López de Prado, and Zhu, the probability of backtest overfitting attacks the selection problem directly. It does not ask whether a strategy’s backtest is good; it asks whether the act of selecting that strategy from among its tried alternatives is likely to have selected noise.
The standard estimator, combinatorially symmetric cross-validation (CSCV), works on the full performance history of every configuration tried — not just the winner. The history is partitioned into an even number of blocks. For each way of designating half the blocks as in-sample, the procedure identifies the configuration that performed best on that half, then observes where that same configuration ranks among all configurations on the other half. Repeating this across all combinations yields a distribution of out-of-sample ranks for in-sample winners. PBO is the fraction of combinations in which the in-sample winner falls below the median out-of-sample — the frequency with which “best in backtest” predicted “below average in practice.”
A low PBO says selection is finding something persistent. A high PBO says the ranking of configurations is unstable across subsamples, which is the signature of fitting noise: shuffle the data and a different lottery winner emerges.
The estimator’s integrity depends entirely on the honesty of its input. PBO is conditioned on the recorded set of trials; every configuration that was tried, failed, and quietly deleted biases the estimate downward, making the surviving strategy look more legitimate than it is. This is why PBO is inseparable from the discipline of a kill ledger and pre-registration: the statistic is only as truthful as the trial count behind it. It pairs naturally with the deflated Sharpe ratio, which corrects the winner’s headline statistic for the same multiplicity.