# Probability of Backtest Overfitting (PBO) > A statistic estimating the probability that the strategy configuration selected as best in-sample will underperform the median of its rivals out-of-sample, typically computed via combinatorially symmetric cross-validation. - Canonical: https://hadalinstruments.com/glossary/probability-of-backtest-overfitting/ - Term set: https://hadalinstruments.com/glossary/ --- The probability of backtest overfitting is the probability that the configuration a backtest selected as best in-sample underperforms the median of its rivals out-of-sample. It was introduced by David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu in "The Probability of Backtest Overfitting" — the [SSRN working paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253), later published in the *Journal of Computational Finance* ([September 2016, doi:10.21314/jcf.2016.322](https://doi.org/10.21314/jcf.2016.322)). The statistic attacks the selection problem directly. It does not ask whether a strategy's backtest is good; it asks whether the *act of selecting* that strategy from among its tried alternatives is likely to have selected noise. ## How it is calculated The standard estimator, combinatorially symmetric cross-validation (CSCV), works on the full performance history of every configuration tried — not just the winner. The history is partitioned into an even number of blocks. For each way of designating half the blocks as in-sample, the procedure identifies the configuration that performed best on that half, then observes where that same configuration ranks among all configurations on the other half. Repeating this across all combinations yields a distribution of out-of-sample ranks for in-sample winners. PBO is the fraction of combinations in which the in-sample winner falls below the median out-of-sample — the frequency with which "best in backtest" predicted "below average in practice." ## How to read a high or low PBO A low PBO says selection is finding something persistent across subsamples. A high PBO says the ranking of configurations is unstable: shuffle the data and a different lottery winner emerges, which is the signature of fitting noise. Neither is a certificate. A PBO below one half means the in-sample winner beat the median of its rivals out-of-sample more often than not — and that is all it means. A strategy with an excellent backtest can sit inside a search whose PBO is high, which is the useful case, because the number grades the procedure rather than the survivor. ## What the paper measures, and what this page adds The paper defines the statistic and proposes the estimator, with the selection of a strategy from a set of trials as the object under test. What this entry adds is the condition the estimator cannot check for itself: the honesty of its input. PBO is conditioned on the *recorded* set of trials; every configuration that was tried, failed, and quietly deleted biases the estimate downward, making the surviving strategy look more legitimate than it is. This is why PBO is inseparable from the discipline of a [kill ledger](/glossary/kill-ledger/) and [pre-registration](/glossary/pre-registration/): the statistic is only as truthful as the trial count behind it. It pairs naturally with the [deflated Sharpe ratio](/glossary/deflated-sharpe-ratio/), which corrects the winner's headline statistic for the same multiplicity rather than grading the selection; the two answer different questions and are read together. On this site the estimator runs inside the [Overfit Auditor](/instruments/overfit-auditor/), whose page publishes its reading on this estate's own research, and the procedure for a backtest you hold is walked through in [how to test a backtest for overfitting](/research/how-to-test-a-backtest-for-overfitting/). ## Commonly confused with Neighbouring concepts that get used interchangeably, and the distinction that actually separates them. - **[Deflated Sharpe ratio](https://hadalinstruments.com/glossary/deflated-sharpe-ratio/)** — Both correct for the same multiplicity and report different things. DSR corrects the winner's headline statistic and returns a probability that its true Sharpe exceeds zero. PBO ignores the headline entirely and asks whether the act of selecting was likely to have selected noise. - **An out-of-sample test** — A single out-of-sample result grades one configuration once. PBO uses the full performance history of every configuration tried, across many in-sample and out-of-sample splits, to grade the selection procedure rather than the survivor. - **[Overfitting itself](https://hadalinstruments.com/glossary/backtest-overfitting/)** — Overfitting is the condition; PBO is an estimate of how likely your selection process is to be producing it. A low PBO is evidence about the procedure, not a certificate for the strategy that came out of it. - **A backtest score** — PBO does not ask whether the backtest is good. It asks whether "best in backtest" has been predicting "above average in practice" across subsamples. A configuration with an excellent backtest can sit inside a search with a high PBO, which is the useful case. ## How to measure it in your own data A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself. - **Records you need** — The full performance history of every configuration tried — not only the winner. That is the demanding part: the estimator cannot run on a survivor, and most research keeps only survivors. - **What you compute** — Combinatorially symmetric cross-validation. Partition the history into an even number of blocks; for each way of designating half as in-sample, find the configuration that performed best on that half and observe where it ranks among all configurations on the other half. PBO is the fraction of combinations in which the in-sample winner falls below the median out-of-sample. - **What the answer tells you** — A low value says selection is finding something persistent across subsamples. A high value says the ranking of configurations is unstable — shuffle the data and a different lottery winner emerges, which is the signature of fitting noise. The estimate is biased downward by every configuration that was tried, failed and quietly deleted, so an unrecorded search makes the survivor look more legitimate than it is. ## Questions and answers ### What question does PBO actually answer? Not whether your strategy is good. It asks whether the act of selecting that strategy from among the alternatives you tried is likely to have selected noise — how often, across subsamples, the in-sample winner turned out to be below average out of sample. It grades the procedure rather than the survivor. ### Why does it need every configuration, not just the winner? Because the statistic is about ranking stability. Combinatorially symmetric cross-validation needs to know where the in-sample winner sits among all configurations on the held-out half, which is impossible if the rivals were not kept. A PBO computed on a survivor is not the statistic. ### What does a high PBO mean in practice? A high PBO means the ordering of your configurations is unstable across subsamples — resample the data and a different candidate comes first. That instability is what fitting noise looks like from the outside, and it is a stronger warning than any single disappointing out-of-sample result. ### Can PBO be gamed? Not deliberately so much as by omission, which is more common. Every configuration tried, failed and deleted biases the estimate downward. The statistic is only as truthful as the trial count behind it, which is why it is inseparable from keeping a record while the search happens. ## If this has already cost you The stability of a configuration ranking across subsamples is computable from the full set of variants you tried. - [Overfit Assay](https://hadalinstruments.com/assays/#overfit-assay) “Is my backtest real, or did I fit it to noise?” Will not establish: Whether the strategy will be profitable. A backtest that survives the battery is a backtest that was not obviously fitted — it is not a forecast, and the report says so on its first page. Intake is not open yet, so none of these can be commissioned today. They are listed here so you know the measurement exists and what it would and would not settle — the [launch list](https://hadalinstruments.com/pricing/#waitlist) hears first. ## Work it out yourself Free calculators that take this concept as an input. Each shows its working, so the number it gives you can be checked rather than taken on trust. - [Sharpe ratio calculator](https://hadalinstruments.com/tools/sharpe-ratio-calculator/) Annualised Sharpe, convention stated · The 95% interval your sample supports ## Related terms Derived from the links this entry makes and the entries that link back to it. - [Backtest Overfitting](https://hadalinstruments.com/glossary/backtest-overfitting/) The condition in which a strategy's historical performance reflects fitting to noise in a particular dataset rather than a persistent market effect, so that live performance regresses toward zero or below. - [Combinatorial Purged Cross-Validation (CPCV)](https://hadalinstruments.com/glossary/combinatorial-purged-cross-validation/) A backtest protocol that partitions a history into groups, holds out every combination of them in turn with purging and an embargo, and so produces many out-of-sample paths instead of a single one. - [Data Snooping](https://hadalinstruments.com/glossary/data-snooping/) The reuse of one dataset to evaluate many hypotheses — sequentially, collectively, or unconsciously — until something fits, at which point the fit is reported as if it were the only hypothesis ever tried. - [Deflated Sharpe Ratio (DSR)](https://hadalinstruments.com/glossary/deflated-sharpe-ratio/) A test statistic that adjusts an observed Sharpe ratio for the number of trials conducted, the length of the track record, and the non-normality of returns, estimating the probability that the true Sharpe ratio exceeds zero. - [Kill Ledger](https://hadalinstruments.com/glossary/kill-ledger/) An append-only research record of every strategy configuration that was tried and abandoned, kept so that multiple-testing corrections can be computed from the true number of trials rather than the remembered one. - [Pre-Registration](https://hadalinstruments.com/glossary/pre-registration/) The practice of recording a hypothesis, its test design, and its success criteria — timestamped — before examining the data that will judge it, so that a confirmation cannot have been shaped by the answer. - [Walk-Forward Validation](https://hadalinstruments.com/glossary/walk-forward-validation/) An out-of-sample testing protocol in which a strategy is fitted on one window of historical data and evaluated on the subsequent, unseen window, with the process rolled forward through time. ## Where the term is used Instrument pages whose published copy uses this term. Each page states what it measures and what it does not establish. - [Overfit Auditor](https://hadalinstruments.com/instruments/overfit-auditor/) Mathematical bounds on how hard you tortured the data. Backtest Honesty - [Epistemic Harness](https://hadalinstruments.com/instruments/epistemic-harness/) Registry-as-a-service: pre-registration, walk-forward enforcement, the kill ledger. The Honesty Stack ## In the research Probability of Backtest Overfitting (PBO) comes up in two research notes on this site. - [Do round numbers matter in forex?](https://hadalinstruments.com/research/do-round-numbers-matter-in-forex/) A 13-month window gave z = minus 2.59. Run the same test across twelve windows and the mean is plus 0.08, none significant, some pointing the other way. - [How do I test a backtest for overfitting?](https://hadalinstruments.com/research/how-to-test-a-backtest-for-overfitting/) You cannot test the backtest. Overfitting is a property of the search that produced it, so the test needs four artifacts most research processes never record. ## Cite This Definition APA BibTeX HTML Hadal Instruments. (2026). Probability of Backtest Overfitting (PBO). Hadal Glossary. https://hadalinstruments.com/glossary/probability-of-backtest-overfitting/ Version 654acee, 2026-09-14. @misc{hadal_2026_probability-of-backtest-overfitting, author = {Hadal Instruments}, title = {Probability of Backtest Overfitting (PBO)}, year = {2026}, url = {https://hadalinstruments.com/glossary/probability-of-backtest-overfitting/}, howpublished = {Hadal Glossary}, version = {654acee}, note = {Pre-launch publication; version dated 2026-09-14} } Source: Hadal Instruments, Probability of Backtest Overfitting (PBO). Original Research Copy Citation **Version 654acee** identifies the commit that last changed this page in Hadal's content repository. That repository is not public, so the identifier does not resolve externally — it is published so a citation pins one specific state rather than a moving page. To obtain the exact version cited, use the [press and research route](https://hadalinstruments.com/press/).