15 / 97
Combinatorial Purged Cross-Validation (CPCV)
Written CPCV throughout.
Working definition
A backtest protocol that partitions a history into groups, holds out every combination of them in turn with purging and an embargo, and so produces many out-of-sample paths instead of a single one.
A conventional backtest produces exactly one out-of-sample path: this strategy, over this history, in this order. Any statistic computed from it — a Sharpe ratio, a maximum drawdown — is a single draw from a distribution nobody has seen, and one draw supports no statement about variance.
Combinatorial purged cross-validation, proposed by López de Prado, generates the distribution. The history is partitioned into N contiguous groups; every combination of k groups is held out as a test set in turn, with purging and an embargo applied at each boundary; the held-out segments are then stitched into whole out-of-sample paths. Choosing N and k fixes how many paths result — a count that grows quickly, which is the method’s cost and its point.
What a distribution supports differs in kind from what one path supports. The spread of Sharpe ratios across paths says how much of a headline figure is sampling variation. The share of paths in which the configuration selected as best in-sample fails to beat the median of its rivals out-of-sample is the probability of backtest overfitting. Neither is available from a single pass, which is why walk-forward validation and CPCV answer different questions rather than competing.
The compute cost is genuine: every path is a full fit and evaluation. The subtler cost is that the paths are not independent — they are drawn from one finite history — so the distribution is narrower than it looks, a point effective sample size makes precise.
Why it matters
A backtest is usually asked did this work? The question worth asking is how often would something like this have worked, and how far apart are the outcomes? CPCV is the machinery that makes the second question answerable — and, like every multiple-testing correction, it is honest only if the trial count fed into it is the true one.
Commonly confused with
Neighbouring concepts that get used interchangeably, and the distinction that actually separates them.
- Purged cross-validation
Purging is the leak repair applied at every fold boundary. CPCV is the protocol that applies it across every combination of held-out groups. The first makes one split honest; the second turns many honest splits into a distribution.
- Walk-forward validation
They answer different questions rather than competing. Walk-forward mirrors deployment — always trading on parameters chosen without the future. CPCV abandons that resemblance to extract many out-of-sample paths from the same history, which is what makes variance estimable.
- A conventional backtest
A conventional backtest produces exactly one out-of-sample path: this strategy, this history, this order. Any statistic from it is a single draw from a distribution nobody has seen, and one draw supports no statement about variance.
- Probability of backtest overfitting
PBO is one of the things a CPCV distribution can be used to compute — the share of paths where the in-sample winner fails to beat the median out of sample. The protocol produces the paths; PBO is a reading taken from them.
How to measure it in your own data
A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself.
- Records you need
The full history, the information interval of every observation so purging can be applied at each boundary, and the performance of every configuration if PBO is the goal rather than a single strategy's variance.
- What you compute
Partition the history into N contiguous groups, hold out every combination of k in turn with purging and an embargo at each boundary, then stitch the held-out segments into whole out-of-sample paths. Choosing N and k fixes how many paths result, and the count grows quickly.
- What the answer tells you
Read the spread, not the mean. The dispersion of Sharpe ratios across paths says how much of a headline figure is sampling variation — which is the whole reason to run it. Two costs are real: every path is a full fit and evaluation, and the paths are not independent, since they are drawn from one finite history. The distribution is therefore narrower than it looks.
Questions and answers
What does CPCV give me that a normal backtest does not?
A distribution instead of a single draw. A conventional backtest produces one out-of-sample path, so any Sharpe ratio or maximum drawdown computed from it is one sample from a distribution you have never seen. CPCV generates many paths from the same history, which is what makes it possible to say how much of the headline number is sampling variation.
How do I choose N and k?
Together they fix the number of paths, and the count grows quickly with both. That growth is simultaneously the method's cost — every path is a full fit and evaluation — and its purpose, since a distribution needs enough paths to have a shape. The practical constraint is usually compute.
Is CPCV better than walk-forward validation?
Neither replaces the other. Walk-forward resembles deployment, always trading on parameters chosen without knowledge of the future, which is the property you want when asking whether a strategy could have been run. CPCV gives up some of that resemblance to buy variance estimates from the same history.
Are the paths independent?
No, and this is the subtle cost. Every path is drawn from one finite history, so they share data and the resulting distribution is narrower than its shape suggests. Treating the spread as if it came from independent samples overstates how much you have learned — the same dependence that effective sample size makes precise.
Related terms
Derived from the links this entry makes and the entries that link back to it.
In the research
Combinatorial Purged Cross-Validation (CPCV) comes up in one research note on this site.