75 / 97
Purged Cross-Validation
Working definition
A cross-validation scheme for time-dependent data in which any training observation whose information horizon overlaps the test window is removed, so that no label can leak across the split.
Standard k-fold cross-validation assumes observations are exchangeable: shuffle the rows, hold out a fifth, train on the rest. Market data violates that assumption twice over, and the violation is not subtle.
The first problem is that a financial label usually spans time. A row timestamped Tuesday may be labelled by what happened over the following ten days; a feature on that row may summarise the previous sixty. The row is therefore not a point but an interval. If a test fold holds Tuesday and a training fold holds the following Thursday, the two are built from overlapping data — the model is trained on part of the answer it is about to be graded on. That is look-ahead bias arriving through the back door of the split rather than the front door of the feature.
Purging is the fix. For every observation in the test fold, compute the interval its information spans, then delete from the training set every observation whose interval intersects it. What remains could not have seen the test period’s outcomes. The cost is real — purging discards rows, and the discarded rows are precisely those adjacent to the test window — but the alternative is a validation score that measures leakage rather than skill.
Purging alone is not sufficient. Serial correlation carries information forwards past the end of the test window, which is what an embargo removes, and the two are applied together. Run across many combinations of held-out folds, the same machinery becomes combinatorial purged cross-validation.
Why it matters
An unpurged k-fold on labelled market data reliably produces validation scores that live trading does not reproduce, and it produces them without the usual symptoms of backtest overfitting: the researcher did nothing wrong except apply a textbook procedure to data the textbook did not have in mind. Purging is what makes an out-of-sample claim mean what it says.
Commonly confused with
Neighbouring concepts that get used interchangeably, and the distinction that actually separates them.
- Standard k-fold cross-validation
Standard k-fold assumes observations are exchangeable and shuffles them. Market rows are intervals rather than points, so shuffling puts overlapping data on both sides of the split. Purging is what the same procedure looks like once that assumption is dropped.
- Embargoing
Purging removes training rows whose information intervals overlap the test window, on both sides. An embargo removes the rows that follow it, closing the leak that serial correlation carries forwards. They are applied together and neither is sufficient alone.
- Walk-forward validation
Walk-forward preserves chronological order and tests forward through time. Purged cross-validation keeps the fold structure and repairs the leak within it. The first mirrors deployment; the second extracts more evaluations from the same history at the cost of resembling deployment less.
- Look-ahead bias
The same leak arriving through a different door. Ordinary look-ahead enters through the feature — a value that did not exist yet. This one enters through the split, with impeccable features and a training set that overlaps the answer.
How to measure it in your own data
A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself.
- Records you need
For every observation, the interval its information actually spans — the label's forward horizon and the feature's lookback — rather than the single timestamp on the row. Most pipelines store the timestamp and not the interval, which is why the leak survives review.
- What you compute
For each observation in the test fold, compute that interval, then delete from the training set every observation whose interval intersects it. What remains could not have seen the test period's outcomes. Apply an embargo after the window in the same pass.
- What the answer tells you
The cost is real and worth naming: purging discards rows, and the discarded rows are precisely the ones adjacent to the test window. If your validation score falls materially after purging, that difference was the leak, and the earlier number was measuring it rather than skill.
Questions and answers
Why can I not just shuffle my data like any other dataset?
Because a financial row is an interval, not a point. A row timestamped Tuesday may be labelled by what happened over the following ten days and carry features summarising the previous sixty. Shuffle it into a training fold while a neighbouring row sits in the test fold and the model has been trained on part of the answer it is about to be graded on.
How much data does purging remove?
Enough to notice, and the loss is concentrated where it hurts — the rows adjacent to each test window are exactly the ones purged. That cost is the price of the score meaning what it says. The alternative is a cheaper number that measures leakage.
Is purging enough on its own?
No. Purging handles intervals that formally overlap; serial correlation carries information past the end of the test window without any formal overlap at all. The embargo removes that, and the two are applied as a pair rather than as alternatives.
Why does unpurged k-fold produce results that live trading never reproduces?
Because the researcher did nothing wrong except apply a textbook procedure to data the textbook did not have in mind. There is no parameter to blame and none of the usual symptoms of an overfitted search — just a validation score that quietly measured leakage. That is what makes it worth naming as its own failure.
Related terms
Derived from the links this entry makes and the entries that link back to it.
In the research
Purged Cross-Validation comes up in one research note on this site.