Overfit Auditor

Every backtest is a claim, and most of them are confessions. The Overfit Auditor puts a number on the difference.

The instrument takes a strategy’s backtest — the equity curve, the parameter history, and crucially the number of trials it took to find it — and computes the statistics the marketing screenshot leaves out: the probability that the selected configuration is an artifact of the search rather than a property of the market, and how much of the reported performance survives once you account for how many things were tried before this one worked.

What it measures

• Trial accounting. How many configurations were evaluated, explicitly or implicitly, before this one was selected. Every downstream statistic conditions on this number, and almost nobody records it. The instrument makes it unavoidable.
• Probability of backtest overfitting. Via combinatorially symmetric cross-validation: how often does the in-sample winner underperform the median of its rivals out-of-sample?
• Deflated performance statistics. The reported Sharpe ratio adjusted for trials, track-record length, and the non-normality of returns — the version of the number a referee would accept.
• Walk-forward degradation. How performance decays from calibration window to unseen window, rolled through time — the degradation curve published, not the single best window.

What it does not do

It does not tell you a strategy works. No statistic can. It bounds the ways in which the evidence for a strategy can be an illusion — that is all, and that is the honest maximum on offer. Every output ships with its can-fail proof and its computation receipt, recomputable from the same inputs.

Who it is for

Anyone about to risk capital on a curve they fitted: independent quants, prop-firm evaluees who paid for an evaluation seat, and desks reviewing external track records. If your backtest cannot survive this instrument, the market will run the same test with your money.