Kill Ledger

The most consequential number in backtest evaluation is one that almost no research process records: how many things were tried. Every multiple-testing correction — the probability of backtest overfitting, the deflated Sharpe ratio — is conditioned on the trial count N, and every one of them degrades into false reassurance when N is understated.

N is always understated, because memory is the default ledger and memory is selective in exactly the wrong direction. Parameter sweeps run and discarded in an afternoon, notebooks deleted because they “didn’t work”, variants a colleague tried on the same data, the quick sanity checks that were — statistically — full trials: all of it vanishes, and what vanishes is precisely the failures. The surviving strategy is then presented as a discovery rather than what it often is: the lottery winner, described without mention of how many tickets were bought.

A kill ledger is the corrective. It is an append-only record, written at the moment a configuration is killed, of what was tried, on what data, with what result, and why it died. Three properties make it load-bearing rather than decorative: it is append-only, so an embarrassing entry cannot be quietly removed; it is contemporaneous, recorded at the time of death rather than reconstructed for the audit; and it is complete, counting a thirty-second experiment the same as a month-long one, because the selection statistics do not care how long the trial took.

Nothing about the practice is mathematically sophisticated — it is a timestamped list. The difficulty is behavioural: a kill ledger is a record of one’s own failures, maintained against one’s own incentive to forget them, which is precisely why it so rarely exists. Without one, corrections for backtest overfitting compute a deflation for a trial count that is fiction, and pre-registration has no companion record of what happened after registration. The Epistemic Harness we are building is intended to make the ledger a service rather than a resolution.