49 / 97
Kill Ledger
Working definition
An append-only research record of every strategy configuration that was tried and abandoned, kept so that multiple-testing corrections can be computed from the true number of trials rather than the remembered one.
The most consequential number in backtest evaluation is one that almost no research process records: how many things were tried. Every multiple-testing correction — the probability of backtest overfitting, the deflated Sharpe ratio — is conditioned on the trial count N, and every one of them degrades into false reassurance when N is understated.
N is always understated, because memory is the default ledger and memory is selective in exactly the wrong direction. Parameter sweeps run and discarded in an afternoon, notebooks deleted because they “didn’t work”, variants a colleague tried on the same data, the quick sanity checks that were — statistically — full trials: all of it vanishes, and what vanishes is precisely the failures. The surviving strategy is then presented as a discovery rather than what it often is: the lottery winner, described without mention of how many tickets were bought.
A kill ledger is the corrective. It is an append-only record, written at the moment a configuration is killed, of what was tried, on what data, with what result, and why it died. Three properties make it load-bearing rather than decorative: it is append-only, so an embarrassing entry cannot be quietly removed; it is contemporaneous, recorded at the time of death rather than reconstructed for the audit; and it is complete, counting a thirty-second experiment the same as a month-long one, because the selection statistics do not care how long the trial took.
Nothing about the practice is mathematically sophisticated — it is a timestamped list. The difficulty is behavioural: a kill ledger is a record of one’s own failures, maintained against one’s own incentive to forget them, which is precisely why it so rarely exists. Without one, corrections for backtest overfitting compute a deflation for a trial count that is fiction, and pre-registration has no companion record of what happened after registration. The Epistemic Harness I am building is intended to make the ledger a service rather than a resolution.
Commonly confused with
Neighbouring concepts that get used interchangeably, and the distinction that actually separates them.
- A research journal
A journal records what you did and can be edited, summarised or tidied. A kill ledger is append-only by design, so an embarrassing entry cannot be quietly removed. The immutability is the feature; without it the count is negotiable.
- Pre-registration
Registration records what a trial was supposed to be, before the data can argue. The ledger records what became of it, after. Neither substitutes for the other, and pre-registration without a ledger has no companion record of what happened next.
- Version control
Git records what the code became; it does not record that a configuration was tried, judged and abandoned. A branch deleted after a disappointing sweep leaves no trial in the count, which is exactly the entry the statistics need.
- A list of your live strategies
The survivors are the least informative part. The ledger exists for the ones that died — those are the trials that inflate the selection bias behind whatever is still running.
How to measure it in your own data
A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself.
- Records you need
Written at the moment a configuration is killed: what was tried, on what data, with what result, and why it died. Count a thirty-second sanity check the same as a month-long study, because the selection statistics do not care how long a trial took.
- What you compute
Nothing sophisticated — it is a timestamped list. The output is N, the true trial count, which feeds every multiple-testing correction you subsequently apply.
- What the answer tells you
Three properties make it load-bearing rather than decorative. Append-only, so entries cannot be removed when they become inconvenient. Contemporaneous, recorded at the time of death rather than reconstructed for an audit. And complete, including the quick checks that were statistically full trials. Fail any one and the deflation you compute later is a deflation for a fictional number.
Questions and answers
Why is the trial count always understated?
Because memory is the default ledger and it is selective in exactly the wrong direction. Parameter sweeps run and discarded in an afternoon, notebooks deleted because they did not work, variants a colleague tried on the same data, quick sanity checks that were statistically full trials — all of it vanishes, and what vanishes is precisely the failures.
Do quick experiments really count as trials?
Yes. The selection statistics condition on how many configurations the data has been asked about, not on how much effort each one cost. A thirty-second check that looked at the same history is a trial in exactly the sense that matters, and excluding it understates N.
Why does it have to be append-only?
Because the entries most worth keeping are the ones you will most want to remove. A ledger you can edit is a ledger whose count is negotiable at the moment the count becomes inconvenient, which is precisely when it needs to be fixed.
If it is just a list, why does almost nobody keep one?
Because the difficulty is behavioural rather than technical. A kill ledger is a record of your own failures, maintained against your own incentive to forget them. Nothing about it is mathematically sophisticated, and that is why its absence is a statement about discipline rather than about tooling.
Related terms
Derived from the links this entry makes and the entries that link back to it.
Where the term is used
Instrument pages whose published copy uses this term. Each page states what it measures and what it does not establish.
In the research
Kill Ledger comes up in five research notes on this site, and this entry lists three of them.