# Kill Ledger > An append-only research record of every strategy configuration that was tried and abandoned, kept so that multiple-testing corrections can be computed from the true number of trials rather than the remembered one. - Canonical: https://hadalinstruments.com/glossary/kill-ledger/ - Term set: https://hadalinstruments.com/glossary/ --- The most consequential number in backtest evaluation is one that almost no research process records: how many things were tried. Every multiple-testing correction — the [probability of backtest overfitting](/glossary/probability-of-backtest-overfitting/), the [deflated Sharpe ratio](/glossary/deflated-sharpe-ratio/) — is conditioned on the trial count N, and every one of them degrades into false reassurance when N is understated. N is always understated, because memory is the default ledger and memory is selective in exactly the wrong direction. Parameter sweeps run and discarded in an afternoon, notebooks deleted because they "didn't work", variants a colleague tried on the same data, the quick sanity checks that were — statistically — full trials: all of it vanishes, and what vanishes is precisely the failures. The surviving strategy is then presented as a discovery rather than what it often is: the lottery winner, described without mention of how many tickets were bought. A kill ledger is the corrective. It is an append-only record, written at the moment a configuration is killed, of what was tried, on what data, with what result, and why it died. Three properties make it load-bearing rather than decorative: it is **append-only**, so an embarrassing entry cannot be quietly removed; it is **contemporaneous**, recorded at the time of death rather than reconstructed for the audit; and it is **complete**, counting a thirty-second experiment the same as a month-long one, because the selection statistics do not care how long the trial took. Nothing about the practice is mathematically sophisticated — it is a timestamped list. The difficulty is behavioural: a kill ledger is a record of one's own failures, maintained against one's own incentive to forget them, which is precisely why it so rarely exists. Without one, corrections for [backtest overfitting](/glossary/backtest-overfitting/) compute a deflation for a trial count that is fiction, and [pre-registration](/glossary/pre-registration/) has no companion record of what happened after registration. The [Epistemic Harness](/instruments/epistemic-harness/) I am building is intended to make the ledger a service rather than a resolution. ## Commonly confused with Neighbouring concepts that get used interchangeably, and the distinction that actually separates them. - **A research journal** — A journal records what you did and can be edited, summarised or tidied. A kill ledger is append-only by design, so an embarrassing entry cannot be quietly removed. The immutability is the feature; without it the count is negotiable. - **[Pre-registration](https://hadalinstruments.com/glossary/pre-registration/)** — Registration records what a trial was supposed to be, before the data can argue. The ledger records what became of it, after. Neither substitutes for the other, and pre-registration without a ledger has no companion record of what happened next. - **Version control** — Git records what the code became; it does not record that a configuration was tried, judged and abandoned. A branch deleted after a disappointing sweep leaves no trial in the count, which is exactly the entry the statistics need. - **A list of your live strategies** — The survivors are the least informative part. The ledger exists for the ones that died — those are the trials that inflate the selection bias behind whatever is still running. ## How to measure it in your own data A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself. - **Records you need** — Written at the moment a configuration is killed: what was tried, on what data, with what result, and why it died. Count a thirty-second sanity check the same as a month-long study, because the selection statistics do not care how long a trial took. - **What you compute** — Nothing sophisticated — it is a timestamped list. The output is N, the true trial count, which feeds every multiple-testing correction you subsequently apply. - **What the answer tells you** — Three properties make it load-bearing rather than decorative. Append-only, so entries cannot be removed when they become inconvenient. Contemporaneous, recorded at the time of death rather than reconstructed for an audit. And complete, including the quick checks that were statistically full trials. Fail any one and the deflation you compute later is a deflation for a fictional number. ## Questions and answers ### Why is the trial count always understated? Because memory is the default ledger and it is selective in exactly the wrong direction. Parameter sweeps run and discarded in an afternoon, notebooks deleted because they did not work, variants a colleague tried on the same data, quick sanity checks that were statistically full trials — all of it vanishes, and what vanishes is precisely the failures. ### Do quick experiments really count as trials? Yes. The selection statistics condition on how many configurations the data has been asked about, not on how much effort each one cost. A thirty-second check that looked at the same history is a trial in exactly the sense that matters, and excluding it understates N. ### Why does it have to be append-only? Because the entries most worth keeping are the ones you will most want to remove. A ledger you can edit is a ledger whose count is negotiable at the moment the count becomes inconvenient, which is precisely when it needs to be fixed. ### If it is just a list, why does almost nobody keep one? Because the difficulty is behavioural rather than technical. A kill ledger is a record of your own failures, maintained against your own incentive to forget them. Nothing about it is mathematically sophisticated, and that is why its absence is a statement about discipline rather than about tooling. ## Related terms Derived from the links this entry makes and the entries that link back to it. - [Backtest Overfitting](https://hadalinstruments.com/glossary/backtest-overfitting/) The condition in which a strategy's historical performance reflects fitting to noise in a particular dataset rather than a persistent market effect, so that live performance regresses toward zero or below. - [Burned Door](https://hadalinstruments.com/glossary/burned-door/) A failure mode in automated systems where the agent destroys the evidence of a failed validation gate in order to falsely assert completion. - [Data Snooping](https://hadalinstruments.com/glossary/data-snooping/) The reuse of one dataset to evaluate many hypotheses — sequentially, collectively, or unconsciously — until something fits, at which point the fit is reported as if it were the only hypothesis ever tried. - [Deflated Sharpe Ratio (DSR)](https://hadalinstruments.com/glossary/deflated-sharpe-ratio/) A test statistic that adjusts an observed Sharpe ratio for the number of trials conducted, the length of the track record, and the non-normality of returns, estimating the probability that the true Sharpe ratio exceeds zero. - [False Discovery Rate (FDR)](https://hadalinstruments.com/glossary/false-discovery-rate/) The expected proportion of rejected null hypotheses that are in fact true — the share of your discoveries that are false, as distinct from the probability of making any false discovery at all. - [Pre-Registration](https://hadalinstruments.com/glossary/pre-registration/) The practice of recording a hypothesis, its test design, and its success criteria — timestamped — before examining the data that will judge it, so that a confirmation cannot have been shaped by the answer. - [Probability of Backtest Overfitting (PBO)](https://hadalinstruments.com/glossary/probability-of-backtest-overfitting/) A statistic estimating the probability that the strategy configuration selected as best in-sample will underperform the median of its rivals out-of-sample, typically computed via combinatorially symmetric cross-validation. - [Sharpe Ratio](https://hadalinstruments.com/glossary/sharpe-ratio/) The ratio of a return series' mean excess return to its standard deviation — how much return was earned per unit of volatility borne, expressed as a single scalar. ## Where the term is used Instrument pages whose published copy uses this term. Each page states what it measures and what it does not establish. - [Epistemic Harness](https://hadalinstruments.com/instruments/epistemic-harness/) Registry-as-a-service: pre-registration, walk-forward enforcement, the kill ledger. The Honesty Stack - [Reproducible Verdict Kernel](https://hadalinstruments.com/instruments/reproducible-verdict-kernel/) Same inputs, same verdict, byte for byte — or it is not a verdict. Backtest Honesty ## In the research Kill Ledger comes up in five research notes on this site, and this entry lists three of them. - [Does COT positioning predict reversals?](https://hadalinstruments.com/research/does-cot-positioning-predict-reversals/) Crowded positioning extremes precede fewer liquidations, not more. The interval excludes zero, p = 0.0066, and the sign is backwards. - [The level that remembered nothing](https://hadalinstruments.com/research/the-level-that-remembered-nothing/) My first real finding said price levels remember their own history. At sixteen years and 183,689 revisits, the memory was recency wearing a costume. - [How do I test a backtest for overfitting?](https://hadalinstruments.com/research/how-to-test-a-backtest-for-overfitting/) You cannot test the backtest. Overfitting is a property of the search that produced it, so the test needs four artifacts most research processes never record. ## Cite This Definition APA BibTeX HTML Hadal Instruments. (2026). Kill Ledger. Hadal Glossary. https://hadalinstruments.com/glossary/kill-ledger/ Version 426b66a, 2026-08-25. @misc{hadal_2026_kill-ledger, author = {Hadal Instruments}, title = {Kill Ledger}, year = {2026}, url = {https://hadalinstruments.com/glossary/kill-ledger/}, howpublished = {Hadal Glossary}, version = {426b66a}, note = {Pre-launch publication; version dated 2026-08-25} } Source: Hadal Instruments, Kill Ledger. Original Research Copy Citation **Version 426b66a** identifies the commit that last changed this page in Hadal's content repository. That repository is not public, so the identifier does not resolve externally — it is published so a citation pins one specific state rather than a moving page. To obtain the exact version cited, use the [press and research route](https://hadalinstruments.com/press/).