72 / 97
Pre-Registration
Working definition
The practice of recording a hypothesis, its test design, and its success criteria — timestamped — before examining the data that will judge it, so that a confirmation cannot have been shaped by the answer.
Pre-registration is a discipline quantitative research borrowed from medicine. Clinical trials began registering their designs in advance because the alternative had a name and a body count: outcomes switched after the data arrived, negative results quietly shelved, analyses re-run until something cleared significance. Markets research has the same disease with fewer regulators.
The mechanism it defends against is not fraud but flexibility. Every choice made after seeing the data — which instruments to include, which window to test, which metric to headline, which “obviously broken” period to exclude — is a researcher degree of freedom, and each one consumed silently converts an out-of-sample test into an in-sample fit. A hypothesis written down before the test binds those choices while they are still cheap to bind: universe, period, parameters, evaluation metric, and the result that will count as failure, all fixed before the data can argue back.
The binding is only as strong as its provability. A registration that could have been backdated proves nothing, so the timestamp must be independently verifiable — which in practice means registering into something append-only, and recording when the data itself arrived relative to the registration. The ordering is the evidence: a hypothesis registered before the data existed is a prediction; the same hypothesis written afterwards is a description wearing a prediction’s clothes. My own Burned Door post-mortem documents what it looks like when that ordering is destroyed rather than preserved.
Pre-registration pairs with the kill ledger: registration records what a trial was supposed to be, the ledger records what became of it, and together they make the trial count behind a surviving strategy auditable. Neither makes a strategy good. They make the evidence for it mean what it claims to mean — which is the property data snooping removes, and the property no amount of after-the-fact statistics can restore to an unregistered result.
Commonly confused with
Neighbouring concepts that get used interchangeably, and the distinction that actually separates them.
- Documentation
Documentation records what you did; pre-registration constrains what you may do. The difference is entirely in the ordering — written before the data can argue back, the same words are a prediction; written afterwards they are a description wearing a prediction's clothes.
- A kill ledger
Registration records what a trial was supposed to be; the ledger records what became of it. Neither substitutes for the other, and together they make the trial count behind a surviving strategy auditable rather than asserted.
- Out-of-sample testing
A held-out sample protects the data; pre-registration protects the decisions. Choices made after seeing results — which instruments, which window, which metric to headline — silently convert an out-of-sample test into an in-sample fit, and no amount of holding data back prevents that.
- A trading plan
A plan states what you intend to do. A registration additionally states what result will count as failure, before the data has an opinion. The failure criterion is the part that binds, and the part most often left out.
How to measure it in your own data
A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself.
- Records you need
Hypothesis, universe, period, parameters, evaluation metric, and — the field that does the work — the result that will count as failure. Plus a timestamp that cannot have been backdated, and a record of when the data itself arrived relative to the registration.
- What you compute
Register into something append-only, then compare the registration timestamp against the data's own arrival. The ordering is the evidence: a hypothesis registered before the data existed is a prediction, and the same hypothesis registered afterwards is not.
- What the answer tells you
The binding is only as strong as its provability. A registration that could have been backdated proves nothing, however sincerely it was written. If the append-only property cannot be demonstrated to somebody who does not trust you, the registration is a note to self rather than evidence.
Questions and answers
What does pre-registration actually protect against?
Not fraud, in most cases — flexibility. Every choice made after seeing the data is a researcher degree of freedom: which instruments to include, which window to test, which metric to headline, which obviously-broken period to exclude. Each one consumed silently converts an out-of-sample test into an in-sample fit, and none of them feels like cheating at the time.
Why does the timestamp have to be independently verifiable?
Because the entire value is in the ordering. A registration you could have written afterwards carries exactly the evidential weight of no registration at all. In practice that means registering into an append-only record, and recording when the data arrived relative to it, so the sequence can be checked by someone with no reason to take your word.
Does pre-registration make a strategy more likely to work?
No, and claiming otherwise would misrepresent it. It makes the evidence for the strategy mean what it claims to mean. A registered strategy that fails has produced an honest negative result; an unregistered one that succeeds has produced a number whose trial count nobody can reconstruct.
Can I fix an unregistered result with statistics afterwards?
Not reliably. Multiple-testing corrections all condition on the number of trials, and the reason an unregistered search is hard to correct is that the count was never kept. This is the property data snooping removes, and it is not one that after-the-fact statistics can restore.
Related terms
Derived from the links this entry makes and the entries that link back to it.
Where the term is used
Instrument pages whose published copy uses this term. Each page states what it measures and what it does not establish.
In the research
Pre-Registration comes up in ten research notes on this site, and this entry lists three of them.