33 / 97
False Discovery Rate (FDR)
Written FDR throughout.
Working definition
The expected proportion of rejected null hypotheses that are in fact true — the share of your discoveries that are false, as distinct from the probability of making any false discovery at all.
When one hypothesis is tested, a p-value below the threshold means what it says. When a thousand are tested, some will clear the threshold by construction, and the question changes from is this result significant? to what fraction of my significant results are noise?
Two answers exist. Family-wise error control — the Bonferroni correction and its relatives — bounds the probability of making even one false rejection. It is the conservative choice, and across a search over thousands of candidates it is so conservative that almost nothing survives, real effects included. The false discovery rate, introduced by Benjamini and Hochberg, bounds instead the expected proportion of rejections that are false. Accepting that a stated share of discoveries will be wrong buys back a great deal of power.
The procedure is mechanical: rank the p-values, compare each against a threshold that rises with its rank, and reject everything up to the largest index that clears. The result is often reported as a q-value — the smallest FDR at which a given finding would be called a discovery. Under general dependence, which is the realistic case for related strategies tested on one history, the Benjamini–Yekutieli variant applies a stricter penalty.
The arithmetic is unremarkable. The input is not. Every FDR procedure conditions on the total number of tests, and in strategy research that total includes the specifications abandoned quietly, the parameters nudged and re-run, and the ideas tried on the same data months earlier. An FDR computed over the tests one remembers is not a correction; it is data snooping with a citation. The honest count comes from a kill ledger kept while the searching happens.
Why it matters
FDR converts a vague worry into a stated budget: this share of these findings is expected to be wrong, and the threshold was fixed before anyone looked. That is a claim a reader can audit — provided the denominator is the true number of trials, which is why the number belongs in the pre-registration rather than in the write-up.
Commonly confused with
Neighbouring concepts that get used interchangeably, and the distinction that actually separates them.
- Family-wise error rate
Family-wise control bounds the probability of making even one false rejection; FDR bounds the expected proportion of rejections that are false. Across a search over thousands of candidates the first is so conservative that almost nothing survives, real effects included. They answer different questions and only one of them suits strategy search.
- The p-value
A p-value belongs to a single test. FDR is a property of a collection of them. Reporting a p-value from the best of a thousand configurations as though it were the p-value of one hypothesis is the error the whole procedure exists to correct.
- The q-value
The q-value is how an FDR result is usually reported — the smallest false discovery rate at which a given finding would still be called a discovery. It is the output of the procedure rather than an alternative to it.
- Data snooping
Snooping is the failure; FDR is one of the corrections. The correction only works if the count of tests is honest, which means an FDR computed over the tests you happen to remember is not a correction at all — it is snooping with a citation attached.
How to measure it in your own data
A definition you cannot test is a definition you have to take on trust. This is the shortest honest route from the concept to a number you computed yourself.
- Records you need
Every p-value from the search, and — the input that actually decides the answer — the true total number of tests. In strategy research that total includes the specifications abandoned quietly, the parameters nudged and re-run, and the ideas tried on the same history months earlier.
- What you compute
Rank the p-values, compare each against a threshold that rises with its rank, and reject everything up to the largest index that clears — the Benjamini-Hochberg procedure. Under general dependence, which is the realistic case for related strategies tested on one history, apply the Benjamini-Yekutieli variant and its stricter penalty.
- What the answer tells you
The arithmetic is unremarkable and the denominator is everything. FDR converts a vague worry into a stated budget — this share of these findings is expected to be wrong, and the threshold was fixed before anyone looked — which is a claim a reader can audit only if the trial count is true. The honest count comes from a ledger kept while the searching happens, not reconstructed afterwards.
Questions and answers
How is FDR different from a Bonferroni correction?
Bonferroni and its relatives control the probability of making any false rejection at all, which is the right goal when a single false positive is costly. FDR controls the expected share of your rejections that are false. Accepting that a stated proportion of discoveries will be wrong buys back a great deal of statistical power, which is what makes it workable over a large search.
What is a q-value?
The smallest false discovery rate at which a given finding would still be called a discovery — the usual way an FDR result is reported. Reading it as "the probability this particular result is false" is close enough to be tempting and wrong enough to matter: it is a property of the threshold, not of the individual finding.
Does FDR control work when my strategies are related?
Not with the standard procedure, which assumes a form of independence that related strategies tested on one history do not satisfy. The Benjamini-Yekutieli variant handles general dependence with a stricter penalty. Using the standard version on a correlated family understates the correction.
Can I compute FDR after the search is finished?
The arithmetic, yes. The meaning, only if you know the true number of tests — and that is the number most searches fail to record. Every FDR procedure conditions on the total, so a correction computed over remembered trials produces a number that looks like rigour and is not.
Related terms
Derived from the links this entry makes and the entries that link back to it.
Where the term is used
Instrument pages whose published copy uses this term. Each page states what it measures and what it does not establish.
In the research
False Discovery Rate (FDR) comes up in two research notes on this site.