Receipt page · Backtest Honesty
How the Backtest Honesty suite measures
This is a receipt page. Dossier §8.2 fixes what one contains, and the five sections below are that anatomy in order: the versioned methodology, the pre-registrations, the proof corpus, the changelog, and the invitation to recompute any published number without asking me for anything. Nothing has been measured yet. Three of the five sections are therefore empty, and they are empty in a way you can inspect — the structure is printed, the slots are named, and each one says NOT YET PUBLISHED rather than showing you something plausible. Looking for what this family will contain instead? That is the Backtest Honesty suite hub.
Receipt anatomy · dossier §8.2Backtest Honesty suiteReceipt version v1.0
- Instruments in scope
- 3
- Metrics defined
- 13
- Pre-registrations posted
- NOT YET PUBLISHED
- Artifacts published
- NOT YET PUBLISHED
- Measurements published
- NOT YET PUBLISHED
- 01Methodology, versionedProse and formal definitions. Every metric carries its name, its definition, the estimator that computes it, its known failure modes, and what it does not establish. The page carries a version, and the version is printed on it.DEFINED · v1.0
- 02Pre-registrationsDeclarations made and dated before any measurement runs: decision thresholds, exclusion rules, completeness bars. Immutable once posted — an amendment appends beneath the original with its reason, and never overwrites it.NOT YET PUBLISHED
- 03Proof corpusThe published artifacts themselves: dataset descriptors, content hashes, download links, and the version tag of the code that generated each one. A result whose inputs are not downloadable is not a receipt.NOT YET PUBLISHED
- 04ChangelogAppend-only, dated, written by a person. What changed on this page, when, and why. A correction is a new entry naming the entry it corrects — entries are never edited away.1 ENTRY
- 05Independent recomputationThe exact numbered steps a stranger follows to re-derive a published number without asking me for anything. Where the recomputation disagrees, the disagreement is the finding.PROCEDURE STATED
Methodology, versioned
version v1.0last changed 2026-08-0113 metrics · 3 instruments
The Backtest Honesty suite measures whether the evidence for a strategy would survive hostile review. It attacks the two places backtests actually fail: the search process that produced the equity curve, and the dataset underneath it. One instrument bounds how much of a reported result could be an artifact of how many things were tried; another audits whether the data itself is telling the truth; a third adjudicates event-study evidence reproducibly enough that a counterparty can re-derive the verdict.
Most backtests are not wrong at the strategy layer. They are wrong at the evidence layer — untracked trials, silent data defects, the best window presented as the typical one. A clean statistic computed on a lying dataset is still a lie, which is why this suite audits both.
Every metric below carries five fields, because four of them are the fields a methodology usually leaves out. Definition states what the quantity is, closely enough that a stranger could implement it. Estimator states how it is computed and which parameters must be fixed before the data is read. Known failure modes states where the estimator breaks — written now, while nothing depends on it, rather than conceded later under questioning.Does not establish states the claim the metric will not support, however natural the reading. The last field is the one that costs something to publish, which is why it is published.
The fidelity battery pointed at a dataset instead of a feed: whether the data under a clean-looking backtest is itself telling the truth.
P10.1Carry-forward and merge artifacts
- Definition
- Places where a stale value has been repeated forward to fill a hole, and seams where two sources have been stitched into a single series.
- Estimator
- Repeat-run detection on the value series against the run-length distribution the instrument would be expected to produce, plus discontinuity tests at candidate seams for shifts in level, variance and timestamp convention.
- Known failure modes
- A genuinely motionless market produces the same signature as a carry-forward. The only discriminators are session context and the behaviour of neighbouring instruments, both of which must be present in the sample or the finding is reported as indeterminate.
- Does not establish
- An artifact does not establish vendor intent, and a series showing none does not establish that the data is correct.
P10.2Per-side staleness inside a dataset
- Definition
- The age of each side of the quote within the dataset, as a distribution rather than an average.
- Estimator
- Per-record elapsed time since that side last changed, computed on the dataset’s own timestamps and conditioned on declared event windows.
- Known failure modes
- A dataset resampled before delivery carries the resampler’s staleness rather than the market’s, and the two are indistinguishable without access to the raw source.
- Does not establish
- It does not establish that a backtest run on this data is wrong — only where the data stopped moving.
P10.3Gap structure
- Definition
- What is missing from the series and where: session boundaries, holidays, outages, and the gaps that appear in one instrument but not in its neighbours.
- Estimator
- Expected-record reconstruction against the instrument’s session calendar; every absence is classified as calendar, cross-instrument, or unexplained, and the three are never pooled.
- Known failure modes
- Missing data is rarely missing at random. Censoring that correlates with volatility flatters every risk statistic computed on what remains, and only the classification makes that visible.
- Does not establish
- A gap ledger does not repair the series, and a series with no unexplained gaps is not thereby complete.
P10.4Bar provenance
- Definition
- Whether bars are what they claim to be: how OHLC was constructed, from which ticks, under what timestamp convention, and whether the series is point-in-time or has been revised.
- Estimator
- Reconstruction of bars from the underlying tick record where one is supplied, plus revision detection by comparing the delivered series against earlier deliveries of the same range.
- Known failure modes
- Without an earlier delivery or a tick source, revision is undetectable — and it is reported as undetectable rather than as absent, because those are different findings.
- Does not establish
- Confirmed provenance does not establish that the data is accurate. It establishes that the data is what it says it is.
Standing limit. It audits the data, not the strategy. A clean dataset does not validate the backtest run on it, and the instrument does not repair, reconstruct, or return a cleaned file.
Full instrument page →
Statistical bounds on how much of a backtest’s reported performance could be an artifact of the search that produced it.
P2.1Trial count
- Definition
- The number of configurations evaluated, explicitly or implicitly, before the reported one was selected. Every downstream correction conditions on this number.
- Estimator
- A count taken from the registered trial book, including trials abandoned early. Implicit trials — parameter sweeps, discarded variants, re-runs after a rule change — are counted under a rule fixed in advance.
- Known failure modes
- The count is self-reported unless it comes from an append-only registry. A trial book assembled after the fact is a reconstruction, and it is marked as one rather than treated as a record.
- Does not establish
- A trial count does not establish dishonesty on the researcher’s part. It establishes the denominator every correction downstream of it requires.
P2.2Probability of backtest overfitting
- Definition
- The probability that the configuration selected as best in sample underperforms the median of its rivals out of sample.
- Estimator
- Combinatorially symmetric cross-validation over the performance matrix of all evaluated configurations; the statistic is the fraction of splits in which the in-sample winner falls below the out-of-sample median.
- Known failure modes
- The procedure requires the performance series of every trial, not just the survivor. Supplied with a single column it cannot be computed, and the instrument refuses rather than estimating from what it was given.
- Does not establish
- A low value does not establish that a strategy works. It bounds one specific way the evidence can be an artifact of selection.
P2.3Deflated performance statistic
- Definition
- A reported Sharpe ratio adjusted for the number of trials, the length of the track record, and the non-normality of the return series.
- Estimator
- The deflated statistic computed against the trial count and the sample moments of the returns, published together with every input it conditions on.
- Known failure modes
- It is only as good as the trial count fed to it, which is the input most likely to be understated. Short track records make the adjustment large and the adjusted estimate itself uncertain.
- Does not establish
- A surviving deflated statistic does not establish an edge. It establishes that this particular deflation did not eliminate the result.
P2.4Walk-forward degradation
- Definition
- How performance decays from calibration window to unseen window, rolled through time and published as the whole curve rather than as its best point.
- Estimator
- Rolling calibration and holdout splits at a cadence fixed in advance. Every window’s result is reported, including the worst one.
- Known failure modes
- Reshaping the strategy between passes consumes the walk-forward’s evidential value. The pass count is carried with the result so the consumption is visible instead of forgotten.
- Does not establish
- A flat degradation curve does not establish out-of-sample viability. The future is not one of the windows.
Standing limit. It cannot certify that a strategy works — no statistic can. It bounds the ways the evidence can be an illusion, and every output ships with its can-fail proof.
Full instrument page →
The event-study verdict engine: effect estimates with honest uncertainty, multiplicity accounting, and verdicts that a counterparty can regenerate byte for byte.
P18.1Effect estimate with block-bootstrap interval
- Definition
- The estimated effect of an event class on a series, with a confidence interval that respects serial dependence rather than assuming it away.
- Estimator
- Effect estimated over the declared event window; interval from the stationary block bootstrap with block length selected by a rule fixed before the run and published with the result.
- Known failure modes
- Block length is a parameter: too short and the interval is optimistic, too long and it is uninformative. Overlapping event windows violate the resampling assumption and are removed by a declared rule rather than tolerated.
- Does not establish
- A non-zero effect does not establish a tradable one. It establishes that the estimate is separated from zero under this interval.
P18.2HAC standard errors
- Definition
- Standard errors that remain valid under autocorrelation and heteroskedasticity in the residuals.
- Estimator
- Heteroskedasticity- and autocorrelation-consistent covariance estimation, with the bandwidth rule declared before the run.
- Known failure modes
- HAC estimators carry finite-sample bias that grows with bandwidth relative to sample length. On short samples the interval’s true coverage is not the nominal level, and the sample length is published so the reader can judge.
- Does not establish
- Corrected standard errors do not establish that the model is correctly specified.
P18.3False-discovery-rate masking
- Definition
- The expected proportion of false declarations when a family of events is tested together, controlled across the family rather than one test at a time.
- Estimator
- The Benjamini-Hochberg step-up procedure applied across the declared family, with family membership fixed before the run.
- Known failure modes
- The correction is only as honest as the family declaration. A family narrowed after seeing which effects survived is the multiplicity problem restated in a more flattering form.
- Does not establish
- Surviving the correction does not establish that an effect is real. It establishes that it survived this correction on this family.
P18.4Byte reproducibility
- Definition
- The property that the same inputs and the same binary produce the identical verdict, byte for byte.
- Estimator
- A recomputation over the content-hashed inputs, compared byte for byte against the published verdict. A mismatch is a finding, not a rounding difference.
- Known failure modes
- Reproducibility is a property of a pinned toolchain. A verdict recomputed under a different binary may differ legitimately, which is why the binary is part of what is published.
- Does not establish
- Reproducibility does not establish correctness. A verdict can be reproducibly wrong, and this metric would not notice.
P18.5Keystone can-fail proof
- Definition
- A test demanding that the engine recover a planted synthetic effect while rejecting a declared set of placebo effects — detection and refusal, both required.
- Estimator
- A planted effect of known magnitude and a fixed placebo set; the proof passes only when the planted effect is recovered and every placebo is killed.
- Known failure modes
- A can-fail proof tuned until it passes proves only that it can be tuned. The planted magnitude and the placebo set are fixed in advance and published with the outcome.
- Does not establish
- Passing the proof does not establish power against effects unlike the planted one.
Standing limit. It renders verdicts on the event studies it is given. It does not generate hypotheses, and a recovered effect is a statistical finding rather than a recommendation.
Full instrument page →
Versioning rule: this page is v1.0. A change to any definition, estimator, failure mode or limit above increments the version and appends an entry to the changelog in §04 naming what changed. Definitions are never edited silently, because a definition that can move after a result is published is not a definition — it is a degree of freedom.
Pre-registrations
A pre-registration is a declaration made and dated before the measurement runs: the thresholds that will decide, the rules that will exclude, and the sample bar below which the honest null publishes instead of a number. Its entire value comes from its ordering. Posted before the answer is known it is a constraint; posted afterwards it is a description of a result, which is a different and much cheaper object wearing the same clothes.
Pre-registration record · Backtest HonestyNOT YET PUBLISHED
No pre-registration has been posted for the Backtest Honesty suite. Not one that is pending review, not one that is drafted and unhashed — none. This block is the structure a registration will occupy, printed empty on purpose, because the alternative is a page that describes a discipline while quietly implying it has already been exercised.
A pre-registration is worth exactly the provability of its ordering. It has to be posted, dated and content-hashed while the answer is still unknown; posted afterwards it is a description of a result, which is a different and much cheaper object. So the first registration cannot be backdated into this slot, and the slot stays visibly empty until one is posted in the only way that counts.
The field schema of a pre-registration record for the Backtest Honesty suite: each field, what it will hold, and its current value. Every value reads NOT YET PUBLISHED because no registration exists.| Field | What it will hold | Value |
|---|
registration_ref | The permanent identifier this registration is cited by. | NOT YET PUBLISHED |
|---|
scope | The instruments and the measurement window the declaration binds. | NOT YET PUBLISHED |
|---|
declared_utc | When the declaration was posted — necessarily before any data was touched. | NOT YET PUBLISHED |
|---|
first_data_utc | When collection began. This must fall after the line above, and the ordering is the evidence. | NOT YET PUBLISHED |
|---|
thresholds | Every decision threshold, fixed while the answer was still unknown. | NOT YET PUBLISHED |
|---|
exclusion_rules | What will be dropped from the sample, and on what stated grounds. | NOT YET PUBLISHED |
|---|
completeness_bar | The minimum sample below which the honest null publishes instead of a number. | NOT YET PUBLISHED |
|---|
document_sha256 | The content hash of the registered document itself. Any later edit changes it, visibly. | NOT YET PUBLISHED |
|---|
amendments | Appended corrections, each with its own date and reason. The original text stays. | NOT YET PUBLISHED |
|---|
What a Backtest Honesty registration must fix in advance. The lists below are classes of declaration, not declarations. They name the decisions that have to be made before the data is touched, because each one is a decision that could otherwise be made afterwards, in the direction that flatters the result. No value below has been registered.
Thresholds
- The significance level and the false-discovery-rate threshold applied across the declared family.
- The combinatorial split count used for cross-validation, and the block-length rule for any resampling.
- The minimum track length below which a deflated statistic is not reported at all.
Exclusion rules
- What counts as a trial, including implicit trials from parameter sweeps and re-runs after a rule change.
- How overlapping event windows are handled before any resampling touches them.
- Which configurations, if any, are removed from the performance matrix, and on what stated grounds.
Completeness bars
- The minimum number of evaluated configurations required before an overfitting probability is reported.
- The minimum record length per walk-forward window.
- The minimum event count in a family below which the honest null publishes instead of a verdict.
Immutability, stated before it is tested. Once a registration is posted it is not edited. If it is wrong, an amendment is appended beneath it carrying its own date and the reason for the change, and the original text stays where it is, readable, above the correction. A registration that quietly improved after the data arrived would be indistinguishable from one that was right all along — which is precisely why the append rule is written here, now, while there is nothing yet to be tempted by.
Amendment rule, stated in advance: a posted registration is never edited. An amendment is appended beneath the original carrying its own date and its reason, and the original text stays above it, readable. This page will show both.
Proof corpus
The corpus is the set of artifacts a published measurement ships with — not a description of them, the artifacts themselves, downloadable, each with the digest that proves you received the bytes I measured and the code tag that produced them. A result whose inputs cannot be downloaded is not a receipt; it is an assertion with better typography.
The corpus for this suite is empty. Every row below is a slot, and every slot is NOT YET PUBLISHED. The table is printed anyway, because a reader should be able to see the exact shape of what will arrive — and because a page that described a corpus without showing how empty it currently is would be making the claim it exists to refuse.
Five columns: artifact, contents, content hash, code tag, download. Scroll sideways if they do not all fit.
Proof corpus for the Backtest Honesty suite: the seven artifact classes a published receipt carries, what each will contain, and its current state. Every content hash, code tag and download reads NOT YET PUBLISHED, because no artifact from this suite's proof corpus has been published yet.| Artifact | What it will contain | Content hash | Code tag | Download |
|---|
| Registered methodology document | The versioned document these definitions are taken from, in the exact form it was registered — estimators, parameters, and the limits stated above. | SHA-256NOT YET PUBLISHED | NOT YET PUBLISHED | NOT YET PUBLISHED |
|---|
| Pre-registration record | The dated declaration: thresholds, exclusion rules and completeness bars, plus any amendments appended beneath the original with their reasons. | SHA-256NOT YET PUBLISHED | NOT YET PUBLISHED | NOT YET PUBLISHED |
|---|
| Trial book and performance matrix | Every evaluated configuration and its performance series — not the survivor alone — together with the dataset the matrix was computed on. | SHA-256NOT YET PUBLISHED | NOT YET PUBLISHED | NOT YET PUBLISHED |
|---|
| Result set | Per-metric results with intervals and effective sample sizes, and the honest nulls wherever a sample could not support a metric. | SHA-256NOT YET PUBLISHED | NOT YET PUBLISHED | NOT YET PUBLISHED |
|---|
| Can-fail proof transcript | For every test in the battery: the planted defect, the refusal that was expected, and the outcome that was observed. | SHA-256NOT YET PUBLISHED | NOT YET PUBLISHED | NOT YET PUBLISHED |
|---|
| Kill-ledger extract | Hypotheses registered against this suite and killed by the data, each with the run that killed it. Published with the same visibility as a registration. | SHA-256NOT YET PUBLISHED | NOT YET PUBLISHED | NOT YET PUBLISHED |
|---|
| Generation code | The tagged commit that produced the result set, with its build receipt. Named here because a result whose code version is unstated cannot be re-run. | SHA-256NOT YET PUBLISHED | NOT YET PUBLISHED | NOT YET PUBLISHED |
|---|
Corpus state, per instrument
Per instrument, so that the emptiness cannot hide behind a suite-level summary.
Changelog
Append-only, dated, written by a person. It records changes to this page — it is not a measurement log, and it will not become one. A correction is a new entry that names the entry it corrects; nothing here is ever edited away, because a changelog you can rewrite is a marketing surface with a monospace font.
2026-08-01 · v1.0
Receipt page established for the Backtest Honesty suite, carrying all five parts of the §8.2 anatomy: the versioned methodology with a formal definition, estimator, failure modes and non-claim for each of the 13 metrics its 3 instruments measure; the pre-registration structure with no registration in it; the proof-corpus table with no artifact in it; this changelog; and the independent-recomputation procedure. Supersedes the earlier per-suite methodology summary at this URL, which carried the battery outline without the receipt anatomy. No measurement, pre-registration, artifact or hash accompanies this version — every receipt slot below is empty as a matter of fact, not of omission.
Independent recomputation
Independent recomputation is what makes the rest of the page checkable rather than merely well-written. It is the exact sequence a stranger follows to re-derive a published number from this suite using only artifacts I published — no account, no request, no conversation with me at any point.
Today the procedure terminates at step 1, because no measurement from the Backtest Honesty suite has been published and there is no receipt to open. The steps are written now, in the specific form they will take for this suite, precisely so that they exist before the first result does and cannot afterwards be shaped to fit one.
Open the receipt and take its four identifiers.
Every published measurement links a receipt carrying four: the registration reference, the methodology-document digest, the input-manifest digest, and the generation-code tag. If any one is missing, stop — the result is not recomputable and should not be treated as though it were, including by me.
Verify the methodology document against its digest.
Download it, hash it, compare. A mismatch means the method you are about to apply is not the method that was registered, and everything after this step would be measuring a different thing.
Check the ordering before you check anything else.
The registration timestamp must precede the first-data timestamp on the manifest. If it does not, the registration is a description of a result rather than a constraint on one, and no statistic downstream can repair that.
Download the trial book and the performance matrix, and verify every digest.
The matrix is every evaluated configuration, not the survivor. A matrix with a single column cannot support the statistics below, and the correct response is to stop rather than to estimate.
Apply the registered exclusion rules yourself.
The trial-inclusion rule, the minimum track length, and the rule for counting implicit trials. Each one changes the denominator, and the denominator is the entire argument.
Check out the auditor at the code tag on the receipt and run it.
Where a procedure resamples, the receipt names the seed. A run without the receipt’s seed is a different computation and will not reproduce byte for byte — which is a difference in your run, not a finding about mine.
Recompute the trial count, the overfitting probability and the deflated statistic.
Then recompute the walk-forward curve whole. The published claim is the curve, not its best window, and a reproduction that reports only the best window has not reproduced the claim.
Run the keystone can-fail proof.
The planted synthetic effect must be recovered and every declared placebo must be rejected. Both halves, or the engine has not demonstrated that it can refuse.
If your number differs, the difference is the finding.
Send it with your inputs and the version you ran. A confirmed discrepancy publishes as a correction appended beside the original — and the original stays exactly where it is, unedited, because the error is the part of the record that proves the discipline is real.
The point of publishing this before there is anything to check: a recomputation procedure written after a result is a procedure written by someone who already knows which steps would be inconvenient.