# Does my regime filter use future data?

> Usually not through a coding bug. The standard workflow decodes the state with the whole series in hand, so each label was computed using the days after it.

- Canonical: https://hadalinstruments.com/research/does-my-regime-filter-use-future-data/
- Published: 2026-08-29
- Author: Hadal Research
- Answers the question: "does my hmm regime filter use future data in a backtest"

---
## The short answer

A regime filter can read the future without a single wrong index in your code, because the leak is in the estimator rather than in the loop. The usual workflow fits a state model on the whole price history, asks it for the state sequence, and trades the result. That state sequence is a **smoothed** estimate: the label for each date was computed with every later observation in hand. What a live system has instead is the **filtered** estimate — the state given the observations up to that date and nothing after. These are two different numbers for the same day, and a backtest that trades the first is trading a label nobody could have held.

The corrective is not a better model. It is to fix the conditioning set and then prove it held: at every simulated date, the label acted on must have been computable from that date's own past. Everything below is a way of testing whether yours was.

  **WHAT THIS IS — AND WHAT IS NOT PUBLISHED.** This article is method. **I have measured and published no estimate of how much apparent edge a smoothed decode manufactures** — no gap figure, no ratio, no per-strategy contamination rate — and no such number appears on this page. The papers named below are cited so the distinction can be checked at its source, never as results of mine. Where I describe how my own instruments will apply this, those instruments are pre-launch and their outputs are `NOT YET PUBLISHED`.

## The Observable Mechanism
The mechanism is a conditioning set, and it is chosen silently. A state estimate is a conditional probability, so it is meaningless until you say what it is conditioned on — and the same fitted model will answer three different questions depending on which conditioning set the call implies.

Conditioned on observations up to and including the date in question, it is a filtered estimate, and it is the only one a participant standing there could have formed. Conditioned on the entire series, it is a smoothed estimate, and it is strictly better informed because it has read the ending. Conditioned on the entire series and additionally required to be jointly consistent, it is a global decode, which is better informed still. The observable consequence is that a label for a past date can change after the fact: run the same code again next month, with a month of new data appended, and yesterday's history is relabelled. A number that moves when the future arrives was a function of the future.

## Filtering and smoothing answer different questions

The distinction is old and it is not controversial. The forward pass of a hidden Markov model accumulates evidence in the direction time runs, and at each step it holds a belief about the current state given everything seen so far. The backward pass starts at the end of the series and propagates evidence in the opposite direction. Combining the two gives the smoothed estimate — the belief about each state given the whole record — and it is the standard product of a forward-backward run. Regime-switching models in economics carry the same pair, with a smoothing recursion that runs from the final observation back through the sample.

For the purpose those methods were built for, smoothing is the right answer and filtering is the impoverished one. A researcher describing a historical business cycle should use every observation available; declining to look at the second half of the sample would make the description worse. The literature is explicit about which is which, and neither is a mistake in itself.

The mistake is transplanting the descriptive answer into a decision rule. A trading rule is a claim about what could have been known at a point in time, so it inherits the opposite preference: the estimate that saw less is the only one that is admissible. When a workflow moves from describing a regime to trading it, the default that was correct silently becomes the defect, and nothing in the tooling announces the change of purpose.

## Three places the future gets in

Each of the three leaves a different trace, and a pipeline can be clean on one and open on the next. They are worth separating because the fixes do not overlap.

### 1. The decode conditioned on the whole file

This is the one that survives every code review, because there is nothing wrong with the code. A single call returns the state sequence, that sequence gets joined onto the price frame, and the strategy reads a column. The column is a smoothed or globally-decoded quantity. The tell is that no error was ever made and the labels are nonetheless better than any live system's, particularly around turning points — a smoothed decode is at its most flattering exactly where a filtered one is least certain, which is the moment a regime changes and the moment a regime filter is supposed to earn its place.

### 2. The parameters were fitted on the whole file

Transition probabilities, state means, variances and any covariance structure are estimated from the sample, and if that sample is the whole history then every parameter used at the start of the backtest was informed by its end. This channel is milder than the first, because parameters are a summary rather than a per-date label, but it is not nothing: a variance estimated across a period containing a crisis carries information about a crisis that had not happened yet at the dates it is being applied to. It is also the channel most people believe they have fixed, and the one a rolling refit genuinely does address.

### 3. The structure was chosen after seeing the outcome

How many states? Which observable — returns, realised volatility, spread, volume, some composite? What threshold turns a state probability into a position? These are decisions, and they are routinely made by trying several and keeping the configuration whose equity curve looked best. That is [backtest overfitting](/glossary/backtest-overfitting/) rather than look-ahead in the narrow sense, and it compounds with the first two: a structure selected on the same history that a smoothed decode already flattered is being chosen partly for how well it exploits the leak. No causal re-run of one configuration can undo a selection made across many.

## How to test your own regime filter

The tests below need nothing but the pipeline you already have and the discipline to run it twice. None of them requires a vendor, and none produces a number I have published — every figure they yield is yours, about your own system.

### The causal re-run

Re-run the strategy taking only the terminal estimate from each fit. Fit on data up to date *t*, read the state at *t* and nothing earlier, act, step forward, refit on the cadence you would genuinely refit at. This is slower and it is the only version whose result describes something achievable. Report the difference between it and the original as its own quantity rather than folding it into a revised headline: the gap is the measurement, and it is the number the original backtest was concealing.

### The two-label diff

Keep both label series for the same dates — the smoothed one your original run used, and the filtered one the causal re-run produced — and compare them date by date. Two things fall out. The disagreement rate tells you how much of the labelling was doing work that a live system could not do, and the distribution of disagreements over time tells you where: if they cluster in the days around regime boundaries, the filter's apparent skill is concentrated in exactly the transitions it is sold on.

### The relabelling test

Freeze a copy of the state series as computed today. Append the next period of data, re-run the identical code, and diff the labels for the dates that both versions cover. A causal pipeline produces no differences on the overlapping dates, because nothing in a past label depends on data that arrived later. Any date whose label moved is a date whose label was a function of the future, and the size of that set is a direct measure of the contamination — with the useful property that it can be run on a system you did not write and cannot read.

## What the wrong answers get wrong

**"Walk-forward validation covers this."** It covers the parameters, on the same argument as channel two, and it says nothing about how each fold is decoded. A fold that is refitted honestly and then smoothed internally leaks inside itself. [Walk-forward validation](/glossary/walk-forward-validation/) is necessary here and it is not sufficient, and the fold boundary is not where the problem lives.

**"The lag is small, so the effect is small."** The gap between a filtered and a smoothed estimate is not uniform across time, so an average lag is the wrong summary. It concentrates at transitions, and transitions are where a regime strategy takes its positions. A leak that is negligible in the flat middle of a state and decisive at its edges will look modest on any whole-sample statistic and still account for most of the strategy's result.

**"I can just shift the labels forward a day."** Shifting the series delays a smoothed estimate; it does not make it causal. The label for Tuesday moved to Wednesday still contains November, and the strategy still trades a quantity that no participant possessed. The property to establish is about the conditioning set, and no amount of shifting changes what a number was computed from.

## What This Does Not Establish (The Limits)
This article establishes nothing empirical. I have published no measurement of the smoothed-versus-filtered gap — not its typical size, not its distribution across strategy families, not the share of published regime results it would account for — and no such figure appears above or is implied by anything that does.

The tests set out are procedures, and every number they produce belongs to the reader's own system rather than to any result of mine. A causal re-run establishes that one configuration was tested honestly; it does not establish that the configuration is any good, and a filtered decode with no edge is a filtered decode with no edge. Nor does the third channel yield to this method at all: a structure chosen across many attempts on one history is a selection problem, and re-running the survivor causally leaves the selection intact and unmeasured. Finally, the distinction described here is a property of estimators generally, and the citations below are the sources for that distinction, not evidence about any market.

## Where the measured version publishes

The awkward part of this method is not the mathematics — there is none here beyond a distinction the sources below set out in the 1980s and 1990s. It is that the honest version costs a second full run of a pipeline that already produced an answer you liked, and produces a number whose only possible direction is downwards. Nobody schedules that voluntarily, which is exactly why it belongs in instrumentation rather than in good intentions.

That is the half I am building. The [Overfit Auditor](/instruments/overfit-auditor/) is intended to carry the causal re-run and the relabelling test as standing checks rather than as an afternoon of scripting — with the smoothed-minus-filtered gap reported as its own line item, beside the selection-stage measures that the third channel above needs and this method does not provide. The [Reproducible Verdict Kernel](/instruments/reproducible-verdict-kernel/) is the other half: it is what makes a result of this kind recomputable by somebody who has no reason to trust the person reporting it, which for a number that only ever embarrasses its author is the property that decides whether it gets published at all. Both are pre-launch. Neither has published an aggregate, and the gap figure this article declines to state is precisely the sort of quantity that publishes with its artifact or does not publish.

  **If you want the battery rather than the afternoon:** the <a href="/suites/backtest/">Backtest Honesty</a> suite is where the validation instruments live, and each instrument page states what it measures and what it does not establish before it says anything else. Pre-launch; nothing is for sale.
---

## Claims examined

### Claim 01 — canonical: https://hadalinstruments.com/refutations/#claim-b2a371f7

> "The regime labels come out of the model, not out of the future, so there is no look-ahead." — our reading: Misleading

The model does produce the labels, and that is precisely why the conditioning set matters. Asking a fitted model for the state sequence normally returns a decode over the entire series — a Viterbi path chosen to be jointly best given every observation, or smoothed marginals from a backward pass that starts at the end of the file. Either way the estimate for a Tuesday in March was informed by the following November. Nothing in that is a coding error; it is the default call doing exactly what it documents.

**What is true:** A state label is a conditional estimate, and the thing it is conditioned on decides whether it leaks: a label computed from observations up to that date is causal, while a label computed from the whole series used the days after it and could not have been held at the time.

Evidence: https://hadalinstruments.com/research/does-my-regime-filter-use-future-data/#claim-b2a371f7

### Claim 02 — canonical: https://hadalinstruments.com/refutations/#claim-40c85b9c

> "I refit the model on a rolling window, so my regime filter is walk-forward clean." — our reading: Misleading

Rolling refit addresses the parameters, and the parameters are rarely where the leak lives. The common shape is a window that is refitted honestly and then decoded in full, with the label read from somewhere inside it rather than at its right-hand edge. Every label except the last one in each window was smoothed by the observations that followed it inside that same window. The discipline that fixes it is not a shorter window; it is taking only the terminal estimate and stepping forward.

**What is true:** Refitting parameters on a rolling window controls one channel and leaves the decode untouched, so the causal question is whether the label traded on a given date was computed without any observation after it, which a window can satisfy only if the label taken from it is the one at its final edge.

Evidence: https://hadalinstruments.com/research/does-my-regime-filter-use-future-data/#claim-40c85b9c

### Claim 03 — canonical: https://hadalinstruments.com/refutations/#claim-ca323299

> "The regime filter lifted my backtest, so the regimes are real." — our reading: Unproven

A lift is evidence about the procedure that produced it, not about the market. Under a smoothed decode, part of the improvement is attributable to knowing where the calm period ended before it ended, which is an advantage no live system possesses. The two candidate explanations make the same backtest, and the honest way to tell them apart is to run the causal version and treat the gap as the measurement rather than as an inconvenience.

**What is true:** An improvement measured under a smoothed decode is consistent with a real regime effect and with a labelling artifact alike, so the comparison that separates them is the same strategy re-run under a causal decode, with the difference between the two reported as its own quantity.

Evidence: https://hadalinstruments.com/research/does-my-regime-filter-use-future-data/#claim-ca323299

## Cite This Article

APA BibTeX HTML

Hadal Research. (2026). Does my regime filter use future data?. Hadal Research. https://hadalinstruments.com/research/does-my-regime-filter-use-future-data/ Version e472963, 2026-09-14.

@misc{hadal_2026_does-my-regime-filter-use-future-data,
author = {Hadal Research},
title = {Does my regime filter use future data?},
year = {2026},
url = {https://hadalinstruments.com/research/does-my-regime-filter-use-future-data/},
howpublished = {Hadal Research},
version = {e472963},
note = {Published: 2026-08-29; version dated 2026-09-14}
}

Source: Hadal Research, Does my regime filter use future data?. <a href='https://hadalinstruments.com/research/does-my-regime-filter-use-future-data/' rel='canonical'>Original Research</a>

Copy Citation

**Version e472963** identifies the commit that last changed this page in Hadal's content repository. That repository is not public, so the identifier does not resolve externally — it is published so a citation pins one specific state rather than a moving page. To obtain the exact version cited, use the [press and research route](https://hadalinstruments.com/press/).

## Explore further

### Instruments

- [Data Forensics](https://hadalinstruments.com/instruments/data-forensics/)
- [Overfit Auditor](https://hadalinstruments.com/instruments/overfit-auditor/)
- [Reproducible Verdict Kernel](https://hadalinstruments.com/instruments/reproducible-verdict-kernel/)

### Concepts

- [Backtest Overfitting](https://hadalinstruments.com/glossary/backtest-overfitting/)
- [Look-Ahead Bias](https://hadalinstruments.com/glossary/look-ahead-bias/)
- [Point-in-Time Data](https://hadalinstruments.com/glossary/point-in-time-data/)
- [Purged Cross-Validation](https://hadalinstruments.com/glossary/purged-cross-validation/)
- [Regime Shift](https://hadalinstruments.com/glossary/regime-shift/)
- [Walk-Forward Validation](https://hadalinstruments.com/glossary/walk-forward-validation/)

### Research

- [Does my backtest have survivorship bias?](https://hadalinstruments.com/research/does-my-backtest-have-survivorship-bias/) Asked as: does my backtest have survivorship bias
- [Is my volatility regime just telling me the time?](https://hadalinstruments.com/research/is-my-volatility-regime-just-telling-me-the-time/) Asked as: is my volatility indicator just measuring time of day
- [Do order blocks predict anything?](https://hadalinstruments.com/research/do-order-blocks-predict-anything/) Asked as: do order blocks actually predict anything

[All Hadal research](https://hadalinstruments.com/research/)[This article as plain markdown](https://hadalinstruments.com/research/does-my-regime-filter-use-future-data.md)

---

## Raw artifact — NOT PUBLISHED FOR THIS PAGE

No downloadable artifact ships with this page. Eight published measurements do, each content-hashed so a reader can verify the figures independently. Where a measurement is published here without one, that is a gap rather than a policy, and it is stated rather than left to be noticed.

[Measurements that ship their data](https://hadalinstruments.com/research/)