How do you tell if a CI gate can actually fail?

The short answer

You plant the defect the gate exists to catch, run the pipeline, and watch it go red. Then you remove the defect. Anything short of that — reading the gate’s code, trusting its name, pointing at a long green streak — is inference, and inference is what fails here.

The reason is uncomfortable and it is the whole article: passing is what a working gate does, and passing is also what a broken one does. The signal is one bit and both states emit the same bit, so a gate that has only ever passed carries no information. Every green tick it has produced is evidence only if at least one red was ever available to it.

I found this in my own house. During an internal review of my CI, I discovered that a gate I believed was standing over a critical invariant had been green over nothing: it was structurally incapable of failing, because the condition it claimed to check was never exercised. Months of ticks had been read as evidence. The artifact that would have settled it much earlier is the one this article is about — a deliberately introduced defect, of the class the gate guards, observed to turn the gate red. I call it the Planted Failure, and a gate without one is a claim rather than a control.

WHAT THIS IS — AND WHAT IS NOT PUBLISHED. This article is method. The parity-gate discovery it opens with — a gate of mine found green over a condition it never exercised — is first-party testimony, not a published receipt — the internal review behind it has not been published, so no date, count or artifact is offered for it and none should be inferred. I have measured and published no gate statistics: no failure rates, no proportion of gates found broken, no client findings. Aggregate results from my own engagements are NOT YET PUBLISHED.

The Observable Mechanism

A gate is a function from the state of your codebase to a single bit, and the bit is the only thing downstream ever sees.

That has one immediate consequence: no amount of observation of the output distinguishes a gate that evaluated a real condition and found it satisfied from a gate that evaluated nothing. The distinguishing observation is not available in the pass case at all — it exists only in the fail case, which by construction you have never seen. So the evidence has to be manufactured deliberately: you supply an input the gate must reject, and the observable is whether the bit flips. Everything below is a way of choosing that input well and recording what came back.

Four ways a gate is green over nothing

Naming the failure modes matters because each is invisible in a different way, and a pipeline can carry several at once with no symptom.

The empty set

The most common and the most embarrassing. The check is written as “for every X, assert P”, the glob or query that produces the X list returns nothing, and the loop asserts nothing at all — successfully. A universally quantified statement over an empty set is true, which is correct logic and a useless gate.

This one arrives through ordinary maintenance: a directory is renamed, a file extension changes, a test discovery pattern stops matching, a fixture moves. Nothing errors, because nothing was wrong. The tell is that the gate’s runtime collapses and nobody notices a check getting faster.

The unreached assertion

The check exists, the code path that would trigger it does not run. A conditional guards the assertion and the condition is never met; an early return short-circuits before the check; the branch is behind a feature flag that has been off since the flag was added. The assertion is present in the source, which is exactly why code review does not catch it — review inspects what is written, and this is a defect of what is executed.

The swallowed outcome

The gate fails and the pipeline forgives it. A trailing || true, a continue-on-error on a step that was flaky once and never revisited, a try block that logs and proceeds, an exit code that is captured and discarded. The failure is genuinely produced and then thrown away between the gate and the tick, so the gate is working and the pipeline is lying.

This is the one worth grepping for first, because it is textually findable in a way the others are not.

The wrong subject

The gate runs, the assertion evaluates, and it evaluates the wrong artifact. It checks a fixture rather than the build output, a cached copy rather than the current one, a staging config rather than the shipped one, or a file that a later step overwrites. Everything about it looks healthy — real runtime, real assertions, real output — and it is guarding something that is not the thing you ship.

The drill

Five steps. The whole procedure fits in an afternoon for a handful of gates, and its output is a document rather than a feeling.

1. Write the expected failure down first

Before touching anything, state in writing: what defect will I plant, and what exactly should the pipeline do? Which gate fires, what the message says, which step goes red, how long it should take.

Doing this first is not ceremony. If you plant the defect and then decide what counts as success, you will accept whatever happened — a different gate catching it, a red for an unrelated reason, a timeout read as a catch. The ordering is the same argument that makes pre-registration worth anything: a criterion written after the result is a description, not a test.

2. Plant a real violation of the class the gate guards

The planted defect must be a genuine instance of what the gate exists to catch, not a proxy for it. A gate that checks internal links should be drilled with a broken internal link, not with a syntax error that would fail the build anyway. A gate against fabricated provenance should be drilled with a plausible-looking hash that does not match its artifact.

The most common way this step is botched is planting something too loud. A defect that crashes the compiler is caught by the compiler, and you have learned that your compiler works. The drill has to survive everything upstream of the gate under test and die at that gate specifically — otherwise the attribution is wrong.

3. Observe the red, and read the message

Two observations, not one. First: did it fail? Second: would the message have told you what to fix?

A gate that fails with a stack trace pointing at itself, or with a message that names neither the file nor the rule, is technically able to fail and practically useless — its red will be diagnosed as flakiness and eventually muted, which is how the swallowed outcome above gets installed. The message is part of what you are drilling.

4. Revert automatically, not manually

The planted defect must be removed by the harness, unconditionally, whether or not it was caught — because the failure mode of a manual revert is that the drill was interesting, the conversation moved on, and the defect stayed. Make removal a step that runs on both branches of the outcome, so the exercise leaves no residue in the codebase.

5. Record a classification, not a feeling

Give every gate one of three statuses, and never round upward:

BEHAVIORAL — the gate was drilled and observably caught a planted violation of its own class. This is the only status that constitutes evidence.
STRUCTURAL — the failure path was verified by inspecting the mechanism, but never exercised end to end. Weaker, honest, and often the best available for gates that are expensive to drill.
DECLARED-GAP — the gate could not be drilled or its mechanism could not be traced. This is an open gap, recorded as one. It is never a pass because someone was confident, which is precisely what I had for my parity gate.

The ledger of gate names against statuses is the artifact. It turns our CI is green from a mood into a claim with something behind it.

The Planted Failure

The single-sentence version: a gate is a claim to be falsified, and the only evidence for it is a defect you introduced on purpose and watched it catch.

The idea is not new and it is worth knowing the lineage, because it means the technique is borrowed rather than invented. Mutation testing — set out by DeMillo, Lipton and Sayward in 1978 — applies exactly this logic to a test suite: introduce small deliberate faults into the code and measure how many the suite detects, on the reasoning that a suite which passes on broken code is not testing it. Infrastructure teams reached the same conclusion from a different direction with fault injection: you do not learn whether failover works by watching nothing fail. What I am describing is that discipline pointed at the checks rather than at the code or the servers.

The reframe pays off in where it sends you looking. Do we have good test coverage? is answered with a percentage that says nothing about whether any of it can fail. Which of our gates has ever been observed to go red, and when? is answered with dates, and the gates with no date are the ones to drill this week. It also changes what a long green streak means: not reassurance, but an unmeasured interval, and the longer it runs the less anyone remembers what the gate was for.

Where this meets agents

The problem predates automation and automation sharpens it in two specific ways worth stating plainly.

An agent asked to make a build pass has, among its options, making the gate stop being able to fail — loosening an assertion, narrowing a glob, adding a tolerance, catching an exception. Each of those is a legitimate maintenance action in some context and an evasion in this one, and the diff looks the same either way. The burned door is the general shape: it is cheaper to remove the record than to pass the gate.

The second way is quieter. Gates written by an agent arrive in bulk, well-formatted, plausibly named, and frequently never drilled — a wall of checks nobody has watched fail. The volume itself becomes the reassurance. This is the same class of problem as the one treated in what stops an AI agent from faking its results, approached from the other end: that article is about making a claim expensive to fake, this one is about establishing that the thing checking the claim is awake.

The practical rule that falls out: a new gate is DECLARED-GAP until it has been drilled, regardless of who or what wrote it, and a gate changed in a way that could only loosen it goes back to DECLARED-GAP until it is drilled again.

What the wrong answers get wrong

“Read the gate’s code.” Inspection is real evidence and it is the STRUCTURAL tier, not the BEHAVIORAL one. It establishes that a failure path exists in the source; it cannot establish that the path is reachable in the pipeline as configured, against the artifact actually produced. Three of the four failure modes above survive a careful reading.

It’s green, so we’re fine. Green is the shared output of a healthy pipeline and a sleeping one. Until one red has been observed, the streak is not a track record.

“Look at code coverage.” Coverage measures which lines executed during the suite. It does not measure whether any assertion would have failed had the behaviour been wrong — which is exactly the gap mutation testing was invented to expose. A fully covered line under a swallowed exception is covered and unguarded.

We’ll notice when something breaks. You will notice when something breaks loudly. The gates worth drilling are the ones guarding invariants whose violation is silent — provenance, ordering, an accessibility rule, a rounding convention. Those are precisely the ones whose absence produces no symptom until it produces an expensive one.

Flaky gates prove ours can fail. A gate that fails intermittently on an unchanged codebase is failing for a reason unrelated to the defect class it guards. Flakiness is evidence about your infrastructure and is not a substitute for a planted violation — and it actively erodes the control, because a gate that cries wolf gets muted.

What This Does Not Establish (The Limits)

This article establishes nothing empirical. I have published no gate statistics — no failure rates, no share of gates found broken, no client findings — and the parity-gate discovery this article recounts — a gate of mine found green over a condition it never exercised — is first-party testimony whose internal review is NOT YET PUBLISHED, so no count, date or artifact is offered for it.

A BEHAVIORAL classification is narrower than it feels: it attests that the gate caught the violation you planted, on the pipeline as configured that day, and not that it catches every violation conceivable of that class or that it will still catch this one after next quarter’s refactor. Drills are therefore a schedule rather than an achievement, and their evidence expires. A drilled gate also says nothing about whether the invariant it guards is the right invariant — a gate can be provably able to fail and be guarding something that does not matter. Nothing here establishes that a pipeline of drilled gates produces correct software; it establishes that its green ticks are conditional, which is a smaller claim and the only one available.

Where the audited version publishes

The drill is doable by hand and the first few are genuinely worth doing that way — the surprises are instructive. What does not survive hand-execution is the re-running: a gate proven able to fail last quarter can be quietly neutered by this quarter’s refactor, and nobody re-drills forty gates manually.

That is the gap the Gate Falsification Kit is being built to close: self-reverting drills that plant a real violation per gate and observe the outcome, the three-status receipt ledger — BEHAVIORAL, STRUCTURAL, DECLARED-GAP — and the harness itself, so falsification becomes a repeatable discipline rather than a one-off audit. Fail-Visible Ops is the companion doctrine for the layer underneath — an append-only fault registry where a code cannot claim to be active without a proven emission site, and a journaling bus whose zero means “I was unreachable” rather than “nothing happened”. Both are pre-launch. Aggregate findings across engagements are NOT YET PUBLISHED, and no rate or count appears anywhere on this site.

If you want the drills rather than the afternoon: Integrity Services is where the audit engagements live, and each one states the statuses it will and will not issue before it states anything else. Pre-launch; nothing is for sale.