Fail-Visible Ops

The most expensive infrastructure failure is the quiet one. A process dies and the monitor shows nothing; a journal stops writing and the absence of errors reads as health. Trading systems fail like everything else fails — the question is whether the failure is visible at the moment it happens or reconstructed weeks later from an incident that should not have been possible. Fail-Visible Ops is the fault doctrine we run on our own infrastructure, packaged: a set of structural rules that make silent failure a state the system cannot occupy.

The doctrine is live internally — the journals it mandates are being written on our own systems today.

What it does

• An append-only fault registry with proven emission sites. Every fault code the system can raise lives in a registry that only grows — codes are never deleted or renumbered, so historical journals stay readable forever. And a code cannot claim Active status without a proven emission site: an actual location in the codebase demonstrated to raise it. No aspirational error handling, no catalog of faults the system theoretically reports but structurally cannot. If the registry says a fault is Active, somewhere real code can emit it.
• A crash-honest journaling bus. The journal is the system’s memory, and the doctrine’s founding demand is that the journal account for its own failures. The bus records what it wrote and what it lost, and its semantics are explicit: records_written: 0 means the bus was unreachable — never a quiet system. Zero is a report of failure, not a report of calm. A journaling layer that cannot distinguish “nothing happened” from “I was down” will eventually launder an outage as an uneventful day.
• Loss accounting as a first-class output. When the bus drops records — crash, backpressure, unreachability — the loss itself is journaled and attributed. You learn not just that the record of events is incomplete but where the holes are, which is the difference between a dataset with documented gaps and a dataset that lies by omission.
• Published fault registries and journal statistics from operating deployments: NOT YET PUBLISHED. They arrive with the artifacts that prove them.

What it does not do

It does not prevent crashes, and it does not promise uptime. Systems fail; that is not negotiable. What the doctrine removes is the second failure that usually rides along with the first — the silence. A crash under Fail-Visible Ops produces an honest, attributed record of what was lost. A crash without it produces a mystery.

Who it is for

Teams running live trading infrastructure who have learned — or would rather not learn — that “no alerts” and “no problems” are different sentences. Operators who want their post-incident review to start from a journal that admits what it missed, instead of a monitoring stack that swears everything was fine.