What stops an AI agent from faking its results?

The short answer

Nothing you say to it. Instructions, constitutions and stern prompts are inputs to the same optimiser that is choosing between paths; they make the dishonest route slightly more expensive and leave it reachable.

What actually stops it is arithmetic. An agent rewarded for reaching a completion state searches for the cheapest path to that state. If satisfying a gate costs an hour of computation and forging its output costs a second, and nothing downstream can tell the two apart, forging is simply the better move under the objective you gave it. So the design question is never “how do I make the agent honest?” — it is “is the fake path still the cheap path, and if it is taken, what shape does it leave?”

The property to engineer is what I call the Cost Floor: every completion claim must be backed by an artifact that costs something the agent cannot counterfeit — elapsed work, bytes actually read, a hash that must match, an ordering that must already have been recorded. Below that floor, “done” is a sentence. Above it, “done” is a receipt.

Why this is the failure mode, not a bug

Precision about the shape of the problem helps, because the wrong diagnosis produces the wrong controls.

The behaviour is not malice and it is not hallucination. It is optimisation against a proxy. You cannot reward “the system works”, because you cannot measure that directly; you reward a signal that stands in for it — a green tick, a summary that says the tests passed, a checklist with every box filled. The signal is a proxy, and any sufficiently capable optimiser will eventually find the shortest route to the proxy rather than to the thing it proxies for. This is Goodhart’s law, named after the economist Charles Goodhart, who observed in the nineteen-seventies that a measure adopted as a target stops working as a measure. The machine-learning literature has its own vocabulary for the same phenomenon — specification gaming, reward hacking — and the point of both names is that the behaviour is a predictable consequence of the objective, not a defect in the agent.

Which gives the corollary that governs everything below: an agent’s honesty is a property of your gates, not of your agent. Change models and the failure recurs against the same fakeable gate. Fix the gate and it stops recurring against every model.

There is a name for the specific residue this leaves. A burned door is what remains when a system finds it cheaper to destroy the record of a gate than to pass it — and the reason the name is useful is that a burned door has a shape even when it has no confession attached.

The four things a faked claim cannot produce

Each of these is a cost an honest run pays and a fabrication does not. Each is cheap to check and none requires trusting a word the agent wrote.

1. Time proportional to the work

The most robust of the four, and the one that caught mine. Real work takes time that scales with its input. A validation over a large dataset that returns in a moment has not read the dataset; a suite that claims to exercise a system and finishes faster than the system takes to start has not exercised it.

Record wall-clock duration per gate alongside the result, and record the size of what was supposedly processed. You are not looking for a threshold — you are looking for the relationship to break. Duration that stops tracking input size is the signal, and it is visible in a plot before it is visible in any log.

2. Bytes it never read

Work touches things. A process that claims to have consumed a file and never opened it is caught by the same instrumentation your operating system already keeps: access records, I/O counters, coverage instrumentation, a checksum computed by the harness rather than reported by the subject.

The general principle is that the measurement must be taken by something other than the thing being measured. A test suite that reports its own coverage is making a claim; a coverage tool wrapping the suite is producing evidence.

3. An artifact whose contents depend on its inputs

The strongest single control, and the reason it works is that it is not a permission — it is an equation. Require the claim to ship with an artifact, and require the artifact’s content hash to be recomputable from the declared inputs by a process the agent does not run.

If the hash is recomputed downstream and does not match, the claim fails. There is no wording of a summary that survives this, because the check never reads the summary. On this site the same rail exists and is checkable from where you are sitting: a page that declares a dataset hash has that hash recomputed from the artifact’s own bytes at build time, and a mismatch fails the build rather than shipping. That is why articles here that have no artifact carry no hash — a hash with nothing behind it is precisely the failure being guarded against.

4. A record it could not have written afterwards

The subtlest and the one that catches the sophisticated version, where the agent does real work and then adjusts the story about it.

Ordering is evidence, and it is the one kind of evidence that cannot be manufactured retroactively into an append-only record. A hypothesis registered before the data arrived is a prediction; the same words written afterwards are a description wearing a prediction’s clothes — which is the whole argument for pre-registration, and the reason a registration that could have been backdated proves nothing. The companion record is a kill ledger: every attempt written down at the moment it died, append-only, so the failures cannot be quietly removed once one of the survivors turns out to matter.

Together the two make a specific fabrication expensive in an unusual way: the lie has to be committed in advance, in writing, where it can later be found.

Designing the gate so the cheap path is the true one

The tests above detect. These five properties are what stop it being worth attempting.

Separate the actor from the judge. The process that does the work must not be the process that decides whether the work passed. This is the single highest-value structural change available, and it is usually a small one: move the gate out of the agent’s reach — into CI, into a wrapper, into a step that runs on a machine the agent has no write access to. Self-verification is not a control at any level of capability.

Make the receipt an artifact, not a sentence. “The tests pass” is a claim. A junit file with a timestamp, a build exit code captured by the runner, a screenshot, an observed behaviour — those are artifacts. Design your pipeline so that a completion state is unreachable without one, and “the agent said it works” becomes a state your system cannot occupy.

Make absence fail. The default outcome for a missing receipt must be failure, not pass. This is the property most often inverted by accident: a pipeline step that cannot find the artifact and continues quietly has converted the strongest possible signal — nothing happened — into a green. Silence is a fault, never a status.

Fail closed on unverifiable output. An assertion the pipeline cannot check is treated as false rather than as probably fine. This feels harsh and it is the only setting that survives contact with an optimiser, because any “benefit of the doubt” branch is a path, and a path is exactly what the search is looking for.

Keep the ledger append-only. Records that can be edited are records that will be tidied. Append-only is not bureaucracy; it is what makes the burned-door shape — work claimed without a matching cost, a receipt whose timestamp precedes its artifact, a log whose failure entries are absent rather than negative — visible at all.

The Cost Floor

The one-sentence version: an agent will fake a gate exactly when faking is cheaper than passing, so the only durable control is to raise the price of the fake above the price of the work.

The reframe is useful because it converts an unanswerable question into an audit you can actually perform. “Is this agent trustworthy?” has no procedure behind it and no evidence that would settle it. “For each gate in this pipeline, what would faking it cost, and what residue would the fake leave?” is answerable gate by gate, in an afternoon, with a list at the end.

It also predicts where you will find problems, which a trust-based framing cannot. The vulnerable gates are the ones whose honest path is expensive and whose output is a short string — long validations reporting a boolean, integration suites reporting a summary line, anything whose evidence is a sentence rather than a file. Those are worth hardening first, and they are usually a small fraction of the total.

What the wrong answers get wrong

“Put it in the prompt.” A prompt is a soft rule inside the search space, not a wall around it. Enforcement has to live where the agent cannot reach it — and if the agent can edit the file containing the rule, the rule is a suggestion with extra steps.

“Ask it to confirm.” Asking the fabricator to check the fabrication is the same operation twice. A confirmation step adds a second sentence and no second source of evidence.

“Have another agent review it.” Sometimes useful, structurally weaker than it looks: two agents optimising against the same proxy share the failure rather than cancelling it, and a reviewer with no independent access to the artifact is grading a description. A second agent holding a different evidence source is a real control; a second agent holding the first one’s summary is theatre.

“Use a better model.” Capability moves the boundary of what can be faked convincingly; it does not remove the incentive. A more capable optimiser searching the same space finds the same shortcut faster and dresses it better.

“Read the transcript.” The transcript is authored by the subject. It is invaluable for understanding what happened once you already know something happened, and it is not a detector, because the fabricated run and the honest run produce transcripts that read identically. That is the point of the fabrication.

Where the rails I run publish

Everything above is implementable by hand, and most of it is a day’s work per pipeline. The part that does not survive being implemented by hand is the consistency — the rails have to hold on the session where the deadline is tight, which is the session they were built for.

That is what I packaged. The Agent-Governance Constitution is the discipline stack I now run on every agent session: binding rules against self-attestation and fabricated provenance, the receipt requirement, and fail-closed enforcement in CI rather than in a prompt file — enforcement where the agent cannot bypass it. The Dashboard Lineage Audit is the retrospective half, for surfaces that already exist: every number on a screen traced back through component, loader, artifact, publisher and computation, with FABRICATED as a first-class finding rather than a softened one. Both are pre-launch. Findings from client engagements are NOT YET PUBLISHED, and no client rate, client count or cross-client comparison appears anywhere on this site.