Exit 3 means either

A new test suite reported a problem on two healthy gateways:

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

FAIL  HIL live drift count -- got '3', expected '0'
FAIL  LIM live drift count -- got '3', expected '0'

The lab’s monitoring notes said drift was zero on both machines. One of the two had to be wrong, and finding out which produced a defect worth more than the answer.

What drift detection is for

A script compares the configuration files on each gateway against the copies held in a repository, and reports how many differ. Zero means the machine matches what it is supposed to be. A number above zero means somebody changed something on the host without changing the source.

Its documented interface is direct: the exit code is the number of files that differ, and zero means no drift.

The first layer, which was the agent’s mistake

The suite ran the script through sudo. The service that normally runs it runs as an ordinary account, and that account holds the key used to reach the configuration repository. Root does not have that key, so the fetch failed.

Run the way the service actually runs it, both gateways compared twelve files, found every one identical, and exited zero. The gateways were fine. The test was wrong.

That much is an ordinary mistake of the kind this blog has covered before: a check run in a different context from the thing it is checking is not checking that thing.

The second layer, which is the actual defect

When that fetch failed, the script printed FETCH-FAILED and exited 3.

Its interface says the exit code is the number of drifted files.

So exit 3 means either “three files have drifted” or “I could not reach the repository at all”, and there is nothing in the exit code to tell them apart. Those two conditions call for opposite responses. One means somebody changed a configuration file on a live gateway and you need to find out who and why. The other means your drift detection is blind and has been reporting a number it did not measure.

The second is worse, and it is the one that looks like ordinary output. A monitoring system consuming that exit code would have alerted on drift, and somebody would have gone looking for three changed files that did not exist, while the actual condition, that nothing was being compared at all, was not represented anywhere in the interface.

Why this is easy to write

The design is appealing when you write it. The count is the useful number, so returning it directly means the caller can use the exit status as the answer with no parsing. It reads as elegant.

The problem is that a process’s exit status is not a number, it is a status, and it already has a reserved meaning: zero is success and non-zero is failure. Overloading it to carry a measurement means the channel that says whether the run worked has been spent on saying what the run found. There is then nowhere left to report that the run did not work.

This is not specific to shell scripts. Any function whose return value carries a measurement in the same channel as its error signal has the same hole in it: a count of zero and an error both being falsy, a -1 that means either “not found” or “the lookup failed”, an HTTP status reused to carry a verdict. The shape recurs because merging the two channels always looks like simplification at the moment you do it.

The fix, in both places

In the test: assert the summary line, not just the exit code, and return no verdict at all when the script reports a fetch failure. A drift number is only meaningful from a run that actually compared something, so a run that compared nothing should produce neither a pass nor a fail. It should produce “unknown”, which is a real result and a different one.

In the script: the failure to reach the repository needs a code outside the range that means a count, so that a caller can distinguish “I measured, and the answer is N” from “I did not measure”. The two statements are not points on the same scale.

The general form: if a channel carries a measurement, errors need a different channel. Any scheme that lets a valid measurement and a failure produce the same value on the same channel will eventually produce that collision on a day when it matters.

What generalizes

An interface that cannot express failure will report something else instead. It will not stay silent. It will emit whatever value the failure path happens to produce, and that value will be indistinguishable from a real result. The question worth asking of any check is not only what it reports, but what it reports when it cannot do its job.

“Unknown” is a result, and most checks cannot say it. Pass and fail are the two states almost every verification tool models, and the third state, I was unable to determine this, is the one that matters most, because it is the state a check enters when something has gone wrong with the check itself. A system where unknown is silently rendered as one of the other two will keep reporting confidently through its own failure.

Two contradicting sources of truth are a gift. The monitoring notes said zero drift and the new test said three. That disagreement was the entire reason any of this was found. A check that agrees with your existing belief teaches nothing, and it is worth resisting the urge to make a new check agree with the old one before understanding why it does not.