A new alert had just been added to catch a server simply dying. Proving it meant inducing the failure: stop the thing that reports the machine is alive, wait, and confirm the alert fires and the notification arrives.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
The test watched for five minutes and reported:
*** HostDown did NOT reach Alerting -- the rule is not proven
The alerting system’s own log, for the same five minutes:
rule_uid=host_down msg="Sending alerts to local notifier" count=1
22:17:32 22:18:32 22:19:32 22:20:32 22:21:32 22:22:32
Six times. The rule was correct. The observer was broken.
What the test was actually reading
It queried a database table for the alert’s current state. The query was syntactically valid, ran without error, and returned no rows.
In this version of the software, that table holds no rows for any rule. Alert state lives in an in-memory cache and does not reach the table by the path the test assumed. So the query was a well-formed question about a place the answer never appears, and it would have returned nothing forever.
Nothing errored. There was no exception, no timeout, no permission problem. The check asked, received an empty result, and translated that into the alert did not fire.
Empty is not false
Those are different findings and the test collapsed them into one.
False means the measurement was taken and the condition was absent. Empty means no measurement was obtained. A check that cannot tell them apart will report the second as the first, confidently, indefinitely.
This project has run into the same distinction from a different direction, where a script’s exit code could not distinguish a real count from a failure to measure. The shape recurs because “nothing came back” is the natural result of both a working system with nothing to say and an instrument that is not connected.
The tell was in the first line
The observer returned empty at baseline, before anything had been induced.
That reading was there, it was visible, and it was passed over, because empty at baseline is what a healthy system is supposed to look like. Of course there is no alert firing; nothing is wrong yet.
But a measurement that reads empty before the experiment cannot distinguish unchanged from not measuring. It has no demonstrated ability to produce a non-empty result, so when it produces empty afterwards, that carries no information either.
This is a rule the project already had, applied somewhere new. Every control pair here is required to have a positive half that must succeed, because a negative result only means something if a positive one was possible. That rule had always been applied to checks on the system. This was the first time it needed applying to the instrument doing the checking.
The guard, worth copying into any induced-failure test
Before inducing anything, confirm the observer can see something at all, and refuse to run if it cannot:
N0=$(observe)
[ -n "${N0:-}" ] || { echo "observer returned NOTHING -- refusing to test with a blind check"; exit 9; }
The important word is refusing. The instinct is to warn and carry on, and that is precisely wrong here.
An induced-failure test running with a blind instrument produces a false negative, and a false negative about a monitor is worse than not testing at all. It reports that a working alert does not work. The natural response to that report is for somebody to go and change the alert, which means a correct rule gets modified on the strength of a broken measurement. Aborting produces an obvious problem in the test harness. Continuing produces a plausible problem in production.
For this particular system the reliable signal turned out to be the service’s own log lines about dispatching the alert, plus the notification counters, both of which move when a rule fires. The general point is that the observable had to be chosen by checking where the software actually records the event, rather than where it seemed like it should.
What generalizes
A test can fail in two directions and only one of them is loud. A test that wrongly fails gets investigated. A test that wrongly passes gets trusted. This one wrongly failed, which sounds like the safe direction, and it was not: the subject was a monitor, so a false negative invites someone to break a working one.
Prove the instrument before trusting the experiment. Any test that induces a condition and watches for a response needs to establish, first, that its watching mechanism can produce a positive reading. Otherwise the entire result rests on an assumption that was never checked.
Look hard at a baseline that reads as nothing. It is the most easily explained reading in any experiment, because a healthy system genuinely does look quiet. It is also exactly what a disconnected instrument looks like, and the two are indistinguishable at the moment you most want to move on to the interesting part.