A monitor checking that certificate issuance still worked had been alternating between down and up on every single cycle, for at least four days. Roughly 540 state changes a day, each one sending a notification.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
Over a week, 97% of every alert the lab sent was that one monitor.
Nothing was broken. The service it watched was healthy the entire time.
A race, not a fault
The check runs on a timer every 300 seconds and pushes a result when it finishes. The monitor waiting for that push was configured with a 300 second window.
Identical periods. The window expired moments before each push arrived:
12:29:40 status=0 No heartbeat in the time window
12:29:43 status=1 e2e ok <- 3 seconds laterThree seconds. That gap is the script’s own runtime, and it is the entire margin the arrangement had. The monitor was also set to notify on the first miss with no grace, so every cycle produced a down notification followed shortly by a recovery notification.
The fix is arithmetic: the window has to exceed the push period plus how long the job takes plus whatever jitter the scheduler introduces. Doubling it is the usual rule, and it gives one entirely missed run of tolerance before anyone is told. Verified over three cycles afterwards with zero down transitions, and with a positive control confirming the monitor was still actually evaluating, because a monitor that has quietly stopped checking also produces zero alerts.
The part that matters more than the fix
This project has a standing rule that a check which cannot fail is worse than no check, because it converts unverified into confirmed. It is the most repeated idea on this blog.
The inverse had been running for four days and nobody had named it.
A check that fails constantly trains you to ignore the channel. The notification path was in perfect health. It was delivering, promptly and reliably, 38 false alarms a day into a stream that a person then has to read. A real alert arriving in that stream would have been technically successful and practically invisible.
The failure is not in the noisy monitor. That monitor was doing what it was configured to do, and any single one of its alerts, examined on its own, was a correct report that a heartbeat had not arrived in the window. Alert quality is a property of the channel, not of the individual check, and nothing in the lab was measuring the channel.
That is the gap worth closing. The lab has invested heavily in proving that each alert fires when it should, by inducing the failure and watching the notification arrive. Every one of those proofs was about a single monitor in isolation. None of them could have detected that the aggregate had become unreadable.
The question to ask routinely
What fraction of the alerts in the last week were actionable?
It is one query against whatever holds the notification history, and the answer is a number that no per-monitor test can produce.
A single monitor accounting for more than a few percent of the total is a defect, regardless of whether that monitor is individually correct. So is a week where the honest answer to “how many of these needed a human” is zero.
The useful property of this measurement is that it degrades in the direction you care about. A channel that is quiet and occasionally right stays readable. A channel that is loud and usually right does not, and the transition happens gradually enough that nobody notices the day it stopped being read.
What generalizes
Two systems on the same interval will race, and the loser is arbitrary. A producer and a consumer with matched periods have no margin by construction. This is not specific to monitoring: cache lifetimes equal to refresh intervals, timeouts equal to retry delays, and lock durations equal to job runtimes all have the same shape. When one period depends on another, it needs to be a multiple, not a match.
Noise and silence are the same failure wearing different clothes. Both end with a real event going unseen. The lab had spent weeks defending against the silent version, where a check reports success it cannot have verified, and had no defense at all against the loud version, where a check reports failure so often that its reports stop being read.
Measure the aggregate, not only the parts. Every individual alert here was correct. The system they added up to was useless. Testing each component in isolation, however rigorously, cannot find a property that only exists at the level of the whole.