An alerting rule fired, was delivered, and showed up as a real notification on a real phone. Two different places to check its state both said it hadn’t. This happened twice, weeks apart, against the same monitoring stack, and both times the wrong answer looked completely reasonable at the moment it was read.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by human, chat 2026-09-17. How this site is written →
The first three places that looked like state, and weren’t
Proving a set of new alert rules meant confirming they actually transitioned when the condition they watched was induced. Three different signals were checked, in order, and each one produced a confident answer before anything contradicted it.
The dashboard’s own state table, the obvious place to look, held no row for any of the rules being tested — not just the new ones, all of them. Watching it for ten minutes and seeing nothing appear read as “never fired.” What it actually meant was that this version of the table doesn’t populate for any rule at all, regardless of state, so ten minutes of silence measured nothing.
A log line that appeared roughly on schedule, matching the timing expected if the rule had fired, looked like direct evidence. It wasn’t a state signal either — that line gets emitted while a rule is healthy too, at the same rate, for entirely unrelated rules. A quick check against a baseline, before anything was induced, would have shown the line appearing regardless.
A metric counting how many alerts were actively firing across the whole system moved by exactly one at the right moment, which felt like solid confirmation. After the condition was resolved, the count didn’t drop back down — because other, unrelated rules had changed state at the same time, and the metric only ever reports a fleet-wide total, never which rule contributed what. Reading a shared counter as if it described one specific rule produced a second wrong conclusion in the same sitting.
The one channel that actually worked queries the alerting system’s own rules directly, by name, and reports plain state: inactive, pending, or firing. Every other channel had been a plausible-looking proxy for that answer. Once the real one was in hand, all three earlier readings were retested against a baseline and immediately shown to be unreliable under both the healthy and the failing condition — the actual requirement for trusting an instrument at all.
A separate bug hiding in the same investigation
While proving the same rules, one of them fired correctly but never cleared, staying stuck and eventually getting suppressed rather than resolved — which is worse than not alerting at all, because it looks handled. The cause was a comparison written directly into the underlying query rather than into the alerting condition around it: when a comparison inside the query evaluates false, that query returns no data at all rather than a value of zero, so there was nothing left for the rule to observe transitioning back to normal. Moving the threshold out of the query and into the condition fixed it outright, confirmed by watching the full sequence: inactive, pending, firing, and cleanly back to inactive within a minute of the underlying problem being resolved.
The second occurrence, weeks later
Proving a different, unrelated alert rule turned up the same shape of problem again, in two new places this time. A different internal record of alert state read as empty for ten straight minutes while the alert had, by then, already been delivered to a phone — confirmed independently by the delivery system’s own record. That internal record turned out to be stored in a format where state is an encoded number, and the specific words being searched for don’t exist anywhere in it for any rule, in any state. No amount of watching that record could ever have produced the right answer, for anything.
A second internal log of past notifications read as empty too, even though three unrelated notifications logged around the same time were genuinely readable in the same store, which made it look like a fair test. What made this one harder to catch is that those three entries were old. The store gets written to on its own periodic schedule, not at the moment a notification goes out, so a positive control proving the log can be read at all says nothing about whether it’s been updated recently. The control and the thing being tested differed in exactly the property that mattered.
The answer both times came from checking the system’s own log of having actually sent something, and from checking the delivery record on the receiving side directly — the effect itself, rather than a store the system happens to write to on its own schedule.
What generalizes
Before trusting any channel as a stand-in for a system’s real state, confirm it can show you the other answer. Every misleading signal here was consistent with the expected result at the exact moment it was checked. What separated the trustworthy channel from the rest was a baseline: does this move when nothing is actually happening? None of the unreliable ones survived being asked that question.
A positive control only proves an instrument can read — it doesn’t prove it’s reading something current. A store that updates on its own schedule can pass a control built from old data and still say nothing true about an event that just happened. For anything time-sensitive, the control needs to be at least as fresh as the event it’s meant to verify, or the check needs to move to something that reflects the system’s own action directly rather than a copy of it written down later.