A critical alert exists to report that authentication on the LLM gateway has been turned off. It sat quietly in a pending state for a while, which read as a scraping problem — plausible, since Prometheus had just been restarted a few times that day. A day later the underlying check was answering correctly and the alert was still pending. The rule itself turned out to be broken in a way that meant it could never have fired, under any circumstance, since the day it was written.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
What the rule actually required
The rule needs its condition to hold true for thirty continuous minutes before it’s allowed to report anything. The check behind it runs every fifteen minutes, and Prometheus only treats a sample as current for five minutes past when it was collected. Put those together and a query against that check has real data for five minutes out of every fifteen, and nothing for the other ten.
one character per minute across an hour ('#' = data, '.' = no data):
####..........#####..........#####..........#####..........#
Data exists roughly a third of the time. A condition that needs to hold continuously for thirty minutes can never survive the gaps in between, regardless of whether the thing it’s checking is actually true. If authentication on the gateway had genuinely been disabled at any point, this alert would not have told anyone.
Confirmed without touching the gateway
Proving this didn’t require actually turning authentication off and risking a real exposure to test an alert. The check’s result was forced to its failing value directly inside the query used to evaluate the rule, leaving the real gateway untouched. Measuring the longest stretch the old query could ever see as continuously true came back at five minutes, against a rule that needs thirty. A version of the same query built to tolerate gaps up to twenty minutes — wider than the fifteen-minute scrape interval, with margin for one late collection — saw a full sixty-minute run instead. Checked against every other rule watching every other check in the fleet, this was the only one evaluating something slower than its own alert could tolerate.
The measurement that was wrong the first time
The first attempt to confirm how long the old rule’s condition held true counted timestamps from an API that quietly omits any point with no data, rather than returning it as a gap. Twenty-one scattered points, with the gaps invisible, counted as one continuous twenty-one-minute run. Re-counted by comparing actual timestamps instead of just counting list entries, the real figure was five minutes. A result with its gaps already removed has a shape of its own, and it needs to be checked before it gets counted, not after.
The second problem the fix nearly caused
Fixing the rule meant editing one query inside a larger configuration file. Reserializing the whole file to make that edit produced a diff of over a hundred and forty added lines against fewer than twenty removed — for a change that should have touched one line. Comparing the freshly generated file against what was actually live turned up why: an unrelated rule’s description had been hand-corrected directly on the live system weeks earlier, to work around how the alerting system parses its own template syntax, and that fix had never made it back into the repository. Shipping the repository’s version, confident it was the same file with one small edit, would have silently reverted a fix that was already working in production.
The file that looked like the outdated copy was actually the correct one. The deploy tooling’s own guard against unreviewable, whole-file diffs is what caught the discrepancy at all — the fix that finally shipped touched exactly one line of the query and carried the live system’s existing correction forward untouched, rather than overwriting it.
What generalizes
Reserializing a structured file to make a small change turns a one-line edit into a diff nobody can meaningfully review, and an unreviewable diff is exactly the condition under which a real regression hides in plain sight. Separately: a rule that watches something slower than the rule itself can tolerate isn’t a rule with a blind spot — it’s a rule that can never fire at all, and nothing about its syntax says so. The only way to know is to check the actual timing against every check it depends on, the same way this one turned out to be the single exception across the entire fleet.