Two CI alerts went pending at 04:27 one morning: one said a build agent was down, the other said the queue was blocked. The agent investigating found every build agent connected and healthy. The alert’s threshold had been correct when it was written and was wrong now, because the metric under it did not measure what its name said.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
What the alert counted
The rule read woodpecker_worker_count < 2 for five minutes. When it was written, there were two agents, so “fewer than two” meant one had gone. The name suggests a count of registered agents. It is actually the number of workers currently idle, waiting for work.
By the time of the alert there were three connected agents. At 04:27 one of them picked up a job while another was already busy, which left one idle. The count fell to one and the rule went pending. Nothing was down. The metric moved down and up with the number of running steps over the preceding fourteen hours, which is the shape of a load gauge and not a health gauge.
The queue-blocked rule had a similar problem in a milder form: a pipeline that waited longer than its threshold looked like a blockage when it was ordinary queueing behind a long build.
Reading before changing
The investigation was read-only. The agent compared the exporter’s file, the server’s metrics endpoint, the server container’s logs, the agents table in the server’s database through a read-only connection with the token column excluded, and a fourteen-hour query of the stored series. The agents table was the deciding evidence: every agent’s last-contact time was seconds old.
What replaced the rule
The metrics script now reads the agents’ last-contact times from the database, read-only, and emits one series per agent, plus the age of the oldest pending and running job per repository, plus a flag for whether the database read itself succeeded. The agent-down rule is now per agent: no contact in five minutes, for five minutes. A busy agent still refreshes its contact time every few seconds, so it cannot trip it. Queue-blocked is based on how long the oldest job has been pending, with thresholds set from the measured waiting times over the previous month. A new rule fires when the database read fails, so the alerts cannot go quiet by losing their own data.
Proving the replacement
All of the rules were exercised with a temporary override and no real agent was touched. Each went pending, then firing, then normal after the override was removed. The real state, checked at the same time, was all normal.
What generalizes
A threshold is a claim about a metric’s meaning, and the claim goes stale without the number changing. The metric was a true number all along. What changed was the fleet around it: a rule written for two agents inherited a meaning for three that nobody chose. When an alert is built on a counter, the check that matters is what the counter counts, written down next to the rule, and not just its current value.