Side quest: systemctl start joined the run that had already read its inputs

A new alert rule was being proven the way every alert on this site is proven: induce the condition, then watch the rule react. The condition was induced correctly and the reading said nothing had happened. For a brand-new rule, “nothing happened” reads as “the rule does not work”, and the next step would have been to start debugging a rule that was fine.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

What the test did

The rule watches for a guest rebuild in progress on the education chassis. A small collector reads a marker file per blade, and a timer runs it every five minutes. To prove the rule, the agent wrote a synthetic in-flight marker, started the collector by hand, and read the metrics file it produced.

The sequence was right and the result was wrong. The unit reported success. The collector’s own health metric said its scrape was fine. The marker was valid and sat on disk where the parser would find it. The metric for that blade still read zero.

What the journal showed

The log held the answer. The five-minute timer had started a run about a minute before the hand-written marker existed. That run read its markers, then spent roughly seventy seconds on slow per-blade reads from the management controllers before writing its output.

systemctl start on a oneshot unit that is already running does not queue a second run. It attaches to the run in flight and returns when that run finishes. So the manual start returned success, and the output file looked fresh, written seconds after the change. It described the world as it had been a minute before the change, because the run that wrote it had read its inputs before the marker existed.

Running the collector again with nothing in flight gave the correct reading immediately.

The wait-for-idle step
while [ "$(systemctl is-active fx2-redeploy-watch.service)" = "activating" ]; do sleep 5; done
sudo systemctl start fx2-redeploy-watch.service

The loop waits until nothing is mid-run, so the next start is guaranteed to be a new run that begins after the change.

What generalizes

A collector’s output timestamp says when it wrote, not when it read. For a collector that does slow work between the two, whether that is controller reads, an SSH fan-out or an API sweep, those are different moments and the gap is as long as the slow part.

So when inducing a condition to prove a monitor, check that the unit is idle before triggering it, because a start on a running oneshot means “wait for the current run” and its success says nothing about whether the change was in scope. A successful run with an unchanged reading is not evidence of no effect until it is known that the run began after the change. A collector that published its read time as well as its write time would have exposed this directly; this one published only the write time, which is why the metric looked trustworthy.