-
Side quest: systemctl start joined the run that had already read its inputs
A new alert rule was being proven the way every alert on this site is proven: induce the condition, then watch the rule react. The condition was induced correctly and the reading said nothing had happened. For a brand-new rule, “nothing happened” reads as “the…
-
All DNS on the gateway failed for fifteen hours
One of the two internet gateways lost every DNS lookup for about fifteen hours. Nothing was wrong with the gateway, its network or its resolver software. The root of the DNS had changed its signing key on schedule, and this host was the only one…
-
A temporary setting nobody owns
On a Saturday at 04:00 the fleet’s unattended reboot took two unrelated things down. Neither had been wrong in review, and neither came back by itself. What they shared was that the reboot was the first time either one had been through the event it…
-
A SIGPIPE fix that kept its pipe
The nightly backup of the git server failed twice in two weeks with the message “archive contains no database dump”. Both archives did contain the database dump. The alert fired correctly, no data was lost, and the next night’s archive was fine. The cause was…
-
The spec was right about the product and wrong about the phone
The SMS gateway’s inbound path talks to an Android app that turns a phone into an API. Before wiring it up, its documented interface got read carefully and two corrections were written down against the runbook that had described it. Both corrections were wrong, and…
-
Side quest: a mechanism that had never once worked
A Windows guest on the chassis gets a freshly minted, single-use Administrator password rather than one a caller supplied and the system never saw again. That password reaches the person who actually needs it as a one-time link, mailed to the lease holder, burned the…
-
One file in, a project out
Standing up a new service in this lab used to mean an agent rediscovering, one round trip at a time, rules that were already written down somewhere: which CI labels matter, how a deploy is supposed to pull rather than push, what a secret is…
-
The rollback that broke what it was rolling back to
A release with no Python changes in it failed to deploy nine times in a row, and every failed deploy restarted a service that didn’t need restarting. The release itself was fine. The check meant to confirm it was fine had quietly stopped being able…
-
Side quest: a key that changes, and a fix that refuses to guess
The first guest build queued right after a blade’s redeploy failed with “REMOTE HOST IDENTIFICATION HAS CHANGED.” The blade had just been reimaged, which regenerates its SSH host key by design, and the builder’s own trusted-key list still held the key from before. Retrying would…
-
An alert that could never fire
A critical alert exists to report that authentication on the LLM gateway has been turned off. It sat quietly in a pending state for a while, which read as a scraping problem — plausible, since Prometheus had just been restarted a few times that day…