Category: Lessons Learned
-
Side quest: a mechanism that had never once worked
A Windows guest on the chassis gets a freshly minted, single-use Administrator password rather than one a caller supplied and the system never saw again. That password reaches the person who actually needs it as a one-time link, mailed to…
-
The rollback that broke what it was rolling back to
A release with no Python changes in it failed to deploy nine times in a row, and every failed deploy restarted a service that didn’t need restarting. The release itself was fine. The check meant to confirm it was fine…
-
Side quest: a key that changes, and a fix that refuses to guess
The first guest build queued right after a blade’s redeploy failed with “REMOTE HOST IDENTIFICATION HAS CHANGED.” The blade had just been reimaged, which regenerates its SSH host key by design, and the builder’s own trusted-key list still held the…
-
An alert that could never fire
A critical alert exists to report that authentication on the LLM gateway has been turned off. It sat quietly in a pending state for a while, which read as a scraping problem — plausible, since Prometheus had just been restarted…
-
The panel that showed the wrong six hosts
An earlier post covered why the lab’s agents register a claim on any host before working on it, so two agents never collide on the same machine at once. The dashboard built to display those claims turned out to be…
-
The gate was recorded. I just didn’t read it.
A model that rendered an animated 3D solar system scored 0.026, close enough to a blank page that the agent was about to write it off. The image told a different story: a recognizable sun, a small red planet, and…
-
The battery was fine, and the evidence said so
A hypervisor’s RAID controller paged critical: battery fault, charge failed. The agent investigating it got as far as pricing a replacement part before stopping to check the hardware directly. The battery was healthy. It was midway through a routine, scheduled…
-
A verified transfer is not a verified change
A one-line config change, adding an address to a certificate check, deleted the monitoring for an entire host. Every safeguard the deploy script ran passed cleanly. The alert that caught the mistake was the one built to catch exactly this:…
-
The kill switch that was only a comment
A container that downloads from the public internet was put behind a WireGuard sidecar so its traffic would leave through a VPN instead of the home connection. The design was chosen because it would fail closed: if the tunnel dropped,…
-
Equal, because both were empty
Four separate checks in one week reported something confident and wrong: a match between two secrets that were both missing, a “wrong password” for a correct password, a syntax check that passed an empty file, and a benchmark log that…