Category: Lessons Learned
-
The credentials that were never literals
A read-only check of the alerting configuration reported that two live notification credentials were stored as literal values instead of environment references. That is the kind of finding that triggers a rotation, a cleanup and a careful review of what…
-
Side quest: the guest was reachable and its host was not
The first guest built on a chassis blade came up fine and answered on the network. The blade it ran on disappeared. No ping, no SSH, no management web interface, while the chassis still reported the blade as powered on.…
-
A counter of idle workers that paged as a dead agent
Two CI alerts went pending at 04:27 one morning: one said a build agent was down, the other said the queue was blocked. The agent investigating found every build agent connected and healthy. The alert’s threshold had been correct when…
-
We set the setting; the outpost never read it
Every service behind the lab’s single sign-on gate redirected an unauthenticated visitor to a plain-HTTP address on a raw internal IP and port. The configuration that was supposed to prevent that was set correctly and had been recorded as fixed.…
-
Unreadable is not absent
The config-deploy tool had already been fixed once for treating “I could not check” as “it is not there”. A verified transfer is not a verified change covers that fix. Within days the same mistake turned up in the same…
-
Side quest: systemctl start joined the run that had already read its inputs
A new alert rule was being proven the way every alert on this site is proven: induce the condition, then watch the rule react. The condition was induced correctly and the reading said nothing had happened. For a brand-new rule,…
-
All DNS on the gateway failed for fifteen hours
One of the two internet gateways lost every DNS lookup for about fifteen hours. Nothing was wrong with the gateway, its network or its resolver software. The root of the DNS had changed its signing key on schedule, and this…
-
A temporary setting nobody owns
On a Saturday at 04:00 the fleet’s unattended reboot took two unrelated things down. Neither had been wrong in review, and neither came back by itself. What they shared was that the reboot was the first time either one had…
-
A SIGPIPE fix that kept its pipe
The nightly backup of the git server failed twice in two weeks with the message “archive contains no database dump”. Both archives did contain the database dump. The alert fired correctly, no data was lost, and the next night’s archive…
-
The spec was right about the product and wrong about the phone
The SMS gateway’s inbound path talks to an Android app that turns a phone into an API. Before wiring it up, its documented interface got read carefully and two corrections were written down against the runbook that had described it.…