Category: Lessons Learned
-
Ninety-seven percent of the alerts were one monitor
A monitor checking that certificate issuance still worked had been alternating between down and up on every single cycle, for at least four days. Roughly 540 state changes a day, each one sending a notification.
-
Nine of nine, out of ten
A change was rolled out across the fleet. The record says it covered nine hosts. The verification sweep afterwards reported nine of nine green.
-
Six seconds was the evidence
A certificate was issued in six seconds. That was taken as proof the new certificate provider worked.. It proved the opposite of what it appeared to, and the speed was the tell.
-
Active, and refusing every query
A routine change to add a forwarding rule took DNS down for the entire lab. Every query from every machine, internal names and external ones alike, came back REFUSED from both resolvers.
-
The alert fired six times and the test said it did not
A new alert had just been added to catch a server simply dying. Proving it meant inducing the failure: stop the thing that reports the machine is alive, wait, and confirm the alert fires and the notification arrives.
-
Forty-three checks passed with a key that did not exist
A test suite for the DNS resolvers had a weakness worth closing: if the SSH key it used went missing, every check would fail, and thirty-seven DNS failures would be reported for what was actually one authentication problem.
-
The off-by-one that answered
A planned feature of this lab lets you text a question to it and get an answer back, so that basic status is reachable when the internet is not. One of the questions is how full a machine’s disks are.
-
Exit 3 means either
A new test suite reported a problem on two healthy gateways. The lab’s monitoring notes said drift was zero on both machines. One of the two had to be wrong, and finding out which produced a defect worth more than…