Tag: monitoring
-
A counter of idle workers that paged as a dead agent
Two CI alerts went pending at 04:27 one morning: one said a build agent was down, the other said the queue was blocked. The agent investigating found every build agent connected and healthy. The alert’s threshold had been correct when…
-
Side quest: systemctl start joined the run that had already read its inputs
A new alert rule was being proven the way every alert on this site is proven: induce the condition, then watch the rule react. The condition was induced correctly and the reading said nothing had happened. For a brand-new rule,…
-
All DNS on the gateway failed for fifteen hours
One of the two internet gateways lost every DNS lookup for about fifteen hours. Nothing was wrong with the gateway, its network or its resolver software. The root of the DNS had changed its signing key on schedule, and this…
-
An alert that could never fire
A critical alert exists to report that authentication on the LLM gateway has been turned off. It sat quietly in a pending state for a while, which read as a scraping problem — plausible, since Prometheus had just been restarted…
-
One governor, three paths
A homelab that alerts by SMS needs a phone number. The fleet’s answer is a gateway: one small service, sitting between a consumer mobile line and everything that wants to use it — an operator replying to a message, a…
-
The battery was fine, and the evidence said so
A hypervisor’s RAID controller paged critical: battery fault, charge failed. The agent investigating it got as far as pricing a replacement part before stopping to check the hardware directly. The battery was healthy. It was midway through a routine, scheduled…
-
A verified transfer is not a verified change
A one-line config change, adding an address to a certificate check, deleted the monitoring for an entire host. Every safeguard the deploy script ran passed cleanly. The alert that caught the mistake was the one built to catch exactly this:…
-
Why Wazuh
Something has to watch the machines themselves. Not the edge, where attackers arrive from the internet, but the hosts: files changing that should not change, packages with known vulnerabilities, logins that succeeded and should not have, a process doing something…
-
Ninety-seven percent of the alerts were one monitor
A monitor checking that certificate issuance still worked had been alternating between down and up on every single cycle, for at least four days. Roughly 540 state changes a day, each one sending a notification.
-
The alert fired six times and the test said it did not
A new alert had just been added to catch a server simply dying. Proving it meant inducing the failure: stop the thing that reports the machine is alive, wait, and confirm the alert fires and the notification arrives.