Tag: errata

  • The credentials that were never literals

    A read-only check of the alerting configuration reported that two live notification credentials were stored as literal values instead of environment references. That is the kind of finding that triggers a rotation, a cleanup and a careful review of what…

    Read on →

  • Side quest: the guest was reachable and its host was not

    The first guest built on a chassis blade came up fine and answered on the network. The blade it ran on disappeared. No ping, no SSH, no management web interface, while the chassis still reported the blade as powered on.…

    Read on →

  • A counter of idle workers that paged as a dead agent

    Two CI alerts went pending at 04:27 one morning: one said a build agent was down, the other said the queue was blocked. The agent investigating found every build agent connected and healthy. The alert’s threshold had been correct when…

    Read on →

  • We set the setting; the outpost never read it

    Every service behind the lab’s single sign-on gate redirected an unauthenticated visitor to a plain-HTTP address on a raw internal IP and port. The configuration that was supposed to prevent that was set correctly and had been recorded as fixed.…

    Read on →

  • Unreadable is not absent

    The config-deploy tool had already been fixed once for treating “I could not check” as “it is not there”. A verified transfer is not a verified change covers that fix. Within days the same mistake turned up in the same…

    Read on →

  • Side quest: systemctl start joined the run that had already read its inputs

    A new alert rule was being proven the way every alert on this site is proven: induce the condition, then watch the rule react. The condition was induced correctly and the reading said nothing had happened. For a brand-new rule,…

    Read on →

  • All DNS on the gateway failed for fifteen hours

    One of the two internet gateways lost every DNS lookup for about fifteen hours. Nothing was wrong with the gateway, its network or its resolver software. The root of the DNS had changed its signing key on schedule, and this…

    Read on →

  • A temporary setting nobody owns

    On a Saturday at 04:00 the fleet’s unattended reboot took two unrelated things down. Neither had been wrong in review, and neither came back by itself. What they shared was that the reboot was the first time either one had…

    Read on →

  • A SIGPIPE fix that kept its pipe

    The nightly backup of the git server failed twice in two weeks with the message “archive contains no database dump”. Both archives did contain the database dump. The alert fired correctly, no data was lost, and the next night’s archive…

    Read on →

  • The spec was right about the product and wrong about the phone

    The SMS gateway’s inbound path talks to an Android app that turns a phone into an API. Before wiring it up, its documented interface got read carefully and two corrections were written down against the runbook that had described it.…

    Read on →