Category: Lessons Learned

  • A notification that asked to be obeyed

    An agent watching one of the lab’s hosts received what looked like its own monitoring tool reporting back, formatted exactly the way that tool always formats its output. Appended to it was an instruction: reset the lab’s ownership-tracking state, immediately,…

    Read on →

  • Side quest: a success that changed nothing

    Every hardware checkout in the guest-builder lab has a hard ceiling — a maximum hold time, set the moment the lease is created, so nobody can hang onto a chassis indefinitely by extending it in a loop. Asking to extend…

    Read on →

  • curl’s braces meant something else

    A routine check reported zero targets on the fleet’s only metrics server. Read literally, that meant every host-level alert in the environment was evaluating nothing. The real number was 22, all healthy. The check wasn’t wrong about the fleet —…

    Read on →

  • The service that was never on the list

    A remote desktop session died the instant it started, three times in sixteen days, on the same workstation. The service was xrdp. The cause was needrestart, the tool that automatically restarts daemons after a library update, and a list it…

    Read on →

  • A wrong answer scored better than no answer

    A benchmark comparing several language models on the same set of coding tasks finished with percentages that shouldn’t have been comparable at all, though nothing about the output made that obvious. One model that answered every task, just often incorrectly,…

    Read on →

  • The alert had already fired

    An alerting rule fired, was delivered, and showed up as a real notification on a real phone. Two different places to check its state both said it hadn’t. This happened twice, weeks apart, against the same monitoring stack, and both…

    Read on →

  • The registration that trusted every address

    Registering a new account with an internal DNS-challenge service, needed to issue a certificate for a newly published tool, was a routine step that had already been done correctly several times before. This time the registration came back with no…

    Read on →

  • The count that looked like thrashing

    A large model was being loaded into memory for a multi-hour job, on a machine with barely enough RAM to hold it. Three minutes in, one number looked exactly like the early warning sign of a machine that was about…

    Read on →

  • The command that killed its own caller

    A short script killed a process and then backed up its log file. The kill worked. The backup never ran, and nothing reported an error, because the thing that would have reported the error was the thing that got killed.…

    Read on →

  • The checks that passed without looking

    Four checks passed in the same working session, on four unrelated parts of the build. All four were wrong, and all four were wrong in the exact sense the project already has a name for: a check that cannot fail…

    Read on →