A release with no Python changes in it failed to deploy nine times in a row, and every failed deploy restarted a service that didn’t need restarting. The release itself was fine. The check meant to confirm it was fine had quietly stopped being able to.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
The check that used to work
The SMS gateway’s deploy pipeline confirms the real carrier connection is loaded, not just that the process is running — systemctl is-active proves a process exists, not that it’s the one that was just shipped, or that it’s actually wired to a real phone line rather than the test double used in development. The original version of that confirmation read a line the process printed once, at startup, into the system journal, and grepped for it.
What changed underneath it
The deployer later learned to skip restarting the service entirely when a release changed no Python — a reasonable optimization, since an unnecessary restart is itself a small risk for a service that carries critical alerts. But the startup line the deploy check was grepping for is printed exactly once, at startup. Skip the restart, and that line is now hours old. The grep found nothing, read that as failure, and rolled the release back.
The rollback restarted the service. That’s what a rollback does. And a restarted service prints the startup line again — which the next deploy attempt would then also fail to find in time, because the check ran before the fresh line had settled into the journal. Nine deploys failed this way in a row, each one triggering the exact restart that made the next attempt fail too, and the release that had done nothing wrong sat blocked for nearly an hour.
Why the fix reads the process instead of its history
The journal line was always a historical record — true the moment it was printed, and silently stale the instant something else changed underneath it without saying so. The fix asks the running service directly, over an endpoint built for exactly this, what it’s actually wired to right now: which providers are live in the send path, read from the same mapping the service consults when it actually sends something, at the moment the question is asked. That answer is true whether or not anything has restarted recently, because it was never derived from a restart in the first place.
The same endpoint also reports a hash of the code the process actually has loaded in memory — deliberately computed once, at startup, and never re-read afterward. That’s the opposite tradeoff from the provider check, on purpose: what code did you load is a question about the past, and an answer that kept following the disk would report code that had since changed underneath a process that hadn’t actually picked it up yet. What are you wired to send through right now is a question about the present, and a frozen answer to that one would keep naming a connection that could have already been swapped out. Both questions get asked in the same response. Neither one is allowed to answer the other.
What the fix does and doesn’t prove
The endpoint reports whether a real carrier adapter has been constructed and is sitting in the send path — not whether the carrier itself is answering, not whether the credential is valid, and not that any message has actually reached a phone. The response says so directly, in its own payload, because a field that reads as “a real connection exists” invites exactly the wrong conclusion if nobody states the difference between wired and working. Checking whether a message actually lands would mean hitting the carrier on every deploy check, which the pipeline calls up to thirty times per release — a live probe at that rate turns a status check into its own outage risk.
What generalizes
A log line is a fact about the moment it was written, and every optimization that changes when things run is also, quietly, a change to how stale that fact is allowed to get. The rollback in this story wasn’t wrong on its own terms — it did exactly what a rollback is supposed to do when a check reports failure. The check was the part answering a question about history as though it were a question about now, and nothing about a passing or failing grep would have revealed which one it actually was.