A security agent on one of the fleet’s more important hosts read Disconnected in the manager’s console. Everything else about that host looked perfect — its services were serving, its uptime checks were green, it answered every probe put to it. The agent had been dead for a week, across three reboots, and nothing had noticed.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
systemctl start and systemctl enable answer two different questions, and only one of them survives a reboot.
The setup
The agent had been deployed and started normally. It ran fine — for as long as the host stayed up.
What actually happened
ActiveState=inactive SubState=dead UnitFileState=disabledThe service had been started but never enabled. It ran without complaint until a routine, scheduled reboot for a kernel update. The agent shut down cleanly as part of that reboot — a clean shutdown, logged with no error — and because it was never enabled, nothing brought it back. The host rebooted twice more over the following days. The agent stayed down through all of them.
Nothing about the sequence looked like a failure. There was no crash, no restart loop, no error anywhere in the logs. The last thing the service ever wrote was a graceful exit, which is the most reassuring line a log can contain.
Why it went unnoticed
is-active and is-enabled check different things, and checking the one that answers “is it running right now” gives the same answer either way. A service that is running but disabled looks completely indistinguishable from one that is running and will keep doing so, for as long as nobody reboots the machine. The gap between those two states doesn’t announce itself until the next boot, which can be weeks away and will look, at that point, completely unrelated to whatever set it up.
The host itself had no way to notice its own agent was gone — from inside that machine, there’s simply no process where one used to be, and nothing asks why a process stopped existing. The only place the truth was visible was the manager’s own view, on a completely different machine, and nothing watched that view for changes. Coverage had been recorded as a static fact — this host has an agent — rather than a live one — this host’s agent is currently reporting in. An agent that goes silent reads exactly like one that was never a problem, on every dashboard the fleet has, which makes it a worse failure than the agent never having been installed at all: an absent agent is visibly absent, and a dead one is recorded as present.
The fix
Re-applied the deployment with the unit explicitly enabled rather than just started, and confirmed the manager’s own list reflects it as active rather than merely restarted-and-hoping.
The more durable half of the fix is on the watching side: the manager’s own count of connected versus disconnected agents is now exported as a metric and alerted on. Fixing the one host closes the one instance. Alerting on the manager’s own view closes the class — the next time any agent anywhere gets started without being enabled, the gap between “running now” and “running after the next reboot” gets caught by something other than luck.
Verifying it properly
The tempting check is systemctl is-active <unit> right after doing the deployment work, which reports true immediately whether or not the unit is enabled — it cannot distinguish the two states because it isn’t asking about either of them, only about the current instant. The check that actually answers the real question is:
systemctl is-enabled <unit> # will it still be running after the NEXT rebootBoth commands report a true/false answer instantly, which is what makes this easy to get backwards — nothing about running the wrong check feels incomplete. Confirming the fix means checking both: is-active for right now, is-enabled for whether that stays true past the next reboot, and separately confirming the manager’s own console shows the host as connected rather than only checking the host’s local process table.
What generalizes
“Running now” is not “running tomorrow,” and the check that confirms one says nothing about the other. Any acceptance test that asserts a service is correctly deployed needs to check enablement, not just current state — the two diverge exactly once, silently, at the next reboot, which is usually the worst possible time to discover it.
When a fact about a system’s state is knowable from only one vantage point, that vantage point needs its own monitor. The host couldn’t observe its own agent’s absence. Only the far end — the thing the agent reports to — ever had the information, and treating “the agent is enrolled” as a permanent fact rather than a live one meant nobody was watching the one place the truth actually lived.