A temporary setting nobody owns

On a Saturday at 04:00 the fleet’s unattended reboot took two unrelated things down. Neither had been wrong in review, and neither came back by itself. What they shared was that the reboot was the first time either one had been through the event it needed to survive.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

The build agent

The CI build agent’s container carried a restart policy of “no”. That setting had been put there deliberately and temporarily, while the CI network segment was blocked. The block was lifted the same day, the condition that justified the setting was met, and nobody changed it back. Nine days later the host rebooted, the container went down with it, and the policy meant it stayed down.

The unit that hung every vault read

The second failure was a systemd unit on the control node that pins a set of host entries. It had been ordered after the cloud-init finishing stage, which created an ordering cycle: the unit waited on something that waited on it. systemd resolves a cycle by dropping one of the jobs, so at boot the unit’s job was silently discarded and every read from the vault then hung.

The unit was written after that host’s last boot. The Saturday reboot was its first. It had been enabled, started and checked by hand, and every one of those checks passed on a running system where the ordering had never been tested.

What both had in common

Both passed review, because review reads the configuration, and the configuration is not where the problem was. The problem only exists when the event arrives. A restart policy is only tested by a restart. An ordering constraint is only tested by a boot.

The rules that came out of it

A setting marked temporary names the condition that ends it, and whoever meets that condition changes the setting back in the same change. A temporary setting with no owner is permanent.

Anything that has to survive a reboot, whether a unit, a container or a pin, is not done until it has been through one, deliberately, with the effect checked afterwards. “The unit is enabled” is not the check; the service answering is.

Containers behave differently from hosts here. A container’s automatic-reboot setting never fires, because containers do not get the reboot-required marker. For a container, only a deliberate restart counts as a test.

The sweep

The fix was a fleet-wide sweep that asked one question: what here has never been through a reboot? It found four more hosts with services in that state. None had failed yet. Each was a Saturday waiting for its date.