If the watcher dies, its last words are reassuring

The monitoring system was switched off for eighteen minutes. Every monitor reported UP for the entire period.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

id | name                | last_status | last_msg
 1 | DNS resolver 01     | 1           | Records: 10.0.10.24
 2 | DNS resolver 02     | 1           | Records: 10.0.10.24

Status 1 is UP. Both rows were real, correct, and eighteen minutes old.

Not a bug

Adding a monitor to this system requires stopping the container. The docker start that should have followed never ran — the SSH session carrying it timed out mid-script — so the container sat exited.

Nothing malfunctioned. A heartbeat table records what was last seen, and the last thing seen was two healthy resolvers. The table answered the question it was asked, correctly.

The question was just the wrong one. Last known status is not current status, and a dead monitor never writes the row that would say otherwise.

This project has met a related shape before: systemctl is-active returning active for a daemon stuck in a reconnect loop. This is worse, because that answer is merely uninformative. A stale UP is affirmatively reassuring. It does not fail to tell you something is wrong; it tells you something is right.

The fix is to ask when, not just what

Select the age of the heartbeat alongside its status:

cast((julianday('now') - julianday(
  (select h.time from heartbeat h
    where h.monitor_id = m.id order by h.id desc limit 1)
)) * 86400 as int) as age_s

A row means something only while age_s is less than roughly twice the check interval. Older than that and you are reading a historical record as a live one.

That single column converts “everything is UP” into “everything was UP, eighteen minutes ago, and nothing has been checked since”.

The wrong turn, which is the more useful half

The first instinct was to set a container restart policy. It was wrong twice over, and both ways are worth recording because the change would have looked like a fix.

The container already had a stronger policy than the one applied. It was set to always; the “fix” changed it to unless-stopped, which is weaker — always also restarts after a Docker daemon restart. The change made things slightly worse and was reverted the same day.

And it would not have helped anyway. The container was not crashed. It had been stopped deliberately by a maintenance script, and Docker honors a manual stop regardless of restart policy.

That is the part worth taking away. The restart policy was a change that touched the right component, sounded like a mitigation, and would have closed the item on paper while leaving the hole exactly as it was. Those are harder to catch than doing nothing, because doing nothing does not generate a commit that looks like progress.

The uncomfortable generalization

If the monitoring container exits on its own, every monitor stops and the last stored state is UP. Nothing alerts, because the thing that would alert is the thing that died.

This site has already written about an observer sharing a failure domain with its subject. That was one host watching its own services. This is worse in scope: it covers the entire monitoring system, and the failure is silent by construction.

The general form applies well beyond this lab. Any watcher that reports state rather than emitting a heartbeat will, on dying, leave behind its most recent opinion — and that opinion was formed while things were working, because if things had been broken someone would have been looking.

Ask a monitoring system when it last checked, not just what it last saw. If it cannot tell you, it is reporting history.

A second defect from the same script, same shape

The maintenance script’s backup step printed an empty confirmation, because it globbed a path inside a root-only directory from the unprivileged shell. The glob matched nothing, the confirmation was empty, and the database was then modified on the strength of a backup nobody had confirmed.

The backup had in fact succeeded — verified afterwards with an integrity check — but that was luck rather than method.

This is the same privilege-boundary trap that once created an empty database and pronounced it intact: sudo command "$(glob)" expands the glob as you and runs the command as root, so anything the unprivileged shell cannot see becomes an empty string. Empty strings are extremely good at looking like valid input.

Let root do the globbing: sudo sh -c 'ls …'.

Where it stands

Unresolved, and recorded as unresolved. The monitors now report their heartbeat age, so a stale reading is visible to anyone who looks. Nothing yet watches the watcher, and no restart policy is pretending to.