Validated from the wrong side of the mount

Two new hosts were added to a monitoring config. The deploy tool said the config was valid. It reloaded the service cleanly. And the two new hosts never showed up as monitored — not once, not after repeating the exact same deploy a second time.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

Every check involved told the truth about the wrong file.

The setup

The metrics collector’s configuration file is bind-mounted into its container as a single file, not a directory — a common enough pattern for a one-file config. Adding two scrape targets meant writing the new config to the host path the mount points at, then signaling the container to reload.

What actually happened

A file bind mount attaches to an inode, not to a path. The tooling used to write the new config — copy a new file into place, then rename it over the old one — creates a new inode and moves the name onto it. The host’s path now points at the new file. The already-running container’s mount does not follow; it stays attached to the old inode, which still exists on disk, unlinked from any name, invisible to anyone looking at the filesystem the normal way.

The reload command succeeded, honestly, against the config the container could actually see — the orphaned one. The two new hosts were never in it.

Why it went unnoticed

Three separate signals all reported success, and each one was individually true:

  • the write to the host path succeeded
  • the config validator reported the file valid
  • the reload signal was accepted

The validator is the one worth pausing on, because it existed specifically to catch exactly this class of mistake, and it still passed. It runs inside the container, checking the config path as the container sees it — which during the failure window was the orphaned file, not the one just written. A safety check that reads the wrong copy of the truth will validate that copy perfectly and prove nothing about the one that matters.

The fix

Two changes, at two different points in the same problem:

Write the new config in place rather than replacing the file. Truncating and rewriting the existing file keeps the original inode, which the running mount is still attached to, so the container picks up the change on its next reload without needing a restart.

Once a mount has already detached, only restarting the container re-resolves it. Writing correctly to the live path after the fact does not help — the container is still attached to the orphan, and no amount of correct writing on the host side reaches it. The fix there is a restart, not a better write.

Verifying it properly

The tempting verification is exactly the one that failed here: check the config from the host, check that the tool accepted it, check that the reload didn’t error. All three can be true while the container runs on a completely different file. The check that actually proves something reads the config from inside the running container and compares it against the host copy directly, by identity rather than by content:

host inode      = 262274
container inode = 262173      <- different. the mount is detached.

After the fix, the same comparison should show the same inode number on both sides. Comparing inodes rather than file contents matters too — two different inodes can hold identical bytes by coincidence during testing, which would pass a content comparison and still leave the real defect undetected.

What generalizes

Verify a config change by reading it from inside the consumer, never from the side you wrote it on. A write, a validator, and a reload signal are three observations of the host’s own view of the world. None of them can see what the running process actually has open, and a stale mount is invisible from that side by construction — there is no error, no warning, nothing to grep for. The only observation that closes the gap is one taken through the boundary itself: read the value back from inside the thing that’s supposed to be using it.

The narrower, mechanical version of the same rule: prefer mounting a directory over mounting a single file, wherever the choice exists. A directory mount is resolved by path on every lookup, so replacing a file inside it behaves the way most people already assume file replacement works. A single-file bind mount carries this exact hazard by default, for every tool that ever replaces the file with a new one instead of writing into the existing one.