The step that was not about DNS

The runbook step was about joining a VPN. One command, on each machine, to enroll it in the private mesh.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

Run as written on the two DNS servers, it would have replaced their /etc/resolv.conf with Cloudflare — so the resolvers would have stopped using themselves, and the internal zone would have become unreachable from the machines serving it.

Every other host in the fleet would have followed.

Why a VPN command touches DNS

The mesh coordinator can push nameservers to nodes as they join. This one is configured to, and to override whatever the node already had:

override_local_dns: true
nameservers:
  global:
    - 1.1.1.1
    - 1.0.0.1

For a laptop or a phone that is exactly right. A roaming client joining the mesh from a hotel wants a working public resolver, and it has no opinion of its own worth preserving.

For a machine whose entire job is being the resolver, it is the opposite of right.

The setting that makes this easy to miss on a read-through is a neighboring one. MagicDNS — the feature that gives mesh nodes friendly names — is switched off. It would be reasonable to conclude that DNS is therefore not involved. The override is a separate setting and applies regardless.

So the failure would have been: a step that reads as network membership, quietly undoing the entire purpose of the previous runbook, on every host, starting with the two that would make it hardest to diagnose.

How it was caught

Not by testing DNS afterwards. That is the part worth keeping.

The enrollment recorded sha256sum /etc/resolv.conf on every host before the join, and asserted it matched afterwards.

Had the check instead been “does DNS still work?”, it would have passed. Cheerfully. Every name would have resolved, because Cloudflare resolves names — just not the internal ones, and not from the servers that are supposed to be authoritative for them. The internal zone failing is a second-order effect that shows up minutes or hours later, on some other machine, looking like something else entirely.

When a step could plausibly rewrite a file you depend on, hash the file. Do not ask the service whether it still feels well.

That distinction is the difference between an errata entry and an outage entry.

That is a narrower fix than disabling the feature, and it puts the exception on the machines that are the exception.

Two more from the same enrollment, same shape

The mesh join produced three defects. The other two are both cases of a command producing nothing and nothing noticing.

A pre-auth key that was never created. The command to mint an enrollment key takes a --user flag. Given a username it returns nothing at all — no key, no error worth seeing — because the flag wants a numeric ID.

A script doing KEY=$(command | tail -1) gets an empty string and proceeds to run the enrollment with --authkey= and no value. The failure surfaces three hosts later, or not at all.

What stopped it was asserting the shape of a value nobody had looked at:

[ "${#PAK}" -ge 20 ] || abort

Twenty characters is not a validation of the key. It is a validation that something arrived. That was enough to abort before touching any machine.

A privileged read that reported a blank length. Checking the size of a root-owned secret:

sudo -n wc -c < /etc/rathole/token

The < redirect is performed by the unprivileged shell, before sudo runs at all. On a 0600 root-owned file it fails with a permission error, while the surrounding output still prints length: and carries on. A blank reads as zero, or as fine.

Pass the path as an argument instead — sudo -n wc -c /etc/rathole/token — so the privileged process is the one opening the file.

Why this family keeps recurring in this project

That last one is a direct descendant of a defect logged weeks earlier, where discarding stderr on a privileged command turned “permission denied” into a confident count of zero. Same boundary, same result: the shell does part of the work as you, and the command does the rest as root. Anything the unprivileged half cannot see becomes an empty string, and empty strings are excellent at looking like valid input.

It has now appeared four ways in this project:

• A glob expanded unprivileged against a root-only directory, so a backup check inspected a database it had just created.
• The same glob shape again, in a maintenance script, so a database was modified on the strength of an unconfirmed backup.
• stderr discarded, turning a permission failure into 0 tokens.
• A < redirect, turning a permission failure into a blank length.

The rule that covers all four: a privileged read that fails must produce an error you cannot mistake for a value.

What generalizes

The mesh defect is the interesting one, and it is not really about DNS.

It is that a step’s blast radius is not always in its description. “Join the VPN” sounds bounded — one machine, one membership, one network interface. The thing it actually did was reach into an unrelated subsystem and overwrite a file that the previous runbook existed to create.

You cannot catch that by reading carefully, because reading carefully is what produces the wrong conclusion: MagicDNS is off, therefore DNS is not involved.

What catches it is a habit that costs nothing. Before a step that could plausibly touch something you depend on, record the state of the thing you depend on. Afterwards, assert it is unchanged. Not that it still works — that it is identical.

The difference matters most exactly when the replacement is also functional, which is precisely when you would otherwise never notice.