Status: alerting that can actually reach you

Two days on the public edge and no new hardware. The alerting path is now proven end to end rather than merely deployed, which is a distinction this project keeps having to relearn. It produced four more errata along the way, three of them the agent’s own.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

What went in

The notification service was verified working, not just running. A successful publish returning a message ID proves the sender’s own credentials work. It says nothing about whether someone without credentials can read or write, and only the second is a security property. The checks that mattered were the two that came back 403.

Uptime monitoring now reaches it. That did not work at first; the firewall was eating the container runtime’s DNS, covered below.

The service was sandboxed. Its systemd security score went from 7.3 (medium) to 1.3 (OK): no new privileges, read-only system paths, private temp, empty capability set, restricted namespaces, and a umask that stops it writing world-readable files.

The verification is the interesting part. systemctl show proves only that the directives were parsed. The real check entered the service’s own mount namespace and tried to write in two places, one that should fail and one that must still succeed:

touch /etc/<service>/.probe      -> fails    (system paths read-only)
touch /var/lib/<service>/.probe  -> succeeds (state still writable)

That second test earns its place. A strict sandbox with the wrong writable-paths list produces a service that starts perfectly and then silently fails to persist anything, which is worse than a crash because a crash tells you.

And it went behind a source allowlist, reachable only from the home network, loopback and the container bridge. It returns 404 rather than 403 to everyone else, since 403 confirms something is there.

Human’s call

Sandbox first, then the allowlist.

Once the allowlist is live the service can no longer be reached from the workstation, so any breakage caused by the sandboxing had to be found while it was still observable. Doing it the other way round would have made a sandbox fault indistinguishable from the allowlist working as intended, at the point where testing had become hardest.

The mesh coordination endpoint was deliberately left world-reachable. A brand-new device has no mesh connection yet and must bootstrap over the ordinary internet, so locking it down would look like hardening and quietly break enrollment from anywhere new.

WireGuard also moved off its default port, which turned into a longer argument about threat models and has its own post.

What broke

Four errata. The pattern across them is more useful than any individual fix.

The firewall was silently overriding the container runtime. A drop rule in the forward chain took precedence over the container engine’s own accept rules, so containers could not resolve DNS. Nothing logged an error; the alerting integration simply failed. The repair script’s flush ruleset then destroyed the container engine’s tables outright — rebuilt in about a second, and baffling to diagnose later.

A guard was checking its input instead of its effect. A script maintaining a firewall address set verified the value it had just computed rather than what the firewall actually held afterwards. It would have reported success while the live rule contained something else. It now compares against the running ruleset, and earned its keep on the first run by exiting non-zero on a genuine mismatch nobody had noticed.

Discarding error output turned a permission failure into a confident wrong answer. A command run without sufficient privilege had its stderr sent to /dev/null. The failure produced no output, the script counted zero results, and reported “0 tokens” — a number sitting comfortably inside the range of plausible answers, which is what made it dangerous.

A readiness check read the wrong column. A test for whether a UDP port was free parsed field 5 of ss -lunp. Field 5 is the peer address, always *:* for a listening socket, so the pattern could never match and the check reported “free” for any port it was ever asked about. The port was genuinely free, by luck rather than because anything established it.

Why these keep being the same bug

All four were structurally incapable of returning the answer they claimed to provide. That has its own post and a standing rule: make a check fail on purpose once before trusting it.

The uncomfortable part is that all four were written to confirm a fix. That is consistently where they appear. A careless check written while debugging gets caught immediately, because the problem is still in front of you. A check written afterwards to prove the fix worked has no such feedback — it says “confirmed” and everyone moves on.

The count across this project is now eight.

One of the four did get a proper negative test, by accident. A reload reproduced the precise combination that defeated the old guard — the wrong value present and the state file intact — and the new guard caught it. That fix is now verified rather than assumed.

Something that is not fixed

The uptime monitor runs on the same host as the services it monitors. If that host dies, the monitor dies with it and no alert is sent. The dashboard reads all-green right up to the moment it reads nothing at all.

This is the same defect class as a finding from weeks earlier: a failover script that would have repointed the alerting service’s own DNS at a server with no backend for it, breaking the alert channel exactly when it was needed. Same blind spot, different mechanism. The earlier review caught the script and missed the deployment, because it was reading the script rather than asking where the observer runs.

An observer sharing a failure domain with its subject is not monitoring. The fix is two gateways watching each other, and the second gateway does not exist yet, so the honest position is that there is no external watchdog. That is recorded as a gap rather than counted as coverage, which is the only part of this currently going right.

Still not started

Everything below the edge. No hardware racked, no hypervisor, no storage, no virtual machines. The full inventory has the detail and the shape has not changed: nine components complete, five in progress, forty-eight not started.

Next

Rack the hypervisors and storage. Two days of hardening the edge, however worthwhile, has not moved it.