The tunnel from the rented edge to the house is up and carrying traffic. That was the last structural piece of the original design never exercised end to end, and it has now been exercised in both directions.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
There is also, at last, something watching the watcher.
The tunnel
Both gateways run the relay server, the home-side client connects outward to each, and requests arriving at either public address reach services inside the house. Seventeen established connections at the time of checking, the health responder answering, and a full end-to-end fetch succeeding through each tunnel independently.
Nothing at the house accepts an inbound connection. The path exists because the inside reached out, which was the point of the design and is now demonstrated rather than described.
Getting there produced two defects worth noting on their own. The relay was configured for a transport the prebuilt binary does not support — that is a compile-time feature, so the runbook specified something that could never work and the fix was to use the transport it does have, which is not a downgrade. And the hardened service unit made the process crash on startup until one directive was relaxed, found by bisecting them one at a time rather than by removing the hardening wholesale.
Something is finally watching the watcher
A post from yesterday described a monitoring system that reported every monitor UP while switched off for eighteen minutes, and closed on the honest note that nothing yet watched the watcher.
Half of that is now fixed.
An external heartbeat runs on unrelated shared hosting, entirely outside this infrastructure. A cron job on the gateway posts a timestamp every five minutes; a small endpoint elsewhere serves it; the monitoring system checks that endpoint like any other. If the gateway dies, the heartbeat goes stale and the check goes down.
The heartbeat endpoint was tested until it failed, which is the only reason to believe it:
| Condition | Response |
|---|---|
| Fresh timestamp | 200 / ok |
| One hour stale | 503 / stale |
| Timestamp in the future (clock skew or corruption) | 503 |
| State file missing entirely | 503, not a PHP error |
Then the monitor was proven by an authentic outage rather than a synthetic one: the cron job was disabled outright and the heartbeat left to age. The monitor went pending at five minutes, down at eight, the alert arrived, and — the part that matters — a neighboring monitor stayed up throughout, so the two discriminate rather than moving together.
The alert mailbox must not live on the infrastructure being watched.
Obvious once stated, easy to get wrong while building. A watchdog that emails you through the mail server it is watching tells you nothing on the day it matters. The heartbeat host, the endpoint and the alert destination are all deliberately outside the lab.
The other half is not proven. The design is mutual: the external host probes the gateway and alerts by email, while the gateway’s monitor checks the heartbeat and alerts by push notification. The push direction is proven. The email direction is the one that matters when the gateway is down, and its arrival has not been confirmed — two attempts to force a failure alert were defeated by throttling. Recorded as unproven rather than counted.
The first Ansible run found five defects
The baseline role had been written, reviewed and never executed against a real host. Running it against one resolver found four defects before it touched anything, and a fifth afterwards.
Two would have severed connectivity. The role enables a firewall with a default deny and had no mechanism to open a service port at all — it would have killed a resolver the whole fleet depends on. It also allowed SSH only from two source ranges, neither of which contained the machine running Ansible, so it would have cut its own connection mid-play.
One would have been silent and slow. A setting named a login account that exists on no host in the fleet, rendered into a config file that would have sat there looking correct until the next reboot.
And one would have created something. A task referenced a user that does not exist, using a module that does not fail on a missing user — it creates one. Every host the role touched would have quietly gained a stray account in a root-equivalent group.
That is the first post on this blog arriving on schedule: reviews find inconsistency, execution finds reality. Four of the five were visible only because something tried to run.
Also
The standby gateway now serves TLS, with certificate issuance proven against the staging authority before production — so a rate-limit mistake could not become an outage.
A dedicated control node exists for configuration management, replacing a Windows-hosted arrangement that had already caused one environmental failure: Ansible silently ignores its own config file when it sits on a world-writable path, which a Windows mount always is.
The multi-minute freezes on one virtual machine were localised, and the earlier explanation was wrong twice over. Two periodic samplers running on different machines settled it: the one inside the guest ran late three times, the one on the host never missed a beat. A post on that follows.
Where that leaves the build
| Public edge | Two gateways, both serving TLS, both on the mesh |
| Tunnel home | Live, carrying traffic, proven end to end |
| Virtual machines | 4 of 16 |
| Internal DNS | Live on both resolvers |
| Monitoring | Eight monitors; external watchdog live, email path unproven |
| Configuration management | Control node live, first role proven against a real host |
| Hypervisor cluster | Still one node. Others racked, still not powered on |
Next
The identity tier, which everything above it has been waiting for, and the mail server — which will finally give this site somewhere to receive corrections.