Three virtual machines exist. After weeks of rented gateways and documents, something is finally running on hardware in the building.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
More importantly, one of them is the machine every blocked item has been waiting on.
What went in
Three VMs, cloned from the validated template, all on the management VLAN with static addresses: a home-side gateway and two DNS resolvers. Each passed its full acceptance run — eight or nine assertions apiece — with the guest agent reporting. No change to cabling, the production bridge, the host address or routing.
The gateway VM is the unblock. Two runbook sections have been recorded as BLOCKED since 28 July for want of exactly this machine. They are buildable now, pending a shared secret that has to come from a human rather than from automation.
Both resolvers are live, serving split-horizon DNS for the internal zone with DNSSEC validation working. Getting there involved every external query silently failing while every internal one worked perfectly, which has its own post.
Four machines joined the mesh, including the US gateway, which now reaches the internal network without exposing anything new to the internet.
Five uptime monitors, and the alert path proven by inducing a real outage rather than assuming it.
Put both DNS resolvers on the same node for now, and write down that it is not redundancy.
The agent’s objection was the obvious one: two resolvers on one hypervisor share a failure domain, so the pair provides no more availability than one. True, and beside the point.
Every VM built from here embeds both resolver addresses in its configuration. Assigning those addresses now means moving the second resolver to another node later is a one-VM change. Deferring until a second node exists would mean reconfiguring eighteen machines. The co-location is temporary and the addressing is permanent, so the addressing is what should drive the decision.
What broke
Four errata, and the pattern is consistent enough to be worth naming: every one of them produced something that looked built.
Three ways to build a VM that looks fine. The VMs were pointed at the two DNS resolvers as their nameservers — before those resolvers existed. Cloud-init could not resolve the package archive, could not install anything, and exited with an error. But the VM still booted, took its static address and set its hostname, because all of that happens before package installation. It looked built while missing the guest agent, time sync, and every baseline package.
The runbook documents this exact ordering dependency, and it had been read earlier the same day.
Regenerating cloud-init does not reach a running VM. After correcting the nameservers, the guest demonstrably re-ran cloud-init — it regenerated its SSH host keys — and still applied the old configuration. The hypervisor only presents the regenerated image on a full stop and start, not a reboot. The evidence, gathered before acting this time:
hypervisor config nameserver: 1.1.1.1
guest netplan says: <the two resolvers>Joining the mesh would have repointed the whole fleet’s DNS. The mesh coordinator is configured to override each node’s local resolvers with its own. On the two DNS servers, that means they stop using themselves, and the internal zone becomes unreachable from the machines serving it.
Caught because the enrollment asserted that /etc/resolv.conf was byte-identical before and after on every host. Checking “does DNS still work” afterwards would have reported success — from the wrong resolver. That is the whole reason this is an errata entry rather than an outage entry.
The monitoring system reported every monitor UP while switched off for eighteen minutes. That has its own post and remains an open gap rather than a fixed one.
The one that was proven rather than assumed
Worth separating out, because it is the opposite of the above.
A new firewall rule was needed to let the monitoring container reach the mesh. Rather than adding it and checking it worked, the path was verified blocked first, then the rule added, then verified working. A before-and-after control rather than an after-only observation.
The rule went in additively at runtime and was then persisted and syntax-checked, with no ruleset reload — a full reload destroys the container runtime’s own tables, which this project has already done once by accident.
The monitors were proven the same way: a resolver was actually stopped and the alert waited for. The useful part was that the other resolver’s monitor stayed green, which proves the two discriminate rather than moving together. A monitor that has never been seen to fail is not known to work.
Where that leaves the build
| Public edge | Two gateways, both hardened, both on the mesh |
| Hypervisor | One node online, carrying unrelated production alongside this |
| Virtual machines | 3 of 16 |
| Internal DNS | Live on both resolvers, external resolution fixed |
| Tunnel home | Buildable at last, pending an operator-supplied secret |
| Monitoring | Five monitors, alert-proven, no external watchdog |
The cluster still needs a second node and the witness commissioned. Both machines exist and are racked; neither has been powered on.
The component inventory has the full count, and the archive shows how it has moved.
Next
The tunnel, once the secret is handed over — that closes the loop between the rented edge and the house, which is the last structural piece of the original design that has never been exercised end to end.