Two things happened that had not happened before. The standby gateway in Europe exists, which completes the pair at the edge. And a physical server was brought online for the first time, which ended the long stretch of this project being entirely about rented virtual machines and documents.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
The hardware immediately disagreed with the documentation, in ways worth recording.
The EU standby is live
Provisioned in Germany, hardened, and asserted against the same regression checks as the US primary. Its runbook carried every correction learned on the first gateway, so the build itself was uneventful.
Provisioning fought back twice before that. The first handover was unusable: ordered without an SSH key, so the provider produced a password-only account flagged as expired, and both SSH and the console reached the forced-change prompt and died on passwd: Authentication token manipulation error. That message is a write failure on the shadow file — a wrong password reports Authentication failure instead — so authentication had been working the whole time and the diagnosis “bad credentials” was never right. Resolved by reinstalling with a key attached, which is now the runbook’s instruction.
Two things worth carrying forward. The image shipped with no swap at all, which the runbook had predicted; on a 4 GB host that will eventually run a spam filter, that is the exact configuration behind an earlier misdiagnosis — no swap means available is not headroom and memory pressure presents as a CPU-bound stall. A swapfile went in. And the IPv6 allocation arrived as a single address rather than a block, so there is nothing to sub-allocate later. Fine for a relay. Worth knowing before the mail server assumes otherwise.
Brute force was live throughout the build: 18 failed password attempts, 9 of them against root. None can succeed now.
The stock image ships the trap
An earlier post covered a server that accepted SSH password logins for weeks because sshd_config is first-match-wins and a hardening file had been numbered 99-.
The new machine confirmed something that post could only infer. The image ships two conflicting drop-ins out of the box:
50-cloud-init.conf PasswordAuthentication yes
60-cloudimg-settings.conf PasswordAuthentication no
EFFECTIVE passwordauthentication yesThe distribution’s own hardening file loses to cloud-init’s, because the lower number wins. Nobody misconfigured anything. That is the default state of a stock Ubuntu 24.04 cloud image, and any hardening file numbered above 50 inherits it silently.
The hypervisor is not what the architecture describes
First contact with the physical Proxmox host produced a table of differences:
| Found | Architecture says | |
|---|---|---|
| Hostname | provider default | the designed name |
| Cluster | single node, no corosync | cluster node 1 |
| GPU | none | “GPU host” |
| Address | on the live management LAN | the planned management VLAN |
| Existing workload | 11 VMs and a container, including unrelated production | blank slate |
The missing GPU is consistent with the record, which lists that purchase as unfinalized. The missing cluster is the bigger finding: high availability is not a later configuration step. It is gated on a second physical node plus the quorum witness, and while both machines are on hand, neither has been powered on or had an operating system installed.
Build VMs on local storage now and add high availability afterwards.
The agent would have followed the runbook order, which assumes the cluster exists before any VM is created. The instinct to invert that was right, and the reason is worth stating: the blocker is commissioning work, not storage layout. The other machines are sitting there unpowered with no operating system on them, and that is a day of work nobody has scheduled yet. A single node can carry the whole build in the meantime, so the sequence has no reason to wait on it.
Two safety rules went in before the agent was given any privilege on that box. The agent’s own VM runs on this host, so stopping it, migrating it or touching its network interface would end the session mid-command — a failure that could not even be observed, let alone reported. And the existing bridge carries the host’s address, the default route and all twelve guests, so it is off limits; the work goes on spare interfaces.
Running someone else’s production on the same machine also decided a smaller question. sudo was not installed, which is normal for Proxmox. Installing it was preferred over adding a root key, purely for the audit trail.
The architecture had no VLAN IDs in it
Forty runbooks. Four full-series reviews. A complete subnet plan with names and CIDRs for every segment.
No 802.1Q tags anywhere in the document.
You cannot configure a switch trunk from that. The gap was invisible for months because every prior task was documentation, review, or a rented VPS with one network interface. It surfaced within an hour of someone needing to tag a port.
Assign the VLAN IDs now, on a mnemonic that matches the third octet of each subnet.
Checked against the tags already live on the host, which belong to unrelated projects and could not be reused. The alternative — picking numbers at configuration time and recording them afterwards — is how you end up with an architecture document that describes a network nobody can rebuild.
The same audit found the plan describing the live management network as a retired segment from an earlier revision. Adopting the plan as written means renumbering the management plane that every host in the building currently sits on, which is a very different job from retiring an unused VLAN. That framing would have misled whoever performed the migration.
Two discovery tools, each blind to the other
A host with six network interfaces, three live, and no record of what was plugged into what.
lldpd answered the physical question in about thirty seconds: switch name, port number and negotiated speed for every cable, including two spare 10 Gb ports on the aggregation switch exactly where they were needed. No amount of host-side inspection produces that.
What it could not answer is the more interesting half. The switch vendor does not advertise the 802.1Q TLVs, so LLDP reported where each cable went but not what it carried. That required the controller’s own API.
Two discovery mechanisms, each blind to what the other sees. Using only one would have produced a confident and incomplete picture — which, given the subject matter of this blog, is the failure mode to expect.
A 10 Gb bond, built without disruption
A bonded pair across the two free 10 Gb ports, with a separate VLAN-aware bridge, configured in a drop-in directory so the working configuration file was never edited, and brought up on the new interfaces only rather than reloading everything.
- Ping to the gateway over the new VLAN interface: success, so the trunk genuinely carries it
- Failover proven rather than assumed: one link downed, active slave moved to the other, traffic continued, link restored
- The production bridge, its address, the default route and its uplink: all asserted unchanged afterwards
active-backup, deliberately not LACP. LACP requires a matching link aggregation group on the switch; if the bond expects one and the switch has not got one, the link comes up and passes nothing — and that would be discovered at the exact moment production traffic moved onto it. Conversion waits until the switch side exists and is proven.
One honest caveat recorded with it: both ports are on the same physical card, so this survives a cable, transceiver or switch-port failure but not a card failure. The cluster heartbeat links on separate 1 Gb ports, on a different chipset, are what preserve quorum in that case — which is precisely why the architecture specifies them as dedicated physical paths.
And one destroyed disk
Storage cleanup reclaimed about 115 GB by moving archive volumes off the SSD pool. During that, a virtual disk was destroyed outright by a recovery script that detected the dangerous condition correctly and then did the dangerous thing anyway.
It was recoverable only because a backup had been taken twenty minutes earlier, against a plan that did not call for one. That has its own post, and it is the most useful thing in this update.
Where that leaves things
The edge is now a pair rather than a single host. One physical server is online, carrying unrelated production, with a 10 Gb path built and proven but not yet in service. The cluster, the storage tier and every service VM remain unbuilt.
Worth being precise about the cluster, because “blocked on hardware” would overstate it. The machines are bought and racked. None of the others has been powered on or had an operating system installed. That is commissioning work that has not been scheduled, which is a different and much smaller problem than waiting on a purchase.
The component inventory has the current count, and now keeps dated snapshots so the trajectory is visible rather than just the present state.
Next
Commission the second node and the witness, or accept single-node operation for longer and start building service VMs on local storage. That decision is the next real fork, and it is a scheduling question rather than a purchasing one.