The second physical server is online. For the first time this project is not one computer with a lot of ambition attached to it.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
Three days of work also landed the entire observability tier: metrics, dashboards, alerting, log aggregation and security monitoring. Seven components moved from designed to running, which is the largest single jump since the build started.
What went in
A second hypervisor, running, monitored, and carrying its own weight. It arrived with the full set rather than just an operating system: metrics collection, log shipping, security agent, mesh membership, RAID monitoring, and a real publicly trusted certificate on its management interface.
It is not clustered, and that is not an oversight. Two servers cannot form a safe cluster on their own, because two votes cannot break a tie. The third vote, a small always-on witness, does not exist yet, and until it does this is two independent servers rather than a high-availability pair. The Proxmox note covers why that third machine is not optional.
The observability tier, all of it. Metrics collection and dashboards are live. Log aggregation is receiving from eight hosts. Security monitoring is running with eight agents reporting, verified from the manager rather than from the agents, because an agent that thinks it is connected is not evidence. Certificate expiry is probed across six targets.
Alerts proven by breaking things. A server was deliberately taken down to prove the host-death alert fired and the notification arrived. A drive failure was simulated to prove the RAID alert. Each liveness monitor was proven by stopping the service it watches.
That last part is the standard this project now holds itself to, and it caught something worth the effort. The fleet had no alert for a host simply dying. Four monitoring rules existed and all four read metrics that a dead machine stops producing, so a dead machine produced no data and one of the rules was configured to treat no data as healthy. The condition being monitored least well was the most total failure available.
The identity restore was rehearsed, not assumed. A real dump was restored and every table accounted for. A backup that has never been restored is a belief.
What is still missing, and one of them is a part
Storage is blocked on hardware, in a way no design document predicted. The integrity-tier machine is recycled enterprise equipment that does not support UEFI boot, so the plan to boot it from an NVMe drive in an expansion slot cannot work: the firmware will not boot from that card. The current path is an adapter that lets a standard SATA disk replace the onboard optical drive and act as the boot volume.
Wait for the adapter rather than redesigning around the problem.
The alternatives were available and each carried a real cost. The machine could have been given a different role, or the integrity tier could have been folded onto hardware that boots the way the design assumed. Both would have worked and both would have quietly changed the storage design to suit a firmware limitation on one box.
The judgment was that a part on order is a better trade than an architecture bent around a boot method. This is the specific tax of building on recycled enterprise hardware: it is inexpensive, it is capable, and its firmware predates the assumptions modern guides are written against. Anyone shopping for second-hand servers should check what the thing can boot from before deciding what it will be.
That blockage reaches further than one machine. The storage tier is what makes offsite backups possible, so every backup in this lab is currently on-host only. Nightly backups run, their guards have been proven to fire, and the restore path has been rehearsed. None of it survives losing the machine it sits on.
Failover still has not been drilled. Unchanged, and still the oldest gap on the list.
The metrics host cannot report the death of the server it runs on. It is a virtual machine on the machine it watches, so a total failure takes the watcher with it. The second physical node now probes the first every minute for exactly this reason, which is a real fix and also a reminder: a monitor inside the thing it monitors can report degradation and never death.
Where that leaves the build
| Public edge | Two gateways, both serving TLS, tunnel live |
| Identity | Live, backed up, restore rehearsed |
| Secrets vault | Live behind single sign-on, local backups only |
| Observability | Metrics, logs, dashboards, alerting, all live |
| Security monitoring | Live, eight agents |
| Physical servers | Two online, not clustered, no witness |
| Virtual machines | 8 of 16 |
| Storage | Integrity tier blocked on a boot adapter; bulk tier part-built |
| Offsite backup | None |
| Failover | Scripts exist, never drilled |
Twenty-three components complete, eight in progress, thirty-four not started. A week ago that was sixteen, six and forty-three.
Next
The witness, which is the cheapest machine in the design and the one that turns two servers into a cluster. Then storage, when the part arrives, and the offsite backups it unblocks.
Full detail is on the Status page.