Where the build actually is, as opposed to where the documentation says it should be. Updated as things change; last revised 18 September 2026.
Two rented gateways are live, both serving TLS, with the tunnel to the house up and carrying traffic. Single sign-on is running, which unblocks every service above it, and the first services on top of it are now running too. The second physical server is online, so the lab is no longer a single machine, though the two are not yet clustered. Eight service virtual machines are built.
The observability tier is running: metrics, dashboards, alerting, log aggregation and security monitoring, with alerts proven by inducing the failures rather than assuming delivery. Offsite backups exist as of 18 August.
The application tiers are filling in fast. A bookmark archive, webmail, news reader, workflow automation, file sync and two independent personal finance tools on the productivity side; continuous integration, a container registry and now software supply-chain scanning on the development side; and a local language model runtime with a chat interface and a routing gateway in front of it, now also reachable through single sign-on rather than only from inside the mesh. Several are complete. Several are running without the monitoring or backups that this project counts as finished, and are listed as in progress for that reason rather than as done. A photo library cleared the manual step that had it blocked and took its first real import. A dedicated GPU host has joined the AI tier and is still being tested, not yet counted as finished.
An internal phone system has made its first real, encrypted call in and out, though nothing yet watches it stay up. An SMS-to-email bridge sends real messages through a carrier line; the reverse direction is still being built. Power draw for the physical gear now has its own dashboard.
Two separate tracks now run alongside the main build: an education lab of blade servers checked out through an API, and an application built by another agent that uses the lab as a tenant rather than changing it.
The design is still far ahead of the deployment, which is a problem this project is aware of and writing about rather than hiding.
Past versions of this page are kept in the status archive, so the trajectory is visible rather than only the present state.
Live
Running and verified. The US gateway carries real traffic; the rest is built and asserted but not yet in service:
| Component | State |
|---|---|
| Hardened SSH (key-only, non-default port) | Live, verified |
| Firewall ruleset | Live, verified across reboots |
| Caddy with real Let’s Encrypt certificates | Live, via DNS challenge |
| Edge intrusion detection and blocking | Live, verified |
| Dynamic home-address tracking for tunnel rules | Live, verified |
| WireGuard (server side) | Live, no peers enrolled yet |
| Mesh coordination service | Live |
| Notification service | Live |
| Uptime monitoring | Live |
| Standby gateway (EU) | Live, hardened, serving TLS |
| First physical hypervisor | Online, carrying unrelated production |
| 10 Gb bonded uplink, failover proven | Built, not yet in service |
| Home-side gateway VM | Live, 8/8 acceptance checks |
| Two internal DNS resolvers | Live, split-horizon + DNSSEC |
| Seven nodes on the mesh | Live |
| Twenty-three uptime monitors | Live, alert paths proven individually |
| Rathole tunnel, both gateways | Live, carrying traffic, proven end to end |
| Caddy on the standby gateway | Live, ACME proven on staging then production |
| External heartbeat watchdog, off-site | Live, negatives tested |
| Configuration-management control node | Live |
| Identity provider (single sign-on) | Live, current stable, backed up nightly, restore rehearsed |
| Second physical hypervisor | Online, monitored, standalone |
| Shared secrets vault | Live behind single sign-on |
| Metrics, dashboards and alerting | Live, host-death alert proven by induced failure |
| Log aggregation | Live, eight hosts shipping |
| Security monitoring | Live, eight agents reporting |
| Certificate expiry probing | Live, six targets |
| RAID and disk health alerting | Live, both alert-proven |
| Git hosting | Live, backed up, restore drill passed |
| Offsite backup chain | Live, encrypted repositories |
| Personal finance, document archive | Live, restore drill passed |
| Continuous integration and container registry | Live, both alert-proven |
| Local language model, chat interface, routing gateway | Live, no cloud providers, monitoring incomplete |
| Bookmark archive, webmail client, news reader | Live behind single sign-on, backed up |
| Notes and wiki | Live, backed up, login proven |
| Supply-chain scanning | Live, images and deployed hosts both scanned |
The mesh coordinator matters more than it looks: it is a prerequisite for any VM joining the private network, so its being live unblocks the entire VM tier.
The edge is now a pair rather than a single host, which is what the failover design has always assumed. Failover itself is still not built.
In progress
- The mail server. Design settled and mailbox automation working; the records and the service itself are not live yet. This is what the site’s corrections route is waiting on.
- VLAN plan applied to one host only. IDs were assigned on 7 August and a trunk proven on the new bonded link. Nothing else has been migrated, and adopting the plan fully means renumbering the live management network.
- Blocklist status of the gateway addresses. Two lists report both addresses clean. One is unusable through a public resolver, and the rest need re-running from a host with working DNS — why the first answer was wrong.
- The photo library’s storage block cleared, and it is now actively serving. A first real import ran end to end — hundreds of gigabytes, tens of thousands of photos, one upload error out of all of them — and the background processing it kicked off is still draining. It is counted as in progress rather than complete because monitoring and backup have not caught up to it yet, not because it isn’t working.
- File sync went from not started to reachable. The setup wizard has run and the service answers requests. Single sign-on, the shared-storage connection to the rest of the file library, and backup are all still outstanding — this is a service that exists, not one that is finished.
- The bulk storage tier is the other part-built piece on the productivity side, started rather than designed and not yet running anywhere permanent.
Known gaps, stated rather than counted as coverage
- The external watchdog is half proven. An off-site heartbeat now exists on unrelated hosting, outside this infrastructure, and its failure modes were tested until they failed — stale, future-dated and missing state all return an error rather than a reassuring answer. The monitor watching it was proven by disabling the heartbeat outright rather than faking a response. The other direction is not proven. The off-site host also probes the gateway and alerts by email, and email is the path that matters when the gateway is down. Its arrival has never been confirmed. Counted as unproven — why that distinction matters.
- Failover has never been drilled. The scripts exist and their alerting was repaired — both referenced a token that does not exist, so a real failover would have changed every public DNS record and then crashed before telling anyone. Fixed, and still never exercised. A capability that has not been performed is a plan.
- Offsite backups now exist, and each one’s key lives on the machine it protects. Eight encrypted repositories were built on 18 August, in four legs that fail independently, with staleness alerts on each. The archives are encrypted at the source, so the copies hold only ciphertext.
The password for each repository currently exists in exactly one place: the host whose data it protects. So losing a host loses the key to that host’s backup. The archive survives, replicates offsite, and is unreadable. The chain was built to make host loss survivable and does not yet do that. It is the outstanding item on the whole backup effort, and it is deliberately a person’s job rather than an automated one, because it is credential handling.
- Nothing measures the alert channel as a whole. Every alert in the lab is proven individually by inducing its failure and watching the notification arrive. None of that could detect the state found on 18 August, where a single misconfigured monitor had produced 97% of every alert sent over a week. The delivery path was healthy; it was carrying 38 false alarms a day. Alert quality turns out to be a property of the channel rather than of any one check, and it was not being measured.
- Two live services have no liveness monitor. The language model runtime and its chat interface both answer requests and neither would raise an alert if it stopped. They are counted as in progress rather than complete for exactly this reason: by this project’s own definition, a service that runs and is not watched is not finished.
- The metrics host cannot report its own host’s death. Monitoring runs as a virtual machine on the server it watches, so if that server dies the watcher dies with it. The second physical node now probes the first every minute precisely because of this, but the general shape is worth naming: a monitor inside the thing it monitors can only report degradation, never death.
Not started
- The hypervisor cluster. Both nodes are now online, monitored, and running independently. They are not clustered, because a two-node cluster cannot break a quorum tie on its own and the third vote does not exist yet. Until the witness is built, there is no high availability, only two servers.
- The storage tiers, and the reason is worth stating because it is the kind of thing that does not appear in a design document. The integrity-tier box is recycled enterprise hardware that does not support UEFI boot, so the plan to boot it from an NVMe drive in a PCIe slot does not work: the firmware cannot boot from that card. The current approach is to source an adapter that lets a standard SATA disk take the place of the onboard optical drive and serve as the boot volume. Waiting on a part, not on a decision.
That blockage no longer holds up offsite backups, which were built on 18 August against the bulk storage box instead. It still holds up the integrity tier itself, which is where the small, precious, checksummed data is meant to live.
- Ten of the sixteen service VMs. Built so far: the home-side gateway, two DNS resolvers, the control node, the identity server, the secrets vault, the observability host, the security monitoring host, and hosts for the productivity and development tiers whose services are still being built.
Verified how, not just verified
Log shipping and the security monitoring stack were both proven on throwaway virtual machines first, and both are now running for real. That rehearsal earned its keep: executing the automation found defects that four separate document reviews had missed entirely, including a case where the stated mitigation for a known vulnerability did not hold, and several where a verification step was structurally incapable of failing.
The standard applied since is that an alert is not counted until the failure has been induced and the notification observed arriving. A host was deliberately taken down to prove the host-death alert. A drive failure was simulated to prove the RAID alert. A service was stopped to prove each liveness monitor. Every alert described as proven on this page was proven that way, and the ones that were not are named as unproven rather than quietly counted.
The gap, stated plainly
Roughly forty runbooks exist. Around a dozen have now been substantially executed on real hosts, up from four a week ago.
Every review pass, every new design document, and every good idea added to the backlog widens that gap. The gap — not any missing feature — is the actual risk to this project.
There is now a rule about it: new ideas get one paragraph in a backlog file and nothing else until the current tier is running. An idea earns its way out only if deferring it would make some decision expensive to reverse later. Two have qualified so far, and neither was a feature — both were addressing decisions that become costly once hardware is physically installed.
What comes next
- Get every backup repository’s key into a second place, so that losing a host does not lose the means to read its own backup.
- The quorum witness, which is what turns two servers into a cluster.
- The integrity storage tier, once the boot adapter arrives.
- The mail server, which gives this site a corrections route.
Progress against this list is what the status updates cover.
Full component inventory
Everything in the design, with an honest status against each. The three values mean specific things:
- Complete — deployed and verified on the running system, not inferred from a config file. This site has written at length about why that distinction matters.
- In progress — partially deployed, or proven on throwaway test hardware but not yet running anywhere permanent.
- Not started — designed and documented, nothing built.
Note how much sits in the third column. That gap is the point.
Public edge
| Component | Purpose | Status |
|---|---|---|
| Hardened SSH | Key-only remote access | Complete |
| nftables firewall | Packet filtering, survives reboot | Complete |
| Caddy | Reverse proxy, TLS termination | Complete |
| Let’s Encrypt via DNS challenge | Real certificates, including internal-only names | Complete |
| CrowdSec | Edge intrusion detection and blocking | Complete |
| Dynamic egress-IP tracking | Keeps tunnel rules following a changing home IP | Complete |
| Headscale | Mesh VPN coordination | Complete |
| ntfy | Push notifications for alerts | Complete |
| Uptime Kuma | External availability monitoring | Complete |
| WireGuard | VPN — server deployed, no peers enrolled | In progress |
| Rathole tunnel | Inbound relay, both gateways, proven end to end | Complete |
| Standby gateway (EU) | Failover target | Complete |
| DNS failover automation | Health-checked cutover, ~90s recovery | Not started |
Core infrastructure
| Component | Purpose | Status |
|---|---|---|
| VM baseline automation | Template validated; three VMs built from it | Complete |
| Physical hypervisors | Both nodes online and monitored, not clustered | In progress |
| 10 Gb bonded uplink | Built and failover-proven, not yet in service | In progress |
| VLAN plan (802.1Q tags) | Assigned and trunked on one host | In progress |
| Proxmox cluster | Two-node with HA — blocked on a second node | Not started |
| Quorum witness | Third vote, breaks cluster ties | Not started |
| ZFS integrity tier | Checksummed storage for small, precious data | Not started |
| BTRFS bulk tier | Capacity storage for media and models | In progress |
| Unbound | Internal DNS, both resolvers live | Complete |
| Home-side gateway VM | Terminates the tunnel inside the network | Complete |
Identity and secrets
| Component | Purpose | Status |
|---|---|---|
| Authentik | Single sign-on for every service | Complete |
| Vaultwarden | Shared and machine secrets | Complete |
Files and productivity
| Component | Purpose | Status |
|---|---|---|
| Nextcloud | File sync and sharing, external storage, single sign-on, offsite backup, restore rehearsed | Complete |
| Outline | Notes and internal wiki | Complete |
| Linkwarden | Bookmark archive | Complete |
| Snappymail | Webmail client, gated by single sign-on | Complete |
| Paperless-ngx | Document scanning, OCR, archive | Complete |
| Firefly III | Personal finance | Complete |
| Actual Budget | Personal finance, second instance, single sign-on only | Complete |
| n8n | Workflow automation | Complete |
| FreshRSS | News reader | Complete |
| Component | Purpose | Status |
|---|---|---|
| Stalwart | Mail server | In progress |
| rspamd + ClamAV | Spam and malware filtering | Not started |
| Mail backup and archive pipeline | Long-term retention | Not started |
Media
| Component | Purpose | Status |
|---|---|---|
| Immich | Photo library with local ML | In progress |
| Jellyfin | Media streaming | Not started |
| Audiobookshelf | Audiobooks and podcasts | Not started |
| Curated photo sharing | Second, public-facing photo instance | Not started |
Communications
| Component | Purpose | Status |
|---|---|---|
| FreePBX / Asterisk | Phone system, internal extensions, voicemail | In progress |
| SMS bridge | Email-to-SMS two-way messaging, outbound proven | In progress |
| LAN-only SMS alert gateway | Critical alerts that survive an internet outage | Not started |
| SMS query gateway | Conversational status queries | Not started |
Development
| Component | Purpose | Status |
|---|---|---|
| Gitea | Git hosting and container registry | Complete |
| Woodpecker CI | Build automation on an isolated VLAN | Complete |
| Dependency-Track | SBOM and vulnerability tracking | Complete |
| Syft + Grype | SBOM generation and scanning | Complete |
| devpi / Verdaccio | Package proxies | Not started |
AI
| Component | Purpose | Status |
|---|---|---|
| Ollama | Local language model runtime, CPU, with one GPU-backed host under test | In progress |
| Open WebUI | Chat interface | In progress |
| LLM gateway | Routing and per-service keys, no cloud providers | Complete |
| Voice pipeline | Local speech-to-text and text-to-speech | Not started |
| GPU tier | One card selected and under test on a new dedicated host | In progress |
Observability and security
| Component | Purpose | Status |
|---|---|---|
| Loki + Alloy | Log aggregation, 8 hosts shipping | Complete |
| Wazuh | Security monitoring, 8 agents reporting | Complete |
| Prometheus | Metrics collection | Complete |
| Grafana | Dashboards and alerting | Complete |
| Blackbox exporter | Certificate expiry and endpoint checks | Complete |
| Disk and hardware health monitoring | RAID and SMART, both alert-proven | Complete |
| UPS / power telemetry | Load and runtime, alert-proven by induced failure | Complete |
Operations
| Component | Purpose | Status |
|---|---|---|
| Ansible automation | Control node live; baseline role proven against a real host | Complete |
| Offsite backup chain | Encrypted at source, four independent legs | Complete |
| Restore procedures | Identity and git rehearsed; the vault is not | In progress |
| Maintenance playbook | Patching and routine operations | Not started |
Thirty-nine components complete. Thirteen in progress. Fourteen not started.
Every one of those sixty-six has a written runbook behind it. None of that counts for anything until it runs, which is the recurring theme here and the subject of the first post.