The deeper version of The Lab. This page covers how traffic actually reaches a service, how the network is divided, and what order things have to be built in.
How a request gets in
Start with what happens when someone outside the house loads a self-hosted page.
- DNS resolves the public name to a rented VPS — the US gateway under normal conditions.
- That VPS runs Caddy, a web server that terminates TLS and reverse-proxies onward. It holds a real Let’s Encrypt certificate.
- Caddy hands the request into a tunnel that was established outward from the home network. The home firewall has no inbound ports open — the connection already exists, initiated from the inside.
- Inside the house, a gateway VM receives it and passes it to whichever service VM owns that name.
- If the service requires a login — nearly all of them do — the request is intercepted by Authentik, the identity provider, before it ever reaches the application.
The load-bearing idea is step 3. Because the tunnel is outbound, there is no port forward, no DMZ host, and nothing at the home address for a scanner to find. The public attack surface is a rented server with no data on it.
Why a self-hosted tunnel rather than Cloudflare Tunnel
Cloudflare Tunnel does the same job with less effort and no VPS to maintain. It was rejected on dependency grounds: it makes a third party a hard requirement for reaching your own data, gives them termination of your TLS, and means an account suspension is an outage you cannot fix.
The tunnel software chosen instead is Rust-based, single-binary, TLS-encrypted, and self-hosted end to end. The cost is two VPS instances to keep patched, and the failover logic below — which is real work that Cloudflare would have done for free. That trade was made deliberately, not by default.
Failover
The second VPS, in Europe, is a standby — it holds the same configuration but takes no traffic while the primary is healthy.
Health checks watch the primary. When it fails, DNS records move to the standby, and because the records carry a short TTL, service resumes in roughly ninety seconds.
Two decisions inside that are worth pulling out:
Failback is manual. Automatic failback sounds like the obvious completion of the feature and is a trap: a primary that is flapping will hand traffic back and forth, and a primary that has recovered partially will attract traffic it cannot serve. Returning is a decision a human makes after looking.
Not everything fails over. Video streaming stays on the US side only — transatlantic latency makes it unusable, and a service that is honestly unavailable beats one that is technically up and miserable.
That second point produced the single best catch of the project so far. The failover script’s list of names to move included two services that only ever ran on the primary — the notification service and the mesh coordinator. A real failover would have repointed them at a server with no backend for them, breaking the exact alerting channel the failover procedure depends on to tell you it happened. It was found by review before it was ever exercised. It has been fixed, and it still is not considered verified, because it has not yet been tested in a real drill.
Network divisions
The network is split by function, with traffic between segments filtered rather than assumed:
| Segment | Carries |
|---|---|
| Management | Hypervisor, storage, and infrastructure interfaces |
| Server | The service VMs themselves |
| Storage | NFS traffic between hypervisors and the NAS tier |
| Cluster interconnect | Two dedicated links for cluster heartbeat only |
| Voice | Phone system and handsets |
| CI | Build runners — isolated, because they execute arbitrary code |
| IoT | Devices that should never reach anything important |
| Lab | The teaching/experiment environment |
Two of those deserve explanation.
The CI segment is isolated because build runners execute untrusted code by design. A continuous integration runner’s entire job is to run whatever arrives in a pull request. Treating it as a trusted host on the server network would be treating an attacker-controlled shell as a trusted host.
The lab segment is numbered in a visibly different range from everything else. Not for any technical reason — purely so that a glance at an address tells you whether you are looking at production or at something disposable. Cheap, and it has already prevented at least one confusion.
Build order
Components depend on each other, so the build runs in tiers. Anything in a tier needs everything in the tiers above it:
| Tier | What | Why here |
|---|---|---|
| 1 | Public gateways, tunnels, DNS | Everything reaches the world through these |
| 2 | Hypervisor cluster, storage, VM baseline | Nothing runs without somewhere to run |
| 3 | Internal DNS | Services address each other by name |
| 4 | Identity and secrets | Every service above assumes single sign-on exists |
| 5+ | Applications | Files, media, mail, finance, and the rest |
| 8 | Monitoring, logging, security | Needs things to watch |
The ordering rule that cost the most to learn, and that the runbooks now state as a hard requirement: do not skip the identity tier. Every service written above it assumes single sign-on is already there. Adding authentication to a running service, with users and data in it, is far more work than configuring it on day one — and the version you bolt on afterwards is usually worse.
Certificates, everywhere, without exception
Every web server in the environment gets a real, publicly-trusted certificate. Including the ones that are only reachable from inside. Including the ones only one person will ever open.
This is unusual for a homelab, where the normal answer is a self-signed certificate and a browser warning you learn to click through, or a private certificate authority you have to install on every device you own.
It is possible because certificates are obtained via a DNS challenge — proving control of the domain by writing a DNS record, rather than by serving a file over port 80. That works for names that are not publicly reachable at all, which is exactly the internal-only case.
The benefit is not really cryptographic. It is that “certificate warning” goes back to meaning something is wrong, instead of meaning you are at home. A warning you are trained to dismiss is a warning that no longer works.
Where it actually is
See Status for what is deployed today versus designed on paper. The gap between those two is the honest subject of this blog.