Tooling

What this lab runs, and — more usefully — why each thing beat the alternative. Choices without reasons are not reusable.

Platform

ChoiceInstead ofWhy
Proxmox, two-node cluster + witnessCloud hosting — a managed provider, or a fleet of rented instancesThe whole point is owning the data and the machines it sits on. At this scale, renting equivalent compute and storage costs more every month than the hardware did once, and egress charges alone would make a family photo library expensive to look at. Owned hardware also means no provider can suspend the account, change the pricing model, or deprecate the service. The honest cost: you become the operations team, and there is no support line
ZFS for the integrity tierOne undifferentiated poolChecksumming and self-healing matter most for small, write-sensitive data: the identity database, the password vault, git repositories
BTRFS on a Synology for bulkMore ZFSCapacity per pound wins for media and reproducible AI model files. Losing a model file means re-downloading it

Edge and network

ChoiceInstead ofWhy
Caddynginx, TraefikAutomatic certificate management is the default rather than a bolt-on, and the config file stays readable at 3 a.m.
Rathole tunnelCloudflare TunnelSelf-hosted end to end. No third party terminating TLS for your own data, and no account suspension that becomes an outage you cannot fix
WireGuardOpenVPN, IPsecSmall enough to audit, fast, and boring in the good sense
HeadscaleHosted TailscaleSame client ecosystem, self-hosted coordination
UnboundThe ISP resolverInternal names have to resolve internally, and DNS is not a service to outsource from a network with private addressing
Cloudflare DNSNamecheapMoved after a genuinely alarming discovery: the previous registrar’s update endpoint replaced the entire zone on every call. One malformed request could erase every record. Per-record APIs are not a nicety

Identity and secrets

ChoiceInstead ofWhy
AuthentikKeycloak, AutheliaOpen source, self-hosted, OIDC/SAML/LDAP, strong MFA, and a proxy for applications with no native SSO support
Vaultwarden for shared secretsEverything in one vaultSplit deliberately: a commercial manager for personal credentials, self-hosted for shared and machine secrets. No migration, and the failure modes stay separate

Observability and security

ChoiceInstead ofWhy
Prometheus + GrafanaThe default for good reason
Loki + AlloyThe ELK stackDramatically lighter for the same job at this scale. The original log shipper was chosen, then swapped before deployment when it reached end of life — see below
WazuhSecurity OnionSecurity Onion’s real value needs a network tap this topology has no natural place for, and its resource floor is roughly 24 GB of memory. Wrong tool for this shape of network, not a worse tool
CrowdSec at the edge onlyCrowdSec everywhereIts value is crowd-sourced attacker reputation, which is meaningful facing the internet and meaningless on an internal VLAN where traffic comes from your own address space
Syft + GrypeTrivyTrivy suffered two supply-chain compromises in March 2026. A vulnerability scanner is an unusually poor place to accept supply-chain risk

Applications

Files and sync: Nextcloud. Photos: Immich. Media: Jellyfin and Audiobookshelf. Documents and OCR: Paperless-ngx. Mail: Stalwart. Phone system: FreePBX/Asterisk, chosen for handset support and because a cloud-dependent PBX defeats the purpose. Finance: Firefly III. Automation: n8n. Reading: FreshRSS. Notes and bookmarks: Outline, Linkwarden. Development: Forgejo with Woodpecker CI and a Zot registry. Local AI: Ollama behind a routing gateway, with Open WebUI.

Nearly every one of those is free software maintained by people who are not paid by you. If this list is useful to you, that has a bill — and it is a small one.

Two decisions worth their own paragraph

Version pinning was being done backwards. The rule had been to pin to the oldest version that fixed all known vulnerabilities. That sounds careful and is precisely wrong: it guarantees you ship every vulnerability discovered after that version. The rule is now to pin to current stable and treat published vulnerability lists as a rejection floor — a reason to reject a candidate, never a reason to select one. The inversion was caught when a pinned security platform turned out to carry a critical remote-code-execution flaw, in the very component whose job was detecting compromise.

Choosing a tool is not the same as verifying it. Three full document reviews passed cleanly over that vulnerability. It was found by a fourth review that checked the runbooks against current upstream documentation and published vulnerability data rather than against the document set’s own internal consistency. A set of documents can be perfectly self-consistent and uniformly wrong. External verification is now a required step, not an optional one.

How it is deployed

Ansible, agentless over SSH, with roles kept in version control alongside the runbooks. The runbook is the human-readable procedure; the role is the executable one. Where they disagree, the role is what actually ran — and several defects have been found precisely by executing a role that a written procedure had described correctly but incompletely.