This lab has about forty step-by-step build documents and one machine actually doing anything.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
That ratio is the most interesting thing about the project right now, and it is not flattering. Starting a public build log by describing the parts that work would set the wrong tone for everything after it.
How it got here
The plan was reasonable. Before building anything, write down the architecture: naming, addressing, network boundaries, hardware inventory, and a build procedure for every component in dependency order. Then execute it.
The first half went extremely well. Forty-odd documents now cover everything from provisioning the public gateway to deploying the security monitoring stack. They cross-reference each other. They have a dependency graph. They have been reviewed four separate times: for internal consistency, for security and privacy design, for resiliency and continuity, and finally against current upstream documentation.
The second half has produced eight completed sections of the first document, on one rented virtual server.
Why writing kept winning
Documentation has a property deployment does not: it always succeeds.
Write a runbook section and you have unambiguously produced something. Nothing refuses to boot, no cable is missing, and there is no moment where the thing you just made turns out not to work — because nothing has been made, only described.
Deploying is the opposite. It fails in specific, tedious ways, and each failure has to be understood before anything else proceeds.
So there is a quiet gradient pulling toward more design. Every new idea can become a document today, and every document feels like progress. Four review passes over an unbuilt system feel like diligence. Some of that is diligence. Some is a comfortable way to avoid racking servers.
The part that makes it worse
The reviews found real defects, which is what made stopping hard.
One found that the automatic failover procedure would have moved two services to a standby with no backend for either. One of those was the notification service, so a real failover would have broken the exact alerting channel that tells you a failover happened, at the moment you need it.
Another found a security platform pinned to a version carrying a critical remote-code-execution flaw, in the component whose job is detecting compromise. Three earlier reviews had passed over it, because they checked the documents against each other rather than against the outside world.
Both are genuine saves. Both also happened to unbuilt infrastructure and could have been found the same way six months later.
What broke the tie
The deciding argument was not about discipline. It was that design and deployment find different classes of defect, and only one finds the interesting class.
When the automation for the base VM configuration was finally executed against a throwaway machine, it found defects four document reviews had missed. Not because the reviewers were careless, but because the defects were of a kind reading cannot detect:
- A verification step structurally incapable of reporting failure, which had been logging “confirmed working” over a broken component.
- A readiness probe returning true the instant a container was created, long before anything inside was listening, so every check sequenced after it was racing.
- A setting that only takes effect at install time, so changing it later did nothing to already-installed machines.
None of those is visible on the page. All three are obvious within seconds of running the thing.
The strongest example came shortly after: an internet-facing host accepting SSH password authentication for weeks while two configuration files and the deployment log all said otherwise. The documents were self-consistent, thorough, and wrong. That has its own post.
New ideas get one paragraph in a backlog file and nothing else until the current tier is running.
An idea earns its way out only by failing a specific test: does deferring this make some decision expensive to reverse later? Not “is it good” — everything in the backlog is good, which is exactly what makes it dangerous.
Two things have passed that test, and neither is a feature. Both are addressing decisions: free to make now, and requiring re-cabling once hardware is physically installed.
The backlog file opens with a line that exists to be re-read: every one of these ideas is good, and none of them is why the lab is not running yet.
Why publish this
Partly discipline. A gap described publicly is harder to keep comfortably ignoring.
Mostly because the failure mode seems common and rarely written up. The homelab posts that get shared are finished-system tours: the rack photo, the service list, the dashboard screenshot. Almost nobody publishes the eight months where they had a beautiful architecture document and an empty rack.
The gap between designed and running is the actual risk here. Not missing features, not the wrong tool choice. Just the distance between a very good plan and a room full of hardware that has not been plugged in.
Where things actually stand, updated as that changes.