Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
This lab has about forty step-by-step build documents. It has one machine actually doing anything.
That ratio is the most interesting thing about the project right now, and it is not a flattering observation. This first post is about it, because starting a public build log by describing the parts that work would set exactly the wrong tone for everything that follows.
How it got here
The plan was reasonable. Before building anything, write down the architecture: naming, addressing, network boundaries, hardware inventory, and a build procedure for every component in dependency order. Then execute it.
The first half went extremely well. Roughly forty documents now cover everything from provisioning the public gateway to deploying the security monitoring stack. They cross-reference each other. They have a dependency graph. They have been reviewed four separate times — for internal consistency, for security and privacy design, for resiliency and continuity, and finally against current upstream documentation.
The second half has produced: eight completed sections of the first document, on one rented virtual server.
Why writing kept winning
Documentation has a property that deployment does not: it always succeeds.
Write a runbook section and you have unambiguously produced something. Nothing refuses to boot. No cable is missing. There is no moment where the thing you just made turns out not to work, because nothing has been made yet — only described.
Deploying is the opposite. It fails, in specific, tedious ways, and each failure has to be understood before anything else can proceed.
So there is a quiet gradient pulling toward more design. Every new idea can be turned into a document today, and every document feels like progress. Four review passes over an unbuilt system feel like diligence. Some of that is diligence. Some of it is a very comfortable way to avoid racking servers.
The part that makes it worse
The reviews found real defects. That is the awkward bit — this was not wasted effort, which makes it harder to stop.
One review found that the automatic failover procedure would have moved two services to a standby server that had no backend for either of them. One of those was the notification service. A real failover would have broken the exact alerting channel that tells you a failover has happened, at precisely the moment you need it.
Another found that a security platform was pinned to a version carrying a critical remote-code-execution vulnerability — in the component whose job is detecting compromise. Three earlier reviews had passed over it, because they checked the documents against each other rather than against the outside world.
Both are genuine saves. Both also happened to unbuilt infrastructure, and could have been found the same way six months from now.
What actually breaks the tie
The argument that ended the debate was not about discipline. It was that design and deployment find different classes of defect, and only one of them finds the interesting class.
When the automation for the base VM configuration was finally executed against a throwaway virtual machine, it found defects that four document reviews had missed entirely. Not because the reviewers were careless — because the defects were of a kind that reading cannot detect:
- A verification step that was structurally incapable of reporting failure, so it had been logging “confirmed working” over a broken component.
- A readiness probe that returned true the instant a container was created, long before anything inside it was listening — so every check sequenced after it was racing.
- A setting that only takes effect at install time, meaning changing it later silently did nothing to already-installed machines.
None of those are visible on the page. All three are obvious within seconds of running the thing.
The strongest single example came a little later: an internet-facing host was accepting SSH password authentication for weeks, while two configuration files and the deployment log all said it was disabled. The documents were self-consistent, thorough, and wrong. That one has its own post.
The rule now in force
New ideas get one paragraph in a backlog file and nothing else until the current tier is actually running.
An idea earns its way out of that file only by failing a specific test: does deferring this make some decision expensive to reverse later? Not “is it good”. Not “is it interesting”. Everything in the backlog is good and interesting; that is why it is dangerous.
Two things have passed that test so far, and neither is a feature. Both are addressing decisions — the kind that are free to make now and require re-cabling once hardware is physically installed. Everything else, including several things there is real enthusiasm for, waits.
The backlog file opens with a line that is there specifically to be re-read:
Every one of these ideas is good, and none of them is why the lab is not running yet.
Why publish this
Partly discipline: a gap you have described publicly is harder to keep comfortably ignoring.
Mostly because this failure mode seems common and rarely written up. The homelab posts that get shared are finished-system tours — the rack photo, the service list, the dashboard screenshot. Almost nobody publishes the eight months where they had a beautiful architecture document and an empty rack.
The gap between designed and running is the actual risk to this project. Not missing features. Not the wrong tool choice. Just the distance between a very good plan and a room full of hardware that has not been plugged in.
Where things actually stand, updated as that changes.