Most of what gets built for this project runs somewhere that other things already depend on. The chassis behind this new series is not that. It exists specifically so that breaking something costs nothing beyond a power cycle, and every design decision underneath it defends that property directly rather than assuming it.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
Separate by number, not just by name
The lab’s own network sits in a numbering space that shares nothing with production, on purpose and by an explicit rule written down before any of this hardware went in: management traffic, ordinary fleet workloads and this lab’s own guests each get a visibly distinct range, so a glance at an address says which world it belongs to without needing to look anything up. That is a small thing to enforce and a large thing to have, because the alternative — one flat numbering scheme everywhere — is exactly the condition under which a typo sends a test build’s traffic somewhere it was never meant to go.
The chassis itself carries the same property physically. It sits powered off except when someone is actively using it, unlike anything that carries a real workload. A machine that is off cannot be accidentally reached, misconfigured, or overloaded, which is a cheaper guarantee than any amount of access control on a machine that is always running.
The decision that mattered most never happened
The single biggest way this lab could have quietly stopped being a lab was never touched: these blades were never added to the fleet’s own Proxmox cluster. It would have been easy to default into — one more set of hosts, added to the pool everything else already shares — and it was the one option ruled out immediately instead. A cluster shares quorum across every member. Adding four hosts that are routinely powered off to a cluster that otherwise runs continuously means the whole cluster loses quorum every time this lab goes dark, which is precisely the moment nobody is around to notice. That is not a tuning problem to work around later; it is a reason the two should never have been the same cluster in the first place, and the record treats it that way — a plan that never got written down, rather than a mistake that had to be corrected.
The guest builder makes the separation structural, not procedural
Once the plan moved from “a chassis to bring online” to “a service that hands out virtual machines on demand,” the isolation had to survive something that had never existed here before: a caller who is not the person who built the system, asking for a network by name rather than being handed one. The design answers that by taking the choice away rather than trusting whoever is asking. A build request can only name a network from a list the service itself publishes, and the network that carries a host’s own management address and its default route is not on that list at any privilege level — it cannot be selected, named, or guessed into, because the field it would go in only ever accepts values the service already approved. The same discipline applies to which virtual machine IDs a caller can even address: builds come from a reserved range set aside for nothing else, and one specific ID is denied by number on top of that, because it belongs to the very workstation running the automation — if that one host doesn’t exist, it doesn’t need a policy to protect it. Belt and braces, on purpose.
Where the isolation was actually tested, not just designed
It held under a real mistake, not just on paper. Configuring a virtual network on one of the chassis’s own blades, before any of this service’s guardrails existed to catch it, ran into a gap between two settings that looked fine independently: the network bridge on that host was never told to expect tagged traffic, so creating a guest with one caused the hypervisor to quietly reassign its own physical network port out from under itself. The blade disappeared from the network entirely and needed a physical power cycle to come back. That is a genuinely bad failure mode, and it happened on hardware that exists to absorb exactly that kind of failure. Recovering from it cost one trip to the chassis and nothing else, which is the entire reason this lab is set up the way it is. The same mistake landing on a host carrying real traffic would have been a production incident with a very different cost. The finding is now a standing check the guest builder runs before creating anything: read the target’s actual bridge configuration first, and refuse the build outright if it is not ready for a tagged network, rather than discover the gap the same way a second time.
What generalizes
A test environment is only as isolated as the thing that would have to fail for it to stop being one, and that thing should be checked for directly rather than assumed. Separate numbering, a chassis that is off by default, and a cluster membership decision that was never made all point at the same question: what specifically would have to happen for a mistake here to become a mistake somewhere that matters, and has that been made structurally difficult rather than merely discouraged.
Letting a real failure happen somewhere safe is worth more than preventing it everywhere in theory. The bridge misconfiguration was not caught by review. It was caught by building the thing, watching it break in the one place built to survive that, and turning what was learned into a check that now runs before it can happen again.