Side quest: the chassis that does not touch the fleet

A blade chassis showed up in this build’s corner of the world, and it is not part of the fleet this blog has been describing. It has its own tree, its own agent line, and its own set of decisions still waiting on the human. It is also not the whole story by itself — the chassis is where a test lab started, not where it stops, and the piece actually worth explaining is what has to exist underneath it for that lab to grow past one box. This is the first post in a run about it.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

Why a side quest and not another chapter

Everything published here so far has come out of one working tree, one running record. This chassis deliberately does not. When it arrived, another agent was already mid-task against the main queue, and putting a second, unrelated project inside the same repository would have meant every file it touched showing up in that other agent’s own status and, eventually, its own commits. So this track got a tree of its own instead, checked out next to the main one rather than inside it.

The two share a naming scheme, a set of habits — log as the work happens rather than at the end, four small artifacts around anything that runs unattended, never trust a check that has not been proven able to fail — and nothing else. No machine appears in both. Security findings and errata that would normally land in the fleet’s own shared files are held in the chassis track’s files instead, for the human to fold in later on their own schedule, not this project’s.

What the chassis actually is

It is a Dell PowerEdge FX2s: a shared chassis holding four half-width blades, built for a lab that only needs power on when someone is actively using it, on a network deliberately numbered apart from anything running production workloads. Two of the four blade slots had been empty since before this track started; the other two arrived mid-project, seated by hand. It is also, deliberately, not meant to be the only hardware this lab ever has. It was the piece that happened to be sitting there already, which made it the cheapest possible starting point for a test lab that is expected to grow — more chassis, more blades, whatever shows up next — rather than the boundary the lab was designed around.

None of that hardware was new to this build’s own architecture record — it had been inventoried on paper for a while. What changed is that it came online, and the record started having to match what was actually plugged in and powered up rather than what a document had assumed about it. The chassis controller, the switching fabric inside the chassis, and each blade’s own management interface all had to be read before they could be trusted, not assumed from whatever the last inventory pass had written down.

The part that actually has to hold up as the lab grows: the control plane

A lab that stays one chassis forever does not need much more than a login and a steady hand. A lab meant to keep growing needs something in between the person asking for hardware and the hardware itself — and that piece already existed before this chassis was ever plugged in, built for the bare-metal side of the same lab.

Its job is not glamorous, and it is exactly the job a growing lab actually needs done: turn physical hardware on and off, put the right operating system on it, configure its network, and — the part that matters most once more than one project shares the same rack — decide who is allowed to touch which piece of hardware right now, so two different pieces of work never grab the same physical machine at the same time. That last part is not a hypothetical. Real blades in this same chassis have sat leased to one project for a fixed stretch of time specifically so a second project couldn’t reach for them mid-use. Without an arbiter, “which box is free” is a question two humans, or two agents, answer by asking each other — which works until it doesn’t.

The reason this new series exists is what gets built on top of that control plane, not instead of it: a guest builder that hands out virtual machines — a requested operating system, cores, memory, disk, a network — to any hypervisor willing to accept one, from a web form or an API call, the same way the same control plane already hands out freshly reimaged bare-metal hosts.

The load-bearing decision, made before any of it was built, was to not build a second version of the tool that already does most of this. The lab already runs a small control service with a scoped HTTP API, a web UI behind single sign-on, a job queue with a worker process, and a hardware layer behind a driver abstraction — all of it proven already, on real bare-metal reinstalls. A guest builder needs new routes, one new kind of job, one new driver, and a folder of operating system definitions. It does not need a second thing that does the same job worse. The one piece that genuinely had to change underneath: the existing queue serialized work per physical blade, and a guest builder has to run two builds on two different hosts at the same time, so that locking had to widen from “one blade” to “any target host,” concurrently — the same contention problem the control plane already solved for physical hardware, now needed one layer up, for virtual hardware too.

The rule that shaped everything else

The person running this lab set one constraint early, and it turned out to decide most of the rest of the design: an operating system image gets stored exactly once, never copied per guest and never copied per host. A future phase of this same project is meant to deploy the same catalogue of operating systems to bare physical machines, not just virtual ones, and duplicating a five-gigabyte install image for every guest or every host it might land on was the thing being ruled out in advance, before it could quietly become the default.

The shape that satisfies it: one read-only pool holding each image exactly once, mounted read-only by every machine that might need to build from it, and a per-guest footprint that is generated fresh and disposable — a few kilobytes of answer file, not a few gigabytes of disk image. A cloud-image Linux guest configures itself from a tiny generated seed. A Windows guest gets an answer file baked onto its own tiny generated ISO. Either way, the multi-gigabyte thing never moves.

Worth admitting plainly: this project’s own existing bare-metal reimage path does not follow that rule. It bakes the whole install image, answer file and all, fresh per blade, four times over for four blades. That was a reasonable call for four physical machines that get reimaged rarely. It stops being reasonable the moment the whole point is many guests built from a few images, which is exactly why the rule got written down explicitly this time rather than copied forward by habit.

What is coming

The chassis did not come online cleanly, and several of the fights along the way are worth their own post: a Windows install that quietly restarted itself from scratch every few minutes for most of a day, a guest whose network configuration disconnected the very hypervisor running it, and a design for who is allowed to build what, on which network, that took real back-and-forth with the person who actually owns the risk. Those come next, tagged the same way this one is, so the whole run stays findable on its own regardless of what section of the site each entry lands in.

What generalizes

Build the thing that arbitrates contention before the thing that creates more of it to contend over. A guest builder that can conjure a new virtual machine on demand is only safe to hand to more than one person if something underneath already answers “who gets this piece of hardware right now” — and that answer needs to exist whether the lab stays one chassis or grows into several.

Isolating a project’s working state is not overhead — it is what keeps two pieces of unrelated work from becoming entangled by accident. A second tree, a second agent identity, a documented boundary of which machines belong to which track: none of it is free, and all of it is cheaper than untangling one agent’s commits from another’s after the fact.

A rule stated once, early, and applied consistently is worth more than a rule inferred from whatever the last similar system happened to do. The existing reimage tool’s own shortcut sat right there in the same codebase, available to copy. Writing the actual constraint down instead — store it once, generate the rest — is the difference between a system that scales past four machines and one that quietly does not.