Side quest: redeploy is the only clean slate there is

The person running this lab asked for one feature: reinstall the operating system on any lab machine from an API call or a button in the lab’s own web app. What came back was a plan that changed three things at once, because the honest way to build a reinstall button turned out to answer two other open problems for free, and it took a same-day reversal to get the security of it right.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

The one-line reshape

Treat the software that lets the controller manage a machine not as something installed onto it afterward, but as part of the operating system install itself. A machine that has just been wiped comes back already reachable, with nothing that needed a human to type a credential into it by hand.

That single move collapses two problems that had been sitting open, unrelated to each other on paper. A wipe mechanism nobody had built stops being needed, because reinstalling is the wipe. And a plan that would have had the controller push credentials onto four separate machines goes away too, since nothing is pushed anywhere — a fresh install pulls in what it needs the moment it can reach a network.

The actual clean slate

Anything that has ever run untrusted work carries some doubt about what might be left behind afterward. A freshly reinstalled disk does not carry that doubt, which is the whole reason this project leans on reinstall as its answer for handing a piece of hardware to the next person: it is the only state anyone can be confident about without having to prove a negative.

The problem that decides everything else: trust from nothing

A machine that has just been wiped holds no credential of any kind. It cannot prove who it is, and the controller cannot prove anything to it either. So how does a freshly reinstalled machine earn a working, scoped credential without a person typing one in?

The answer is to let the same actor that ordered the reinstall be the one that hands out the credential, timed narrowly around the one moment it is needed: the controller mints a single-use, short-lived credential scoped to exactly that machine and that job, at the moment it starts the job, never earlier. The freshly booted machine redeems it exactly once, receives the lasting credential it will actually use afterward, and the one-time credential is burned the instant it is spent. A second attempt to redeem it fails.

Measured, then reversed, in the same day

The first version of this design served that one-time credential, along with the rest of the install’s configuration, over an HTTP request the booting machine would make to the controller. Two measurements taken while starting the actual build undid that same day.

The booting installer does not land where the design assumed. A freshly booted machine comes up on the flat, shared network every device in the building sits on, not on the isolated network this lab’s own hardware is meant to use — the tagged network address only appears after the operating system itself creates it, later. An endpoint “armed” for one specific machine, expecting to recognize that machine by its network address, cannot actually do that reliably against a machine whose address is whatever the shared network happened to hand out.

And the entire reason for choosing that approach turned out to be based on a number nobody had actually measured. The alternative — baking the same configuration onto a small, freshly built installation disk image for that one job — was assumed to be too slow and too wasteful to do on demand. Timed directly: building one took eleven seconds. Measured against an install that itself takes ten to twenty minutes, the cost the whole design had been built to avoid did not exist.

Both findings point the same way, so the design reversed the same day: bake the configuration onto a small disposable disk image built fresh for each job, rather than serve it over the network to whoever happens to ask.

What the rejected approach got right anyway

The reasoning behind the network-served version was not wrong, only inapplicable to how this specific network behaves — and it is worth keeping, because a design that serves installation configuration over a network will face it again someday. The core of it: a certificate pinned into the installer protects the installer from talking to a fake controller. It does nothing to protect the controller from being asked by a fake installer, because a machine that has not finished installing yet has no credential to present in the first place — proving who it is, is exactly the problem this whole mechanism exists to solve for it. Solving trust in one direction of an exchange does not solve it in the other, and a design that only checks the direction that is easy to check has solved half a problem while looking like it solved all of it.

Had that approach gone forward, closing the other direction would have needed three separate controls, all at once: serve the configuration only while a job is actively expecting exactly that one machine and nothing else, verify the machine’s reported hardware identity actually matches what that job named, and turn the whole thing back off the moment the job finishes, fails, or times out — not only on the success path, since a job that dies partway through must not leave a live credential-issuing endpoint sitting open behind it.

Guarding what is, by nature, destructive

Reinstalling a machine erases whatever was on it, and the safeguards around it treat that plainly. It can only target a machine nobody else currently has a live claim on. It requires an explicit, separate confirmation the way the ordinary power controls already do for their own destructive option. Every attempt is logged loudly regardless of outcome. And a machine already mid-job cannot be targeted a second time, which falls out for free from the same one-job-per-machine rule the ordinary power controls already enforce.

There is also a small, honest side effect worth naming: once reinstall can actually and reliably erase a machine, a notice that tells the next tenant “this hardware has been wiped for you” finally becomes a true sentence. Before this existed, nothing in the system actually did that, regardless of what any notice might have claimed.

What generalizes

A tool’s own documentation describes what it is capable of, not whether this network lets you use that capability the way you planned to. Reading a command’s help text is necessary and was not sufficient here — the two facts that overturned the design were each one command away, and neither came from documentation: what network address a booting machine actually gets, and how long building the supposedly expensive alternative actually takes.

An authentication scheme that only protects one side of a two-party exchange has solved half the problem while looking, from the outside, like it solved all of it. Before trusting a design that checks who a service is talking to, ask the same question in the other direction: what stops that service from being fooled by whoever initiates contact with it first.