Side quest: a key that changes, and a fix that refuses to guess

The first guest build queued right after a blade’s redeploy failed with “REMOTE HOST IDENTIFICATION HAS CHANGED.” The blade had just been reimaged, which regenerates its SSH host key by design, and the builder’s own trusted-key list still held the key from before. Retrying would have failed identically forever — this wasn’t a flaky connection, it was two systems that disagreed about the blade’s identity and neither would budge.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

Why the obvious fix was the wrong one

The one-line fix for “the recorded key doesn’t match” is StrictHostKeyChecking=no: stop checking, connect anyway. That trades a blocked build for a build that will connect to anything answering on that address, including something that isn’t the blade at all. A chassis whose entire design premise is running untrusted guests on hardware other tenants will later reuse is exactly the setting where “connect to whoever answers” is the wrong failure mode to build in.

Where the fix actually gets its answer from

Not the network. Scanning the blade’s key off the wire and trusting whatever comes back is trust-on-first-use by a third party — it would record an interceptor’s key with exactly the same confidence as the real one. The fix instead asks the blade’s own management agent, which reads the key material directly off the blade’s disk and reports it over a channel that’s already authenticated. A redeploy clears that report when the reimage is committed, so “the agent reported this key” always means “reported by the operating system that is running right now,” never a stale answer left over from before the reimage.

A key scanned off the network only gets written to the trusted list if its fingerprint is one the agent already vouched for. No match, and nothing gets written — the build stops rather than guesses, which is the shape of what an interception would also look like, and neither is something worth connecting through on the hope that it’s the harmless explanation.

The second half of the same bug

Withdrawing trust in a stale key matters as much as pinning a new one. A build queued immediately after a redeploy was found trusting a pre-reimage confirmation that no longer meant anything, because nothing reports a key as gone while the blade is mid-reinstall — there’s no agent running yet to say so. The fix marks a blade’s key confirmation withdrawn the moment a redeploy is committed, not when the new key eventually shows up, so there’s no window where an old confirmation is still treated as current.

Proving the fix resists the thing it’s supposed to resist

A positive test that pins a real key proves the happy path. It says nothing about whether the fix would also pin a key it shouldn’t. The test suite includes a deliberately wrong version of the same logic — one that pins whatever the network presents, the naive fix this design exists to avoid — run through the same sixteen checks as the real code. The real version passes all sixteen. The naive version fails five, every one of them a test built to simulate interception. A test that only exercises the correct path can’t tell a careful fix from a naive one; feeding it the mistake it was built to prevent is what makes the pass mean something.

The fix was then proven outside the test suite too: the next real tenant build after a blade’s redeploy went through end to end, exercising the same pinning path on real hardware rather than only the simulated one.

What generalizes

Trust that comes from scanning the network and trust that comes from an authenticated report look identical in a log line that just says “key verified.” Only one of them is actually anchored to something that can’t be spoofed by whatever answers on an address. When a system’s job is to connect to hardware other tenants will reuse, the question worth asking isn’t whether a key check exists — it’s where the thing being checked against actually came from.