Forty-three checks passed with a key that did not exist

A test suite for the DNS resolvers had a weakness worth closing: if the SSH key it used went missing, every check would fail, and thirty-seven DNS failures would be reported for what was actually one authentication problem.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

So a positive control was added at the top. Before testing anything, prove the connection works, and abort clearly if it does not.

Then the control was tested, by pointing the suite at a key that does not exist:

SSH_KEY=/nonexistent/key tests/rb05-unbound.sh
  exit=2, 43 checks ran, all PASS

Forty-three checks passed. The key was not real.

Why the key did not matter

The suite reused connections. Opening an SSH session takes a noticeable moment, and a suite making dozens of connections to the same host spends most of its time on that rather than on the checks, so it was configured to open one master connection and route everything else through it.

That is a sound optimization and it was added deliberately, to fix a real slowness problem.

The master connection was still alive from an earlier run.

When a master exists, a new invocation hands its session to that master, and the key argument is never consulted, because authentication already happened. It happened earlier, for the master, with a key that did exist.

So a check written to answer does this key work was actually answering is there already a socket open. Those are different questions with the same output.

The kind of bug this is

The control was correct when it was written and could not fail in the environment it shipped into.

Consider how it behaves over time. On a cold machine, with no master connection, it works exactly as intended: a bad key produces a clean abort. On a warm machine, minutes after any previous run, it passes regardless.

That combination is unpleasant, because it means the control would have passed its own first test. Write it, run it in a fresh shell, watch it correctly refuse a bad key, and ship it. Every subsequent verification, done while iterating quickly with a warm socket, would be worthless, and nothing would announce the change.

It is the same defect class this project has written about before, with a delay on it. Not a check that could never fail, but a check that stops being able to fail once the environment warms up.

The fix, which keeps the speed

The problem is not connection reuse. It is reuse across runs. The control socket path was fixed, so every invocation inherited the previous one’s authenticated session.

Making the path unique per run is enough:

ControlPath=${TMPDIR:-/tmp}/.cc-mux-$$-%r@%h:%p

The process ID in the path means one run still shares a single master across all of its own connections, which is the entire benefit the reuse was added for, while no run can inherit another’s authentication.

Re-tested afterwards with the nonexistent key: exit=99, zero checks run, and a distinct abort message. The control now fails when it should.

For a test whose subject is authentication, the stronger option is to disable reuse for that one connection entirely, so nothing is inherited under any circumstances.

What generalizes

Connection reuse invalidates any test of connection establishment. Keys, agent forwarding, source-address restrictions, jump hosts, multi-factor prompts: none of them are exercised by a session that was handed to an existing master. If the thing under test is how a connection gets authenticated, the test has to force a new connection.

Test a new check in the state the suite actually runs in, not in a clean shell. Making every check fail on purpose once is necessary and this project already does it. This is the refinement: doing it in a pristine environment proves the check works in a state it will rarely be in again. The warm, half-cached, ran-it-twenty-minutes-ago state is the one that matters, because that is where the check will spend its life.

An optimization can silently change what a test means. Caching, pooling, reuse and memoization are all introduced to make things faster while keeping results identical, and they usually do. The exception is any test whose subject is the work being skipped. Nothing warns you when a speedup has quietly removed the thing you were measuring.