The checks that passed without looking

Four checks passed in the same working session, on four unrelated parts of the build. All four were wrong, and all four were wrong in the exact sense the project already has a name for: a check that cannot fail is worse than none. What made the session worth writing up is that the four checks failed to be checks in four genuinely different ways, not one bug repeated.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by human, chat 2026-09-17. How this site is written →

The restore drill that never opened the archive

A nightly script proves a database backup restores by unpacking it and reading the dump inside. One backup ships wrapped in a tarball rather than as a bare file, and the script’s pattern matching only recognized the bare shape. Against the wrapped one, it confirmed the archive was a valid tarball with the right number of entries inside and stopped there. An archive holding two completely empty files with the right names would have produced the identical passing result. The drill had been proving that a compression tool could list a file’s contents, which is not the same claim as “the backup restores.”

The fix extracts and actually parses the database dump from inside the archive rather than stopping at the wrapper, and it was verified against every other host running the same drill to confirm nothing regressed.

The test step that only echoed its own conclusion

A different verification script had a step that printed a sentence describing what had supposedly been confirmed, rather than actually confirming it. A line of the shape “(verified: the two groups match)” looks, read quickly, exactly like the output of a real comparison. It cannot fail, because nothing runs before it prints. It was rewritten to query both sides and compare them for real, with one extra step: a positive control proving the comparison can actually detect a match when one is genuinely there, since a comparison that always reports “no overlap” is the same defect one layer further in.

The false alarm from a page that renders itself in the browser

Confirming that a login option had survived a configuration change meant searching the login page’s raw HTML for a marker specific to that option. The marker wasn’t there, and the honest-looking conclusion was that the option had broken. It hadn’t: the page in question renders that part of its form with client-side script, so the marker is genuinely absent from the raw page on a completely healthy install too. The search had only ever proven that a particular string wasn’t present in one snapshot of HTML — it had never been shown capable of finding the marker on a working install, which would have caught the problem immediately.

The negative test satisfied by the wrong layer

A firewall rule was supposed to block one specific path from unauthorized addresses. Testing it from an address that should have been refused returned an empty, connection-refused-shaped response, which read as the rule doing its job. It wasn’t the rule. A separate, broader firewall layer was dropping the traffic before it ever reached the service the rule was written for, so the test had never actually exercised the thing under test. Only a request sent from a point that bypasses the broader layer produced the real, specific refusal the narrower rule was meant to issue.

What ties the four together

Each of these had a different shape:

  • one examined a container instead of its contents;
  • one asserted nothing at all;
  • one searched for a marker that is absent on a healthy system, not only a broken one;
  • one was satisfied by a different layer than the one it claimed to test.

One thing caught all four, and it’s the same thing the project has leaned on before: make the check fail on purpose before trusting it. If there’s no input you can name that would flip a given check’s result, it isn’t producing evidence — it’s just producing output that happens to look like evidence when nothing is wrong.