Two new DNS resolvers went live. The configuration validated. Both services reported active. Every internal name resolved correctly, on both machines.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
And every single query to the outside world returned SERVFAIL.
The fault
The resolvers forward external queries over DNS-over-TLS, which means the upstream’s certificate gets verified. The runbook specified forwarders in this shape:
forward-addr: 1.1.1.1@853#cloudflare-dns.comThat #hostname suffix is an instruction to verify the upstream certificate against that name. Verification needs a list of certificate authorities to check against, and the configuration named none.
So every upstream connection died at the TLS handshake:
error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
notice: ssl handshake failed 1.1.1.1 port 853Four forwarders were configured for redundancy. All four failed identically, because it was not four independent failures — it was one defect, four times. A fallback pool protects against a forwarder being down. It does nothing when the reason for failure is on your side.
One line fixes it:
tls-cert-bundle: /etc/ssl/certs/ca-certificates.crtExternal resolution and DNSSEC validation both came back on restart.
Why it presented as a network problem
Three separate things said everything was fine, and each was telling the truth about a different question.
The config checker reported no errors. It validates syntax. Whether a TLS handshake can complete is not a syntax question.
The service reported active, and genuinely was — listening, accepting queries, answering them. A theme this project keeps returning to: started is not working.
Internal resolution worked perfectly. Every internal name, every reverse lookup, every access rule behaved exactly as designed, because none of them touch a forwarder. Split-horizon DNS answers internal names locally.
That third one is what made it hard. A resolver that fails outright gets noticed immediately. A resolver where the half you use constantly works, and the half you use occasionally is entirely dead, reads as flakiness somewhere else on the network. The healthy half masked the dead half.
The test that found it, and the three checks that could not have
The defect was caught on the first real run of the project’s first durable acceptance test — written the same day, specifically to catch this family of problem, after three manual spot-checks had passed over it.
The test found the fault. Three of its own checks were incapable of doing so.
| Check | What was wrong with it | ||
|---|---|---|---|
upstream is DoT (port 853 out) | Ended in `\ | \ | true`. Could never fail. Decorative. |
DNSSEC: bad signature is REFUSED | The test domain returns SERVFAIL when validation works and when nothing resolves at all. It passed during the total outage. | ||
both resolvers agree | Ran dig from a machine that cannot resolve anything. Reported none vs none and blamed the resolvers for its own blindness. |
The test suite’s own README forbids all three patterns by name. Writing that document did not stop the same person reproducing all three within hours.
Control pairs have an order
The middle check is the instructive one, because the instinct behind it was right.
It was half of a deliberate control pair: a signed name must validate, and a deliberately broken signature must fail. Asserting both directions is exactly how you avoid a check that always passes.
But the two halves are not equal. “A broken signature fails” passes trivially when nothing resolves at all. It carries no information until the positive half is already passing. So during a total outage it sat there reporting success, which is precisely the moment a control is supposed to earn its keep.
Assert the positive discriminator first. The negative half means nothing until it does.
Two measurement artifacts from the same run, neither a real failure
A flaky check is not a check. A sample for established TLS sessions passed only when a query happened to have run recently, because the resolver closes idle sessions. Fixed by forcing a guaranteed cache miss — a random label — immediately before sampling. A result that depends on timing is not a measurement.
The reported exit code was never the script’s. Running bash tests/… ; echo $? through PowerShell had $? expanded by PowerShell, which printed True, and on an earlier run 0, while the script was correctly exiting with its failure count. The test was right and the harness was misreporting it.
And a label matched its own search. The check named DNSSEC: bad signature is SERVFAIL contains the substring FAIL, so counting failures with grep -c FAIL found two on a completely passing run. The label is now …is rejected — the same shape as an earlier defect where a status file’s own comment header matched the grep looking for its status.
What this does not mean
It would be easy to read this as an argument against the test, since the test was riddled with the defects it was written to prevent.
The opposite. It found a real runbook defect on its first run, one that three passing manual checks had walked straight past, and it found it in under a minute. A test with three broken checks and one working one is still infinitely better than four careful glances.
The lesson is the one already written down and evidently not yet automatic: a check has to be exercised against a known-broken state before its PASS means anything. Writing that rule down does not make it happen. Neither does writing the tests on the same day you write the rule.