A routine change to add a forwarding rule took DNS down for the entire lab. Every query from every machine, internal names and external ones alike, came back REFUSED from both resolvers.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
The configuration that did it was three lines, and each of them was correct.
access-control: 0.0.0.0/0 allow
access-control: ::/0 allow
access-control: 0.0.0.0/0 refuse
Why that is an outage
Unbound decides who may query it by matching the client’s address against these rules, most specific first. Where two rules carry the identical prefix, the last one wins.
The first line opens the resolver to everything, which was a deliberate change made the day before. The third line is a default-deny that the configuration template always emits at the end. Same prefix, so the refuse silently overrode the allow, for every source on the network.
Neither half was wrong. They had simply never been rendered into the same file before, because the live configuration had been written by hand and the automation had not been applied since.
Everything said it was fine
This is the part worth keeping.
| Signal | Said | Was actually true |
|---|---|---|
| The configuration checker | passed | The syntax is perfect. It does not evaluate policy |
| The service status | active | The daemon is healthy and refusing everyone |
| The automation’s own summary | failed=0 | It did exactly what it was told |
| A line-by-line review of the diff | looked right | The added lines were read. Their interaction with the existing ones was not |
Four independent confirmations, all accurate, none of them about whether DNS worked.
The configuration checker is the sharpest of them. It is the tool whose entire job is to say whether the configuration is good, and it is a syntax validator. It will confirm that a policy is well-formed while that policy locks out the whole network, because “well-formed” and “correct” are different properties and only one of them is cheap to check.
The service status is the same trap this project has hit repeatedly. active did not mean degraded here. It meant answering, promptly and correctly, with a refusal, to every query in the fleet.
What actually caught it
The apply script resolved four real names, plus an external control, on both resolvers, before and after the change.
The “after” block came back empty. The outage was visible within seconds.
That is the whole difference. Had the script asserted that the service was active, or that the configuration parsed, or that the automation reported no failures, the lab would have had no DNS until something else broke loudly enough to start an investigation that began in the wrong place.
Assert the effect, not the status. For a resolver that means resolving names. For anything else it means whatever the service is for, observed from outside it.
The fix, and the review habit that would have caught it
The template now emits its default-deny only when the allow list does not already contain the default route:
{% if '0.0.0.0/0' not in unbound_allowed_subnets %}
access-control: 0.0.0.0/0 refuse
{% endif %}
The review habit is harder and more useful. A diff shows what changed, and the failure here was not in what changed. It was in how the change combined with what was already there.
Reading a diff answers “are these new lines right”. The question that matters is “what does this file mean once these lines are in it”, and for anything with ordering or precedence rules, those are different questions. Rendering the whole resulting file and reading it as a document catches a class that diffs structurally cannot.
The second finding, which is worse
The automation had never been applied since the live file was last edited by hand.
That made its first run dangerous in two directions at once.
Forward, it applied a combination of settings nobody had ever rendered together, which is the outage above.
Backward, it would have deleted two DNS records that existed only in the hand-written file and had never been added to the automation’s variables. The change preview caught those before anything ran, and they were reconciled first.
A configuration role that has not been applied since somebody edited the live file is a loaded gun. It does not just fail to know about the local changes, it actively reverts them, and it does so in the same run where it applies whatever untested interactions have accumulated in the template. Both halves fire at once, and the resulting mess is hard to attribute because two independent things went wrong simultaneously.
What generalizes
A syntax validator is not a policy validator. Every configuration language with a checker invites the belief that passing it means the configuration is correct. It means the file can be parsed. Whether the resulting policy does what you want is a question no parser can answer.
Last-match-wins and first-match-wins are both fine, and knowing which is not optional. Firewall rules, resolver access lists, and web server routes all have precedence semantics, and a rule that is correct in isolation can be completely inert or completely destructive depending on what sits near it. This project has been here before with a first-match-wins file where a later rule never applied. This is the same lesson from the opposite end.
Measure the same thing before and after. Not the service, not the config, not the tool’s own report of its work: the actual capability, from outside, with identical commands on both sides of the change. It is the cheapest verification available and it is the only one here that told the truth.