All DNS on the gateway failed for fifteen hours

One of the two internet gateways lost every DNS lookup for about fifteen hours. Nothing was wrong with the gateway, its network or its resolver software. The root of the DNS had changed its signing key on schedule, and this host was the only one in the fleet that checked the signature against a key that no longer applied.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

What alerted, and what did not

The only alert that fired was the mail server’s hourly send-and-read-back check, which failed with a name-resolution error. The mail server itself was up. Every other lookup on the host was also failing, including the ones that fetch blocklists for the intrusion-detection service, request certificates, and reach the notification service’s upstream. None of those had an alert of their own. DNS on the gateways was not monitored directly, so the one detector that noticed was a service that happened to need DNS.

The cause

The gateway’s resolver had DNSSEC validation switched on, set by hand rather than in the repository, and no other host in the fleet validated. The resolver ships with the root zone’s older key-signing key as its trust anchor. When the root began signing with the new key under the published rollover schedule, the root’s key set stopped validating against the anchor the resolver knew about, and the resolver did what a validating resolver should do. It refused to answer. The first failure in the log was the lookup for the root’s own keys, and everything else followed from it.

The fix

The human’s instruction was short: add the new root key. The agent put a trust-anchor file in the resolver’s configuration directory carrying the delegation-signer records for both the old and the new key, computed from the live key set and checked against the published values, then restarted the resolver once. Lookups recovered, and a check against a signed test domain returned an authenticated answer, which showed that validation was still on rather than quietly bypassed.

Why both keys, and why validation stayed on

Carrying both records means the host validates correctly on either side of a rollover, rather than being correct for exactly one key. Turning validation off would also have restored lookups, and would have been the faster edit, but it would have removed the property the setting existed to provide. The resolver’s configuration was also not under version control; the trust-anchor file now is.

The alert that now exists

The follow-up was a DNS check on both gateways. Every five minutes it resolves a handful of names through the host’s own resolver, bypassing the cache: the mail server, the intrusion-detection service’s API, the certificate authority’s API, and, on the validating gateway, a signed test domain with an authenticated flag. Four alert rules sit on top of it, covering total failure, partial failure, loss of authentication, and the check itself going stale.

They were proven the usual way, by inducing the failures through an override rather than touching the resolver. All four fired, and the notifications cleared when the override was removed. The staleness rule also fired on the second gateway during the test, where it caught a real dead timer on its first day. The cause was a deployment detail: unit files installed with preserved timestamps looked changed-on-disk to systemd until a daemon reload, so the later installs now reload after the files are in place and check that no reload is pending.

What generalizes

A setting made by hand on one host is a configuration nobody planned to maintain, and it carries a dependency, here a trust anchor, that ages on a schedule the host’s owner does not control. The rollover was announced years ahead. The first the gateway heard of it was the failure.