A certificate was issued in six seconds. That was taken as proof the new certificate provider worked.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
It proved the opposite of what it appeared to, and the speed was the tell.
The setup
This lab gives every internal web interface a real, publicly trusted certificate, obtained by proving control of the name through DNS rather than by serving a file over the public internet. The reasoning is in the Caddy note; the short version is that it is the only way an internal-only service can hold a genuine certificate.
The certificate provider was being changed. To confirm the new one worked, a certificate was re-issued for a host. It came back almost immediately.
Why fast was bad news
Certificate authorities cache the fact that you have proven control of a name. Once an authorization succeeds, it stays valid for around thirty days, and any re-issuance inside that window skips the challenge entirely.
The host had been issued a certificate earlier the same day, through the old provider. So the re-issuance rode on that still-valid authorization, never touched DNS, and produced a perfectly good certificate without exercising a single line of the new configuration.
Six seconds is far too fast for a DNS challenge, which involves writing a record, waiting for it to propagate, and having the authority query it from the outside. The log confirmed it plainly, by containing nothing at all where the challenge would have been recorded.
The speed was the evidence, and it was read as success.
The consequence is the part worth carrying: a host can be switched to a completely non-functional certificate provider, issue certificates on demand for a month, and then fail its first real renewal, roughly thirty days after anyone last looked at it.
The actual bug underneath
Separately, and this is what the test would have caught, the DNS challenge genuinely was broken on internal hosts. The reason is a direct consequence of a design decision this lab made deliberately.
The internal resolvers are authoritative for the internal namespace. That is the whole point of them: they hold the records for internal names and answer definitively. But being authoritative for a namespace means answering “no such name” for anything in it that is not in their records.
The challenge records live in the public zone. So an internal host asking its own resolver about a challenge record is told, authoritatively, that it does not exist.
The certificate client checks that its own record has propagated before telling the authority to go and look. It performs that check using the host’s own resolver. Which says no such name. So the client waits for propagation that, as far as it can see, never happens, and eventually gives up.
The record was correct. It was publicly resolvable the entire time. The client was asking the one resolver in the world that was configured to deny its existence.
The fix, and how to test a challenge properly
Point the propagation check at public resolvers, which are what the certificate authority will actually query:
tls {
dns acmedns /etc/caddy/acmedns.json
resolvers 1.1.1.1 9.9.9.9
}The client’s own view was never the relevant one. The question is whether the record is visible to the authority, and the authority is on the public internet.
Testing this properly means defeating the authorization cache, and there are two reliable ways. Use a name that has never been issued for, so no cached authorization can exist. Or issue against the authority’s staging environment, which keeps its own separate cache and is the right place to burn attempts anyway, because production rate limits are easy to exhaust while a configuration is still wrong.
The general form of the check: when verifying that a mechanism works, make sure the run you are watching cannot take a shortcut around that mechanism. If a cache, a session, or a prior success can satisfy the request, the test proves only that the shortcut exists.
What generalizes
A success that arrives too quickly is a result, not a relief. Any operation with a known cost, a network round trip, a propagation delay, a handshake, that completes far below that cost did not do the work. The instinct on seeing something finish fast is satisfaction. The correct instinct is to ask which step was skipped.
Caching turns a broken configuration into a delayed failure. This is the same shape as a control that stopped working once the environment warmed up: a prior success is satisfying the request, so the thing under test is never reached. The difference here is the fuse length. Thirty days is long enough that the eventual failure looks unrelated to the change that caused it.
A deliberate design decision will collide with something eventually, and that is not a reason to reverse it. Split-horizon DNS is correct for this lab and it broke certificate issuance, because being authoritative means denying what you do not know. The fix was to stop asking the wrong resolver, not to give up the design. Worth expecting for any architecture choice with real teeth: somewhere down the line, something will assume the default you deliberately did not take.