This one never failed. It was found by reading, before anything ran, and that is the only reason it is a short story rather than a long one.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
A runbook section set up an API token for the lab’s DNS provider. The prose above the configuration explained that certificates would be obtained through a DNS challenge, and stated the benefit in as many words: no port 80 exposure required for issuance.
The configuration file underneath it did not contain the directive that turns that on.
Why that matters here
This lab issues real, publicly trusted certificates for every human-facing web interface, including the ones that are only reachable from inside the network. The reasoning is in the Caddy note; the short version is that a DNS challenge proves control of a name by writing a DNS record, so nothing has to be publicly reachable for a certificate to be issued.
Without an explicit instruction to use it, the web server falls back to the default: prove control by serving a file over HTTP at that name, from the public internet.
For a service that is publicly reachable, that default works. For an internal-only service it cannot work, because the certificate authority has no route to the host. Being unreachable is the entire point of an internal-only service, and it is also exactly what makes the default challenge impossible.
Why it would have gone unnoticed
The failure would have been partial, and partial failures are worse than total ones.
Had this been deployed and left to run, the public-facing services would have obtained certificates normally. Everything would have looked like it worked. The internal-only services would have failed to obtain certificates, and would have kept failing quietly on every renewal attempt, in a way nobody looks at until a browser complains.
A total failure announces itself. A configuration that works for the things you check first, and fails for the things you check last, gets to survive for a long time.
What actually found it
Reading the configuration file against the prose that described it.
The runbook was internally inconsistent in a specific and detectable way: one section provisioned a credential for a mechanism, the next section’s prose claimed the mechanism was in use, and the configuration between them never referenced it. Each part was individually plausible. Only the combination was wrong.
That is not a subtle bug once it is stated. It is very hard to see when reading a document top to bottom, because each section reads as correct and the error lives in the space between them.
The fix, and the trap it leaves behind
A single directive in the file’s global options block, naming the challenge method and the provider, and reading the API token from the environment rather than sitting in the file:
acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}Being in the global block is the important part. It makes the DNS challenge the default for every site in the file, so individual site definitions need no certificate configuration at all.
That convenience is also the trap. Because a site definition carries no certificate configuration of its own, a site definition is not self-contained. Copying one into a fresh configuration file that lacks the global block produces something that looks complete, parses cleanly, starts successfully, and quietly falls back to the wrong challenge method for that site.
The failure will look identical to this one: fine for anything publicly reachable, broken for anything internal.
What generalizes
Read config files against the prose that introduces them. Execution proves that a file is syntactically valid and that the happy path runs. It does not prove that the file does what the paragraph above it says it does. Those are different claims, and only one of them is checked by running the thing.
Inherited defaults make snippets unsafe to copy. Any configuration where behavior comes from a global block is a configuration where a fragment cannot be judged on its own. That is a good design for reducing repetition and a bad one for copy and paste, and the two facts do not cancel out. Wherever a component relies on inherited configuration to be correct, that dependency is worth stating at the point where somebody is most likely to copy it.
Ask what happens on the path you did not test. The reason this defect was worth catching early is not that it was hard to fix, because it was one line. It is that the natural verification, bring up the service and load the public site in a browser, would have returned a clean result while the internal half of the design was broken.