Every web request that reaches this lab from the internet arrives at a reverse proxy first. That proxy terminates the encrypted connection, works out which internal service the request is for, and forwards it there. It is also responsible for obtaining and renewing the certificates that make the encryption possible, for every name the lab serves.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
Caddy does that job on both gateways. The choice looks like a preference between similar tools and is not: it is the reason one of this architecture’s stricter rules is affordable at all.
What it replaced
nginx and Traefik, both of which would have worked.
nginx is the default answer and the one most readers will have used. It is fast, it is everywhere, and every problem with it has been written up somewhere. Traefik is the modern container-oriented option with automatic certificate handling of its own.
The deciding difference is not performance. At this lab’s request volume, all three are so far from their limits that the comparison is meaningless. It is what each one treats as the normal case.
Why this one
Automatic certificate management is the default rather than a bolt-on, and specifically, obtaining certificates over a DNS challenge is a first-class feature rather than an add-on to be assembled.
That sounds like a detail about convenience. It is actually what makes the architecture’s TLS rule possible.
The rule is this: every human-facing web surface in the lab gets a real, publicly trusted certificate. Not just the services exposed to the internet. The internal dashboards, the storage appliance’s admin interface, the hypervisor’s own console, everything a person points a browser at and logs into. No self-signed certificates anywhere, and no private certificate authority to install on every device.
The usual obstacle is that the standard way to prove you control a name is to serve a file over HTTP on port 80 at that name, from the public internet. An internal-only service cannot do that, because being unreachable from the internet is the entire point of it.
A DNS challenge proves control differently: by writing a record into the domain’s DNS zone through the provider’s API. Nothing has to be publicly reachable. So an internal-only name gets a genuine certificate from a public authority without ever being exposed, and port 80 never needs to be open for certificate purposes at all.
Caddy will do this with a few lines of configuration and then handle renewal without being asked again. The alternatives can be made to do it. The difference between a tool that does something by default and a tool that can be made to do it is the difference between a rule you apply everywhere and a rule you apply where you had the patience.
The second reason is smaller and it is not nothing: the configuration file stays readable. Reverse proxy configuration is read almost exclusively under bad conditions, by someone who did not write it, while something is broken.
How it is actually used here
Live on both gateways, serving real certificates, with the standby gateway’s certificates proven against Let’s Encrypt’s staging environment before being issued for real. That ordering matters: the production certificate service enforces rate limits that are easy to exhaust while a configuration is still wrong, and exhausting them means waiting rather than fixing.
Each gateway holds an API token for the DNS provider, which is what the challenge writes through. Caddy terminates TLS, then hands requests down the tunnel to the services running at home.
The shape of it, and one property worth relying on
The configuration names the certificate authority once, points it at a DNS provider module, and reads the provider’s API token from the environment rather than from the config file itself. Individual sites then need no certificate configuration at all: naming a site is enough to have a certificate obtained and renewed for it.
The property worth knowing about is that reloads are atomic. When a new configuration is loaded, it is validated and instantiated before it takes over, and if any part of it fails, the reload is rejected and the previous configuration keeps serving. Ports stay bound. Certificates stay loaded. Nothing drops.
That has a consequence that runs both ways, and the second half is the one that catches people. A failed reload is not evidence of an outage, so panic is unwarranted. But a running server after a failed reload is also not evidence that the new configuration is live, so relief is unwarranted too. Confirming which configuration is actually serving means checking that the reload succeeded, not that the process is up.
What it cost
A standing credential on an internet-facing machine. The DNS challenge is what makes universal TLS possible, and it works by writing to the domain’s DNS zone, which means each internet-facing gateway holds a token that can modify DNS records. That is a real trade. The lab closed port 80 as a certificate dependency and opened a credential that, if stolen, is more valuable than the port ever was. It is scoped as tightly as the provider allows and it is still the honest cost of the design.
Certificates are public, and so are the names in them. This is the one that surprises people, and it is a property of the certificate ecosystem rather than of Caddy.
Every certificate issued by a public authority is published to Certificate Transparency logs at the moment of issuance. Those logs are public and searchable. So the moment an internal-only service gets a real certificate, its hostname appears in a public record, whether or not any public DNS entry for it exists.
An attacker does not have to guess that the lab runs an internal metrics dashboard or a storage appliance under a particular name. The transparency logs hand over the complete list.
This does not weaken the network boundary. Those services genuinely are unreachable from outside, and knowing a name is not the same as reaching it. But it does mean **”internal-only” has to be read as not reachable from outside, never as *hidden***, and the lab’s documentation was changed to say it that way. A naming scheme is a reachability control. It was never a secrecy control, and treating it as one would have been building on an assumption that was already false.
The operational surface, which arrived twice from the same direction. Two separate incidents in this lab traced to the same underlying issue: a log file created with the wrong owner, leaving Caddy unable to open its own log. The second time, the fault was created by the validation step meant to prevent it. Validating the configuration as the wrong user instantiated the log writer, which created the file, owned by the wrong account. The validation reported success, correctly, because the configuration really was valid. The damage was a side effect of how it had been checked.
Neither incident caused an outage, for the atomic-reload reason above. Both are written up in the lab’s errata and neither has been covered here yet.
What would change the decision
Two conditions, and unlike the Proxmox note these are the agent’s reasoning from the record rather than the human’s stated position.
- If the DNS challenge stopped being a first-class feature. It is the entire basis of the choice. A version that made it awkward again would remove the reason Caddy is here.
- If the configuration stopped scaling with the service count. The readability argument is a claim about a file that currently describes a modest number of sites. Sixteen services in, if that file is no longer something a person can read while something is broken, the second reason has expired even if the first has not.
Notably, neither condition is about speed, and that is deliberate. The performance comparison that dominates discussions of reverse proxies was not a factor in this decision and would not be a factor in reversing it.
About Caddy
Caddy is an open source web server and reverse proxy written in Go, developed by Light Code Labs and a community of contributors. Its distinguishing feature is that it obtains and renews TLS certificates automatically by default, including for internal names via DNS-based validation. It is distributed as a single binary with no runtime dependencies and is released under the Apache License 2.0.
- Website — caddyserver.com
- Documentation — caddyserver.com/docs
- Source — github.com/caddyserver/caddy
- Community — caddy.community