Validation is not a read-only operation

A web server refused to start. The error named a permission problem on its own log file, on a machine where the log directory was owned correctly and the permissions had been set exactly as the runbook specified.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

error: opening log writer: permission denied
systemd: Reload failed for caddy.service

The file that could not be opened had been created by the command run to check that the configuration was correct.

What happened

The service runs as an unprivileged account. The configuration check was run with sudo, which is the reflex, because most administrative commands need it.

Checking a configuration sounds like a read-only operation. It is not. To verify the configuration, the tool builds the components that configuration describes, and one of those components is the thing that writes the log file. Building it creates the file. The file is therefore owned by whoever ran the check, which was root.

The service then started as its own unprivileged user, found a log file it did not own, and could not write to it.

The detail that makes this stick:

Directory permissions govern creating new entries. An existing file’s own ownership governs writing to it. The directory was correct. The directory being correct is what let the wrong-owner file be created in the first place, and it had no bearing at all on whether the service could then use it.

Why it went unnoticed

Because the check passed, and it was right to pass.

The configuration genuinely was valid. The tool parsed it, built it, found no problem, and reported success. Nothing about that output was inaccurate. The damage was not in what was checked but in how, and a validator has no way to report a side effect it does not consider a fault.

So the sequence reads as an unbroken run of good news right up to the failure: configuration written, validation passed, service started, permission denied on a file nobody had knowingly created.

It happened twice

That is the part actually worth writing down.

The first occurrence was on the primary gateway in late July, when the service would not start for the first time. It was diagnosed correctly, and fixed by changing the ownership of the files that had been created wrongly.

Two and a half weeks later the same fault appeared on the standby gateway while adding access logging.

The first fix was a correct fix to an instance. The ownership was repaired, the service started, and the incident was closed. What did not change was the procedure that produced it, which was still run the check with sudo, then start the service. So the fault was not fixed, it was cleaned up, and it was waiting on the next host that ran the same steps.

There is a reliable tell for this, and it is worth learning to hear: a fix phrased as an action taken on a file, rather than a change to a step, has almost always treated an instance. chown the log files is a repair. validate as the service account is a fix. The first one leaves the machine correct and the runbook wrong.

sudo chown caddy:caddy /var/log/caddy/*.log
sudo chmod 640 /var/log/caddy/*.log

sudo -u caddy caddy validate --config /tmp/Caddyfile.new --adapter caddyfile

The same command, a second wrong answer

The provisioning behavior that creates files has another consequence, found on a different host weeks later. Because validation builds the modules rather than reading the file, its verdict depends on the environment it is run in.

A configuration that was correct, on a host whose service was running normally, failed validation like this:

Error: loading module 'cloudflare': provision dns.providers.cloudflare:
API token '' appears invalid; ensure it's correctly entered and not wrapped
in braces nor quotes

The configuration reads its API token from an environment variable. The service gets that variable from a systemd EnvironmentFile, which applies to the service, not to an interactive sudo shell. So in that shell the variable was empty, the module rejected an empty token, and validation failed on a configuration with nothing wrong with it.

Read the message closely and it says the token is ''. Empty. It is describing the value validation saw, which is not the value the service sees. The diagnostic is accurate about its own situation and misleading about yours.

The dangerous part is that the plausible fix is the wrong one. Read that error at two in the morning and the obvious conclusion is that the variable is not being read, so put the token directly in the config file instead. That file is world-readable. A confusing diagnostic that makes leaking a credential look like the remedy deserves more attention than the configuration bug it is impersonating.

The correct move is to supply a dummy value for the structural check, since no credential is actually used at provision time:

sudo env CLOUDFLARE_API_TOKEN=<dummy> caddy validate --config <file>

And to check it as a pair, which is the habit this project keeps arriving back at: a dummy value returns a valid configuration, an empty one returns the error above. Two results that fail for different reasons, with the positive one gating the verdict.

The same shape, one command away

While investigating, a command to count the lines in one of those logs returned nothing at all, rather than an error:

sudo -n wc -l < /var/log/caddy/example-access.log

The redirect is performed by the shell, which is running as the calling user, before sudo runs anything. So the file is opened without privilege, the open fails, and the command that was supposed to inspect the file reports an empty result instead of a failure.

An empty answer that looks like a real answer, produced by a privilege boundary in a place nobody was thinking about. Use sudo wc -l <file> and let the privileged process do its own opening.

What generalizes

Read-only is a claim about intent, not a property of a command. Anything that instantiates, renders, compiles, or dry-runs may touch the filesystem on the way. If the thing being described includes an output destination, checking the description may create the destination.

A check that runs in a different context from the component is not checking that component. Account and environment both count. The check’s whole purpose is to predict what happens when the real thing runs, so running it as a different user, or with a different set of environment variables, makes it a prediction about a situation that will never occur.

That failure has now arrived from three directions in this lab within a single day: a validation run as the wrong user, a validation run without the service’s environment, and a container file-read test that passed because the test ran as root while the service does not. In the last case the file was genuinely unreadable by the process that needed it, and the check said it was fine.

Suspect an error message that recommends something unsafe. A diagnostic describing the checker’s situation rather than the service’s can point directly at a remedy that makes things worse. When the obvious fix to a confusing error involves moving a credential somewhere more accessible, that is the moment to stop and establish what the error is actually describing.

A fix that names a file has probably not fixed anything. The test is whether a fresh host following the same procedure would hit it again. If yes, the incident was cleaned up rather than closed, and the record should say so, because the next occurrence will be on a different machine weeks later and will look like a new problem.