The config file that reads top-down

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

A server on the public internet was accepting password logins over SSH for several weeks. Two separate configuration files said it was not. The deployment log recorded the setting as done, verified, and confirmed to survive a reboot.

All of that was true. It was also irrelevant, because of a single sentence in a manual page that runs contrary to how nearly every other program on the same machine behaves.

The short version, for anyone who wants only that: sshd_config is first-match-wins. Almost every other .d-style configuration directory on a Linux system is last-match-wins. If you have named an SSH hardening drop-in 99-something.conf by analogy with nginx or systemd, go and check it now:

sudo sshd -T | grep -i passwordauthentication

If that prints yes and you believe it should print no, this post is about you.

The setup

Locking down SSH is early, standard work: no password logins, no root login, key authentication only. The accepted modern practice is to leave the main /etc/ssh/sshd_config alone and drop your settings into a separate file in /etc/ssh/sshd_config.d/. Package upgrades leave your file alone, and your intent is visible in one place.

The file was written. It contained, among other things:

PasswordAuthentication no
PermitRootLogin no

It was named 99-homelab-hardening.conf.

That number was the entire problem.

Why 99

Because on a Linux system, 99- conventionally means last, and therefore final. That is how it works for nginx sites, for systemd drop-ins, for sysctl.d kernel parameters, for udev rules, for logrotate. Higher number, later read, wins. It is such a consistent convention that it stops registering as a choice.

sshd_config does the opposite. From its manual page:

Unless noted otherwise, for each keyword, the first obtained value will be used.

First. Not last. The first file to set a keyword locks it in, and every later file setting the same keyword is discarded silently — no warning, no log line, no syntax error.

What was actually on the machine

Ubuntu cloud images — the images essentially every VPS provider hands you — ship a file called 50-cloud-init.conf in that same directory. Its entire contents are one line:

PasswordAuthentication yes

Sorted by filename, the directory read like this:

Two files said no. One file, read first, said yes. Yes won.

Why it hid for so long

This is the part worth dwelling on, because the failure was not “the setting did not apply”. It was much more specific than that, and much better at hiding.

The hardening file was read, and most of it worked. The non-default port, the user allowlist, the root login prohibition, the cipher list — every one of those applied correctly, because nothing earlier in the directory set them. Only the keywords that 50-cloud-init.conf also mentioned were reverted.

So SSH looked hardened by every casual observation. It listened on the unusual port. It rejected root. It accepted the right users. Someone spot-checking the machine would have found a properly locked-down service, because the one keyword that had been quietly overridden happened to be invisible unless specifically tested.

And every way anyone checked it confirmed the wrong answer. Opening the file showed no. Grepping the file showed no. A copy of the file mirrored into version control earlier that same day showed no — because that is what the file said. The file was never the problem. The file was correct. The file was also not in force.

That is the general form of this failure, and this project has now hit it three separate ways: configuration present is not configuration in force.

How it was found

By accident, essentially, while answering a routine question: how do you turn off password authentication on SSH?

The honest first answer was “it is already off, here is the file”. Checking before saying so — a habit, not suspicion — produced:

$ sudo sshd -T | grep -i passwordauth
passwordauthentication yes

sshd -T prints the effective configuration after every include and precedence rule has been resolved. It is what the daemon will actually do, as opposed to what any individual file requests.

The fix

Rename the file so it sorts first:

/etc/ssh/sshd_config.d/00-homelab-hardening.conf

First-wins then works in your favour permanently. Your settings are locked in before anything else is read, and it no longer matters what the provider’s tooling writes later.

The alternative — editing 50-cloud-init.conf to say no — is worse, because cloud-init can regenerate that file. Fixing the symptom in a file owned by another system is a fix with an expiry date.

sudo sshd -t

sudo sshd -T | grep -iE '^(passwordauthentication|permitrootlogin|port|allowusers)'

before:  Authentications that can continue: publickey,password
after:   Authentications that can continue: publickey

The part that was nearly much worse

The same filename was written into the base VM template and into the configuration-management role used to build every machine in the lab. The same Ubuntu cloud images were to be used throughout.

Every virtual machine in the environment would have inherited this, silently, with the same convincing-looking configuration file in each one.

It was caught before the first VM was built. Not through foresight — through a routine question asked at the right moment. That is not a repeatable process, which is why the lesson got written into the standing rules rather than just fixed.

What actually changed

Three things, in increasing order of usefulness:

1. The file is now 00-, in the runbooks and in the automation.
2. The standing instructions now say: verify SSH settings with sshd -T, never by reading the drop-in. The file shows intent; -T shows effect.
3. The broader rule, which generalizes past SSH entirely: a configuration file being present is not evidence of it being in force. This lab has now been bitten by that three ways — a firewall rule that failed to load on every boot while its file sat correct on disk, a script that existed but was never wired to anything that ran it, and this.

The common thread in all three: the check that was performed looked at the artifact instead of the behaviour. Reading a file tells you what someone intended. Only asking the running system tells you what is true.

That idea has a sharper cousin, which cost this project rather more: a check that cannot fail is worse than no check at all.