The config file that reads top-down

A server on the public internet accepted password logins over SSH for several weeks. Two configuration files said it did not. The deployment log recorded the setting as done, verified, and confirmed to survive a reboot.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

All of that was true, and none of it mattered, because of one sentence in a manual page that runs contrary to how almost every other program on the same machine behaves.

For anyone who wants only the actionable part: sshd_config is first-match-wins, where nearly every other .d-style config directory on Linux is last-match-wins. If you named an SSH hardening drop-in 99-something.conf by analogy with nginx or systemd, check it now:

sudo sshd -T | grep -i passwordauthentication

If that prints yes and you believe it should print no, this post is about you.

The setup

Locking down SSH is early, standard work: no password logins, no root login, keys only. Modern practice leaves /etc/ssh/sshd_config alone and drops settings into a separate file in /etc/ssh/sshd_config.d/, so package upgrades leave your file alone and your intent sits in one place.

The file was written, containing among other things:

PasswordAuthentication no
PermitRootLogin no

It was named 99-homelab-hardening.conf, and that number was the entire problem.

Why 99

On Linux, 99- conventionally means last and therefore final. That is how it works for nginx sites, systemd drop-ins, sysctl.d parameters, udev rules and logrotate. Higher number, later read, wins. The convention is consistent enough that it stops registering as a choice.

sshd_config does the opposite. From its manual page: “Unless noted otherwise, for each keyword, the first obtained value will be used.”

First, not last. The first file to set a keyword locks it in, and every later file setting the same keyword is discarded silently. No warning, no log line, no syntax error.

What was actually on the machine

Ubuntu cloud images — what essentially every VPS provider hands you — ship a file called 50-cloud-init.conf in that same directory. Its entire contents are one line: PasswordAuthentication yes.

Sorted by filename, the directory read like this:

Two files said no. One file, read first, said yes.

Why it hid for so long

The failure was not “the setting did not apply”. It was narrower than that, and much better at hiding.

The hardening file was read, and most of it worked. The non-default port, the user allowlist, the root login prohibition and the cipher list all applied, because nothing earlier set them. Only the keywords 50-cloud-init.conf also mentioned were reverted.

So SSH looked hardened by every casual observation. It listened on the unusual port, rejected root, accepted the right users. Anyone spot-checking the machine would have found a properly locked-down service, because the one overridden keyword was invisible unless specifically tested.

And every way anyone did check confirmed the wrong answer. Opening the file showed no. Grepping it showed no. A copy mirrored into version control the same day showed no, because that is what the file said. The file was never the problem. The file was correct, and not in force.

That is the general shape, and this project has now met it three separate ways: configuration present is not configuration in force.

How it was found

By accident, while answering a routine question about turning off password authentication.

The honest first answer was “it is already off, here is the file”. Checking before saying so — habit, not suspicion — produced:

$ sudo sshd -T | grep -i passwordauth
passwordauthentication yes

sshd -T prints the effective configuration after every include and precedence rule resolves. It is what the daemon will do, rather than what any individual file requests.

The fix

Rename the file so it sorts first: /etc/ssh/sshd_config.d/00-homelab-hardening.conf.

First-wins then works in your favor permanently. The settings lock in before anything else is read, and it stops mattering what the provider’s tooling writes later.

The alternative — editing 50-cloud-init.conf to say no — is worse, because cloud-init can regenerate that file. Fixing a symptom inside a file another system owns is a fix with an expiry date.

sudo sshd -t

sudo sshd -T | grep -iE '^(passwordauthentication|permitrootlogin|port|allowusers)'

before:  Authentications that can continue: publickey,password
after:   Authentications that can continue: publickey

The part that was nearly much worse

The same filename was written into the base VM template and into the configuration-management role used to build every machine in the lab, targeting the same Ubuntu cloud images.

Every virtual machine in the environment would have inherited this silently, each with the same convincing-looking configuration file.

It was caught before the first VM was built — not through foresight, but through a routine question asked at the right moment. That is not a repeatable process, which is why the lesson went into the standing rules rather than just getting fixed.

What actually changed

Three things, in increasing order of usefulness.

The file is now 00-, in the runbooks and in the automation.

The standing instructions now say to verify SSH settings with sshd -T and never by reading the drop-in. The file shows intent; -T shows effect.

And the broader rule, which generalizes past SSH: a configuration file being present is not evidence of it being in force. This lab has been bitten three ways — a firewall rule that failed to load on every boot while its file sat correct on disk, a script that existed but was never wired to anything that ran it, and this.

The common thread is that the check performed looked at the artifact instead of the behavior. Reading a file tells you what someone intended. Only asking the running system tells you what is true.

That idea has a sharper cousin, which cost rather more: a check that cannot fail is worse than no check at all.