Single sign-on is running. That is the tier everything above it has been waiting on since the architecture was written, and the one the runbooks warn against skipping in the strongest terms available.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
Every service still to come assumes it exists. It now does.
What went in
The identity provider is live on its own virtual machine, behind a native reverse proxy, with both access paths proven rather than assumed.
It was deployed at current stable, not the version the runbook pins — which is about twenty months old. That is the pinning correction from an earlier review finally being applied rather than merely written down: pin to current stable and treat published vulnerability lists as a rejection floor, never as a selection criterion. A runbook that names a version ages badly by default.
Nightly backups exist, with three guards, each one proven to fire. Not “backups are configured” — each guard was made to trigger deliberately before being trusted. This project has already published what happens when a backup check inspects a file it invented, so the standard is now that a guard is not a guard until it has been seen to stop something.
The machine baseline ran clean and idempotent — 64 tasks, 44 changed, and then zero changed on the re-run. That second number is the one that matters. A configuration run that is not idempotent is a script, not a description of state.
What broke
The VM template was handing out a dead key. Its cloud-init configuration still carried a key whose private half had been destroyed earlier the same day, so every clone was born unreachable and said nothing about it until someone tried to log in.
Fixed, and the fix was accepted the right way: clone to a throwaway machine, boot it once, connect with the replacement key under strict host key checking and with no post-boot correction. Negative control — a real but unauthorized key was refused. Then the test machine was destroyed. That is a much stronger acceptance than re-reading the configuration file, which is what would have “confirmed” the broken version too.
The failover scripts could not send an alert. Both reference a token that is not defined on either gateway, and both run with strict error handling, so the first reference aborts the script.
The consequences are worth stating separately, because one is much worse:
- The failover script would have changed every public DNS record and then crashed before notifying anyone. A real failover completing in silence.
- The failback script correctly refused to run, then crashed while sending the message saying it had refused.
Both fail at the exact moment the operator most needs to hear something, and both are silent by construction — the thing that would have told you is the thing that died.
Found by running the refusal path rather than reasoning about it. A missing token now degrades to logged, not sent, and a failed send is logged rather than swallowed.
And turning off nested virtualization did not turn it off. The flag was removed, the machine power-cycled, the configuration confirmed — and the guest’s hypervisor still started one second after every boot. Half a day, two reboots and a forced power-off later: the CPU model in use is a passthrough model, so removing the flag stops asking for the feature while doing nothing to stop the host handing it over anyway. It has to be masked explicitly. That is a coda to the post on those freezes.
Where that leaves the build
| Public edge | Two gateways, both serving TLS, tunnel live |
| Identity | Single sign-on live, backed up, guards proven |
| Virtual machines | 5 of 16 |
| Internal DNS | Live on both resolvers |
| Monitoring | Eight monitors; external watchdog live, email path unproven |
| Failover automation | Scripts exist, alert path repaired, never drilled |
| Hypervisor cluster | Still one node |
That failover row is the honest one. The scripts are written and their alerting now works, but a real failover has never been performed. Until it has, the capability is a plan.
Next
Services on top of identity, which is the first time in this project that the next step has been an ordinary one — build a thing, put it behind the login that already exists.
And the mail server, which is what this site’s corrections route is waiting on.