Everything built so far has been infrastructure for the lab’s own benefit: gateways, tunnels, DNS, identity, monitoring, backups. Useful, and none of it something a person opens on purpose.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
That changed overnight. A bookmark archive, a webmail client and a news reader are live, all behind single sign-on. A notes and wiki service is running alongside them and is the most interesting one, because it is not finished and the record says so plainly.
What went in
Three services, complete by this project’s own definition. Each has an acceptance test that can be re-run, a monitor proven by stopping the service and watching the alert arrive, nightly local backups, and an offsite copy. The test suites between them run to seventy checks with nine induced failures exercised.
The webmail client is worth a note because it has no single sign-on support of its own. It is gated by the identity provider acting as a proxy in front of it, which is exactly the capability the Authentik note argued was the deciding reason to choose it. Eight of the planned services cannot do SSO natively. This is the first one to actually need the workaround, and it works.
Log shipping was re-measured and the old claim retired. The monitoring document had said for six days that log shipping was active everywhere and delivering to nothing. That was true when written and stopped being true when the log aggregator was built. Thirteen hosts are now confirmed shipping, verified by querying the aggregator for which hosts it has actually heard from, rather than by asking each host whether it thinks it is sending.
What is not finished, stated rather than counted
The notes service has no backup of any kind. Its database sits on local disk, in no repository and under no dump timer, on a machine where every other service reached that same state during its build and then had it closed.
Nobody has ever logged into it. Its single sign-on client is configured and has never had a login attempted, so whether a user can get in, and with what permissions, is unknown.
That second one is sharper than it sounds, and the reason is in the record: an earlier service’s login succeeded and silently granted the wrong role. So the standard is not that a login works. It is that a login produces the permissions it was supposed to produce, verified. Until that has been done, the service has an authentication path that has never been exercised in either direction.
Two backup findings from the same day, in opposite directions
A backup that would have backed up nothing. The news reader was assumed to store its data in the database container running beside it. The backup was written accordingly. The container holds zero tables: the application actually uses a file-based database on disk, and a dump of the database server would have produced a valid, complete, empty archive every night.
Nothing would have failed. The backup would have run, succeeded, reported success, and contained none of the data.
A perfect backup that failed nightly for two days. The opposite case, on the secrets vault. The check that verified the offsite backup asked for the latest snapshot and got an answer it did not expect: the tool returns one snapshot per group of paths, not one snapshot overall. A deliberately frozen canary file, kept unchanged on purpose, was returned as “the latest snapshot” and looked stale, so a backup that was working correctly reported failure every night.
Both were found on the same day. One is a check that could not fail, the other a check that could not pass, and between them they are most of what this blog is about.
Where that leaves the build
| Public edge | Two gateways, TLS, tunnel live |
| Identity | Live, backed up, restore rehearsed |
| Observability | Metrics, logs, dashboards, alerting, thirteen hosts |
| Security monitoring | Live, eight agents |
| Offsite backups | Live, eight encrypted repositories |
| Physical servers | Two online, not clustered, no witness |
| Productivity services | Three complete, one live and unfinished |
| Storage | Integrity tier blocked on a boot adapter |
| Failover | Scripts exist, never drilled |
Twenty-eight components complete, eight in progress, twenty-nine not started. Four days ago that was twenty-three, eight and thirty-four.
Next
Close the notes service: a backup, and a login actually performed and checked for the role it grants. Then the backup keys, which still exist only on the machines they protect, and the quorum witness that turns two servers into a cluster.
Full detail, including what is verified and what is merely asserted, is on the Status page.