The mail server’s backup had run every night for weeks, succeeded every time, reported real byte counts, and passed its size floor. MONITORING.md counted mail as covered. None of that turned out to mean the mail server could actually come back.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
A backup that has never been restored is not a backup. It’s a claim, and this project spent a week finding out how many ways a claim can look true.
The setup
The mail server stores its data in an embedded database, backed up nightly by tarring the data directory into an encrypted, offsite repository. The same pattern already worked for two other services on the same gateway, so it was written by following that example.
What actually happened
The database is written continuously while the mail server runs, compacting files in the background — creating and deleting them, rewriting its own manifest — while tar walks the directory reading whatever it finds when it gets there. The archive captured files from different moments of that process: a torn copy. It might restore. It might restore missing recent mail. It might not open at all, and nothing in the pipeline could tell which, because every component — the pull, the encryption, the offsite copy, the size check — did its job correctly. There was no symptom. That’s what makes this worth writing up.
The fix looked done once it used the mail server’s own documented export command instead of a raw file copy: stop the server, export, restart, archive the export. Byte-identical on restore, verified through the backup chain. Then someone actually tried loading that export into a fresh install, which the same vendor documentation describes as the restore procedure, and it failed on the second of nineteen internal tables:
Cannot import ...: the target database already contains data in the
key range being imported. This usually means the server was started
before the import ran, which can create duplicate entries.The store had never been started. It didn’t exist until the import command created it. The importer’s own first action collides with the store it just initialized, then blames the operator for a step nobody took. Because the run aborted partway, most of what had already imported was never flushed — not a partial restore, effectively a failed one.
Why it went unnoticed
Every signal along the way was green, and each one measured something real that simply wasn’t the thing that mattered. The job ran. The bytes moved. The export matched the source byte-for-byte. The alert path was proven by stopping the service and watching the notification arrive. A consistent copy and a restorable one are different properties, and only one of them can be checked without actually performing a restore onto a machine that didn’t already have a working copy.
The same question, asked of every other database on the fleet once this one turned up a defect, found a second, quieter version of it. Every backup script used the right mechanism — a proper database dump, not a file copy — and a routine audit of the scripts themselves would have shown a clean sweep. Listing what the resulting archives actually contained instead turned up a stray copy of the live database sitting beside the good dump in several of them, pulled in by a generic “tar this whole directory” helper that had no way to know one of the directories it was given contained a database. Nothing had been lost — the good dump was still there — but a person restoring later would reach for the archive named after the service, not the one two lines down in a file listing, and get a copy indistinguishable from the real one until it wasn’t.
The fix
Two changes, aimed at two different failure modes:
The restore path no longer uses the vendor’s export/import pair at all. The server is stopped anyway for a consistent copy, so the backup now also takes a cold tar of the store directory in the same window — free, since the service is already down for the two seconds the export takes. Both are kept: the cold tar is what actually restores, proven on a separate machine; the export is kept alongside it as a portable format for the day the on-disk format itself changes and a raw copy stops being readable by a newer version. Neither replaces the other.
The generic file-backup helper now excludes database files globally, rather than being fixed per service, because the point generalizes: any service on that host with its own consistent dump has no business also appearing inside a wholesale directory tar.
Verifying it properly
The tempting verification is a checksum match between the backup and the source — which is exactly what passed the whole time this was broken. The check that actually proves something is a full restore drill run on different hardware from the one still holding a working copy: rebuild the store from the offsite archive alone, start the real service against it, and compare what it reports to what was there at backup time. Nineteen of nineteen internal tables present, matched byte-for-byte against a fresh export of the restored store — including the specific table the documented import path silently lost.
The file-tar fix was verified the same way, by effect rather than inspection: entry counts in each archive dropped by exactly one file per affected service, and the archive that is supposed to contain a database copy on purpose — because tarring it deliberately, with a written reason, is different from tarring it by accident — was checked to confirm that one still came through unchanged.
What generalizes
A backup pipeline reports on itself, not on its restorability. Every stage between “copy the data” and “an offsite repository holds the bytes” can be instrumented and can pass. None of those stages can tell you whether the result opens on a machine that has never seen the original. Two rules follow:
- For any backup of something that’s being written continuously, ask whether the source is a database. If it is, the only acceptable copy is a consistent dump or a stop, never a file walk mid-write — and a generic “back this whole directory up” helper has to know that, not just the author of the one script that happens to call it correctly.
- A backup nobody has restored is a claim, not a backup, and drilling it on the same machine that already has a working copy doesn’t test the case that matters. The case that matters is the one where the original is gone. That drill is also the only way to find out that a vendor’s own documented restore command doesn’t do what its own documentation says — which happened here, and would otherwise have been discovered during an actual outage rather than a twenty-minute rehearsal.
The tool built to catch the second failure across the rest of the fleet shipped, on its first run, with its own broken positive control — a transport layer that silently swallowed its results, so “no service has this defect” and “the check never actually looked” produced the identical output. That’s the same defect one level up: a check that cannot fail, or cannot even report that it looked, is worse than no check, because it is trusted more.