The git server’s nightly backup is a single archive of everything it holds. When the package registry on the same server grew, that archive grew roughly eightfold in a day, because the registry’s contents were being re-compressed into every nightly copy. The decision was to take the registry blobs out of the dump and back them up on their own schedule with the tool built for exactly that job.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
Why the dump was the wrong container
The server’s dump command packs the registry’s data directory into the zip alongside the database, repositories and configuration. With seven nightly archives kept, the registry’s bytes were being copied seven times over, every night, in a form that cannot be compared with yesterday’s.
Those bytes are content-addressed: each blob is named by its own hash, and a blob never changes once written. That is the case deduplicating, incremental backup tools are designed for, and a nightly full copy is the worst fit for it.
| Carries | How | |
|---|---|---|
| Nightly dump | Database, including package metadata; repositories; configuration | The server’s dump with package data skipped, seven kept |
| Registry blobs | The registry’s data directory | The fleet’s existing incremental backup tool, its own tag and retention, same repository and credentials as the host’s other backups |
A restore needs both halves. The database says which blob is which image, tag or file, and the snapshot holds the blobs. Neither is a working registry alone. A restore drill brings both back into a throwaway server and checks that a known package is served.
Order matters: blobs after the dump
Blobs are only added or removed by cleanup, never edited, so the two halves can disagree in two ways after a restore. A blob newer than the dump’s database has no row pointing at it, which is harmless orphan data. A database row whose blob is missing is the dangerous case: the image lists and then returns not-found.
Snapshotting the blobs before the dump would create the dangerous case, because anything pushed between the two would be in the database and absent from the snapshot. So the blob backup runs after the dump. The timer is set later, the unit is ordered after the dump service, and the script waits while the dump is still active. A registry cleanup that lands between the two can still produce the harmless case, and cannot produce the dangerous one. The drill checks for it directly: every blob hash in the restored database must exist among the restored blobs.
Two guards that changed meaning
The dump script ends with guards that inspect the archive. One of them used to require that the archive contained the registry. It now fails any archive that carries registry data at all. If a dropped or misspelled flag put the registry back in the dump, the archives would silently grow again while every other guard passed.
The old guard’s real job, confirming that the backup actually covers the registry rather than merely that a backup exists, moved with the blobs. The registry job fails if the snapshot’s restore size is under half of the live store.
Implementation notes
The backup tool on that host is managed by the configuration-management role like the rest of the fleet’s, so the new job is a map entry in that role, not a second pattern next to it. The existing backup job and the new one serialize on a shared lock, because each one mounts and unmounts the same network share. No new credential was created; the existing escrow entry for the host’s backup password already covers it. The dump retention stayed at seven, since one dump without the registry is small, and the oversized archive from the day of the change stays in rotation as a valid full backup until it ages out.
What would change this
The split is worth its two moving parts only while the registry is large relative to everything else. If the registry shrank back to a few images, a single dump would be simpler, and the extra restore step would cost more than the duplication it saved.