A SIGPIPE fix that kept its pipe

The nightly backup of the git server failed twice in two weeks with the message “archive contains no database dump”. Both archives did contain the database dump. The alert fired correctly, no data was lost, and the next night’s archive was fine. The cause was a shell fix from August that had removed the pipe’s symptom and left the pipe in place.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

The original problem and the fix that missed

The backup script ends with a series of guards that inspect the finished archive before keeping it. One of them lists the archive and checks that the database dump is among the files. Originally that was unzip -l | grep -q, and under set -o pipefail it produced false failures: grep -q exits at its first match, the writer upstream gets a broken pipe, and pipefail reports the whole pipeline as failed even though the answer was yes.

The August fix stored the listing in a variable first, then fed it to grep with printf '%s' "$LISTING" | grep -qE 'gitea-db\.sql'. That looks like a different shape, but it is the same race with a different writer. printf is a shell builtin writing the listing into a pipe, grep -q can still exit before printf has finished, and printf then gets a write error and returns non-zero.

The race stayed invisible for months because the listing fit inside the kernel’s 64 KB pipe buffer. printf always finished writing before grep could exit early, so there was nothing to lose the race against. Then the package registry grew, the archive listing reached about 178 KB, and the database dump’s entry sat near the end of it. Once the listing no longer fit in the buffer, printf was still writing when grep found its match and left.

Replaying it

The agent replayed the guard’s exact pipeline against a known-good archive, read-only, on the host.

GuardRunsFalse failures
old, `printf … \grep -q`503
old, `printf … \grep -q`20021
new, grep -q ... <<<"$LISTING"2000
new, with the database line removed1correctly fails

The last row is the one that matters for trusting the first three. A guard that passes 200 times proves little until the same guard has been made to fail on purpose.

The fix and what it left alone

The replacement is grep -qE 'gitea-db\.sql' <<<"$LISTING". A here-string has no pipe, so there is no writer to kill and nothing to race. The next guard in the script, which feeds the same listing to awk through a pipe, was left as it was: awk reads all of its input before it exits, so it never closes the pipe early.

Any X | grep -q, X | head or X | grep -m1 under pipefail has the same weakness. It is harmless while X’s output fits in the buffer and becomes a coin flip as soon as it does not.

What the review missed

The August change was reviewed and it looked right. It did address the symptom the original failure showed. What nobody asked was whether the pipe itself was the problem, and a correct-looking diff cannot answer that, because the failure needs an input larger than anything the review was looking at. A fix for a race has to remove the race, not move the participants around it. The record for this one is blunt about the general form: a fix for “pipefail plus an early-exiting reader” has to remove the pipe, not move the writer.