A script set out to answer a simple question after a large photo import: how many photos got stuck partway through processing and never finished? It read half a million log lines and answered: zero. Nothing stuck. A clean result, exactly the number anyone would have hoped for.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
It was wrong by several hundred, and the reason is a better lesson than the number itself: the bug didn’t produce an error. It produced a completely plausible, reassuring answer.
The setup
Tens of thousands of photos had just been imported in one batch. Confirming they’d all made it all the way through processing meant comparing two sets pulled from the import tool’s own log: every file the tool had discovered, against every file that had reached some final, terminal outcome. Whatever was in the first set but missing from the second was stuck.
What actually happened
The comparison came back empty. Discovered and finished matched exactly, every time, for every file. Read at face value, that’s the best possible outcome — a completely successful import with nothing left hanging.
The pattern used to pull a filename out of each log line stopped at the first space it found. The real filenames in this log were not quoted, and plenty of them legitimately contained spaces — folder names, trip names, ordinary photo album naming. Every one of those got cut off at its first word. A folder full of photos named with a shared prefix collapsed into a single repeated string, over and over, because the script only ever extracted the part before the first space, and enormous numbers of these paths shared that much. Ninety-five thousand distinct files became twenty-five identical strings. Every one of those twenty-five appeared to have both a discovery event and a finishing event, so the difference between the two sets came back empty — not because nothing was stuck, but because almost everything had been silently merged into the same handful of look-alike entries.
Why it went unnoticed
A bug like this doesn’t fail loudly, and it doesn’t produce a number that looks suspicious on its own. A crash or a clearly wrong result invites scrutiny. This produced a zero — precisely the number a successful import should produce — built from real data that had simply lost its identity along the way. The totals were still derived from genuine log lines; only which file each line was actually about had gone missing.
The number that should have raised the alarm was never the zero itself. It was the twenty-five — the count of distinct paths the script had actually extracted, against a source log the import tool’s own summary said held tens of thousands of individual files. A results set that small, drawn from a source that large, is a number that cannot possibly be right on its own terms, independent of whatever conclusion it’s being used to support.
The fix
The extraction pattern was corrected to capture a filename to the end of its line rather than stopping at the first space, which immediately raised the discovered-file count from twenty-five to the true total — a number that matched the import tool’s own internally reported count exactly. The real backlog of stuck files turned out to be real, just far smaller than the original scope of the investigation, and measurable for the first time now that files were being told apart correctly.
Verifying it properly
The fix wasn’t trusted on the strength of “it returned a bigger, more plausible-looking number” either — a second wrong regex could just as easily have produced a different wrong number that also looked reasonable. What actually closed the question was building the check directly into the script: pass in an independently produced total — the import tool’s own summary line, a count that the script itself had no part in generating — and have the script compare its own result against that number and refuse to report a result if the two don’t match. Run this way, the original, broken version of the script would have failed its own control immediately, rather than quietly handing back a reassuring zero for someone else to trust.
What generalizes
A key that silently merges distinct records doesn’t produce an error. It produces a smaller, cleaner-looking dataset, and every number derived from it stays inside a completely believable range — because the arithmetic afterward is honest; only the identity feeding into it was already wrong. A crash or an obviously malformed value invites a second look almost by reflex. A quietly-collapsed dataset does not, which makes it more dangerous in direct proportion to how reasonable its output looks.
Before trusting a set comparison, check the size of the sets, not just the size of the difference between them. A suspiciously small or suspiciously round result on one side is often the first and only visible symptom, appearing well before the final answer does anything to earn suspicion on its own.
Wire an independently-produced number into a derivation as a control, wherever one already exists — a tool’s own summary line, a total from a completely different source — and have the derivation refuse to be trusted when it doesn’t match. A calculation that can’t be checked against anything outside itself isn’t a measurement. It’s an assertion wearing a measurement’s clothes, and the only way to tell the difference is to give it something real to be checked against.