The check that worked, and was ignored

A 120 GB virtual disk was being moved from SSD to spinning disk. The SSH session running it timed out and dropped. The move kept going on the server, as it should.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

A recovery script then ran. Its first action was a check, and the check was correct:

move still running server-side: 5

Its second action deleted that move’s target directory.

The disk was destroyed. This one is worth writing up not because a check failed, but because it did not.

What the check was, and what it wasn’t

The script both detected and acted. The detection worked perfectly: it found the running process, counted it, and printed the truth.

Then the next line ran regardless.

sudo rm -rf /mnt/pve/hdd/images/105          # the in-flight target
sudo qm move-disk 105 sata0 hdd --delete 1 & # a duplicate move

The destructive step was never gated on the answer the script had just obtained. The check was an echo, not an if. qemu-img carried on writing into a now-unlinked inode, which is to say into nothing.

Eleven earlier entries in this project’s errata file are checks that could not fail. This is the inverse: a check that worked and was not wired to anything. Both produce the same outcome — an action taken on a false picture — so the rule was never just write better checks. It is make the check load-bearing.

The second failure, which belongs to Proxmox

With its output file unlinked, the original move logged this:

failed to stat '/mnt/pve/hdd/images/105/vm-105-disk-0.raw'
removing source volume 'ssd:vm-105-disk-1' after successful copy
  Logical volume "vm-105-disk-1" successfully removed.

qm move-disk failed to stat its own output, and then deleted the source anyway, announcing “after successful copy”. The VM config was left pointing at a file that does not exist, with the source gone.

That is a genuine upstream defect: a failed stat of the target must abort before --delete touches the source. It is not the cause here — removing a file from under a running process is — but it converted a recoverable mess into a destroyed volume.

Why this was an inconvenience rather than a loss

A vzdump of all four archive VMs had run twenty minutes earlier, before any disk was touched, purely as insurance ahead of storage surgery.

Human’s call

Back everything up before touching storage, even though nothing was being deleted.

The plan was a move, not a delete, and the VMs were stopped. Nothing in the procedure called for it. It went in as belt-and-braces, and it is the entire reason this episode had a recovery path at all.

The archive verified good: zstd -t returned 0 across 73 GB, and vma config extracted the full VM configuration from inside it. The VM had been stopped for the whole session, so the archive had zero delta from the destroyed disk. The EFI disk survived independently, since only the data disk was in flight.

The step that felt redundant at the time is the only reason there was anything to restore.

The rules that came out of it

A check that detects a dangerous condition must gate the action. Printing it is not gating it. Where a script both detects and acts, the detection has to be an if with an exit, never an echo. Worth grepping your own tooling for checks whose output goes only to the screen.

Never remove a path a running process may hold open. Confirm with lsof or fuser first, and prefer stopping the writer over removing its target.

Long-running storage operations get backgrounded from the start — nohup, tmux, or a systemd unit — not run in an SSH foreground where a client-side timeout leaves an orphan the next invocation cannot see.

A dropped SSH session tells you nothing about the remote process. Assume it is still running until pgrep says otherwise, and then act on what pgrep said.

The wider family this belongs to

This project now has twelve logged instances of the same shape, and the distinction between them is worth keeping straight:

  • Checks that could not fail — a pipeline without pipefail, a grep matching a file’s own comment header, a readiness probe returning true at container creation, a backup verification that created its own empty database and pronounced it intact. All covered in an earlier post.
  • A check that could not fail because the mechanism was blind — a blocklist sweep reporting clean because DNS did not work at all, where “no answer” happens to be the success condition. That one has its own post.
  • A check that worked and was ignored — this one.

The first two are about instrumentation. The third is about wiring. A codebase can pass an audit of every individual check and still contain this, because each check, read alone, is correct.

What changed

The recovery scripting now gates on its own detection. The rules above went into the project’s standing instructions rather than only into the errata file, because this is a class that recurs rather than a single bug.

The upstream move-disk behavior is worth reporting: failing to stat the target and then deleting the source is a data-loss path that does not depend on anyone else’s mistake.