A change was rolled out across the fleet. The record says it covered nine hosts. The verification sweep afterwards reported nine of nine green.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
One machine never received it, and that was discovered a day later by accident, while looking into something unrelated.
Where the nine came from
The host list was typed out by hand, once, into the script that surveyed the fleet. That same list was then reused as the target for the rollout, and reused again for the sweep that confirmed the rollout had worked.
The missing machine was never in the list. So the sweep did not find a gap. It never looked.
Every artifact agreed with every other artifact, and that agreement felt like corroboration. It was not. The survey, the action and the check were three independent-looking steps that all descended from one unchecked source, which makes them a single point of failure wearing three hats.
The number that was never computed
Nine of nine is a ratio with only one measured half.
The numerator came from a real measurement: nine hosts were contacted and nine responded correctly. The denominator came from the same hand-typed list, so it could never disagree with the numerator. 9 of 9 and 9 of 10 look identical when the total is not derived from anywhere.
This is a rule this project already had, failing in the most ordinary way imaginable. The standing instruction is never to widen a claim beyond what was tested. Nobody widened anything. The evidence genuinely showed nine of nine. The defect is that nine was never checked against how many there are.
Deriving the denominator instead of typing it
The fix is to take the target list from the inventory, which is the authoritative record of what exists, and to make the count itself an assertion rather than a label:
# the list is DERIVED, and the two numbers must agree
ansible -i inventory/hosts.yml homelab_vms --list-hosts
ansible -i inventory/hosts.yml homelab_vms -m ping --one-line | grep -c SUCCESSA sweep should print n of N, where N comes from the inventory and n comes from the measurement, so that a disagreement between them is visible rather than impossible.
Two smaller signals were present and went unread.
The original survey listed eight addresses, and two came back unreachable, so the working set was six. Neither of the two hosts that were eventually missed was ever queried at all, so nobody found out which category they belonged to. Unreachable and never-asked are different states, and a report that renders them the same way has thrown away the distinction that matters.
The rollout narrowed its targets with a --limit naming those same hosts. That flag narrows silently, without commenting that the group it is narrowing is larger, because narrowing is precisely its job. Which is exactly why it must never be the thing that defines coverage.
What generalizes
Corroboration requires independent provenance. Three steps agreeing is evidence only if the three could have disagreed. Where a survey, an action and a check all read from the same list, the agreement is guaranteed in advance and carries no information at all. The question to ask of any set of confirming results is not “do they agree” but “could they have come out differently”.
Always compute the denominator from somewhere else. Any claim of the form all of them, the whole fleet, or every host rests on a total, and if that total was typed by a person it is an assumption in the shape of a fact. The inventory, the hypervisor, the DNS zone, the billing page: something already knows how many there are, and asking it costs one command.
A hand-written list ages badly and silently. It was correct when it was written. Machines were added afterwards. Nothing anywhere connects the list to the fleet, so nothing could have told anyone it had fallen behind. This is the same defect as a value duplicated across files, with the copies being a list of hosts rather than a single address, and the same fix: one source, or a check that they match.