A planned feature of this lab lets you text a question to it and get an answer back, so that basic status is reachable when the internet is not. One of the questions is how full a machine’s disks are.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
The code that answers it held a small table mapping each machine’s name to the address to ask. The entry for the storage appliance pointed at the hypervisor.
Nobody would have noticed. That is the whole point of writing it down.
Why it would not have failed
Look up the wrong address and the ordinary result is nothing: a refused connection, no route, a timeout. Every one of those is loud, and this lab’s checks are built to catch exactly that kind of silence.
The management addresses here are assigned in a contiguous block. The storage appliance sits at one address, the first hypervisor at the next, the second hypervisor after that, then the bulk storage box, then the quorum witness. Five machines, five consecutive numbers, all of them real, all of them online, all of them running the same metrics agent on the same port.
So an off-by-one in that block does not produce an error. It produces a prompt, correct, well-formed answer about the wrong machine.
Ask how full the storage appliance is, and you would have been told the disk usage of the hypervisor, labeled with the storage appliance’s name. The number would be plausible. It would move over time. It would look exactly like the truth, and the only way to notice would be to already know the real answer.
How it was actually found
Not by a test. Not by monitoring. Not by any of the verification this project has spent weeks building.
The human asked a plain question: which network and address is this machine supposed to be on?
Answering it meant assembling the value from six different files that each state it in some form, and five of them agreed. The one that did not was this table. The defect surfaced because somebody wanted to know something ordinary and the sources disagreed while the answer was being put together.
That is worth sitting with. The check that found this was a person being curious, and it beat everything automated in the lab, because none of the automation was looking for two documents to disagree. Cross-referencing a value against every file that states it is cheap, unusually productive, and nothing here runs it automatically.
The underlying mistake
An address written into application code, duplicating addressing that is defined authoritatively somewhere else.
The moment a value exists in two places, they are one edit away from disagreeing, and nothing announces the divergence. The authoritative source was updated as the design evolved. The copy was not, because a copy has no relationship to its original beyond somebody remembering.
The fix, and why the comment matters as much as the correction
The address was corrected. A comment was added next to it naming the machine that lives at the neighboring address, and saying explicitly that it is a live, responding host.
That comment is doing real work. A future reader who finds this table has to re-derive the trap from scratch otherwise: they must notice the block is contiguous, realize that a wrong entry lands on something that answers, and only then understand why the value deserves care. The comment gives them that in one line at the place they are already looking.
The better fix is to not have the table, and instead read the addressing from the one place that defines it. That is a larger change than this defect justified on its own, and it is the direction to move whenever a value like this is touched again.
What generalizes
Contiguous allocation converts typos into plausible answers. Sequential numbering is good practice for almost every reason: it is legible, it is easy to reason about, and it makes gaps obvious. It also guarantees that an off-by-one lands on a neighbor rather than on nothing, and neighbors respond. Anywhere a block of identifiers is dense and the things behind them are similar, a wrong value stops being self-announcing.
A wrong answer is a worse failure than no answer, and it is much easier to ship. No answer trips every alarm you have. A wrong answer trips none of them, and if the thing being asked is something you cannot readily verify by other means, which is the entire reason for asking, then the wrong answer is indistinguishable from the right one at the moment you need it.
Duplicated values need either one source or a check that they agree. Both work. What does not work is the version this lab had, where the duplication was known, informal, and trusted to be kept in step by attention. Attention is not a mechanism.
Some defects are only findable by asking an ordinary question. This project has invested heavily in checks that prove things, and that investment keeps paying. It did not find this one. A person wondering where something lives, and having to look it up in more than one place to be sure, found it in the course of not looking for it at all.