The battery was fine, and the evidence said so

A hypervisor’s RAID controller paged critical: battery fault, charge failed. The agent investigating it got as far as pricing a replacement part before stopping to check the hardware directly. The battery was healthy. It was midway through a routine, scheduled 90-day cycle that every RAID controller battery runs to verify it can still hold a charge — designed behavior, not a fault.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by human, chat 2026-09-25. How this site is written →

What the tool actually said

Reading the controller’s own diagnostic told a different story than the alert: battery state optimal, voltage and temperature normal, replacement not required, and a learn cycle explicitly reported as active. During that cycle the controller deliberately drops its write cache to write-through, since the cache is briefly unprotected while the battery discharges and recharges on purpose. Every symptom fit: the fault flapped on and off within twenty minutes, matching the discharge and recharge phases, and the chassis’s own hardware sensor never reported a failure at any point.

A monitoring script that wasn’t wrong, exactly

The collector generating the alert used an allowlist: any battery status it recognized as healthy passed, anything else failed. That is the right default. Failing closed on an unrecognized state is what makes the alert trustworthy when a battery is genuinely dying, and widening the allowlist to silence this case would have converted the one check built to catch a dead battery into one that catches nothing.

The actual defect was narrower: the tool doing the reporting collapses “temporarily discharged for a scheduled test” and “permanently dead” into the same status string, and the collector had no way to tell them apart because nothing telling them apart was ever exposed. A different, more detailed diagnostic tool reports the learn cycle as an explicit, separate flag. The fix moves the battery health signal to that tool, with the routine cycle reported on its own, non-critical line instead of folded into the failure state.

Human’s call

The more detailed tool is closed-source, runs as root, and can delete storage volumes on a hypervisor carrying more than a dozen other services — a real, permanent widening of what that box depends on, just to fix one false alarm. The operator decided to keep it, deliberately rather than by default, with explicit limits: only read-only diagnostic commands, ever, in both automation and by hand; the binary’s checksum verified against the vendor’s own published value; and if the tool ever goes missing, the battery reading must go stale rather than default to healthy. The tradeoff was real enough to be worth stating rather than inheriting silently.

The more useful mistake

The tool’s summary string reported “charge failed,” and that phrase matched what a genuinely dead battery produces. The agent read it, concluded the battery was dying, and started pricing a replacement. The evidence that argued otherwise was already available at that point: a real hardware failure does not recover and then fail again on a fifteen-minute cycle, and the chassis’s independent sensor had never flagged a fault at all. The flapping alert was explained away as “intermittent failure” instead of prompting the obvious question, which was what routine process would make a battery cycle down and up on a schedule.

What generalizes

A vendor tool’s status string is a summary written for a purpose the tool’s author had in mind, not a direct report of the hardware’s state, and treating it as one will occasionally be wrong in a way that reads as confident. The evidence that would have caught this mistake was already in hand before the conclusion was reached. The habit worth keeping is asking what routine, scheduled process could produce this exact pattern before asking what part needs replacing.