The diagnosis was wrong, so it got measured

A test machine running the security-monitoring stack became unusable. The agent recorded the cause in the deployment log as an I/O ceiling — the storage could not keep up — and added that memory was never the constraint.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

Both were wrong, and the second was the exact opposite of the truth.

Human’s call

Go back and measure it.

The log entry read as a finished conclusion, and the agent had moved on. There was no new evidence, just an unwillingness to accept a diagnosis that had been reasoned to rather than measured. The machine was still running and still misbehaving, so the numbers were available for the cost of asking.

What was actually happening

Measured on the machine while it was still in the failure state:

MeasurementValueWhat it means
kswapd0 CPU52%, top processKernel stuck reclaiming memory
Free memory / swap147 MB free, no swap configuredNothing to page out to
CPU breakdown0.0 id, 0.0 wa, 58.6 sy, 17.6 siPinned in the kernel, not waiting on disk
Processes in D statenoneNothing ever blocked on I/O
Block I/O over 20s0 MiB read, 0 MiB written, while burning 4.7 coresNo disk activity at all

An I/O-bound machine has processes in uninterruptible sleep, measurable disk throughput and a visible wa figure. This had none of them, and was consuming nearly five cores while touching no disk.

The mechanism: the working set — a 2.4 GB JVM, a Node process, the container runtime, a log shipper, and the search indexer’s memory-mapped files — exceeded available memory with no swap configured. The kernel had nowhere to evict pages to, so it fell into permanently discarding page cache and immediately re-reading it, spinning in reclaim.

Why the wrong answer looked right

The guest’s re-reads of that discarded cache never reached a disk. They were served from the host’s page cache, one layer down, because the machine underneath was caching the same file contents.

So block I/O measured zero. The thrashing was real and furious, and invisible at the layer anyone thought to look.

That produces a confusing signal: a machine obviously struggling, with obviously no disk activity. “High load, no I/O” reads as CPU-bound from one angle and as storage so slow requests never start from another. The first diagnosis picked a story that fit some of the evidence and stopped.

Two things worth taking away

available in free -m is not headroom when there is no swap. The original diagnosis cited it as evidence memory was fine, but available counts reclaimable page cache — and reclaiming that cache was the entire problem. The number said “there is memory you can reclaim” while the machine was dying of reclaiming it.

With swap configured this degrades gracefully: pages go to disk, things get slow, someone notices. Without it, the kernel’s only lever is discarding cache it immediately needs back, and it will keep pulling that lever.

wa near zero rules out an I/O explanation. If a machine is slow, wa is near zero and nothing sits in D state, storage is not the problem. That one observation would have redirected the first diagnosis in under a minute.

The measurements that separate the two cases

Nothing exotic, which is the point. About thirty seconds of work:

top          # is kswapd near the top? that is reclaim, not I/O
free -m      # check SWAP, not just available
vmstat 1     # si/so: swap in/out
             # wa:    actual I/O wait
             # b:     processes blocked on I/O

Then check from outside the machine — the hypervisor’s own view of block device throughput for that guest. If the guest looks like it is thrashing and the layer beneath reports no disk activity, there is a caching boundary in between and the guest’s numbers need reinterpreting.

The general shape: when a measurement is surprising, check whether something underneath is quietly satisfying the request. The guest was not lying about doing no I/O. It genuinely was not doing any. The work was real and happening one layer down.

Why the correction is struck through rather than edited away

The original entry still says “I/O ceiling” and “memory was never the constraint”, with a correction block beneath it carrying the measurements that disprove both.

That costs something. The record is untidier, and anyone reading it meets a confidently wrong conclusion before the right one. The alternative is worse in two ways.

Quietly fixing it produces a log that has always been right, which is not a record of anything. Its value is being a description of what happened, and what happened includes a day of being wrong.

And the wrong diagnosis is the transferable part. “It was memory pressure” is a fact about one machine on one afternoon. “It looked exactly like an I/O ceiling, here is why, and here is the measurement that distinguishes them” is useful to anyone who meets the same signal. Delete the mistake and the reusable content goes with it.

What it changed

The security-monitoring machine’s memory allocation was confirmed rather than trimmed. The design figure was right, and reducing it on the strength of the first diagnosis would have guaranteed the same failure in production. It also gained a requirement for configured swap, so memory pressure degrades into slowness rather than a pinned CPU.

The rebuild used for the testing that followed ran at a fraction of the load doing comparable work, which is the confirmation the first diagnosis never had.