A virtual machine kept freezing for minutes at a time. Remote sessions dropped and took several minutes to come back. On one occasion the hypervisor’s console showed the guest pinned at 100% CPU for the duration; on an earlier occasion it had not.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
Three explanations were offered in turn. The first two were wrong, and the way the third was established is more useful than the answer.
First explanation: the hypervisor paused it
Virtualization platforms briefly freeze a guest for certain operations — snapshots, some disk changes. That is a real mechanism with a real name, and it produces exactly this symptom.
It did not happen. The guest’s own uptime counter showed roughly seventy hours unbroken, and the host’s records contained zero suspend, resume, pause or freeze events for that machine across seven days. A pause is not something you can have without the counter noticing.
Second explanation: an idle session was being reset
The next hypothesis came from the guest’s own logs, which showed a remote session idle for twenty-seven minutes having its connection reset by the far end, then reconnecting eight seconds later. The multi-minute wait to get back in was session arbitration and graphics re-initialization — not a paused machine at all.
That was a good finding and it explained a real thing. It was recorded as a leading hypothesis rather than a conclusion, because it needed one observation from the client end that had not been made — and this site deliberately does not publish a diagnosis the record has not committed to.
It was also not the whole story, because it does not explain a guest sitting at 100% CPU.
The measurement that settled it
Two independent samplers were already running, on two different machines, for unrelated reasons. One on the hypervisor every five minutes; one inside the guest every fifteen.
Neither was built to diagnose this. But a periodic sampler has a property worth knowing about:
A sampler running late is a fingerprint of its own machine not executing.
If the interval between two samples is much longer than the interval it was configured for, the machine it runs on was not scheduling it. Nothing else explains a gap.
| Sampler | Runs on | Late intervals |
|---|---|---|
| Host health sampler | hypervisor | zero — never missed a beat |
| Port sampler | inside the guest | three — one of them nearly twenty minutes |
The host kept running through every freeze. The stall was inside the guest.
That is a decisive result from data nobody collected for this purpose, and it cost nothing to obtain. It also rules out the entire family of “the host is overloaded” explanations without needing to argue about capacity.
And capacity was not the story anyway
Worth stating, because it was the next thing that would have been blamed:
40 threads, 188 GB RAM
committed across 9 running guests: 33 vCPUs, 67 GB
host RAM available 116 GB | swap in use 19 MB
worst load average ever recorded: 15.93 of 40
CPU pressure: 0.00 throughoutNothing is short of anything. A host with a quarter of its load average available and zero pressure is not the reason one guest freezes.
What actually distinguished it
The guest sits alongside two near-identical siblings, and the difference is nested virtualization: a hypervisor running inside the guest, on virtualization extensions the outer hypervisor has to emulate for it. That cost lands as guest-side CPU, which the host sees as ordinary utilization.
One correction worth making precisely, because the obvious version is wrong.
A sibling guest also has those extensions exposed to it, and shows none of the symptoms — because nothing inside it ever uses them. The cost comes from a guest hypervisor actually running, not from the capability being available. Exposure is not expense. So “this VM has nested virtualization enabled” is not the diagnosis; “this VM is running a hypervisor” is.
A postscript, because the fix was harder than the diagnosis
Turning it off cost roughly half a day, two reboots and one forced power-off.
The obvious move is to remove the flag that asks for nested virtualization. That was done, the machine was hard power-cycled, the configuration read back correctly — and the guest’s hypervisor still launched one second after every boot.
The CPU model in use is a passthrough model: it hands the guest every feature the host has that can be virtualized. With nesting enabled host-wide, the virtualization extensions are in that set. Removing the flag stops asking for them. It does nothing to stop passthrough handing them over anyway.
Necessary, not sufficient, and no number of power cycles changes that. The fix is to mask the feature explicitly rather than merely stop requesting it.
Two details from that hunt are worth keeping:
A summary property lied, in the helpful direction. One system reading reported the extensions absent while they were still exposed and in use. It pointed at the right answer for the wrong reason, and building on it would have declared victory about two hours early. What settled it was an event written by the guest’s own hypervisor at the moment it failed to start — an artifact written by the component that acted beats every summary property.
And the platform logs a silent no-op. An unresolvable CPU flag is logged and then simply not applied, leaving a configuration that reads correct while doing nothing. That message appears when the machine starts, not when the setting is changed — so looking for it in the output of the command you just ran finds nothing.
Why this is worth writing down
Not for the answer. Nested virtualization being expensive is unremarkable.
It is worth writing down because of the shape of the investigation, which went wrong twice in a way that is easy to repeat.
The first explanation named a real mechanism that produces the observed symptom. That is the most seductive kind of wrong answer, because it survives a plausibility check perfectly — the symptom matches, the mechanism exists, the story is coherent. It failed only against a counter that had been quietly incrementing the whole time.
The second explanation was true and insufficient. The session really was being reset; that really did explain the delay in reconnecting. It just did not explain everything, and a partial explanation that fits most of the evidence is much harder to discard than one that fits none.
What broke the tie was not cleverness. It was two clocks on two machines, ticking for other reasons, whose lateness could be compared.
This lab has now been through the same arc three times — an I/O ceiling that was memory pressure, a blocklist result that was a blocked resolver, and this. Each time the wrong answer was coherent, and each time it was settled by a measurement rather than an argument.
The generalizable habit is cheap: run a periodic sampler on anything you might later need to reason about, and record the timestamp rather than only the value. The gaps in the record are a measurement of their own, and they are free.