A vulnerability-scanning job started timing out. The fix looked obvious three separate times, and each fix was wrong. The real cause turned out to be a memory limit nobody had noticed setting, and chasing it down honestly ended somewhere stranger than where it started: a limit that, on one machine, cannot be found at all.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
A crash names its cause. A throttle does not, and that difference is the whole story.
The setup
Software-bill-of-materials scanning runs as a scheduled service, cataloguing every container image on a host. It started missing its deadline, catching only a handful of images before being killed for taking too long.
What actually happened
Three explanations were tried, in order, and each one was wrong in a specific and instructive way.
First: “the image is too big.” A 13-gigabyte image was excluded by name. The job timed out anyway on a completely different image — 790 megabytes, nowhere near the excluded one’s size. Size was never the variable.
Second: “the memory ceiling is too low.” The service’s memory limit was raised. The job barely changed. Checking the actual memory use against the new ceiling showed the process using a fraction of what was now available — the ceiling being raised was never the one the job was hitting.
Third, and correct: a second, lower limit on the same service, one line below the one already being adjusted, had never been noticed.
memory.max 4294967296 <- what kept getting raised
memory.high 1572864000 <- what was actually biting
Two settings that sound like they do the same thing bound completely different behavior. One kills a process outright when exceeded. The other throttles it — forces the kernel to reclaim memory on every allocation above the line — and the process never dies. It just gets slower, sometimes by orders of magnitude, and eventually loses a timeout race it was never going to win.
Why it went unnoticed
Every one of the wrong diagnoses had a benchmark behind it, and each benchmark genuinely succeeded — which is exactly what made it convincing. Reproducing the job in isolation, with a simplified version of its resource limits, ran fast every time. What that isolated run actually proved was something else: it silently dropped three of the real service’s directives — the throttle limit itself, a low CPU-scheduling priority, and a low I/O priority — and measured an environment the real job never runs in. A benchmark that succeeds by leaving out the constraint that matters proves the opposite of what it looks like it proves.
The throttle itself is what made this genuinely hard to see from the outside: the standard signal for “this process ran out of memory” stayed at zero throughout, because nothing was ever killed. A process that is technically running, technically making progress, and taking six hours instead of two minutes produces no error anywhere. It just quietly fails to finish.
The fix
The limit that was actually biting was raised, and the fix was verified the way the earlier ones weren’t: by running the real job, at full scale, and watching it complete in minutes rather than hours, with the throttle counter reading zero afterward instead of the hundreds of thousands it had been accumulating.
Measuring the same constraint across every host running this kind of job found the single host that had prompted the investigation was not unusual — four more hosts, elsewhere in the fleet, were throttled the same way and had simply never accumulated enough work to time out and get noticed. And image size did not predict which ones: an 83-megabyte image ran clean, a 308-megabyte image throttled, and a multi-gigabyte image throttled badly enough to take nearly six hours. What actually drives the memory a scanner needs is what it has to catalogue inside the image — file counts, package ecosystems — which is invisible from the outside no matter how the image is sized.
Verifying it properly
The tempting verification, after raising a limit, is “the throttle counter stopped climbing.” That’s necessary and not sufficient — a throttle can stop firing because the ceiling now sits comfortably above what the job needs, or because raising it moved the ceiling and nothing else changed about how the job behaves. The verification that actually closes the question is end-to-end runtime, measured before and after, on the real job doing real work, not a proxy for it.
To reproduce a resource-constrained service honestly, run it as the actual unit rather than approximating its limits by hand — every directive that shapes execution carries equal weight, and dropping any one of them turns a measurement into a guess wearing a measurement’s clothes.
What generalizes
Two limits that sound like the same knob can enforce completely different consequences, and only one of them produces an error. Any resource control worth checking is worth checking in pairs: a memory ceiling that kills and one that throttles, a bandwidth cap and a scheduling weight, a soft file-descriptor limit and a hard one. The one that fails silently is the dangerous one, precisely because failing silently looks identical to succeeding.
Repeatedly raising the same knob without success is itself a signal — not that the knob needs to go higher, but that it’s probably the wrong knob.
A partial reproduction that succeeds proves nothing about why the real thing fails. It only demonstrates that whatever got left out wasn’t load-bearing for the smaller test — which says nothing about whether it was load-bearing for the real one, and is very easy to misread as the opposite.
And underneath all of it, the sharpest version of the same idea, discovered on a second host in the same investigation that kept behaving strangely no matter how high its limit was set: raising a ceiling repeatedly and watching usage rise to match it exactly, every time, is not measuring the workload — it’s reading back the configuration. If a process’s memory use tracks whatever ceiling you give it across several different ceilings, there may not be a real “peak” to find, and the honest conclusion is that the measurement technique has reached its limit before the workload has.
One more correction earned along the way, worth keeping separately: the throttle counter itself is not a harm signal. It was tempting to alert on it directly — any throttling sounds bad — but a job that throttles a hundred thousand times and finishes in the same time it always does is fine, while a job that throttles a fraction as often and takes six hours instead of two minutes is not. The counter tells you reclaim is happening. It does not tell you whether that reclaim is free or expensive, and only one of those is worth paging anyone about. The signal that actually distinguishes them is the plainest one available: how long the job took.