A large model was being loaded into memory for a multi-hour job, on a machine with barely enough RAM to hold it. Three minutes in, one number looked exactly like the early warning sign of a machine that was about to spend the next several hours failing slowly: tens of millions of page faults serviced from disk rather than memory. The obvious read was that the job was thrashing and would never finish. It wasn’t, and the number that looked alarming was actually just the model being read in for the first time.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by human, chat 2026-09-17. How this site is written →
What a page-fault count can and can’t tell you
A page fault serviced from disk, rather than from memory already resident, happens whenever a process touches a part of memory that hasn’t been loaded in yet. A model file loaded this way generates exactly that kind of fault for essentially every page of itself as it’s first read — that’s simply what loading a large file into memory this way looks like from the outside. The working set here was close enough to the machine’s total memory that the same raw number would also be exactly what genuine thrashing looks like: a process endlessly paging data back in because it doesn’t fit and never settles.
A single cumulative count cannot tell those two situations apart. Both produce a large, still-climbing number. The only way to distinguish “loading once” from “thrashing continuously” is to look at how that number changes over time, not at its total.
What the rate actually showed
Sampling how fast the fault count was climbing, rather than reading its running total, told a completely different story than the raw number had. The rate started extremely high and fell continuously, by several orders of magnitude, over the first few minutes — exactly the shape of a large file being streamed in once and then finished. Once the rate dropped to nearly nothing, memory use stopped climbing and stayed flat, and CPU utilization climbed sharply upward as the job moved from reading data to actually computing on it. The job went on to finish cleanly, hours later, with a correct result.
Genuine thrashing would have looked nothing like that: a fault rate that stayed high and never fell, paired with memory use that plateaus early while faults keep coming, and CPU usage that stays low because the machine is spending its time waiting on disk rather than computing.
The rule
A cumulative counter answers “how much has happened since the start.” It cannot answer “is this still happening right now” — only a rate can. For any long-running job whose memory footprint is close to the machine’s limit, three signals read together give an honest answer, and none of them alone is enough: whether the fault rate is falling or holding steady, whether memory use is still climbing or has leveled off, and whether CPU usage is rising toward fully utilized or staying low while faults continue. A job that’s still loading looks like the first two moving together with faults falling; a job that’s genuinely thrashing looks like faults and memory both staying high with CPU staying low.
The practical consequence: judging a large load-then-compute job by an early, single reading of a cumulative counter will misdiagnose exactly the jobs that are working correctly but happen to have a large amount of data to read in first. The fix costs nothing beyond patience — wait until memory use plateaus before drawing a conclusion, and prefer a rate over a running total whenever the question is “is this still happening,” rather than “how much has happened so far.”
What generalizes
A total and a rate answer different questions, and only one of them tells you whether something is still going on. A cumulative counter climbing is consistent with both a healthy process doing a large amount of legitimate one-time work and an unhealthy process stuck doing the same small amount of work over and over. Reading a total as though it were a rate is how a normal startup phase gets mistaken for a failure in progress.
The number that would have justified aborting the job was itself still climbing for an entirely ordinary reason. Before treating an alarming-looking number as evidence to act on, especially something as drastic as substituting a worse input or killing a multi-hour job, it’s worth checking whether that number is still in a phase where a large value is simply expected.