The lab runs several open-weight language models locally and wants to know which one is actually worth using for a given job. That means a benchmark, and a benchmark means a harness — and the harness has been wrong more often than any model it’s measured.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by human, chat 2026-09-21. How this site is written →
The setup
Eight coding tasks, each graded by running the model’s generated code against hidden tests, not by asking another model to judge the answer. Every model runs each task multiple times at temperature 0, and a handful of guardrails exist specifically to keep the harness honest: a --selftest mode that scores a known-correct reference implementation and refuses to proceed unless it gets full marks, and a rule that any task scoring zero for every model gets read as a harness bug until proven otherwise.
That rule has fired four times so far.
What’s actually been learned about the models
Architecture matters more than parameter count. Two models with similar memory footprints — one Mixture-of-Experts, one dense — can differ by an order of magnitude in throughput, because the MoE model only activates a fraction of its weights per token. A 17 GB dense model measured at 5.8 tokens/second; an 18 GB MoE model at the same footprint measured 53.2.
Quantization loses less than expected, until it doesn’t. Dropping a model from 16-bit to 4-bit weights cost under 4% on a standard perplexity measure — close to free. Going one step further, to 3-bit, cost far more, and an importance-weighted quantization scheme that made no measurable difference at 4-bit recovered over 2% at 3-bit. The lesson: the “acceptable precision loss” line isn’t smooth, it’s a cliff, and where the cliff sits has to be measured per model rather than assumed from one data point.
A GPU is worth roughly 2x on identical weights, not more. Running the same model CPU-only versus GPU-accelerated on this hardware produced about double the throughput on the GPU — a real, worthwhile difference, but nowhere near the gap the raw compute specs would suggest, because token generation here is bound by memory bandwidth, not arithmetic.
The four ways the harness lied first
A template that looked broken wasn’t. One model’s chat template was a bare passthrough — no role markers, no formatting — which reads exactly like a packaging failure. It wasn’t: the model still produced usable, well-formed answers through it. Assuming the template was the problem and “fixing” it wasted real time reproducing a score that had already been correct.
A template that looked fine was actually corrupted. A different model’s template contained a literal typo in its role-delimiter syntax. This one really was broken, and grafting a corrected template on made scores worse, not better — the honest fix was switching to a different build of the same model entirely.
Joining every code block the model emitted, instead of picking one. Two capable models were scored at effectively zero because they emit a draft code block before their real answer, and the extractor concatenated both into one syntactically invalid file. Fixed by trying several candidate extractions — the joined text, the last block, the longest block — and keeping the first one that actually parses.
No code fence at all. One model answers in plain prose with no markdown fences around its code, so the extractor, finding nothing to extract, fell back to returning the entire response — including the prose — as “code.” That reliably fails to parse. The fix cost one more candidate in the same ladder: strip a closing </think> tag and try the remaining text directly.
Each of these produced a confident, wrong zero. Each looked, from the summary table alone, exactly like a model that simply couldn’t do the work.
What generalizes
A zero is a claim about the harness until it’s a claim about the model. The rule that emerged from all four incidents is the same one: before a low score gets reported as a finding about a model’s capability, pull one raw generation for that task and read it. If the model wrote working code and the score is still zero, the harness is lying. Every one of the four cases above was caught exactly this way, and none of them would have been caught by staring harder at the percentages.
The corollary is more useful than the rule itself: a benchmark result and a broken harness produce the identical number. Nothing in a summary table distinguishes them. The only way to tell them apart is to go look at what actually happened underneath the score, every single time a result looks surprising — especially when it looks surprisingly bad.