A wrong answer scored better than no answer

A benchmark comparing several language models on the same set of coding tasks finished with percentages that shouldn’t have been comparable at all, though nothing about the output made that obvious. One model that answered every task, just often incorrectly, was scored fairly. A different model that stayed silent on several tasks came out looking better than it should have, purely because of how silence and wrong answers were being counted underneath the percentage.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by human, chat 2026-09-17. How this site is written →

What the numbers were actually hiding

A percentage only means what it looks like it means when every model is being divided by the same total. Reading the raw counts alongside the percentages, two of them didn’t add up against the number of tasks and repeats the benchmark was supposed to be running — the denominators themselves were different from model to model, silently.

The line that caused it

When a model produced no usable code at all for a task, the scoring function counted that as a fixed, minimal penalty — the smallest possible loss, one point out of a maximum of one. A model whose code was present but wrong for the same task was scored against the task’s real maximum, which could be five or ten times larger. Both are failures. Only one of them was actually being treated as a failure worth counting.

The practical effect: a model that mixed silent non-answers with wrong answers ended up with a smaller effective total than every other model, and its percentage was calculated against that smaller total rather than against the benchmark’s real maximum — inflating its apparent score by several points purely as a side effect of failing quietly instead of failing loudly.

The fix, and recovering the run without restarting

The scoring function was changed so a silent non-answer is scored zero out of the task’s actual maximum, the same maximum a wrong answer would have been measured against. Because the bug only ever affected the denominator and never touched the numerators that had already been recorded, the run in progress didn’t need to be restarted — every model’s true score could be recalculated directly from the real, constant maximum, preserving hours of completed work. Only the model that had produced silent non-answers moved when the numbers were recalculated; every other model’s percentage was already correct, because it had never triggered the shortcut in the first place.

A second problem the same numbers exposed

One model in the same run scored zero on every single task, which read, at a glance, like a model that simply couldn’t do the work. The generation logs told a different story: that model had been actively producing substantial output, taking real time to do it, for every task in the run. The model had answered. The scoring harness’s method for pulling code out of that model’s response was built around a different output shape than what this particular model actually produces, and was extracting nothing from it every single time.

A benchmark score of zero and a model that never responded look identical from the summary table. They are not the same finding, and reporting the first as if it were the second would have published a false statement about that model’s actual capability, not a real result about its performance.

What generalizes

A score whose denominator changes depending on the answer isn’t a score. Whenever a harness has more than one way for something to fail, each of those paths needs to be checked against the same total, explicitly — and any total that isn’t the expected maximum for every entry is worth treating as suspicious before trusting the percentage built from it.

A benchmark result and a broken harness produce the identical output. A model that never got a fair chance to answer, because the tool extracting its answer didn’t understand its output format, will report the same zero as a model that genuinely couldn’t solve anything. Before reporting a low score as a finding about a model, it’s worth checking whether the model actually got asked the question in a form its own answer could survive.