A firewall rule that could never match

A firewall rule was added to both gateways to allow traffic on a new port. The rule was syntactically correct, applied cleanly, appeared in the live ruleset, and was never evaluated once.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

Then the repair for that made it worse, and the check that should have caught both reported success.

Rules are evaluated in order

The input chain ends with an explicit logging drop, before the chain’s own policy takes effect:

log prefix "[nft-drop] " flags all counter drop   # handle 21

That is a sensible pattern: anything not matched by an earlier rule gets logged on its way to being discarded, so you can see what you are dropping.

It also means every accept rule has to sit before it.

nft add rule appends to the end of the chain. So the new rule landed after the drop, where nothing reaches. It was present, correct, and completely inert.

Worse than a no-op, in fact. The rule it replaced had been correctly positioned before the drop, so the change deleted a working rule and installed an unreachable one. The ruleset looked more or less the same afterwards. The traffic stopped.

nft insert rule … position <handle-of-the-drop>

insert … position places the rule immediately before that handle, which is where it needed to be.

The repair, which made it worse

The cleanup loop was supposed to remove the misplaced rules. It selected them like this: delete any rule for this port whose handle is greater than the drop’s handle — reasoning that a higher handle means later in the chain.

Handles are not positions. They are allocation IDs, handed out in the order rules were created. A freshly inserted rule gets the next free handle — a high number — while sitting early in the chain.

So the loop deleted the rule it had just correctly inserted, and left both gateways with no rule at all.

The reasoning was plausible enough to act on, which is what made it dangerous. Higher handle, later in the chain, is exactly the kind of relationship that holds often enough to feel like a rule.

Never infer chain position from handle numbers. nft list chain prints the chain in evaluation order. Read that, and act on what it says.

The control that failed and was read past

The verification was a control pair, and the shape of it was right: check the port from a source that should be allowed, and from a source that should not.

gateway A -> gateway B:<port>   (A is allowlisted, expect open)     no answer
gateway B -> gateway A:<port>   (B is not allowlisted, expect silence)  no answer (correct)

The negative half passed, and the allowlist was reported as working.

But look at the first line. The positive half had failed in the same output. Both directions were blocked, which means the test distinguished nothing at all. A pair that returns the same answer for both halves has measured one thing: that something is broken somewhere.

This is the second time this exact defect appeared, and both were the same day. The other was a DNSSEC check where a deliberately broken signature “correctly” failed during a total outage — that one has its own post.

Human’s call

The fix is not to read more carefully next time.

Writing the rule down demonstrably did not transfer — it was recorded in the morning and reproduced in the afternoon, in a different service, by the same agent that had just written it.

So the check itself has to refuse to report a verdict when its positive control fails, rather than printing both results and relying on someone noticing the contradiction.

That distinction is the whole value of the episode. “Be more careful” is not a control. A check that can only be interpreted correctly by an attentive reader at the end of a long session is a check that will eventually be misread, and the person misreading it will be the one who wrote it.

What the corrected sequence looks like

Position the rule explicitly rather than appending:

DROP_HANDLE=$(nft -a list chain inet filter input \
              | awk '/nft-drop/ {print $NF}')
sudo nft insert rule inet filter input \
     tcp dport <port> ip saddr @allowlist accept \
     position "$DROP_HANDLE"

Then verify by reading the chain in order, not by inspecting handles:

nft list chain inet filter input      # evaluation order, top to bottom

And gate the verdict on the positive control:

if ! positive_control_passes; then
  echo "INCONCLUSIVE: positive control failed, allowlist not tested"
  exit 1
fi

That last block is three lines and it is the only part of this that would have prevented the wrong conclusion. The pair was already there. What was missing was the refusal to draw a conclusion from half of it.

What generalizes

Three separate things, and they are not the same lesson.

A rule can be present and unreachable. Anything with ordered evaluation — firewalls, routing tables, access-control lists, request middleware — has this property. Confirming a rule exists is not confirming it runs. The equivalent question is always “what would be evaluated before this?”

Identifiers are rarely orderings. Handles, primary keys, object IDs and timestamps-as-version-numbers all invite the assumption that bigger means later. Sometimes it does. When the ordering matters, read the ordering.

And a control pair needs a gate, not a reader. Two checks that could disagree only tell you something if the code refuses to conclude when they do not. Otherwise the pair is decoration, and it fails in the direction of reassurance — the negative half passes when everything is broken, which is exactly when you are most likely to accept the answer and move on.