A check that cannot fail is worse than no check at all

Skip a check and you know you skipped it. The uncertainty stays visible, sits there being uncomfortable, and eventually gets dealt with.

Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →

A check that cannot report failure is worse, because it converts that uncertainty into a written record of success. The uncertainty does not go away. It stops being visible, and now there is a log entry disagreeing with reality.

This lab has found four. Every one was in a check written specifically to prove a fix had worked.

One: the pipeline that swallowed its own failure

A shell check, roughly:

some_command | grep -q "expected-value" && echo "ok"

That looks fine, and is, right up until some_command fails.

A failing command produces no output. grep then searches nothing, finds nothing, and because of how exit status propagates through a pipeline in a shell without pipefail set, the surrounding logic reports success anyway.

The check passes hardest precisely when the thing it examines is most broken. A partial failure might produce unexpected output and get caught. Total failure produces silence, and silence looks exactly like a clean pass.

set -o pipefail

That one line makes a pipeline fail if any stage fails rather than only the last. Its absence is the most common version of this bug in shell.

Two: the check that failed on correct configuration

A container stack needed a memory setting applied. The check searched the rendered configuration for OPENSEARCH_JAVA_OPTS=.

The setting was applied correctly. The check reported failure anyway.

The reason is dull and instructive: the tool that renders that configuration does not emit KEY=value. It emits a YAML map.

environment:
  OPENSEARCH_JAVA_OPTS: -Xms2g -Xmx2g

A colon, not an equals sign. The check was searching for a format the tool never produces, so it could only ever fail.

This is the inverse of the others — false alarm rather than false comfort — and it belongs here because the root cause is identical. The check was never run against a known-good system. Nobody confirmed it could pass. It was written, it looked right, and it was trusted.

A check that always fails at least gets investigated. That is the lucky version.

Three: the status file that matched its own documentation

A monitoring agent writes its connection state to a status file. The check:

grep connected /var/ossec/var/run/wazuh-agentd.state

The file opens with a comment header explaining what the fields mean, and that header contains the words connected, disconnected and pending, because documentation lists the possible values.

So grep connected matches the explanatory comment. It matches on a healthy agent, on an agent that has never reached the server, and on an agent whose service is stopped, as long as the file exists.

The check does not report the agent’s status. It reports the file’s existence, in language that reads exactly like a status report.

Two problems compounded. The pattern was unanchored, when it should have matched a specific field on a specific line. And disconnected contains connected as a substring, so even a correctly-targeted search matches the opposite of what it wants without a word boundary.

Four: the port that opened before anything was listening

The most subtle, and the one that quietly poisoned everything downstream.

A container needed to be ready before subsequent steps ran, so the probe checked whether its port was listening:

ss -ltn | grep :9200

For an ordinary service that is sound. For a container-published port it is not, and the difference is not obvious.

When a container publishes a port, the runtime’s networking proxy binds it on the host the instant the container is created — not when the application inside has started, loaded its configuration, opened its data files, or become able to answer a request.

So the probe returned true almost immediately, every time, on a stack needing a couple of minutes to start. Every check sequenced after it raced container startup, and their results depended on machine speed and load. Some passed. Some failed intermittently and got re-run until they passed, which is its own kind of wrong.

Probe the thing you care about instead. Ask the application a question and require a sensible answer:

curl -fsS http://localhost:9200/_cluster/health

A bound port tells you a socket exists. It says nothing about whether anything behind it can serve a request.

The related trap: “started” is not “working”

Adjacent enough to belong here, and trusted far too widely:

systemctl is-active some-service

That returns active for a daemon sitting in an indefinite reconnect loop, failing to reach its server, retrying forever, doing nothing useful. The process runs. systemd is satisfied. The service is not working.

Any component whose real success condition is reached something over the network has to be checked against its own view of that connection, not against whether its process exists.

The rule

Make a check fail on purpose once before trusting it.

Break the thing deliberately. Stop the service, corrupt the setting, unplug the dependency, and watch the check go red. Then fix it and watch it go green.

A check only ever seen passing is not a check. It is a hopeful assertion printed in green.

This costs under a minute and is the only method that reliably catches all four failures above. Reading a check does not catch them: every one was written by someone competent, reviewed, and looked correct on the page. The pipeline bug is invisible without knowing about pipefail. The container-port race needs an implementation detail of container networking. The status-file grep needs somebody to have actually opened the file and read its comment header.

None survives thirty seconds of deliberate sabotage.

Applied to this blog

The site you are reading has an automated gate that scans every page for real host names, addresses and credentials before it can publish. Its whole job is preventing one specific, embarrassing, irreversible mistake.

So it ships with a self-test that seeds known-bad input — a real address, a real host name, a private key header — and asserts every rule fires:

$ python scripts/sanitize.py --self-test
  PASS  real gateway IP    caught by real-gateway-public-ip
  PASS  real domain        caught by real-primary-domain
  PASS  host token         caught by location-token
  PASS  ssh port           caught by real-ssh-port
  PASS  deploy user        caught by real-deploy-user
  PASS  internal subnet    caught by mgmt-vlan-range
  PASS  private key        caught by private-key-block
  PASS  clean sample       sanitized text passes cleanly

That last line matters as much as the others. It guards against the failure from case two: a gate so aggressive it blocks correct content and gets disabled out of irritation.

Writing the scanner and trusting it would have been faster. Publishing a post about checks that cannot fail, protected by a check never proven able to fail, seemed like the wrong way to start.