Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
If you do not verify something, you know you have not verified it. The uncertainty is at least visible — it sits there being uncomfortable, and eventually you deal with it.
A verification step that is incapable of reporting failure is worse, because it converts that honest uncertainty into a written record of success. The uncertainty does not go away. It just stops being visible, and now there is a log entry disagreeing with reality.
This lab has found four of these. Every one was in a check written specifically to prove a fix had worked.
One: the pipeline that swallowed its own failure
A shell check, roughly:
some_command | grep -q "expected-value" && echo "ok"This looks fine. It is fine, right up until some_command fails.
When it fails it produces no output. grep then searches nothing, finds nothing, and — because of how exit status propagates through a pipeline in a shell without pipefail set — the surrounding logic reports success anyway.
The check passes hardest precisely when the thing it examines is most broken. A partial failure might produce unexpected output and get caught. Total failure produces silence, and silence looks exactly like a clean pass.
set -o pipefail # the entire fixThat one line makes a pipeline fail if any stage fails, not merely the last. Its absence is the single most common version of this bug in shell.
Two: the check that failed on correct configuration
A container stack needed a memory setting applied. The check searched the rendered configuration for:
OPENSEARCH_JAVA_OPTS=The setting was applied correctly. The check reported failure anyway.
The reason is dull and instructive: the tool that renders that configuration does not emit KEY=value. It emits a YAML map:
environment:
OPENSEARCH_JAVA_OPTS: -Xms2g -Xmx2gA colon, not an equals sign. The check was searching for a format the tool never produces, so it could only ever fail.
This is the inverse of the others — a false alarm rather than false comfort — and it belongs here because it has the identical root cause. The check was never run against a known-good system. Nobody confirmed it could pass. It was written, it looked right, and it was trusted.
A check that always fails at least gets investigated. That is genuinely the lucky version.
Three: the status file that matched its own documentation
A monitoring agent writes its connection state to a status file. The check:
grep connected /var/ossec/var/run/wazuh-agentd.stateThe file begins with a comment header explaining what the fields mean. That header contains the words connected, disconnected, and pending — because it is documentation, and documentation lists the possible values.
So grep connected matches the explanatory comment. It matches on a healthy agent. It matches on an agent that has never once reached the server. It matches on an agent whose service is stopped, as long as the file exists.
The check does not report the agent’s status. It reports the file’s existence, in language that reads exactly like a status report.
Two problems compounded there. The pattern was unanchored — it should have matched a specific field on a specific line. And disconnected contains connected as a substring, so even a correctly-targeted search would match the opposite of what it was looking for without a word boundary.
Four: the port that opened before anything was listening
The most subtle of the four, and the one that quietly poisoned everything downstream.
A container needed to be ready before subsequent steps ran. The readiness probe checked whether its port was listening:
ss -ltn | grep :9200For an ordinary service, that is a sound test. For a container-published port, it is not — and the difference is not obvious.
When a container publishes a port, the container runtime’s networking proxy binds that port on the host the instant the container is created. Not when the application inside has started. Not when it has loaded its configuration, opened its data files, or become able to answer a request. Creation.
So the probe returned true almost immediately, every time, on a stack that needed a couple of minutes to actually start. Every check sequenced after it was racing container startup, and their results depended on machine speed and load. Some passed. Some failed intermittently and got re-run until they passed, which is its own kind of wrong.
The fix is to probe the thing you actually care about — ask the application a question and require a sensible answer:
curl -fsS http://localhost:9200/_cluster/healthA port being bound tells you a socket exists. It says nothing about whether anything behind it can serve a request.
The related trap: “started” is not “working”
Adjacent enough to belong here. This is treated as reliable far too widely:
systemctl is-active some-serviceIt returns active for a daemon sitting in an indefinite reconnect loop, failing to reach its server, retrying forever, and doing nothing useful. The process is running. systemd is perfectly satisfied. The service is not working.
Any component whose actual success condition is “reached something over the network” has to be checked against its own view of that connection — not against whether its process exists.
The rule that came out of it
When you write a check, make it fail on purpose once before trusting it.
Break the thing deliberately. Stop the service, corrupt the setting, unplug the dependency. Watch the check go red. Then fix it and watch it go green.
A check you have only ever seen pass is not a check. It is a hopeful assertion that happens to be printed in green.
This is cheap — usually under a minute — and it is the only method that reliably catches all four failures above. Reading a check does not catch them: every one of those four was written by someone competent, reviewed, and looked correct on the page. The pipeline bug is invisible without knowing about pipefail. The container-port race requires knowing an implementation detail of container networking. The status-file grep requires having actually looked at the file’s comment header.
None of them survives thirty seconds of deliberate sabotage.
Applied to this blog
The site you are reading has an automated gate that scans every page for real host names, addresses, and credentials before it can be published. Its whole job is preventing a specific, embarrassing, irreversible mistake.
So it ships with a self-test that seeds known-bad input — a real address, a real host name, a private key header — and asserts that every rule fires:
$ python scripts/sanitize.py --self-test
PASS real gateway IP caught by real-gateway-public-ip
PASS real domain caught by real-primary-domain
PASS host token caught by location-token
PASS ssh port caught by real-ssh-port
PASS deploy user caught by real-deploy-user
PASS internal subnet caught by mgmt-vlan-range
PASS private key caught by private-key-block
PASS clean sample sanitized text passes cleanlyThat last line matters as much as the others — it guards against the failure from case two, a gate so aggressive it blocks correct content and gets disabled out of irritation.
It would have been faster to write the scanner and trust it. Writing a blog post about checks that cannot fail, protected by a check that had never been proven able to fail, seemed like the wrong way to start.