A test asserted that the tunnel between two servers was up. It failed on both tunnels, while both were demonstrably carrying traffic — seventeen established connections, health endpoints answering, and a full request succeeding through each one.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
The check was not broken in the ordinary sense. It was inverted.
The check
journalctl -u rathole-client --since "-24h" | grep -q "Control channel established"
Search the last 24 hours of logs for the line that says the connection came up. Reasonable on its face, and it works — for about a day.
That message is emitted once, at connect time. Both tunnels had connected roughly forty-eight hours earlier and had been continuously up since. The twenty-four hour window had rolled past the only line that could ever satisfy it.
So:
- A tunnel up for two days: FAIL
- A tunnel that dropped and reconnected an hour ago: PASS
The check rewarded exactly the behavior you do not want. The longer a system ran without incident, the more certain its failure became.
Worse than a check that cannot fail
This project has accumulated a collection of checks that could not report failure. This one is the opposite, and it is worse.
A check that cannot fail costs you a false sense of security — bad, but it fails silently and quietly. A check that reports health as failure costs you something more expensive: it sends someone to debug a working system, and it does so specifically on the systems that have been most reliable.
Repeat that a few times and it trains people to ignore the check. At which point it has become worse than not existing, because now a real failure is also ignored.
The fix, and the rule behind it
Assert the connection that exists now, rather than evidence that one was established at some point:
ss -tnH state established | grep -q "<peer-a>:<port>"
ss -tnH state established | grep -q "<peer-b>:<port>"
Proven both directions: it matches against the live peers, and a deliberately bogus peer address correctly does not match. A control pair, with the positive half asserted first — a lesson this project learned twice in one day.
The general rule is worth stating carefully, because it is about the kind of thing being asserted:
When a condition is continuous — “the tunnel is up”, “the service is listening”, “the mount is present” — assert current state.
Reach for a log window only for genuine events — “a reconnect happened”, “a certificate renewed” — and when you do, be explicit that a quiet log is the good outcome.
Any check whose pass depends on something having happened recently will eventually fail on a system that has been working too long to have done it.
The three ways this family has now appeared here
All three confuse a record of the past with the state of the present, and they differ in which direction they lie:
- Stale data read as fresh. A monitoring system reported every monitor UP while it was switched off, because a heartbeat table records what was last seen. That post covers it. Fails toward reassurance.
- Absence read as success. A blocklist sweep reported clean because the resolver was blocked and every query returned empty, which is exactly what “not listed” looks like. That one too. Also fails toward reassurance.
- Presence-in-a-window read as health. This one. Fails toward alarm, and in proportion to how well the system is running.
The first two get discovered during an incident, when you look closely and find the instrument was never working. The third gets discovered when someone stops believing the instrument — which takes longer and does more damage on the way.
The uncomfortable part
The check was written to catch a specific failure: the tunnel silently dropping without anyone noticing. That is a real risk and a sensible thing to guard.
The author reached for the log because the log is where the event lives, and the event is what a dropped tunnel produces. The mistake was assuming the inverse — that the presence of a connect event is evidence of a current connection.
It is not. It is evidence that a connection was made once, within whatever window you happened to choose. The window is arbitrary, and the arbitrary choice is what determines whether a healthy system passes.
Choosing twenty-four hours felt generous. It was, in effect, an assertion that the tunnel must have restarted within the last day — which is close to the opposite of what anybody wanted to be true.