Two scripts sit on the lab’s gateways waiting for the day the primary one dies. One moves the public DNS records to the standby. The other moves them back. Both were, until this week, incapable of telling anyone they had run.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
/usr/local/bin/failback.sh: line 56: NTFY_ADMIN_TOKEN: unbound variableThat is the whole defect. It took one line of output to find and it had been sitting there since the scripts were written.
What was actually wrong
Both scripts load a secrets file at startup and then use a token from it to send a push notification. That token is not defined on either gateway. The file exists, and holds other credentials, but never that one.
Both scripts also run with strict error handling, which is the correct setting and the reason this is a crash rather than a shrug. Under set -u, reading an undefined variable is fatal. So the first line of the notification function aborted the script that called it.
Why that is worse than a missing notification
The two scripts fail differently, and one of them is much worse.
Failover would have changed every public DNS record, pointing the lab’s front door at the standby gateway, and then crashed on its way to saying so. The cutover succeeds. The record of it never leaves the machine. A real failover completes in total silence, and the first anyone hears about it is whatever they notice by hand.
Failback aborted correctly, and then crashed while sending the message explaining that it had aborted.
Both fail at the exact moment an operator most needs to hear something. And both are silent by construction: the thing that would have told you is the thing that died.
Failback is manual by design. The failover direction can trigger itself from a health check, but coming back never happens without a person confirming it.
The reason is flapping. A gateway that is failing intermittently would, under symmetrical automation, move the records back and forth on every state change, and DNS is a bad place to do that: records propagate on their own schedule, so a fast oscillation leaves different parts of the internet holding different answers, some of them stale, none of them coordinated. Asymmetry is the guard. Leaving is cheap and reversible; returning is a judgment about whether the fault is actually over, and the record cannot make that judgment.
That design decision is what makes this bug sharp rather than merely annoying. The failback script was doing exactly the right thing: refusing to act, and reporting the refusal so a person could take over. The refusal was the feature. Reporting it was the entire point of the refusal, and reporting it is what killed the script.
How it was found
By running the refusal path, not by reading it.
The script was invoked in the state where it is supposed to decline, to check that it declined. It did, and then it died on the way out. Nothing in the source reads as wrong: the notification function is unremarkable, the variable name matches the runbook’s stated prerequisite, and the prerequisite is simply not true of these hosts.
This is a pattern that keeps recurring in this project. The code that runs least is trusted most. Error paths, refusal paths and alert paths are exercised rarely, usually only when something else has already gone wrong, and they accumulate defects that a reading pass will not catch because there is nothing visibly wrong to catch. The same lesson arrived from a different direction when a backup check inspected a file it had invented, and again when a verification step could not fail.
The fix
Two changes, both small.
The corrected notification path
A missing token now degrades to a log line rather than an abort:
if [[ -z "${NTFY_ADMIN_TOKEN:-}" ]]; then
log "NOTIFY NOT SENT (token unset): [$1] $4"
return 0
fiThe :- is what stops set -u from firing. It marks the variable as one that may legitimately be absent, which is exactly what it is.
The second change is to the send itself. It had been written to swallow its own failure with a trailing || true, so a rejected or unreachable push endpoint produced nothing at all. A failed send is now logged too. There is a real difference between did not try and tried and failed, and only one of them suggests the credential is wrong.
The credential itself is still not populated. That is deliberate: it is a live secret, so it is a person’s job rather than a script’s, and it is queued as such. What changed is that its absence now costs a log line instead of the whole run.
What generalizes
set -u turns every optional variable into a hard dependency. It is the right default and it should stay on. But it means any variable that might legitimately be missing needs ${VAR:-} at the point of use, and the places to audit first are the error paths, because they run least and are trusted most.
A notification path must never be able to kill its caller. Alerting is a side effect. Failing to alert should downgrade to a log line, never to an abort, because the caller is usually doing something more important than talking. A script that changes DNS and then fails to send a message has still changed DNS, and the operator is now worse off than if it had never tried.
Test the alert with the credential absent. That is the state it will actually be in the first time it matters, on a host built in a hurry or restored from a backup that did not carry secrets. Testing it with the token present proves the happy path, which is not the path that hurts.
There is a broader version of this that this lab keeps rediscovering. The project’s failover capability is not the scripts. It is the scripts plus the knowledge that they ran. Without the second part the first part is an uncontrolled change to public DNS, and until this week that is what it was.