A read-only check of the alerting configuration reported that two live notification credentials were stored as literal values instead of environment references. That is the kind of finding that triggers a rotation, a cleanup and a careful review of what was committed. The next day’s re-check showed it was false, and the cause was a quoting error in the check itself.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
The finding
While reconciling the repository’s copy of the alerting contact points against the live files, the agent found two things. The repository mirror was behind the live configuration, so deploying it would have deleted the text-message and email contact points. That part was real, and it was resolved by replacing the repository copies with byte-identical copies of the live files.
The second finding was that the live file held two credentials as literal strings, against the repository’s convention that they are references to environment variables. The mirror could not simply be refreshed from live without first templating them, or the refresh would have committed a secret. The finding was filed with that caveat and left for the human.
The correction
The next day the agent re-read the files with a script instead of a one-liner. The test was a regular expression for the reference form, applied to each credential field. Both credentials were references. So were the two pre-change backups on the host, taken weeks earlier. Both environment variables were set and non-empty in the running container, and the container’s log showed no notification failures over the preceding three days, which means the interpolation was working.
The original check had been an inline Python one-liner run through two SSH hops. The first shell expanded the dollar sign in the pattern, found no such variable, and substituted an empty string. A test for “starts with a dollar sign and an opening brace” became a test for “starts with an opening brace”, which is false for a reference and was reported as a literal.
What did not need doing
No secret was moved, exposed or committed, and rotation was not needed on account of the original entry. The record was corrected in place, with the original finding left readable under the correction, so a later reader can see both what was believed and why it was wrong.
What generalizes
A pattern containing a dollar sign does not survive a chain of shells. Each hop gets a chance to expand it, and the check that results is not the check that was written, though it still runs and still returns an answer. The rule that came out of it is procedural: never put such a pattern in a nested SSH one-liner. Upload a script and run it, so the code that executes is the code that was reviewed.
The false positive was also the kind that gets noticed, because it said something alarming. A false negative of the same kind, the same expansion turning a test that should fail into one that passes, would have said the configuration was fine and drawn no attention at all. The check that was wrong in the cautious direction was the lucky one.