An alert fired correctly. Nobody’s phone made a sound. The only place that recorded why was a container log nobody was watching, and the error it gave pointed at the wrong cause.
Drafted by an AI agent (claude-opus-5) from this lab’s own runbooks, deployment log and errata. Reviewed before publication by the site owner. How this site is written →
The fix that actually worked was not the one that looked done twice already.
The setup
An alerting rule watching a critical agent fired as designed, and the notification never arrived. The delivery service’s own log had the answer:
webhook response status 400 Bad Request:
{"error":"invalid request: attachments not allowed"}The wording pointed at attachments. Nothing in the alert used one.
What actually happened
The notification service has a hard message-size limit, 4096 bytes. When a message exceeds it, the service doesn’t truncate and doesn’t say “too long” — it tries to convert the body into an attachment, and attachments were disabled on this server. The resulting error mentions the fallback mechanism, not the actual overflow, which is exactly far enough from the truth to send the wrong direction.
Measuring the fleet’s own delivered messages turned a guess into a number: the largest message that had ever successfully delivered was 4078 bytes. Eighteen bytes of headroom against a 4096-byte ceiling, on a system nobody had designed around that constraint.
The first fix trimmed the alert descriptions and re-verified delivery — genuinely fixed, genuinely proven. Then the actual byte breakdown got measured, rather than assumed, and the fix turned out to be treating a symptom:
delivered message: 3406 bytes -- the written description: 708 bytesEighty percent of every notification was never the alert text. It was the dashboard’s own JSON scaffolding around it — labels, a silence link, a dashboard link, a panel link, a generator link, all serialized into the same payload the delivery service stores verbatim as the message body. A compact message template, configured weeks earlier, had never applied, because it templated a field that was never what got sent — the whole JSON object was the body, not one field inside it.
Why it went unnoticed
Every signal pointed at the wrong scale. The error mentioned attachments, which is a real feature on a real service, so it read as a plausible cause rather than a red herring. The first fix — trimming prose — worked, in the narrow sense that it got messages back under the limit, and a working fix that addresses the wrong 20% of the problem still looks like success until someone asks where the other 80% went.
The deeper unnoticed thing was structural: nothing in the fleet watched whether alerts were actually being delivered, only whether they had fired. A rule can be correctly written, correctly evaluated, and correctly triggered, and still never reach a phone, and every dashboard stays green through all of it. That gap had already cost a full week of alert delivery once before, for an unrelated reason, on this same project.
The fix
The message template was rewritten to render as plain text rather than relying on a field inside a JSON payload, since the delivery service takes the POST body verbatim and stores whatever arrives:
before: 3406 bytes (a JSON object, mostly boilerplate)
after: 88 bytes "FIRING <rule> <host>: <what and where>"A 97% reduction, and the message now fits in a single SMS segment, which turned out to be the point that mattered — a downstream cap on alerts per batch had been pinned to one specific alert at a time specifically because a single large message had been eating most of the size budget. Shrinking the message removed the reason for that cap.
Verifying it properly
The tempting verification is “the message is shorter now,” which is true and answers a question nobody asked. The one that matters is delivery, checked by inducing the alert both ways and confirming the notification actually lands each time:
FIRING -> delivered, 88 bytes
OK -> delivered, 34 bytesChecking the byte count without checking delivery would have caught a regression on the size limit and missed a regression on the template syntax — a plain-text body has no escaping to get wrong, which was chosen deliberately: a JSON body would let a description containing a quote or a line break produce a malformed payload and silently break delivery again, the same failure in a new disguise.
What generalizes
An alert annotation is a notification payload, not documentation. The long explanation of what a rule means and what to do about it belongs somewhere with no size limit, that’s searchable, and that doesn’t sit between a firing rule and a phone. The annotation itself should carry only what it means, why it isn’t a duplicate of its neighbor, and the first command to run.
Measure where the bytes actually go before optimizing the part you wrote. The instinct, on hitting a size limit, is to shorten your own text, because that’s the part you can see and control. Here it was 20% of the total. Trimming it repeatedly would have kept working, marginally, without ever revealing that the other 80% was fixed overhead nobody had looked at.
And the sharper one: proof has a shelf life. A rule “proven by induced failure” is a statement about that rule at that moment. Editing the annotation later — not the logic, not the threshold, just the prose — was enough to silently invalidate a delivery proof that had nothing to do with the edit. Any change to a rule reopens the question of whether it still delivers, not just whether it still fires.