fix(obs): the deploy notification was inhibited before it could send

FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.

kube-prometheus-stack ships general.rules -> InfoInhibitor:

  group by (namespace) (ALERTS{severity="info"} == 1)
    unless on (namespace) group by (namespace)
      (ALERTS{alertname!="InfoInhibitor", alertstate="firing",
              severity=~"warning|critical"} == 1)

which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.

For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.

Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.

Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.

Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform/monitoring/kube-prometheus-stack.yaml

@@ -139,11 +139,19 @@ spec:
             #
             # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
             # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
-            # real platform fault, which is how alert mail stops being read at all. `severity: info`
-            # is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this
-            # route claims exactly the deploy notices and nothing else. Keep it that way — giving a
-            # genuine fault severity: info would silently route it here and re-label it a deploy.
-            - matchers: ['severity = "info"']
+            # real platform fault, which is how alert mail stops being read at all.
+            #
+            # MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read
+            # `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it
+            # was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing
+            # info alert with no warning/critical beside it, and its inhibit_rule
+            # (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops
+            # exactly that alert. A healthy deploy in a healthy flux-system is precisely that case,
+            # so the notification was inhibited on every occasion it was wanted. Measured live on
+            # prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing.
+            # Routing on a non-severity label sidesteps all three of the chart's inhibit_rules,
+            # which match on severity values only. See the label's own comment in alerts.yaml.
+            - matchers: ['notification = "deploy"']
               receiver: email-deploy
               # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
               # Kustomization (16 possible), and grouping on alertname alone collapses them into a