fix(obs): the deploy notification was inhibited before it could send

FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.

kube-prometheus-stack ships general.rules -> InfoInhibitor:

  group by (namespace) (ALERTS{severity="info"} == 1)
    unless on (namespace) group by (namespace)
      (ALERTS{alertname!="InfoInhibitor", alertstate="firing",
              severity=~"warning|critical"} == 1)

which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.

For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.

Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.

Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.

Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform/monitoring/alerts.yaml

@@ -839,6 +839,14 @@ spec:
         # that has never applied anything — or a GitRepository with no artifact yet — publishes the
         # series with no `revision` label. Without the matcher those group together under the empty
         # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
+        #
+        # A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
+        # because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
+        # pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
+        # InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
+        # it is the only thing firing in its namespace, which is every healthy deploy. Measured on
+        # prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
+        # was inhibited in between. The full reasoning is on the `notification: deploy` label below.
         # ===================================================================================
         - alert: FluxRolloutInProgress
           expr: |
@@ -857,7 +865,31 @@ spec:
           # `flux reconcile`, a change touching one Kustomization), and a deploy notification that
           # only sometimes arrives is worse than no notification at all.
           labels:
-            severity: info    # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied"
+            # ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
+            # This alert originally used `severity: info`. That was WRONG, and wrong in the silent
+            # direction — measured on prod within minutes of the first real deploy.
+            # kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
+            #     group by (namespace) (ALERTS{severity="info"} == 1)
+            #       unless on (namespace) group by (namespace)
+            #         (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+            # which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
+            # it; paired with the chart's inhibit_rule
+            #     source_matchers: [alertname="InfoInhibitor"]
+            #     target_matchers: [severity="info"]
+            #     equal: [namespace]
+            # the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
+            # happening in its namespace. For a deploy notification that is the only case that
+            # matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
+            # critical alert, so InfoInhibitor fired and this alert never reached the receiver.
+            # Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
+            # alert began firing for 12 Kustomizations.
+            # Upstream's intent is explicit — `severity: info` means "only worth showing next to a
+            # real problem". A deploy notice is not that, so it must not claim that severity.
+            # `notification` is a label no inhibit_rule in the chart targets (all three match on
+            # severity values), and omitting `severity` also stops InfoInhibitor firing for
+            # flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
+            # `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
+            notification: deploy
           annotations:
             summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
             description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."