--- date: 2026-09-30 branch: fix/flux-deploy-notification-inhibited title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack --- ## What changed `FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real deploy and **its mail was suppressed before Alertmanager sent it**. One label changed: - `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and **no `severity` label at all**, where it previously carried `severity: info`. - `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches `notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged. - `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels` blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected. **The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules` ships: InfoInhibitor: group by (namespace) (ALERTS{severity="info"} == 1) unless on (namespace) group by (namespace) (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1) which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in it**. The chart's third `inhibit_rule` then drops exactly that alert: source_matchers: [alertname="InfoInhibitor"] target_matchers: [severity="info"] equal: [namespace] For a deploy notification this is the worst possible interaction, because the suppressed case is the *only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been delivered only when something else in `flux-system` was already broken. Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping `severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names the severity label as a third thing that must not regress — "do not tidy this by adding a severity" — because the next reader's instinct will be to add one back. ## What I verified Against live prod, after PR #25 merged as `64c72fd` and Flux converged: - **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16, `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` / `status.artifact.revision` paths are correct — previously UNVERIFIED. - **The `podAnnotations` roll mechanism works.** The KSM pod carries `homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod (`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was genuinely closed rather than merely documented. - **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and `revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert **cleared**. Firing and self-clearing are both confirmed on real data, which is the completion notification's whole mechanism. - **The routing works.** `amtool config routes` shows `{notification="deploy"} → receiver: email-deploy` under the default route, and `amtool config routes test notification=deploy` returns `email-deploy`. - **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}` is 0 for every reason, so nothing failed to send; it was never dispatched. - promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox. ## What I did NOT verify - **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven (`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be proven by merging this and watching the next commit produce `[FLUX] applying` and then `[FLUX] applied`. Same structural gap as PR #25, one layer further along. - **Whether the single email during the window was ours.** `alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts (`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the attribution is genuinely unavailable from this metric. - **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`; inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs. - **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be `firing`/`resolved` in the notification template data. - **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only. - **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every 10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.