home-network-k3s-impl/decisions/2026-09-30-flux-deploy-notification-inhibited.md

---

date: 2026-09-30 branch: fix/flux-deploy-notification-inhibited title: The deploy notification routed correctly and was then inhibited before sending — severity: info is a suppressed class in kube-prometheus-stack

What changed

FluxRolloutInProgress (shipped hours earlier in PR #25) fired correctly on the very first real deploy and its mail was suppressed before Alertmanager sent it. One label changed:

• platform/monitoring/alerts.yaml — FluxRolloutInProgress now carries notification: deploy and no severity label at all, where it previously carried severity: info.
• platform/monitoring/kube-prometheus-stack.yaml — the deploy route matches notification = "deploy" instead of severity = "info". Receiver, subject and grouping unchanged.
• platform/monitoring/tests/flux-rollout.test.yaml — the three FluxRolloutInProgress exp_labels blocks updated. FluxRolloutStuck keeps severity: warning and is unaffected.

The mechanism, measured rather than reasoned about. kube-prometheus-stack's general.rules ships:

InfoInhibitor:
  group by (namespace) (ALERTS{severity="info"} == 1)
    unless on (namespace) group by (namespace)
      (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)

which fires for any namespace holding a firing severity="info" alert and nothing else wrong in it. The chart's third inhibit_rule then drops exactly that alert:

source_matchers: [alertname="InfoInhibitor"]
target_matchers: [severity="info"]
equal:           [namespace]

For a deploy notification this is the worst possible interaction, because the suppressed case is the only case that matters: on a healthy rollout of a healthy cluster, flux-system holds no warning or critical alert, so InfoInhibitor fires and the deploy mail is inhibited. It would have been delivered only when something else in flux-system was already broken.

Upstream's intent is explicit and reasonable — severity: info means "only worth showing next to a real problem". A deploy notice is not that, so it must not claim that severity. notification is a label no inhibit_rule in the chart targets (all three match on severity values), and dropping severity also stops InfoInhibitor firing for flux-system at all rather than merely working around it. Both files carry the full reasoning inline, and alerts.yaml's design header now names the severity label as a third thing that must not regress — "do not tidy this by adding a severity" — because the next reader's instinct will be to add one back.

What I verified

Against live prod, after PR #25 merged as 64c72fd and Flux converged:

• The metrics work. kube_flux_kustomization_info 16, kube_flux_kustomization_suspended 16, kube_flux_gitrepository_info 1 = 33 series, exactly as designed. The revision label IS populated (main@sha1:296d6d3… / …64c72fd…), so the status.lastAppliedRevision / status.artifact.revision paths are correct — previously UNVERIFIED.
• The podAnnotations roll mechanism works. The KSM pod carries homelab.keithjackson.dev/crs-config-rollout: 1 and is a new pod (…-7b56d98f46-56n8j, replacing the 2026-09-19 one), so the config-only-change trap was genuinely closed rather than merely documented.
• The rule works, in both directions. It fired for 12 lagging Kustomizations while 4 had converged (count by (revision) → 12 old / 4 new), each alert carrying its own name and revision; then the rollout converged (count(count by (revision) (…)) → 1) and the alert cleared. Firing and self-clearing are both confirmed on real data, which is the completion notification's whole mechanism.
• The routing works. amtool config routes shows {notification="deploy"} → receiver: email-deploy under the default route, and amtool config routes test notification=deploy returns email-deploy.
• The inhibition was real. InfoInhibitor went active 2026-09-30T04:58:48Z — the same moment the alert began firing — and amtool alert query showed no FluxRolloutInProgress in Alertmanager, active or --inhibited. alertmanager_notifications_failed_total{integration=email} is 0 for every reason, so nothing failed to send; it was never dispatched.
• promtool: SUCCESS, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.

What I did NOT verify

• No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end. The suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven (amtool config routes test → email-deploy), but the delivered message is not. It can only be proven by merging this and watching the next commit produce [FLUX] applying and then [FLUX] applied. Same structural gap as PR #25, one layer further along.
• Whether the single email during the window was ours. alertmanager_notifications_total{integration="email"} went 226 → 227 while other alerts (Watchdog, CNPGPrimariesCoLocated) were also active, and that counter carries no receiver label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the attribution is genuinely unavailable from this metric.
• Whether InfoInhibitor interacts with FluxRolloutStuck. Stuck is severity: warning; inhibit_rule 1 (critical → warning|info) requires an equal alertname, so it should not apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
• The [FLUX] applying|applied subject template is still untested; .Status is assumed to be firing/resolved in the notification template data.
• required_approvals disagreement, unresolved and not mine to settle. Live branch protection reads required_approvals: 1, while the ci/gate.sh comment introduced by PR #17 describes the model as required_approvals: 0 ("the gate is the approver"). ci/gitea-settings.sh may report this as drift against infra/gitea/expected-branch-protection.json. Flagged only.
• The drift loop is still unexplained. Five Kustomizations re-applying the same object every 10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.