From 8427b28b0ecb6f285417c0e5d89cd672ce74f800 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 00:05:10 -0500 Subject: [PATCH] fix(obs): the deploy notification was inhibited before it could send MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FluxRolloutInProgress fired correctly on the first real deploy and its mail was suppressed before Alertmanager dispatched it. One label was wrong. kube-prometheus-stack ships general.rules -> InfoInhibitor: group by (namespace) (ALERTS{severity="info"} == 1) unless on (namespace) group by (namespace) (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1) which fires for any namespace holding a firing info alert AND NOTHING ELSE wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor -> target severity="info", equal [namespace]) then drops exactly that alert. For a deploy notification that is the worst possible interaction, because the suppressed case is the ONLY case that matters: on a healthy rollout of a healthy cluster flux-system holds no warning or critical alert, so the notification is inhibited precisely when it is wanted, and would arrive only when something else in flux-system was already broken. Measured on prod: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment the alert began firing for 12 Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager either active or --inhibited, with zero send failures. Upstream's intent is explicit and fair: severity "info" means "only worth showing next to a real problem". A deploy notice is not that, so it must not claim that severity. FluxRolloutInProgress now carries `notification: deploy` and no severity at all, and the route matches on that label — no inhibit_rule in the chart targets anything but severity values, and omitting severity also stops InfoInhibitor firing for flux-system rather than merely working around it. FluxRolloutStuck keeps severity: warning and is unaffected. Also verified on prod in the process, all previously UNVERIFIED: the 33 metric series exist exactly as designed, the `revision` label IS populated (so the lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and then CLEARED on convergence — the completion signal's whole mechanism, on real data. Routing proven by `amtool config routes test notification=deploy` -> email-deploy. Still unverified: no mail observed in the inbox. The 226->227 email counter increment during the window cannot be attributed (no receiver label, other alerts active), and I am not claiming it either way. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) --- ...9-30-flux-deploy-notification-inhibited.md | 95 +++++++++++++++++++ platform/monitoring/alerts.yaml | 34 ++++++- .../monitoring/kube-prometheus-stack.yaml | 18 +++- .../monitoring/tests/flux-rollout.test.yaml | 6 +- 4 files changed, 144 insertions(+), 9 deletions(-) create mode 100644 decisions/2026-09-30-flux-deploy-notification-inhibited.md diff --git a/decisions/2026-09-30-flux-deploy-notification-inhibited.md b/decisions/2026-09-30-flux-deploy-notification-inhibited.md new file mode 100644 index 0000000..f3315f2 --- /dev/null +++ b/decisions/2026-09-30-flux-deploy-notification-inhibited.md @@ -0,0 +1,95 @@ +--- +date: 2026-09-30 +branch: fix/flux-deploy-notification-inhibited +title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack +--- + +## What changed + +`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real +deploy and **its mail was suppressed before Alertmanager sent it**. One label changed: + +- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and + **no `severity` label at all**, where it previously carried `severity: info`. +- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches + `notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged. +- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels` + blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected. + +**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules` +ships: + + InfoInhibitor: + group by (namespace) (ALERTS{severity="info"} == 1) + unless on (namespace) group by (namespace) + (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1) + +which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in +it**. The chart's third `inhibit_rule` then drops exactly that alert: + + source_matchers: [alertname="InfoInhibitor"] + target_matchers: [severity="info"] + equal: [namespace] + +For a deploy notification this is the worst possible interaction, because the suppressed case is the +*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning +or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been +delivered only when something else in `flux-system` was already broken. + +Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a +real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a +label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping +`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working +around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names +the severity label as a third thing that must not regress — "do not tidy this by adding a severity" +— because the next reader's instinct will be to add one back. + +## What I verified + +Against live prod, after PR #25 merged as `64c72fd` and Flux converged: + +- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16, + `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS + populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` / + `status.artifact.revision` paths are correct — previously UNVERIFIED. +- **The `podAnnotations` roll mechanism works.** The KSM pod carries + `homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod + (`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was + genuinely closed rather than merely documented. +- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had + converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and + `revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert + **cleared**. Firing and self-clearing are both confirmed on real data, which is the completion + notification's whole mechanism. +- **The routing works.** `amtool config routes` shows + `{notification="deploy"} → receiver: email-deploy` under the default route, and + `amtool config routes test notification=deploy` returns `email-deploy`. +- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment + the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in + Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}` + is 0 for every reason, so nothing failed to send; it was never dispatched. +- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox. + +## What I did NOT verify + +- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The + suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven + (`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be + proven by merging this and watching the next commit produce `[FLUX] applying` and then + `[FLUX] applied`. Same structural gap as PR #25, one layer further along. +- **Whether the single email during the window was ours.** + `alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts + (`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver + label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the + attribution is genuinely unavailable from this metric. +- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`; + inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not + apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs. +- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be + `firing`/`resolved` in the notification template data. +- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection + reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the + model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report + this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only. +- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every + 10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch. diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index 41bb8c2..e007f2d 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -839,6 +839,14 @@ spec: # that has never applied anything — or a GitRepository with no artifact yet — publishes the # series with no `revision` label. Without the matcher those group together under the empty # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged. + # + # A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered + # because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this + # pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's + # InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever + # it is the only thing firing in its namespace, which is every healthy deploy. Measured on + # prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and + # was inhibited in between. The full reasoning is on the `notification: deploy` label below. # =================================================================================== - alert: FluxRolloutInProgress expr: | @@ -857,7 +865,31 @@ spec: # `flux reconcile`, a change touching one Kustomization), and a deploy notification that # only sometimes arrives is worse than no notification at all. labels: - severity: info # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied" + # ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30). + # This alert originally used `severity: info`. That was WRONG, and wrong in the silent + # direction — measured on prod within minutes of the first real deploy. + # kube-prometheus-stack ships `general.rules` -> InfoInhibitor: + # group by (namespace) (ALERTS{severity="info"} == 1) + # unless on (namespace) group by (namespace) + # (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1) + # which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in + # it; paired with the chart's inhibit_rule + # source_matchers: [alertname="InfoInhibitor"] + # target_matchers: [severity="info"] + # equal: [namespace] + # the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing + # happening in its namespace. For a deploy notification that is the only case that + # matters: on a healthy rollout of a healthy cluster, flux-system has no warning or + # critical alert, so InfoInhibitor fired and this alert never reached the receiver. + # Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this + # alert began firing for 12 Kustomizations. + # Upstream's intent is explicit — `severity: info` means "only worth showing next to a + # real problem". A deploy notice is not that, so it must not claim that severity. + # `notification` is a label no inhibit_rule in the chart targets (all three match on + # severity values), and omitting `severity` also stops InfoInhibitor firing for + # flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches + # `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied". + notification: deploy annotations: summary: "Flux is applying a new revision — {{ $labels.name }} is behind main" description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete." diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index 047b128..1f75c1c 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -139,11 +139,19 @@ spec: # # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a - # real platform fault, which is how alert mail stops being read at all. `severity: info` - # is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this - # route claims exactly the deploy notices and nothing else. Keep it that way — giving a - # genuine fault severity: info would silently route it here and re-label it a deploy. - - matchers: ['severity = "info"'] + # real platform fault, which is how alert mail stops being read at all. + # + # MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read + # `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it + # was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing + # info alert with no warning/critical beside it, and its inhibit_rule + # (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops + # exactly that alert. A healthy deploy in a healthy flux-system is precisely that case, + # so the notification was inhibited on every occasion it was wanted. Measured live on + # prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing. + # Routing on a non-severity label sidesteps all three of the chart's inhibit_rules, + # which match on severity values only. See the label's own comment in alerts.yaml. + - matchers: ['notification = "deploy"'] receiver: email-deploy # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging # Kustomization (16 possible), and grouping on alertname alone collapses them into a diff --git a/platform/monitoring/tests/flux-rollout.test.yaml b/platform/monitoring/tests/flux-rollout.test.yaml index dff9581..ec01525 100644 --- a/platform/monitoring/tests/flux-rollout.test.yaml +++ b/platform/monitoring/tests/flux-rollout.test.yaml @@ -89,7 +89,7 @@ tests: alertname: FluxRolloutInProgress exp_alerts: - exp_labels: - severity: info + notification: deploy job: kube-state-metrics namespace: flux-system name: platform @@ -148,7 +148,7 @@ tests: alertname: FluxRolloutInProgress exp_alerts: - exp_labels: - severity: info + notification: deploy job: kube-state-metrics namespace: flux-system name: platform @@ -229,7 +229,7 @@ tests: alertname: FluxRolloutInProgress exp_alerts: - exp_labels: - severity: info + notification: deploy job: kube-state-metrics namespace: flux-system name: conformance