fix(obs): the deploy notification was inhibited before it could send #26

2026-09-30T05:07:03Z

gitea-admin commented

2026-09-30 05:07:03 +00:00

The defect this fixes

FluxRolloutInProgress (PR #25) fired correctly on the first real deploy and its mail was suppressed before Alertmanager dispatched it. One label was wrong.

kube-prometheus-stack ships general.rules → InfoInhibitor:

group by (namespace) (ALERTS{severity="info"} == 1)
  unless on (namespace) group by (namespace)
    (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)

It fires for any namespace holding a firing severity="info" alert and nothing else wrong in it. The chart's third inhibit_rule then drops exactly that alert:

source_matchers: [alertname="InfoInhibitor"]
target_matchers: [severity="info"]
equal:           [namespace]

For a deploy notification this is the worst possible interaction, because the suppressed case is the only case that matters: on a healthy rollout of a healthy cluster flux-system holds no warning or critical alert, so the notice is inhibited precisely when it is wanted — and would have arrived only when something else in flux-system was already broken.

Measured on prod: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment the alert began firing for 12 Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager — neither active nor --inhibited — with alertmanager_notifications_failed_total at 0 for every reason. Nothing failed to send; it was never dispatched.

Upstream's intent is explicit and fair: severity: info means "only worth showing next to a real problem." A deploy notice is not that, so it must not claim that severity.

The fix

FluxRolloutInProgress now carries notification: deploy and no severity at all, and the route matches that label. No inhibit_rule in the chart targets anything but severity values, and omitting severity also stops InfoInhibitor firing for flux-system at all rather than merely working around it. FluxRolloutStuck keeps severity: warning and is unaffected. Receiver, subject and grouping unchanged.

Both files carry the reasoning inline, and the design header in alerts.yaml now names the severity label as a third thing that must not regress — "do not tidy this by adding a severity" — because the next reader's instinct will be to add one back.

Verified on prod while diagnosing this — all previously UNVERIFIED

The metrics work: kube_flux_kustomization_info 16, _suspended 16, kube_flux_gitrepository_info 1 = 33 series, exactly as designed.
The revision label IS populated (main@sha1:296d6d3… / …64c72fd…), so the status.lastAppliedRevision / status.artifact.revision paths are correct.
The podAnnotations roll worked — KSM carries crs-config-rollout: 1 on a new pod, replacing the 2026-09-19 one. The config-only-change trap was genuinely closed, not just documented.
The rule works in both directions on real data: fired for 12 lagging Kustomizations while 4 had converged, each with its own name and revision; then convergence (count(count by (revision) (…)) → 1) and the alert cleared. Firing and self-clearing are the completion notification's whole mechanism.
Routing proven: amtool config routes test notification=deploy → email-deploy.
promtool: SUCCESS, 53 rules, 6 scenarios, 3.14.0 in the pinned toolbox.

Gate

rc 3 (UNDETERMINED), reported as-is. Green: spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1. Did not run: gitea-settings (no local SETTINGS_API_TOKEN) and L2/L3 conformance (no cluster). Touched requirements R-OBS-01 R-OBS-02.

This branch touches no sensitive path (only platform/monitoring/** and decisions/**), so under the new policy it should merge on a green gate without a human approval.

Still not verified

No deploy mail observed in an inbox. Suppression was proven and routing was proven; the delivered message is not. Only the next commit after this merges can prove it.
The 226 → 227 email counter increment during the window cannot be attributed — no receiver label on that metric, and Watchdog/CNPGPrimariesCoLocated were also active. I am not claiming our mail was or wasn't sent.
InfoInhibitor vs FluxRolloutStuck (severity: warning) is reasoned safe, not measured.
The [FLUX] applying|applied subject template is still untested.

🤖 Generated with Claude Code

## The defect this fixes `FluxRolloutInProgress` (PR #25) **fired correctly on the first real deploy and its mail was suppressed before Alertmanager dispatched it.** One label was wrong. kube-prometheus-stack ships `general.rules` → `InfoInhibitor`: ``` group by (namespace) (ALERTS{severity="info"} == 1) unless on (namespace) group by (namespace) (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1) ``` It fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in it**. The chart's third `inhibit_rule` then drops exactly that alert: ``` source_matchers: [alertname="InfoInhibitor"] target_matchers: [severity="info"] equal: [namespace] ``` For a deploy notification this is the worst possible interaction, because **the suppressed case is the only case that matters**: on a healthy rollout of a healthy cluster `flux-system` holds no warning or critical alert, so the notice is inhibited precisely when it is wanted — and would have arrived only when something else in `flux-system` was already broken. Measured on prod: `InfoInhibitor` went active `2026-09-30T04:58:48Z`, the same moment the alert began firing for 12 Kustomizations, and `amtool` showed **no** `FluxRolloutInProgress` in Alertmanager — neither active nor `--inhibited` — with `alertmanager_notifications_failed_total` at 0 for every reason. Nothing failed to send; it was never dispatched. Upstream's intent is explicit and fair: `severity: info` means *"only worth showing next to a real problem."* A deploy notice is not that, so it must not claim that severity. ## The fix `FluxRolloutInProgress` now carries **`notification: deploy` and no `severity` at all**, and the route matches that label. No `inhibit_rule` in the chart targets anything but severity values, and omitting `severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working around it. `FluxRolloutStuck` keeps `severity: warning` and is unaffected. Receiver, subject and grouping unchanged. Both files carry the reasoning inline, and the design header in `alerts.yaml` now names the severity label as a third thing that must not regress — *"do not tidy this by adding a severity"* — because the next reader's instinct will be to add one back. ## Verified on prod while diagnosing this — all previously UNVERIFIED - **The metrics work:** `kube_flux_kustomization_info` 16, `_suspended` 16, `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed.** - **The `revision` label IS populated** (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` / `status.artifact.revision` paths are correct. - **The `podAnnotations` roll worked** — KSM carries `crs-config-rollout: 1` on a new pod, replacing the 2026-09-19 one. The config-only-change trap was genuinely closed, not just documented. - **The rule works in both directions on real data:** fired for **12** lagging Kustomizations while 4 had converged, each with its own `name` and `revision`; then convergence (`count(count by (revision) (…))` → **1**) and the alert **cleared**. Firing and self-clearing are the completion notification's whole mechanism. - **Routing proven:** `amtool config routes test notification=deploy` → `email-deploy`. - promtool: **SUCCESS**, 53 rules, 6 scenarios, 3.14.0 in the pinned toolbox. ## Gate `rc 3 (UNDETERMINED)`, reported as-is. Green: `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. Did not run: `gitea-settings` (no local `SETTINGS_API_TOKEN`) and L2/L3 `conformance` (no cluster). Touched requirements `R-OBS-01 R-OBS-02`. This branch touches **no sensitive path** (only `platform/monitoring/**` and `decisions/**`), so under the new policy it should merge on a green gate without a human approval. ## Still not verified - **No deploy mail observed in an inbox.** Suppression was proven and routing was proven; the *delivered message* is not. Only the next commit after this merges can prove it. - **The 226 → 227 email counter increment during the window cannot be attributed** — no receiver label on that metric, and `Watchdog`/`CNPGPrimariesCoLocated` were also active. I am not claiming our mail was or wasn't sent. - `InfoInhibitor` vs `FluxRolloutStuck` (`severity: warning`) is reasoned safe, not measured. - The `[FLUX] applying|applied` subject template is still untested. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-09-30 05:07:04 +00:00

fix(obs): the deploy notification was inhibited before it could send

gate / gate (pull_request) Successful in 1m39s

Details

gate / conformance (pull_request) Has been skipped

Details

8427b28b0e

FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.

kube-prometheus-stack ships general.rules -> InfoInhibitor:

  group by (namespace) (ALERTS{severity="info"} == 1)
    unless on (namespace) group by (namespace)
      (ALERTS{alertname!="InfoInhibitor", alertstate="firing",
              severity=~"warning|critical"} == 1)

which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.

For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.

Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.

Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.

Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes

2026-09-30 15:18:51 +00:00

jakman85 merged commit 0fbf544d5e into main

2026-09-30 15:18:54 +00:00

jakman85 referenced this issue from a commit

2026-09-30 15:18:56 +00:00

Merge pull request 'fix(obs): the deploy notification was inhibited before it could send' (#26) from fix/flux-deploy-notification-inhibited into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#26