jakman85andClaude Opus 5
8427b28b0e
FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.
kube-prometheus-stack ships general.rules -> InfoInhibitor:
group by (namespace) (ALERTS{severity="info"} == 1)
unless on (namespace) group by (namespace)
(ALERTS{alertname!="InfoInhibitor", alertstate="firing",
severity=~"warning|critical"} == 1)
which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.
For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.
Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.
Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.
Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
96 lines
6.2 KiB
Markdown
96 lines
6.2 KiB
Markdown
---
|
|
date: 2026-09-30
|
|
branch: fix/flux-deploy-notification-inhibited
|
|
title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
|
|
---
|
|
|
|
## What changed
|
|
|
|
`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
|
|
deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
|
|
|
|
- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
|
|
**no `severity` label at all**, where it previously carried `severity: info`.
|
|
- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
|
|
`notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
|
|
- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
|
|
blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
|
|
|
|
**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
|
|
ships:
|
|
|
|
InfoInhibitor:
|
|
group by (namespace) (ALERTS{severity="info"} == 1)
|
|
unless on (namespace) group by (namespace)
|
|
(ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
|
|
|
|
which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
|
|
it**. The chart's third `inhibit_rule` then drops exactly that alert:
|
|
|
|
source_matchers: [alertname="InfoInhibitor"]
|
|
target_matchers: [severity="info"]
|
|
equal: [namespace]
|
|
|
|
For a deploy notification this is the worst possible interaction, because the suppressed case is the
|
|
*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
|
|
or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
|
|
delivered only when something else in `flux-system` was already broken.
|
|
|
|
Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
|
|
real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
|
|
label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
|
|
`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
|
|
around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
|
|
the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
|
|
— because the next reader's instinct will be to add one back.
|
|
|
|
## What I verified
|
|
|
|
Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
|
|
|
|
- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
|
|
`kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
|
|
populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
|
|
`status.artifact.revision` paths are correct — previously UNVERIFIED.
|
|
- **The `podAnnotations` roll mechanism works.** The KSM pod carries
|
|
`homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
|
|
(`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
|
|
genuinely closed rather than merely documented.
|
|
- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
|
|
converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
|
|
`revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
|
|
**cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
|
|
notification's whole mechanism.
|
|
- **The routing works.** `amtool config routes` shows
|
|
`{notification="deploy"} → receiver: email-deploy` under the default route, and
|
|
`amtool config routes test notification=deploy` returns `email-deploy`.
|
|
- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
|
|
the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
|
|
Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
|
|
is 0 for every reason, so nothing failed to send; it was never dispatched.
|
|
- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
|
|
suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
|
|
(`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
|
|
proven by merging this and watching the next commit produce `[FLUX] applying` and then
|
|
`[FLUX] applied`. Same structural gap as PR #25, one layer further along.
|
|
- **Whether the single email during the window was ours.**
|
|
`alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
|
|
(`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
|
|
label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
|
|
attribution is genuinely unavailable from this metric.
|
|
- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
|
|
inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
|
|
apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
|
|
- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
|
|
`firing`/`resolved` in the notification template data.
|
|
- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
|
|
reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
|
|
model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
|
|
this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
|
|
- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
|
|
10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.
|