diff --git a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh index 141d349..ed4332c 100755 --- a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh +++ b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh @@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [ "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); " "Velero itself publishes nothing until a backup is attempted", }, + { + "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions", + "prefixes": ("kube_flux_",), + "file": KPS, + "require": [ + r"customResourceState:\s*\n\s*enabled:\s*true", + r"metricNamePrefix:\s*kube_flux_kustomization", + r"metricNamePrefix:\s*kube_flux_gitrepository", + # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io + # above: without read access to these two API groups KSM publishes the + # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would + # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured. + r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]', + r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]', + # A config-only edit does not restart KSM (the chart sets no checksum annotation over + # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop + # it and a future change to the flux collector set ships green and never takes effect. + r"homelab\.keithjackson\.dev/crs-config-rollout", + ], + "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"], + "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no " + "revision label anywhere (measured 2026-09-29) and the Flux event stream emits " + "ReconciliationSucceeded on every interval tick whether or not anything changed, so " + "without these metrics nothing can tell a deploy starting from an idle cluster", + }, { "owner": "kube-prometheus-stack: bundled kube-state-metrics", "prefixes": ("kube_",), diff --git a/decisions/2026-09-29-flux-deploy-notifications.md b/decisions/2026-09-29-flux-deploy-notifications.md new file mode 100644 index 0000000..f17e3d2 --- /dev/null +++ b/decisions/2026-09-29-flux-deploy-notifications.md @@ -0,0 +1,144 @@ +--- +date: 2026-09-29 +branch: feat/flux-deploy-notifications +title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes +--- + +## What changed + +The operator was getting **no notification when Flux began applying a git change, and none when it +finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running +(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but +there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get +providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*` +rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`, +`FluxResourceSuspended`) — nothing covered the healthy path. + +**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be +filtered into a deploy signal: + +- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on + `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in + 324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or + not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour. +- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event + counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`, + `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` → + `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail + five times every ten minutes, forever. +- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series, + `gotk_suspend_status` 40, not one with a revision. + +The applied **revision** is the only signal that moves when and only when git moves, and it lives in +object status alone. So: + +1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing* + `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled): + `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from + `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from + `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a + `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so + a config-only edit would have left KSM running the old collector set with the new metrics absent + and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason + `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects. + Cost: 33 series. +2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress` + (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied + revision is not the revision the GitRepository fetched from `main`, and **clears when all have + caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion + half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity: + warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and + never converged. +3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject + `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined + `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault. + `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout; + `group_interval: 2m` so completion is not held 4m behind the event. +4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions. +5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in + `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns + pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so + deleting any of them fails the gate instead of silently re-inerting the alerts. + +Two guards in the expression, each with a test whose only job is to fail if that clause is removed: +`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes +`unless on (revision) ` remove nothing and **all 16 Kustomizations alert forever** — the +permanently-firing mode this repo shipped once already), and `unless on (namespace, name) +(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design +and would hold the rollout open, so the "applied" mail would never arrive). + +## What I verified + +Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as +measured: + +- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`; + `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage); + `get providers,alerts -A` → **`No resources found`**. This is the defect. +- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io` + involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15 + `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`. +- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job + `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40 + (`{kind="Kustomization"}` = 16) — and none carries a revision. +- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused, + protects_volume_data,info}` = 2 series each. +- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system` + `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval` + `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend` + (which is why `nilIsZero: true` is mandatory). +- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and + the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system → + `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`). +- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the + pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that + proves the alert empties itself (the completion mail). +- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`, + `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`, + `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests + and the new runbook all cleared their own checks. Coverage 51/51. + +## What I did NOT verify + +- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the + rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim + ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED** + and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and + CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run). +- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage + versions and that each status field is populated on the live objects, but I have not seen KSM emit + `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""` + then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the + first thing to check after merge: + `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names. +- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds + two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi + limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])` + after this settles rather than assuming the headroom holds. +- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on + `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still + arrives with a poorer subject — cosmetic, but unproven. +- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut + deliberately: its unique value over the rules above is the human-readable **commit subject** + (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit ''`, + revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no + SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at + `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not + establish **what label set that provider actually emits** (whether it sets `alertname`, and whether + `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So + the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off + Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization** + (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that + cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped + cold-start deadlock this repo already documents. +- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been + re-applying the same object every 10 minutes for ~25 days: something outside git is mutating + `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and + `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** — + I only measured the event counts. It is a real defect (constant write load on the API server and + etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The + alerts here are immune to it by construction: they read revisions, not events. +- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED** + (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is + UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage + that ran is green, and one stage did not run. diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index 27590f1..41bb8c2 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -799,6 +799,91 @@ spec: summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}" description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off." runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended" + # =================================================================================== + # DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to + # send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a + # git change and when it finishes. Read the two together — they share one expression and + # differ only in `for` and severity. + # + # WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a + # deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded + # on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message + # "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply + # changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply + # the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed + # on it mails five times every ten minutes forever. The applied REVISION is the only signal + # that moves when and only when git moves. See the kube_flux_* block in + # kube-prometheus-stack.yaml for how it reaches the TSDB. + # + # THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied + # revision is absent from the set of revisions the GitRepository has fetched — i.e. while it + # is behind main. It empties itself as each Kustomization catches up, which is what makes + # the RESOLVED notification mean "the rollout completed" rather than "the alert timed out". + # Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason, + # and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout. + # + # TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml): + # * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled + # without the ConfigMap, the flux RBAC lost, customResourceState reverted) then + # `unless on (revision) ` removes NOTHING and all 16 Kustomizations alert + # forever, for no reason. That is the permanently-firing failure mode this repo already + # shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT, + # which is the correct direction for a notification — a broken exporter is what + # FluxReconciliationFailure and the platform-health.monitoring-self rules are for. + # * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended + # Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open + # for as long as it stays suspended, so the resolved mail (the "it finished" half of the + # feature) would never arrive. FluxResourceSuspended above is what reports that state. + # + # `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization + # that has never applied anything — or a GitRepository with no artifact yet — publishes the + # series with no `revision` label. Without the matcher those group together under the empty + # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged. + # =================================================================================== + - alert: FluxRolloutInProgress + expr: | + ( + ( + kube_flux_kustomization_info{revision!=""} + unless on (namespace, name) (kube_flux_kustomization_suspended == 1) + ) + unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""} + ) + and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0) + # NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule + # reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the + # operator asked to be told about, so the failure mode to avoid is the opposite: a `for:` + # of even 1m would silently swallow any rollout that converged faster than that (a manual + # `flux reconcile`, a change touching one Kustomization), and a deploy notification that + # only sometimes arrives is worse than no notification at all. + labels: + severity: info # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied" + annotations: + summary: "Flux is applying a new revision — {{ $labels.name }} is behind main" + description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete." + runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout" + # The escalation: a rollout that STARTED and never finished. Same expression, so it cannot + # disagree with the notification above about what "behind main" means; 30m is comfortably + # past the slowest legitimate convergence (the longest Kustomization interval is 1h for + # cilium, but its dependents are 10m and a wave chain converges well inside 30m). + # severity: warning, so this one takes the normal email-warning path and reads as a fault. + - alert: FluxRolloutStuck + expr: | + ( + ( + kube_flux_kustomization_info{revision!=""} + unless on (namespace, name) (kube_flux_kustomization_suspended == 1) + ) + unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""} + ) + and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0) + for: 30m + labels: + severity: warning + annotations: + summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision" + description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies." + runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout" # ===================================================================================== # Certificates — cert-manager diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index 8940a80..047b128 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -131,6 +131,36 @@ spec: # page-worthy — notify fast, remind often group_wait: 10s repeat_interval: 1h + # DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires + # the moment the applied revision diverges from main and RESOLVES when every + # Kustomization has caught up — so the FIRED mail is "a deploy started" and the + # RESOLVED mail is "it finished". Both halves are the product, which is why + # send_resolved must stay true on the receiver below. + # + # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a + # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a + # real platform fault, which is how alert mail stops being read at all. `severity: info` + # is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this + # route claims exactly the deploy notices and nothing else. Keep it that way — giving a + # genuine fault severity: info would silently route it here and re-label it a deploy. + - matchers: ['severity = "info"'] + receiver: email-deploy + # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging + # Kustomization (16 possible), and grouping on alertname alone collapses them into a + # single mail that lists all of them. The default group_by would have split them by + # namespace as well — harmless today (all 16 are in flux-system) and wrong the moment + # a Kustomization lives elsewhere. + group_by: ["alertname"] + group_wait: 30s + # 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED + # notification waits, and holding "the deploy finished" four minutes behind the + # deploy finishing makes the completion half of this feature feel broken. + group_interval: 2m + # A deploy notice is a statement about one revision, so re-sending it on the 12h + # default would just repeat old news. 24h is long enough never to fire twice for one + # rollout, while still re-nagging if a rollout is genuinely stuck for a whole day + # (FluxRolloutStuck, severity warning, is the real escalation for that case). + repeat_interval: 24h receivers: - name: email-warning email_configs: @@ -144,6 +174,19 @@ spec: send_resolved: true headers: subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})' + # Flux deploy notices (the `severity = "info"` route above). + - name: email-deploy + email_configs: + - to: "keithallenjackson@gmail.com" # alert inbox (operator: change if a different inbox is wanted) + # BOTH transitions are the deliverable here: firing = "Flux started applying a new + # revision", resolved = "every Kustomization reached it". Turning this off would + # delete the completion notification, not merely reduce noise. + send_resolved: true + headers: + # `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview + # without opening the mail, and never the word WARNING for a healthy deploy. + # .Status is "firing" or "resolved" in Alertmanager's notification template data. + subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}' # --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO --- grafana: @@ -272,11 +315,33 @@ spec: # `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout # restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time # changes the pod spec (new arg + volume), so the initial rollout is automatic. + # FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that + # config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation + # over it — documented in the block above, and the reason the Velero metrics needed a manual + # `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the + # ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set: + # the new metrics never appear and every alert reading them is inert, while the PR is merged + # and green. That is the exact shape of defect this repo keeps getting bitten by, so it is + # closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE + # SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you. + # (Same mechanism as grafana.podAnnotations/oidc-rollout above.) + podAnnotations: + homelab.keithjackson.dev/crs-config-rollout: "1" rbac: extraRules: - apiGroups: ["velero.io"] resources: ["schedules"] verbs: ["get", "list", "watch"] + # Flux applied-revision metrics (the kube_flux_* block at the end of `config` below). + # MANDATORY for the same measured reason as velero.io above: without read access KSM + # publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden + # error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently. + - apiGroups: ["kustomize.toolkit.fluxcd.io"] + resources: ["kustomizations"] + verbs: ["get", "list", "watch"] + - apiGroups: ["source.toolkit.fluxcd.io"] + resources: ["gitrepositories"] + verbs: ["get", "list", "watch"] customResourceState: enabled: true config: @@ -328,6 +393,89 @@ spec: labelsFromPath: cron: [spec, schedule] phase: [status, phase] + # --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) -------------------------- + # WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on + # main?", so "Flux started applying a git change" and "Flux is idle" were + # indistinguishable and a deploy could start, stall halfway and never finish with + # nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail: + # * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere — + # gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a + # revision. It can say a Kustomization is unhealthy, never which commit it holds. + # * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits + # ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across + # 16 Kustomizations — with the message "Reconciliation finished in 324ms, next run + # in 10m0s", byte-identical whether the apply changed anything or not. + # * `Progressing` is worse, not better: five Kustomizations re-apply the same object + # on every single tick (2505/2504/2501/2495/2490 on one event object each), so + # anything keyed on it fires five times every ten minutes, forever. + # The applied REVISION is the one signal that moves only when git moves, and it lives + # in these two objects' status and nowhere else: + # GitRepository/flux-system .status.artifact.revision = what main is (poll 1m) + # Kustomization/* .status.lastAppliedRevision = what is applied + # Exporting both makes "applied != desired" a PromQL question, which is what + # FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml). + # + # NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above. + # These are kube-state-metrics series ABOUT Kubernetes objects and must never be + # mistaken for the gotk_* family the Flux controllers themselves export. + # + # VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the + # STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io, + # gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated + # — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that + # same revision as lastAppliedRevision. + # + # Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33. + - groupVersionKind: + group: kustomize.toolkit.fluxcd.io + version: v1 + kind: Kustomization + metricNamePrefix: kube_flux_kustomization + labelsFromPath: + name: [metadata, name] + namespace: [metadata, namespace] + metrics: + - name: info + help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)." + each: + type: Info + info: + # KSM OMITS a label whose path is absent (the same behaviour noted for the + # Velero `phase` label above), so a Kustomization that has never applied + # anything publishes this series with NO `revision` label at all rather than + # an empty one. Every rule that reads it therefore matches revision!="" — + # otherwise those would collapse together under the empty string and a + # never-applied Kustomization would compare equal to a converged one. + labelsFromPath: + revision: [status, lastAppliedRevision] + - name: suspended + help: "1 if the Flux Kustomization is suspended (spec.suspend)." + each: + type: Gauge + gauge: + # nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above: + # measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at + # all. Without it this metric would not exist, and the `unless` that stops a + # suspended Kustomization holding a rollout notification open forever would + # silently match nothing. + path: [spec, suspend] + nilIsZero: true + - groupVersionKind: + group: source.toolkit.fluxcd.io + version: v1 + kind: GitRepository + metricNamePrefix: kube_flux_gitrepository + labelsFromPath: + name: [metadata, name] + namespace: [metadata, namespace] + metrics: + - name: info + help: "Revision of the artifact a Flux GitRepository has fetched (what main is)." + each: + type: Info + info: + labelsFromPath: + revision: [status, artifact, revision] # MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB # with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike, # roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert diff --git a/platform/monitoring/tests/flux-rollout.test.yaml b/platform/monitoring/tests/flux-rollout.test.yaml new file mode 100644 index 0000000..dff9581 --- /dev/null +++ b/platform/monitoring/tests/flux-rollout.test.yaml @@ -0,0 +1,268 @@ +# promtool rule-unit tests for FluxRolloutInProgress / FluxRolloutStuck — the deploy-notification +# pair in the platform-health.flux group (platform/monitoring/alerts.yaml). +# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via +# conformance/tests/l0-static/alert-rules-unit-tested) +# +# WHY THESE TESTS CARRY MORE WEIGHT THAN USUAL. This pair is a NOTIFICATION, so both failure +# directions are silent in production and neither is visible on a healthy cluster: +# * fire when it should not -> a permanently-firing "deploy in progress" that mails on every +# repeat_interval and trains the operator to ignore Flux mail. Shipped once already in this +# repo (CNPG, 2026-09-12). +# * fail to fire -> no deploy notification at all, which is the exact complaint this pair was +# written to answer. Indistinguishable from "no deploys happened". +# So every clause is pinned in both directions here, and the two GUARDS in the expression (the +# missing-exporter guard and the suspended-Kustomization exclusion) each get a test whose ONLY +# job is to fail if that clause is deleted. +# +# MODELLING A REVISION CHANGE. `revision` is a LABEL, so a Kustomization moving from revision A to +# revision B is modelled as series A ending and series B beginning — not as one series changing +# value. Prometheus' 5m lookback then means series A is still resolvable for 5m after its last +# sample, so every eval_time below is placed at least 5m clear of a handoff. That is deliberate: +# it keeps these tests about the RULE and not about staleness semantics. +# +# Sample interval 1m; the longest `for` under test is 30m. +rule_files: + - generated.rules.yaml + +evaluation_interval: 1m + +tests: + # ============================================================================================= + # 1. CONVERGED — the steady state this cluster is in almost all of the time. + # Both rules MUST be silent. This is the "not permanently firing" half of the proof, and it + # is the test that fails if `unless on (revision)` is mis-spelled: the left-hand side is + # non-empty here, and only the revision match suppresses it. + # ============================================================================================= + - interval: 1m + name: converged (every Kustomization on the GitRepository revision) - silent + input_series: + - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}' + values: '1x40' + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x40' + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x40' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x40' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x40' + alert_rule_test: + - eval_time: 35m + alertname: FluxRolloutInProgress + exp_alerts: [] + - eval_time: 35m + alertname: FluxRolloutStuck + exp_alerts: [] + + # ============================================================================================= + # 2. A COMMIT LANDS — the notification this whole change exists to produce. + # GitRepository moves to revision bbbbbbb at t=11m. apps-wave-0 catches up at t=14m; + # `platform` never does. + # t=25m -> FluxRolloutInProgress fires for `platform` ONLY (apps-wave-0 has converged), + # and FluxRolloutStuck is still quiet because its 30m has not elapsed. + # t=50m -> both fire. ("started and never finished" is a real, separate condition.) + # ============================================================================================= + - interval: 1m + name: new revision on main - InProgress fires for the lagging Kustomization only + input_series: + # The old artifact revision ends at t=10m, the new one starts at t=11m. + - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}' + values: '1x10' + - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}' + values: '_x10 1x70' + # `platform` stays on the old revision for the whole test. + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x80' + # apps-wave-0 catches up at t=14m. + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x8' + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '_x13 1x67' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x80' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x80' + alert_rule_test: + # The deploy notification, ~14m after the commit was fetched. Exactly ONE alert: a rule that + # ignored the revision match would also list apps-wave-0, which has already converged. + - eval_time: 25m + alertname: FluxRolloutInProgress + exp_alerts: + - exp_labels: + severity: info + job: kube-state-metrics + namespace: flux-system + name: platform + revision: "main@sha1:aaaaaaa" + customresource_group: kustomize.toolkit.fluxcd.io + customresource_kind: Kustomization + customresource_version: v1 + exp_annotations: + summary: "Flux is applying a new revision — platform is behind main" + description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete." + runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout" + # `for: 30m` on the escalation is real — not yet firing 14m in. + - eval_time: 25m + alertname: FluxRolloutStuck + exp_alerts: [] + # ...and it does fire once the rollout has genuinely hung. + - eval_time: 50m + alertname: FluxRolloutStuck + exp_alerts: + - exp_labels: + severity: warning + job: kube-state-metrics + namespace: flux-system + name: platform + revision: "main@sha1:aaaaaaa" + customresource_group: kustomize.toolkit.fluxcd.io + customresource_kind: Kustomization + customresource_version: v1 + exp_annotations: + summary: "Flux rollout has not completed: platform stuck on an old revision" + description: "Kustomization flux-system/platform has been behind the flux-system GitRepository revision for 30m (applied main@sha1:aaaaaaa) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies." + runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout" + + # ============================================================================================= + # 3. THE ROLLOUT COMPLETES -> the alert clears, which is what sends the "applied" mail. + # Without this test the completion half of the feature is unproven: an expression that fired + # correctly but never emptied itself would pass test 2 and still never deliver a resolved + # notification. + # ============================================================================================= + - interval: 1m + name: rollout completes - InProgress clears (this is the resolved/completion notification) + input_series: + - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}' + values: '1x10' + - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}' + values: '_x10 1x70' + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x20' + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '_x23 1x57' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x80' + alert_rule_test: + # Mid-rollout: firing. + - eval_time: 20m + alertname: FluxRolloutInProgress + exp_alerts: + - exp_labels: + severity: info + job: kube-state-metrics + namespace: flux-system + name: platform + revision: "main@sha1:aaaaaaa" + customresource_group: kustomize.toolkit.fluxcd.io + customresource_kind: Kustomization + customresource_version: v1 + exp_annotations: + summary: "Flux is applying a new revision — platform is behind main" + description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete." + runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout" + # Converged (5m clear of the old series' last sample at t=20m): silent, and it never had + # 30m of continuous firing, so the escalation must not have fired either. + - eval_time: 32m + alertname: FluxRolloutInProgress + exp_alerts: [] + - eval_time: 32m + alertname: FluxRolloutStuck + exp_alerts: [] + + # ============================================================================================= + # 4. GUARD 1 — kube_flux_gitrepository_info ABSENT (KSM rolled without the ConfigMap, the flux + # RBAC was lost, customResourceState reverted). `unless on (revision) ` removes + # NOTHING, so without the `and on () (count(...) > 0)` guard EVERY Kustomization alerts + # forever. This test exists to go red the moment that guard is deleted. + # Correct behaviour is SILENCE: a broken exporter is FluxReconciliationFailure's job and the + # platform-health.monitoring-self rules', not a deploy notification's. + # ============================================================================================= + - interval: 1m + name: GUARD - GitRepository metric absent - silent, NOT 16 alerts forever + input_series: + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x60' + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x60' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x60' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x60' + promql_expr_test: + # Stated explicitly so the reason for the silence is unambiguous in the test output: the + # guard's own count is empty, not merely false. + - expr: 'count(kube_flux_gitrepository_info{name="flux-system", revision!=""})' + eval_time: 45m + exp_samples: [] + alert_rule_test: + - eval_time: 45m + alertname: FluxRolloutInProgress + exp_alerts: [] + - eval_time: 45m + alertname: FluxRolloutStuck + exp_alerts: [] + + # ============================================================================================= + # 5. GUARD 2 — a SUSPENDED Kustomization pins an old revision by design. It must not hold the + # rollout notification open (or the "applied" mail never arrives), while a non-suspended + # Kustomization on the same old revision MUST still alert. + # Both halves are asserted in one test on purpose: an `unless` written with the wrong labels + # would suppress EVERYTHING and would pass a suppression-only test. + # ============================================================================================= + - interval: 1m + name: GUARD - suspended Kustomization excluded, non-suspended one still alerts + input_series: + - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}' + values: '1x40' + # Suspended and behind -> excluded. + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x40' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x40' + # Not suspended and behind -> must alert. + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="conformance",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x40' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="conformance",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x40' + alert_rule_test: + - eval_time: 20m + alertname: FluxRolloutInProgress + exp_alerts: + - exp_labels: + severity: info + job: kube-state-metrics + namespace: flux-system + name: conformance + revision: "main@sha1:aaaaaaa" + customresource_group: kustomize.toolkit.fluxcd.io + customresource_kind: Kustomization + customresource_version: v1 + exp_annotations: + summary: "Flux is applying a new revision — conformance is behind main" + description: "Kustomization flux-system/conformance has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete." + runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout" + + # ============================================================================================= + # 6. A Kustomization that has NEVER applied anything publishes no `revision` label at all (KSM + # omits absent paths). `revision!=""` must drop it rather than let it group under the empty + # string and compare equal to the GitRepository revision. + # ============================================================================================= + - interval: 1m + name: never-applied Kustomization (no revision label) is not counted as converged or lagging + input_series: + - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}' + values: '1x40' + # No `revision` label at all — this is what KSM publishes for status.lastAppliedRevision="". + - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '1x40' + - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}' + values: '0x40' + alert_rule_test: + # Silent: FluxReconciliationFailure (Ready=False for 15m) is what covers a Kustomization + # that cannot apply at all. This pair only ever speaks about revisions it can compare. + - eval_time: 30m + alertname: FluxRolloutInProgress + exp_alerts: [] + - eval_time: 30m + alertname: FluxRolloutStuck + exp_alerts: [] diff --git a/runbooks/flux-rollout.md b/runbooks/flux-rollout.md new file mode 100644 index 0000000..1f17c16 --- /dev/null +++ b/runbooks/flux-rollout.md @@ -0,0 +1,118 @@ +# flux-rollout — a git change is rolling out, or has stopped halfway + +**Alerts:** `FluxRolloutInProgress` (info, no `for:`) and `FluxRolloutStuck` (warning, 30m) +**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux` +**Metrics:** `kube_flux_kustomization_info` / `kube_flux_kustomization_suspended` / +`kube_flux_gitrepository_info` — kube-state-metrics custom-resource-state, configured in +`platform/monitoring/kube-prometheus-stack.yaml` + +## What this means here + +These two are the **deploy notification**, not a fault report. On this cluster merging a PR into +`main` IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event +stream emits `ReconciliationSucceeded` on every 10-minute tick whether or not anything changed (101 +such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical. + +The pair reads the one signal that moves only when git moves — the **applied revision**: + +- `GitRepository/flux-system` `.status.artifact.revision` — what `main` is (polled every 1m) +- `Kustomization/*` `.status.lastAppliedRevision` — what each tier has actually applied + +**`FluxRolloutInProgress` fires** the moment any non-suspended Kustomization's applied revision is +not the revision fetched from `main`, and **clears when every one of them has caught up**. So: + +| Mail | Subject | Means | +|---|---|---| +| firing | `[FLUX] applying — FluxRolloutInProgress` | a commit started rolling out | +| resolved | `[FLUX] applied — FluxRolloutInProgress` | every Kustomization reached it; rollout complete | + +Both are expected, routine, and not a problem. The `severity: info` label routes them to the +dedicated `email-deploy` Alertmanager receiver so they never arrive subject-lined `[WARNING]`. + +**`FluxRolloutStuck` is the one that matters.** Same expression, `for: 30m`, `severity: warning`. It +means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail +never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds +every tier below it on "dependency not ready". + +## First commands + +`kubectl` does not work on the workstation — the toolchain lives only in the pinned container, and +the context must always be named explicitly. + +Which tiers are behind, and on what: + +```bash +./infra/devcontainer/dev.sh run bash -c ' + kubectl --context k3s-prod -n flux-system get gitrepository flux-system \ + -o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}" + kubectl --context k3s-prod get kustomizations -A \ + -o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"' +``` + +Any row whose `APPLIED` differs from `WANTED` is a tier this alert is naming. Then ask **why** that +tier is not moving — its Ready condition carries the reason: + +```bash +./infra/devcontainer/dev.sh run bash -c ' + kubectl --context k3s-prod -n flux-system get kustomization \ + -o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"' +``` + +Confirm the alert is reading live data rather than a stale exporter: + +```bash +./infra/devcontainer/dev.sh run bash -c ' + POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1) + kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \ + /bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"' +``` + +Three names must come back (`kube_flux_kustomization_info`, `kube_flux_kustomization_suspended`, +`kube_flux_gitrepository_info`). If that query is **empty**, the alerts are inert — go to the last +cause below. + +## Likely causes, most likely first + +1. **Nothing is wrong — the rollout is simply in flight.** Most Kustomizations reconcile on a 10m + interval and the chain is sequential, so several minutes between the "applying" and "applied" + mails is normal. `cilium` is on a 1h interval and can legitimately be the last to converge. +2. **A gating tier is unhealthy, so everything below it is parked.** Look for `DependencyNotReady` + in the stuck tier's message and walk *up* the chain to the first tier that is not Ready; that one + is the real fault. Its own alert (`FluxReconciliationFailure`) and runbook `flux-reconcile` own it. +3. **A Kustomization is suspended.** A suspended tier is excluded from this alert by design (so it + cannot hold the "applied" mail open forever), but anything that `dependsOn` it still stalls and + *will* show here. `FluxResourceSuspended` / runbook `flux-suspended` covers it. +4. **A health check is timing out.** A `wait: true` tier only records `lastAppliedRevision` once its + objects pass health checks, so a workload that never goes Ready keeps the tier on the old + revision even though the manifests applied fine. +5. **kube-state-metrics is not publishing the `kube_flux_*` family.** Then the rules are inert and + you would get *no* mail at all rather than a stuck one — so this is the cause when the complaint + is silence. The metrics come from `customResourceState` plus a read-only RBAC rule for + `kustomize.toolkit.fluxcd.io` / `source.toolkit.fluxcd.io`; without that RBAC, KSM emits the set + **empty** and logs a forbidden error. Check its logs: + ```bash + ./infra/devcontainer/dev.sh run bash -c ' + kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \ + --tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"' + ``` + Note the chart puts **no checksum annotation** over that ConfigMap, so a change to the config + alone does not restart the pod — that is why `podAnnotations` carries a + `homelab.keithjackson.dev/crs-config-rollout` counter that must be bumped in the same commit. + +## When to escalate or stop + +- **`FluxRolloutInProgress` alone, cleared within ~15 minutes: do nothing.** That is the feature + working. Do not suspend, reconcile or apply anything in response to it. +- **`FluxRolloutStuck` firing: fix the upstream fault, not this symptom.** This alert names the + tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use + `flux-reconcile`. +- **Never `kubectl apply`, `edit` or `patch` the change through by hand.** Flux reconciles `main`; + a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes + the applied-revision signal these alerts read a lie. `flux reconcile` (an operator action, never + an agent's) is the only nudge that is safe, and it only *hurries* the same pull. +- **Escalate to a revert if a stuck rollout is degrading service.** Reverting the offending commit + on `main` through the normal PR flow is the rollback path; see runbook `update-rollback`. +- **If both alerts are silent during a deploy you know happened,** the notification itself is broken, + not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these + rules is indistinguishable from "no deploys happened", which is the exact defect they were + written to close. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index deafa5f..08793d7 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -65,6 +65,7 @@ configMapGenerator: - crafty-admin-credential=crafty-admin-credential.md - denied-flows=denied-flows.md - flux-reconcile=flux-reconcile.md + - flux-rollout=flux-rollout.md - flux-suspended=flux-suspended.md - image-automation=image-automation.md - image-policy-stalled=image-policy-stalled.md