Merge pull request 'feat(obs): notify on Flux deploy start and completion' (#25) from feat/flux-deploy-notifications into main

Reviewed-on: #25
Reviewed-by: Keith Jackson <[email protected]>

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
                "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
                "Velero itself publishes nothing until a backup is attempted",
     },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
     {
         "owner": "kube-prometheus-stack: bundled kube-state-metrics",
         "prefixes": ("kube_",),

decisions/2026-09-29-flux-deploy-notifications.md

@@ -0,0 +1,144 @@
+---
+date: 2026-09-29
+branch: feat/flux-deploy-notifications
+title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
+---
+
+## What changed
+
+The operator was getting **no notification when Flux began applying a git change, and none when it
+finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
+(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
+there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
+providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
+rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
+`FluxResourceSuspended`) — nothing covered the healthy path.
+
+**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
+filtered into a deploy signal:
+
+- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
+  `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
+  324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
+  not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
+- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
+  counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
+  `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
+  `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
+  five times every ten minutes, forever.
+- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
+  `gotk_suspend_status` 40, not one with a revision.
+
+The applied **revision** is the only signal that moves when and only when git moves, and it lives in
+object status alone. So:
+
+1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
+   `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
+   `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
+   `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
+   `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
+   `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
+   a config-only edit would have left KSM running the old collector set with the new metrics absent
+   and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
+   `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
+   Cost: 33 series.
+2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
+   (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
+   revision is not the revision the GitRepository fetched from `main`, and **clears when all have
+   caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
+   half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
+   warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
+   never converged.
+3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
+   `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
+   `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
+   `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
+   `group_interval: 2m` so completion is not held 4m behind the event.
+4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
+5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
+   `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
+   pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
+   deleting any of them fails the gate instead of silently re-inerting the alerts.
+
+Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
+`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
+`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
+permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
+(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
+and would hold the rollout open, so the "applied" mail would never arrive).
+
+## What I verified
+
+Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
+measured:
+
+- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
+  `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
+  `get providers,alerts -A` → **`No resources found`**. This is the defect.
+- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
+  involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
+  `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
+- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
+  `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
+  (`{kind="Kustomization"}` = 16) — and none carries a revision.
+- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
+  protects_volume_data,info}` = 2 series each.
+- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
+  `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
+  `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
+  (which is why `nilIsZero: true` is mandatory).
+- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
+  the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
+  `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
+- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
+  pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
+  proves the alert empties itself (the completion mail).
+- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
+  `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
+  `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
+  and the new runbook all cleared their own checks. Coverage 51/51.
+
+## What I did NOT verify
+
+- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
+  rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
+  ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
+  and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
+  CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
+- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
+  versions and that each status field is populated on the live objects, but I have not seen KSM emit
+  `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
+  then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
+  first thing to check after merge:
+  `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
+- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
+  two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
+  limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
+  after this settles rather than assuming the headroom holds.
+- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
+  `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
+  arrives with a poorer subject — cosmetic, but unproven.
+- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
+  deliberately: its unique value over the rules above is the human-readable **commit subject**
+  (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
+  revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
+  SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
+  `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
+  establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
+  `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
+  the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
+  Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
+  (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
+  cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
+  cold-start deadlock this repo already documents.
+- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
+  re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
+  `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
+  `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
+  I only measured the event counts. It is a real defect (constant write load on the API server and
+  etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
+  alerts here are immune to it by construction: they read revisions, not events.
+- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
+  (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
+  UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
+  that ran is green, and one stage did not run.

platform/monitoring/alerts.yaml

@@ -799,6 +799,91 @@ spec:
             summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
+        # ===================================================================================
+        # DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to
+        # send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a
+        # git change and when it finishes. Read the two together — they share one expression and
+        # differ only in `for` and severity.
+        #
+        # WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a
+        # deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded
+        # on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message
+        # "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply
+        # changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply
+        # the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed
+        # on it mails five times every ten minutes forever. The applied REVISION is the only signal
+        # that moves when and only when git moves. See the kube_flux_* block in
+        # kube-prometheus-stack.yaml for how it reaches the TSDB.
+        #
+        # THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied
+        # revision is absent from the set of revisions the GitRepository has fetched — i.e. while it
+        # is behind main. It empties itself as each Kustomization catches up, which is what makes
+        # the RESOLVED notification mean "the rollout completed" rather than "the alert timed out".
+        # Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason,
+        # and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout.
+        #
+        # TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml):
+        #   * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled
+        #     without the ConfigMap, the flux RBAC lost, customResourceState reverted) then
+        #     `unless on (revision) <empty>` removes NOTHING and all 16 Kustomizations alert
+        #     forever, for no reason. That is the permanently-firing failure mode this repo already
+        #     shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT,
+        #     which is the correct direction for a notification — a broken exporter is what
+        #     FluxReconciliationFailure and the platform-health.monitoring-self rules are for.
+        #   * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended
+        #     Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open
+        #     for as long as it stays suspended, so the resolved mail (the "it finished" half of the
+        #     feature) would never arrive. FluxResourceSuspended above is what reports that state.
+        #
+        # `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization
+        # that has never applied anything — or a GitRepository with no artifact yet — publishes the
+        # series with no `revision` label. Without the matcher those group together under the empty
+        # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
+        # ===================================================================================
+        - alert: FluxRolloutInProgress
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          # NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule
+          # reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the
+          # operator asked to be told about, so the failure mode to avoid is the opposite: a `for:`
+          # of even 1m would silently swallow any rollout that converged faster than that (a manual
+          # `flux reconcile`, a change touching one Kustomization), and a deploy notification that
+          # only sometimes arrives is worse than no notification at all.
+          labels:
+            severity: info    # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied"
+          annotations:
+            summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+        # The escalation: a rollout that STARTED and never finished. Same expression, so it cannot
+        # disagree with the notification above about what "behind main" means; 30m is comfortably
+        # past the slowest legitimate convergence (the longest Kustomization interval is 1h for
+        # cilium, but its dependents are 10m and a wave chain converges well inside 30m).
+        # severity: warning, so this one takes the normal email-warning path and reads as a fault.
+        - alert: FluxRolloutStuck
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
 
 # =====================================================================================
 # Certificates — cert-manager

platform/monitoring/kube-prometheus-stack.yaml

@@ -131,6 +131,36 @@ spec:
               # page-worthy — notify fast, remind often
               group_wait: 10s
               repeat_interval: 1h
+            # DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires
+            # the moment the applied revision diverges from main and RESOLVES when every
+            # Kustomization has caught up — so the FIRED mail is "a deploy started" and the
+            # RESOLVED mail is "it finished". Both halves are the product, which is why
+            # send_resolved must stay true on the receiver below.
+            #
+            # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
+            # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
+            # real platform fault, which is how alert mail stops being read at all. `severity: info`
+            # is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this
+            # route claims exactly the deploy notices and nothing else. Keep it that way — giving a
+            # genuine fault severity: info would silently route it here and re-label it a deploy.
+            - matchers: ['severity = "info"']
+              receiver: email-deploy
+              # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
+              # Kustomization (16 possible), and grouping on alertname alone collapses them into a
+              # single mail that lists all of them. The default group_by would have split them by
+              # namespace as well — harmless today (all 16 are in flux-system) and wrong the moment
+              # a Kustomization lives elsewhere.
+              group_by: ["alertname"]
+              group_wait: 30s
+              # 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED
+              # notification waits, and holding "the deploy finished" four minutes behind the
+              # deploy finishing makes the completion half of this feature feel broken.
+              group_interval: 2m
+              # A deploy notice is a statement about one revision, so re-sending it on the 12h
+              # default would just repeat old news. 24h is long enough never to fire twice for one
+              # rollout, while still re-nagging if a rollout is genuinely stuck for a whole day
+              # (FluxRolloutStuck, severity warning, is the real escalation for that case).
+              repeat_interval: 24h
         receivers:
           - name: email-warning
             email_configs:
@@ -144,6 +174,19 @@ spec:
                 send_resolved: true
                 headers:
                   subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
+          # Flux deploy notices (the `severity = "info"` route above).
+          - name: email-deploy
+            email_configs:
+              - to: "[email protected]"    # alert inbox (operator: change if a different inbox is wanted)
+                # BOTH transitions are the deliverable here: firing = "Flux started applying a new
+                # revision", resolved = "every Kustomization reached it". Turning this off would
+                # delete the completion notification, not merely reduce noise.
+                send_resolved: true
+                headers:
+                  # `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview
+                  # without opening the mail, and never the word WARNING for a healthy deploy.
+                  # .Status is "firing" or "resolved" in Alertmanager's notification template data.
+                  subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}'
 
     # --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
     grafana:
@@ -272,11 +315,33 @@ spec:
       #     `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
       #     restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
       #     changes the pod spec (new arg + volume), so the initial rollout is automatic.
+      # FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that
+      # config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation
+      # over it — documented in the block above, and the reason the Velero metrics needed a manual
+      # `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the
+      # ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set:
+      # the new metrics never appear and every alert reading them is inert, while the PR is merged
+      # and green. That is the exact shape of defect this repo keeps getting bitten by, so it is
+      # closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE
+      # SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you.
+      # (Same mechanism as grafana.podAnnotations/oidc-rollout above.)
+      podAnnotations:
+        homelab.keithjackson.dev/crs-config-rollout: "1"
       rbac:
         extraRules:
           - apiGroups: ["velero.io"]
             resources: ["schedules"]
             verbs: ["get", "list", "watch"]
+          # Flux applied-revision metrics (the kube_flux_* block at the end of `config` below).
+          # MANDATORY for the same measured reason as velero.io above: without read access KSM
+          # publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden
+          # error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently.
+          - apiGroups: ["kustomize.toolkit.fluxcd.io"]
+            resources: ["kustomizations"]
+            verbs: ["get", "list", "watch"]
+          - apiGroups: ["source.toolkit.fluxcd.io"]
+            resources: ["gitrepositories"]
+            verbs: ["get", "list", "watch"]
       customResourceState:
         enabled: true
         config:
@@ -328,6 +393,89 @@ spec:
                         labelsFromPath:
                           cron: [spec, schedule]
                           phase: [status, phase]
+              # --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) --------------------------
+              # WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on
+              # main?", so "Flux started applying a git change" and "Flux is idle" were
+              # indistinguishable and a deploy could start, stall halfway and never finish with
+              # nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail:
+              #   * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere —
+              #     gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a
+              #     revision. It can say a Kustomization is unhealthy, never which commit it holds.
+              #   * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits
+              #     ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across
+              #     16 Kustomizations — with the message "Reconciliation finished in 324ms, next run
+              #     in 10m0s", byte-identical whether the apply changed anything or not.
+              #   * `Progressing` is worse, not better: five Kustomizations re-apply the same object
+              #     on every single tick (2505/2504/2501/2495/2490 on one event object each), so
+              #     anything keyed on it fires five times every ten minutes, forever.
+              # The applied REVISION is the one signal that moves only when git moves, and it lives
+              # in these two objects' status and nowhere else:
+              #   GitRepository/flux-system  .status.artifact.revision   = what main is (poll 1m)
+              #   Kustomization/*            .status.lastAppliedRevision = what is applied
+              # Exporting both makes "applied != desired" a PromQL question, which is what
+              # FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml).
+              #
+              # NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above.
+              # These are kube-state-metrics series ABOUT Kubernetes objects and must never be
+              # mistaken for the gotk_* family the Flux controllers themselves export.
+              #
+              # VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the
+              # STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io,
+              # gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated
+              # — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that
+              # same revision as lastAppliedRevision.
+              #
+              # Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33.
+              - groupVersionKind:
+                  group: kustomize.toolkit.fluxcd.io
+                  version: v1
+                  kind: Kustomization
+                metricNamePrefix: kube_flux_kustomization
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)."
+                    each:
+                      type: Info
+                      info:
+                        # KSM OMITS a label whose path is absent (the same behaviour noted for the
+                        # Velero `phase` label above), so a Kustomization that has never applied
+                        # anything publishes this series with NO `revision` label at all rather than
+                        # an empty one. Every rule that reads it therefore matches revision!="" —
+                        # otherwise those would collapse together under the empty string and a
+                        # never-applied Kustomization would compare equal to a converged one.
+                        labelsFromPath:
+                          revision: [status, lastAppliedRevision]
+                  - name: suspended
+                    help: "1 if the Flux Kustomization is suspended (spec.suspend)."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above:
+                        # measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at
+                        # all. Without it this metric would not exist, and the `unless` that stops a
+                        # suspended Kustomization holding a rollout notification open forever would
+                        # silently match nothing.
+                        path: [spec, suspend]
+                        nilIsZero: true
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: GitRepository
+                metricNamePrefix: kube_flux_gitrepository
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision of the artifact a Flux GitRepository has fetched (what main is)."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          revision: [status, artifact, revision]
       # MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
       # with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
       # roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert

platform/monitoring/tests/flux-rollout.test.yaml

@@ -0,0 +1,268 @@
+# promtool rule-unit tests for FluxRolloutInProgress / FluxRolloutStuck — the deploy-notification
+# pair in the platform-health.flux group (platform/monitoring/alerts.yaml).
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE TESTS CARRY MORE WEIGHT THAN USUAL. This pair is a NOTIFICATION, so both failure
+# directions are silent in production and neither is visible on a healthy cluster:
+#   * fire when it should not -> a permanently-firing "deploy in progress" that mails on every
+#     repeat_interval and trains the operator to ignore Flux mail. Shipped once already in this
+#     repo (CNPG, 2026-09-12).
+#   * fail to fire -> no deploy notification at all, which is the exact complaint this pair was
+#     written to answer. Indistinguishable from "no deploys happened".
+# So every clause is pinned in both directions here, and the two GUARDS in the expression (the
+# missing-exporter guard and the suspended-Kustomization exclusion) each get a test whose ONLY
+# job is to fail if that clause is deleted.
+#
+# MODELLING A REVISION CHANGE. `revision` is a LABEL, so a Kustomization moving from revision A to
+# revision B is modelled as series A ending and series B beginning — not as one series changing
+# value. Prometheus' 5m lookback then means series A is still resolvable for 5m after its last
+# sample, so every eval_time below is placed at least 5m clear of a handoff. That is deliberate:
+# it keeps these tests about the RULE and not about staleness semantics.
+#
+# Sample interval 1m; the longest `for` under test is 30m.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. CONVERGED — the steady state this cluster is in almost all of the time.
+  #    Both rules MUST be silent. This is the "not permanently firing" half of the proof, and it
+  #    is the test that fails if `unless on (revision)` is mis-spelled: the left-hand side is
+  #    non-empty here, and only the revision match suppresses it.
+  # =============================================================================================
+  - interval: 1m
+    name: converged (every Kustomization on the GitRepository revision) - silent
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 35m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. A COMMIT LANDS — the notification this whole change exists to produce.
+  #    GitRepository moves to revision bbbbbbb at t=11m. apps-wave-0 catches up at t=14m;
+  #    `platform` never does.
+  #      t=25m -> FluxRolloutInProgress fires for `platform` ONLY (apps-wave-0 has converged),
+  #               and FluxRolloutStuck is still quiet because its 30m has not elapsed.
+  #      t=50m -> both fire. ("started and never finished" is a real, separate condition.)
+  # =============================================================================================
+  - interval: 1m
+    name: new revision on main - InProgress fires for the lagging Kustomization only
+    input_series:
+      # The old artifact revision ends at t=10m, the new one starts at t=11m.
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      # `platform` stays on the old revision for the whole test.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x80'
+      # apps-wave-0 catches up at t=14m.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x8'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x13 1x67'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # The deploy notification, ~14m after the commit was fetched. Exactly ONE alert: a rule that
+      # ignored the revision match would also list apps-wave-0, which has already converged.
+      - eval_time: 25m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              severity: info
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # `for: 30m` on the escalation is real — not yet firing 14m in.
+      - eval_time: 25m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+      # ...and it does fire once the rollout has genuinely hung.
+      - eval_time: 50m
+        alertname: FluxRolloutStuck
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux rollout has not completed: platform stuck on an old revision"
+              description: "Kustomization flux-system/platform has been behind the flux-system GitRepository revision for 30m (applied main@sha1:aaaaaaa) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 3. THE ROLLOUT COMPLETES -> the alert clears, which is what sends the "applied" mail.
+  #    Without this test the completion half of the feature is unproven: an expression that fired
+  #    correctly but never emptied itself would pass test 2 and still never deliver a resolved
+  #    notification.
+  # =============================================================================================
+  - interval: 1m
+    name: rollout completes - InProgress clears (this is the resolved/completion notification)
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x20'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x23 1x57'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # Mid-rollout: firing.
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              severity: info
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # Converged (5m clear of the old series' last sample at t=20m): silent, and it never had
+      # 30m of continuous firing, so the escalation must not have fired either.
+      - eval_time: 32m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 32m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. GUARD 1 — kube_flux_gitrepository_info ABSENT (KSM rolled without the ConfigMap, the flux
+  #    RBAC was lost, customResourceState reverted). `unless on (revision) <empty set>` removes
+  #    NOTHING, so without the `and on () (count(...) > 0)` guard EVERY Kustomization alerts
+  #    forever. This test exists to go red the moment that guard is deleted.
+  #    Correct behaviour is SILENCE: a broken exporter is FluxReconciliationFailure's job and the
+  #    platform-health.monitoring-self rules', not a deploy notification's.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - GitRepository metric absent - silent, NOT 16 alerts forever
+    input_series:
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+    promql_expr_test:
+      # Stated explicitly so the reason for the silence is unambiguous in the test output: the
+      # guard's own count is empty, not merely false.
+      - expr: 'count(kube_flux_gitrepository_info{name="flux-system", revision!=""})'
+        eval_time: 45m
+        exp_samples: []
+    alert_rule_test:
+      - eval_time: 45m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 5. GUARD 2 — a SUSPENDED Kustomization pins an old revision by design. It must not hold the
+  #    rollout notification open (or the "applied" mail never arrives), while a non-suspended
+  #    Kustomization on the same old revision MUST still alert.
+  #    Both halves are asserted in one test on purpose: an `unless` written with the wrong labels
+  #    would suppress EVERYTHING and would pass a suppression-only test.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - suspended Kustomization excluded, non-suspended one still alerts
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # Suspended and behind -> excluded.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      # Not suspended and behind -> must alert.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="conformance",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="conformance",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              severity: info
+              job: kube-state-metrics
+              namespace: flux-system
+              name: conformance
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — conformance is behind main"
+              description: "Kustomization flux-system/conformance has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 6. A Kustomization that has NEVER applied anything publishes no `revision` label at all (KSM
+  #    omits absent paths). `revision!=""` must drop it rather than let it group under the empty
+  #    string and compare equal to the GitRepository revision.
+  # =============================================================================================
+  - interval: 1m
+    name: never-applied Kustomization (no revision label) is not counted as converged or lagging
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # No `revision` label at all — this is what KSM publishes for status.lastAppliedRevision="".
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      # Silent: FluxReconciliationFailure (Ready=False for 15m) is what covers a Kustomization
+      # that cannot apply at all. This pair only ever speaks about revisions it can compare.
+      - eval_time: 30m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: FluxRolloutStuck
+        exp_alerts: []

runbooks/flux-rollout.md

@@ -0,0 +1,118 @@
+# flux-rollout — a git change is rolling out, or has stopped halfway
+
+**Alerts:** `FluxRolloutInProgress` (info, no `for:`) and `FluxRolloutStuck` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
+**Metrics:** `kube_flux_kustomization_info` / `kube_flux_kustomization_suspended` /
+`kube_flux_gitrepository_info` — kube-state-metrics custom-resource-state, configured in
+`platform/monitoring/kube-prometheus-stack.yaml`
+
+## What this means here
+
+These two are the **deploy notification**, not a fault report. On this cluster merging a PR into
+`main` IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event
+stream emits `ReconciliationSucceeded` on every 10-minute tick whether or not anything changed (101
+such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical.
+
+The pair reads the one signal that moves only when git moves — the **applied revision**:
+
+- `GitRepository/flux-system` `.status.artifact.revision` — what `main` is (polled every 1m)
+- `Kustomization/*` `.status.lastAppliedRevision` — what each tier has actually applied
+
+**`FluxRolloutInProgress` fires** the moment any non-suspended Kustomization's applied revision is
+not the revision fetched from `main`, and **clears when every one of them has caught up**. So:
+
+| Mail | Subject | Means |
+|---|---|---|
+| firing | `[FLUX] applying — FluxRolloutInProgress` | a commit started rolling out |
+| resolved | `[FLUX] applied — FluxRolloutInProgress` | every Kustomization reached it; rollout complete |
+
+Both are expected, routine, and not a problem. The `severity: info` label routes them to the
+dedicated `email-deploy` Alertmanager receiver so they never arrive subject-lined `[WARNING]`.
+
+**`FluxRolloutStuck` is the one that matters.** Same expression, `for: 30m`, `severity: warning`. It
+means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail
+never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds
+every tier below it on "dependency not ready".
+
+## First commands
+
+`kubectl` does not work on the workstation — the toolchain lives only in the pinned container, and
+the context must always be named explicitly.
+
+Which tiers are behind, and on what:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get gitrepository flux-system \
+    -o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}"
+  kubectl --context k3s-prod get kustomizations -A \
+    -o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"'
+```
+
+Any row whose `APPLIED` differs from `WANTED` is a tier this alert is naming. Then ask **why** that
+tier is not moving — its Ready condition carries the reason:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get kustomization <name> \
+    -o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"'
+```
+
+Confirm the alert is reading live data rather than a stale exporter:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1)
+  kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \
+    /bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"'
+```
+
+Three names must come back (`kube_flux_kustomization_info`, `kube_flux_kustomization_suspended`,
+`kube_flux_gitrepository_info`). If that query is **empty**, the alerts are inert — go to the last
+cause below.
+
+## Likely causes, most likely first
+
+1. **Nothing is wrong — the rollout is simply in flight.** Most Kustomizations reconcile on a 10m
+   interval and the chain is sequential, so several minutes between the "applying" and "applied"
+   mails is normal. `cilium` is on a 1h interval and can legitimately be the last to converge.
+2. **A gating tier is unhealthy, so everything below it is parked.** Look for `DependencyNotReady`
+   in the stuck tier's message and walk *up* the chain to the first tier that is not Ready; that one
+   is the real fault. Its own alert (`FluxReconciliationFailure`) and runbook `flux-reconcile` own it.
+3. **A Kustomization is suspended.** A suspended tier is excluded from this alert by design (so it
+   cannot hold the "applied" mail open forever), but anything that `dependsOn` it still stalls and
+   *will* show here. `FluxResourceSuspended` / runbook `flux-suspended` covers it.
+4. **A health check is timing out.** A `wait: true` tier only records `lastAppliedRevision` once its
+   objects pass health checks, so a workload that never goes Ready keeps the tier on the old
+   revision even though the manifests applied fine.
+5. **kube-state-metrics is not publishing the `kube_flux_*` family.** Then the rules are inert and
+   you would get *no* mail at all rather than a stuck one — so this is the cause when the complaint
+   is silence. The metrics come from `customResourceState` plus a read-only RBAC rule for
+   `kustomize.toolkit.fluxcd.io` / `source.toolkit.fluxcd.io`; without that RBAC, KSM emits the set
+   **empty** and logs a forbidden error. Check its logs:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c '
+     kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \
+       --tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"'
+   ```
+   Note the chart puts **no checksum annotation** over that ConfigMap, so a change to the config
+   alone does not restart the pod — that is why `podAnnotations` carries a
+   `homelab.keithjackson.dev/crs-config-rollout` counter that must be bumped in the same commit.
+
+## When to escalate or stop
+
+- **`FluxRolloutInProgress` alone, cleared within ~15 minutes: do nothing.** That is the feature
+  working. Do not suspend, reconcile or apply anything in response to it.
+- **`FluxRolloutStuck` firing: fix the upstream fault, not this symptom.** This alert names the
+  tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use
+  `flux-reconcile`.
+- **Never `kubectl apply`, `edit` or `patch` the change through by hand.** Flux reconciles `main`;
+  a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes
+  the applied-revision signal these alerts read a lie. `flux reconcile` (an operator action, never
+  an agent's) is the only nudge that is safe, and it only *hurries* the same pull.
+- **Escalate to a revert if a stuck rollout is degrading service.** Reverting the offending commit
+  on `main` through the normal PR flow is the rollback path; see runbook `update-rollback`.
+- **If both alerts are silent during a deploy you know happened,** the notification itself is broken,
+  not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these
+  rules is indistinguishable from "no deploys happened", which is the exact defect they were
+  written to close.

runbooks/kustomization.yaml

@@ -65,6 +65,7 @@ configMapGenerator:
       - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
+      - flux-rollout=flux-rollout.md
       - flux-suspended=flux-suspended.md
       - image-automation=image-automation.md
       - image-policy-stalled=image-policy-stalled.md