feat(obs): notify on Flux deploy start and completion

The operator got no mail when Flux began applying a git change and none when
it finished. Measured cause: notification-controller v1.4.0 is running and its
CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the
cluster — nothing in git ever created any.

The obvious fix does not work. Neither Flux event can be filtered into a deploy
signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16
Kustomizations, message byte-identical whether the apply changed anything), and
Progressing is worse (five Kustomizations re-apply the same object on every tick
— event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at
all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one).

The applied revision is the only signal that moves when and only when git moves,
and it lives in object status alone. So export it via the kube-state-metrics
custom-resource-state path that already serves the Velero Schedule metrics —
kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and
ask the question in PromQL:

  FluxRolloutInProgress (severity: info, no `for:`) fires the moment a
  non-suspended Kustomization's applied revision is not the revision the
  GitRepository fetched from main, and CLEARS when all have caught up. Fired
  mail = a deploy started; RESOLVED mail = it completed. The completion half is
  send_resolved, not a second rule.

  FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the
  escalation for a rollout that started and never converged.

Two guards, each with a test whose only job is to fail if the clause is removed:
`and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes
`unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert
forever, the permanently-firing mode this repo shipped once already — and the
suspended-Kustomization exclusion, without which a suspended tier holds the
rollout open and the "applied" mail never arrives.

A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy
notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because
the chart sets no checksum over the custom-resource-state ConfigMap, so a
config-only edit would leave KSM on the old collector set with the new alerts
silently inert.

Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test
that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/
spec-immutability green in the pinned toolbox. NOT verified: no mail has been
sent — the metrics do not exist until this merges. Full gap list, and the
deliberately-cut notification-controller half, in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform/monitoring/kube-prometheus-stack.yaml

@@ -131,6 +131,36 @@ spec:
               # page-worthy — notify fast, remind often
               group_wait: 10s
               repeat_interval: 1h
+            # DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires
+            # the moment the applied revision diverges from main and RESOLVES when every
+            # Kustomization has caught up — so the FIRED mail is "a deploy started" and the
+            # RESOLVED mail is "it finished". Both halves are the product, which is why
+            # send_resolved must stay true on the receiver below.
+            #
+            # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
+            # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
+            # real platform fault, which is how alert mail stops being read at all. `severity: info`
+            # is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this
+            # route claims exactly the deploy notices and nothing else. Keep it that way — giving a
+            # genuine fault severity: info would silently route it here and re-label it a deploy.
+            - matchers: ['severity = "info"']
+              receiver: email-deploy
+              # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
+              # Kustomization (16 possible), and grouping on alertname alone collapses them into a
+              # single mail that lists all of them. The default group_by would have split them by
+              # namespace as well — harmless today (all 16 are in flux-system) and wrong the moment
+              # a Kustomization lives elsewhere.
+              group_by: ["alertname"]
+              group_wait: 30s
+              # 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED
+              # notification waits, and holding "the deploy finished" four minutes behind the
+              # deploy finishing makes the completion half of this feature feel broken.
+              group_interval: 2m
+              # A deploy notice is a statement about one revision, so re-sending it on the 12h
+              # default would just repeat old news. 24h is long enough never to fire twice for one
+              # rollout, while still re-nagging if a rollout is genuinely stuck for a whole day
+              # (FluxRolloutStuck, severity warning, is the real escalation for that case).
+              repeat_interval: 24h
         receivers:
           - name: email-warning
             email_configs:
@@ -144,6 +174,19 @@ spec:
                 send_resolved: true
                 headers:
                   subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
+          # Flux deploy notices (the `severity = "info"` route above).
+          - name: email-deploy
+            email_configs:
+              - to: "[email protected]"    # alert inbox (operator: change if a different inbox is wanted)
+                # BOTH transitions are the deliverable here: firing = "Flux started applying a new
+                # revision", resolved = "every Kustomization reached it". Turning this off would
+                # delete the completion notification, not merely reduce noise.
+                send_resolved: true
+                headers:
+                  # `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview
+                  # without opening the mail, and never the word WARNING for a healthy deploy.
+                  # .Status is "firing" or "resolved" in Alertmanager's notification template data.
+                  subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}'
 
     # --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
     grafana:
@@ -272,11 +315,33 @@ spec:
       #     `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
       #     restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
       #     changes the pod spec (new arg + volume), so the initial rollout is automatic.
+      # FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that
+      # config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation
+      # over it — documented in the block above, and the reason the Velero metrics needed a manual
+      # `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the
+      # ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set:
+      # the new metrics never appear and every alert reading them is inert, while the PR is merged
+      # and green. That is the exact shape of defect this repo keeps getting bitten by, so it is
+      # closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE
+      # SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you.
+      # (Same mechanism as grafana.podAnnotations/oidc-rollout above.)
+      podAnnotations:
+        homelab.keithjackson.dev/crs-config-rollout: "1"
       rbac:
         extraRules:
           - apiGroups: ["velero.io"]
             resources: ["schedules"]
             verbs: ["get", "list", "watch"]
+          # Flux applied-revision metrics (the kube_flux_* block at the end of `config` below).
+          # MANDATORY for the same measured reason as velero.io above: without read access KSM
+          # publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden
+          # error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently.
+          - apiGroups: ["kustomize.toolkit.fluxcd.io"]
+            resources: ["kustomizations"]
+            verbs: ["get", "list", "watch"]
+          - apiGroups: ["source.toolkit.fluxcd.io"]
+            resources: ["gitrepositories"]
+            verbs: ["get", "list", "watch"]
       customResourceState:
         enabled: true
         config:
@@ -328,6 +393,89 @@ spec:
                         labelsFromPath:
                           cron: [spec, schedule]
                           phase: [status, phase]
+              # --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) --------------------------
+              # WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on
+              # main?", so "Flux started applying a git change" and "Flux is idle" were
+              # indistinguishable and a deploy could start, stall halfway and never finish with
+              # nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail:
+              #   * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere —
+              #     gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a
+              #     revision. It can say a Kustomization is unhealthy, never which commit it holds.
+              #   * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits
+              #     ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across
+              #     16 Kustomizations — with the message "Reconciliation finished in 324ms, next run
+              #     in 10m0s", byte-identical whether the apply changed anything or not.
+              #   * `Progressing` is worse, not better: five Kustomizations re-apply the same object
+              #     on every single tick (2505/2504/2501/2495/2490 on one event object each), so
+              #     anything keyed on it fires five times every ten minutes, forever.
+              # The applied REVISION is the one signal that moves only when git moves, and it lives
+              # in these two objects' status and nowhere else:
+              #   GitRepository/flux-system  .status.artifact.revision   = what main is (poll 1m)
+              #   Kustomization/*            .status.lastAppliedRevision = what is applied
+              # Exporting both makes "applied != desired" a PromQL question, which is what
+              # FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml).
+              #
+              # NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above.
+              # These are kube-state-metrics series ABOUT Kubernetes objects and must never be
+              # mistaken for the gotk_* family the Flux controllers themselves export.
+              #
+              # VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the
+              # STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io,
+              # gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated
+              # — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that
+              # same revision as lastAppliedRevision.
+              #
+              # Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33.
+              - groupVersionKind:
+                  group: kustomize.toolkit.fluxcd.io
+                  version: v1
+                  kind: Kustomization
+                metricNamePrefix: kube_flux_kustomization
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)."
+                    each:
+                      type: Info
+                      info:
+                        # KSM OMITS a label whose path is absent (the same behaviour noted for the
+                        # Velero `phase` label above), so a Kustomization that has never applied
+                        # anything publishes this series with NO `revision` label at all rather than
+                        # an empty one. Every rule that reads it therefore matches revision!="" —
+                        # otherwise those would collapse together under the empty string and a
+                        # never-applied Kustomization would compare equal to a converged one.
+                        labelsFromPath:
+                          revision: [status, lastAppliedRevision]
+                  - name: suspended
+                    help: "1 if the Flux Kustomization is suspended (spec.suspend)."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above:
+                        # measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at
+                        # all. Without it this metric would not exist, and the `unless` that stops a
+                        # suspended Kustomization holding a rollout notification open forever would
+                        # silently match nothing.
+                        path: [spec, suspend]
+                        nilIsZero: true
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: GitRepository
+                metricNamePrefix: kube_flux_gitrepository
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision of the artifact a Flux GitRepository has fetched (what main is)."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          revision: [status, artifact, revision]
       # MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
       # with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
       # roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert