home-network-k3s-impl/decisions/2026-09-29-flux-deploy-notifications.md

date, branch, title

date	branch	title
2026-09-29	feat/flux-deploy-notifications	Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes

What changed

The operator was getting no notification when Flux began applying a git change, and none when it finished. Root cause, measured rather than inferred: notification-controller v1.4.0 is running (1/1 ready) and the Provider/Alert/Receiver CRDs are installed and serving v1beta3, but there are zero Provider and zero Alert objects in the entire cluster (kubectl get providers,alerts -A → No resources found). Nothing in git ever created any. The five gotk_* rules in platform/monitoring/alerts.yaml only cover failures (FluxReconciliationFailure, FluxResourceSuspended) — nothing covered the healthy path.

The obvious fix does not work, and this is the substance of the change. Neither Flux event can be filtered into a deploy signal:

• ReconciliationSucceeded is a heartbeat, not a change signal. 16 Kustomizations, 15 on interval: 10m → 101 such events in one hour, with the message Reconciliation finished in 324.161192ms, next run in 10m0s — byte-identical in shape whether the apply changed anything or not. An Alert at eventSeverity: info would have mailed ~100 times an hour.
• Progressing is worse. Five Kustomizations re-apply the same object on every tick — event counts 2505 / 2504 / 2501 / 2495 / 2490 (apps-vms → NetworkPolicy/unifi/default-deny, platform → CustomResourceDefinition/cdis.cdi.kubevirt.io, apps-wave-0/1/2 → NetworkPolicy/{gitea,nextcloud,seerr}/default-deny). Anything keyed on Progressing would mail five times every ten minutes, forever.
• gotk_* carries no revision label anywhere — gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a revision.

The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So:

1. platform/monitoring/kube-prometheus-stack.yaml — two resources added to the existing kube-state-metrics.customResourceState config (the Velero-Schedule path, already enabled): kube_flux_kustomization_info{revision} + kube_flux_kustomization_suspended (from status.lastAppliedRevision / spec.suspend) and kube_flux_gitrepository_info{revision} (from status.artifact.revision). Plus the two mandatory read-only rbac.extraRules entries, and a podAnnotations roll-counter — the chart sets no checksum annotation over that ConfigMap, so a config-only edit would have left KSM running the old collector set with the new metrics absent and the new alerts silently inert. Named kube_flux_*, never gotk_*, for the same reason kube_velero_schedule_* is not velero_*: these are KSM series about Kubernetes objects. Cost: 33 series.
2. platform/monitoring/alerts.yaml, group platform-health.flux — FluxRolloutInProgress (severity: info, no for:) fires the moment any non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and clears when all have caught up. Fired mail = "a deploy started"; resolved mail = "it completed" — the completion half is delivered by send_resolved, not by a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged.
3. Alertmanager — a dedicated severity = "info" route and email-deploy receiver, subject [FLUX] applying|applied — …. Without it a routine deploy would have arrived subject-lined [WARNING] via the catch-all receiver and become indistinguishable from a real fault. group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout; group_interval: 2m so completion is not held 4m behind the event.
4. platform/monitoring/tests/flux-rollout.test.yaml — 6 promtool scenarios, both directions.
5. runbooks/flux-rollout.md (+ its configMapGenerator entry) and a kube_flux_ entry in conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh whose require patterns pin customResourceState, both metricNamePrefixes, both apiGroups and the roll-counter — so deleting any of them fails the gate instead of silently re-inerting the alerts.

Two guards in the expression, each with a test whose only job is to fail if that clause is removed: and on () (count(kube_flux_gitrepository_info…) > 0) (without it, a missing exporter makes unless on (revision) <empty> remove nothing and all 16 Kustomizations alert forever — the permanently-firing mode this repo shipped once already), and unless on (namespace, name) (kube_flux_kustomization_suspended == 1) (a suspended Kustomization pins an old revision by design and would hold the rollout open, so the "applied" mail would never arrive).

What I verified

Measured against live prod (--context k3s-prod) before writing anything, and quoted here as measured:

• notification-controller 1/1, image ghcr.io/fluxcd/notification-controller:v1.4.0; alerts/providers/receivers.notification.toolkit.fluxcd.io all serve v1beta3 (storage); get providers,alerts -A → No resources found. This is the defect.
• Event counts above, from kubectl get events -A -o json filtered to toolkit.fluxcd.io involvedObjects: 101 Kustomization/ReconciliationSucceeded, 10 Progressing, 15 DependencyNotReady, 4 GitRepository/GitOperationSucceeded, 1 NewArtifact.
• The gotk_* family is in the TSDB now (flux-podmonitor works): 6 targets up in job flux-system/flux-system, gotk_reconcile_condition 120 series, gotk_suspend_status 40 ({kind="Kustomization"} = 16) — and none carries a revision.
• customResourceState already works end to end: kube_velero_schedule_{created,paused, protects_volume_data,info} = 2 series each.
• Both CRDs serve v1 as the storage version; GitRepository/flux-system .status.artifact.revision = main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de, spec.interval 1m; all 16 Kustomizations report that same lastAppliedRevision; none sets spec.suspend (which is why nilIsZero: true is mandatory).
• Reachability for the follow-up below: flux-system/allow-egress is egress: [{}] (allow-all) and the monitoring namespace has no NetworkPolicy at all, so nothing blocks flux-system → kube-prometheus-stack-alertmanager (10.43.60.242:9093).
• promtool test rules on the new suite: SUCCESS, 53 rules parsed, promtool 3.14.0 from the pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that proves the alert empties itself (the completion mail).
• Full gate in the pinned toolbox: lint, l0, render-golden, l1, decisions-log, spec-immutability all green — l0 includes alert-metrics-have-scrape-targets, alert-rules-unit-tested and alert-runbooks-resolvable, i.e. the new owner entry, the new tests and the new runbook all cleared their own checks. Coverage 51/51.

What I did NOT verify

• No mail has been sent. Nothing here has run on the cluster: the metrics do not exist yet, the rules are not loaded, and the email-deploy receiver has never delivered. The end-to-end claim ("a merge produces an [FLUX] applying mail and then an [FLUX] applied mail") is UNVERIFIED and can only be proven by merging this and watching the next commit. Agents cannot apply, and CI has no cluster (vars.CLUSTER_AVAILABLE unset → L2/L3 did not run).
• The KSM custom-resource-state paths are asserted, not observed. I verified the CRD storage versions and that each status field is populated on the live objects, but I have not seen KSM emit kube_flux_*. If a path is wrong KSM emits the metric with the label omitted, revision!="" then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the first thing to check after merge: count by (__name__) ({__name__=~"kube_flux.*"}) must return three names.
• KSM memory was not re-measured. Limit is 384Mi against a working set of ~111–123Mi; this adds two small informers (16 + 1 objects). 91 OOMKills followed the last CRS addition at a 128Mi limit, so re-measure max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h]) after this settles rather than assuming the headroom holds.
• Alertmanager's .Status subject template is untested. [FLUX] applying|applied relies on .Status being firing/resolved in the notification template data. Wrong, and the mail still arrives with a poorer subject — cosmetic, but unproven.
• The notification-controller Provider/Alert half is designed but NOT shipped. It was cut deliberately: its unique value over the rules above is the human-readable commit subject (GitRepository/NewArtifact, count 1 per commit, message stored artifact for commit '<subject>', revision in source.toolkit.fluxcd.io/revision metadata), and notification-controller has no SMTP provider, so it must post into Alertmanager via type: alertmanager at http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts. I could not establish what label set that provider actually emits (whether it sets alertname, and whether severity is info) without applying it, and the email-deploy route matches on severity. So the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off Alertmanager, then wire the route — and put it in its own wait: false Flux Kustomization (never platform or platform-config, both wait: true), because a notification mechanism that cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped cold-start deadlock this repo already documents.
• The 2500-count drift loop is NOT fixed, and is not mine. Five Kustomizations have been re-applying the same object every 10 minutes for ~25 days: something outside git is mutating NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny and CustomResourceDefinition/cdis.cdi.kubevirt.io after Flux applies them. Cause UNVERIFIED — I only measured the event counts. It is a real defect (constant write load on the API server and etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The alerts here are immune to it by construction: they read revisions, not events.
• Branch protection / runner state re-measured? No. The gitea-settings stage SKIPPED (SETTINGS_API_TOKEN unset), so whether main is still protected and the gate still required is UNVERIFIED by this run. The gate's overall verdict is therefore rc 3, not rc 0 — every stage that ran is green, and one stage did not run.