home-network-k3s-impl/decisions/2026-09-29-flux-deploy-notifications.md

---
date: 2026-09-29
branch: feat/flux-deploy-notifications
title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
---

## What changed

The operator was getting **no notification when Flux began applying a git change, and none when it
finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
`FluxResourceSuspended`) — nothing covered the healthy path.

**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
filtered into a deploy signal:

- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
  `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
  324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
  not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
  counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
  `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
  `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
  five times every ten minutes, forever.
- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
  `gotk_suspend_status` 40, not one with a revision.

The applied **revision** is the only signal that moves when and only when git moves, and it lives in
object status alone. So:

1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
   `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
   `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
   `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
   `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
   `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
   a config-only edit would have left KSM running the old collector set with the new metrics absent
   and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
   `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
   Cost: 33 series.
2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
   (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
   revision is not the revision the GitRepository fetched from `main`, and **clears when all have
   caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
   half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
   warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
   never converged.
3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
   `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
   `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
   `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
   `group_interval: 2m` so completion is not held 4m behind the event.
4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
   `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
   pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
   deleting any of them fails the gate instead of silently re-inerting the alerts.

Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
and would hold the rollout open, so the "applied" mail would never arrive).

## What I verified

Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
measured:

- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
  `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
  `get providers,alerts -A` → **`No resources found`**. This is the defect.
- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
  involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
  `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
  `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
  (`{kind="Kustomization"}` = 16) — and none carries a revision.
- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
  protects_volume_data,info}` = 2 series each.
- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
  `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
  `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
  (which is why `nilIsZero: true` is mandatory).
- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
  the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
  `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
  pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
  proves the alert empties itself (the completion mail).
- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
  `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
  `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
  and the new runbook all cleared their own checks. Coverage 51/51.

## What I did NOT verify

- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
  rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
  ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
  and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
  CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
  versions and that each status field is populated on the live objects, but I have not seen KSM emit
  `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
  then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
  first thing to check after merge:
  `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
  two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
  limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
  after this settles rather than assuming the headroom holds.
- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
  `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
  arrives with a poorer subject — cosmetic, but unproven.
- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
  deliberately: its unique value over the rules above is the human-readable **commit subject**
  (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
  revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
  SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
  `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
  establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
  `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
  the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
  Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
  (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
  cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
  cold-start deadlock this repo already documents.
- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
  re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
  `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
  `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
  I only measured the event counts. It is a real defect (constant write load on the API server and
  etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
  alerts here are immune to it by construction: they read revisions, not events.
- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
  (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
  UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
  that ran is green, and one stage did not run.