feat(obs): notify on Flux deploy start and completion #25

2026-09-30T04:42:56Z

gitea-admin commented

2026-09-30 04:42:56 +00:00

The defect

No notification arrived when Flux began applying a git change, and none when it finished.

Measured cause, not inferred: notification-controller v1.4.0 is running (1/1) and the
Provider/Alert/Receiver CRDs are installed and serving v1beta3 — but there are zero
Provider and zero Alert objects in the entire cluster (kubectl get providers,alerts -A →
No resources found). Nothing in git ever created any. The five gotk_* rules in alerts.yaml
only cover failures; nothing covered the healthy path.

Why the obvious fix does not work

Neither Flux event can be filtered into a deploy signal. Measured on prod 2026-09-29:

Candidate	Why it fails
`ReconciliationSucceeded`	A heartbeat. 16 Kustomizations, 15 on `interval: 10m` → 101 events in one hour, message `Reconciliation finished in 324.161192ms, next run in 10m0s` — identical in shape whether the apply changed anything or not.
`Progressing`	Worse. Five Kustomizations re-apply the same object on every tick — event counts 2505 / 2504 / 2501 / 2495 / 2490. Anything keyed on it mails 5× every 10 minutes, forever.
`gotk_*`	Carries no revision label anywhere — `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40, not one with a revision.

The applied revision is the only signal that moves when and only when git moves, and it lives in
object status alone.

What this does

kube-state-metrics — two resources added to the existing customResourceState config (the
path already serving the Velero Schedule metrics): kube_flux_kustomization_info{revision},
kube_flux_kustomization_suspended, kube_flux_gitrepository_info{revision}. Plus the two
mandatory read-only rbac.extraRules, and a podAnnotations roll-counter — the chart sets no
checksum over that ConfigMap, so a config-only edit would leave KSM on the old collector set
with the new alerts silently inert. Named kube_flux_* not gotk_*, same rule as
kube_velero_schedule_*. Cost: 33 series.
FluxRolloutInProgress (severity: info, no for:) fires the moment a non-suspended
Kustomization's applied revision is not the revision the GitRepository fetched from main, and
clears when all have caught up. Fired mail = a deploy started; resolved mail = it
completed. The completion half is send_resolved, not a second rule.
FluxRolloutStuck (severity: warning, for: 30m) — same expression, as the escalation for
a rollout that started and never converged.
Alertmanager — a dedicated severity = "info" route + email-deploy receiver, subject
[FLUX] applying|applied — …. Without it a routine deploy arrives [WARNING] via the catch-all
and becomes indistinguishable from a real fault. group_by: [alertname] → one mail per rollout.
Runbook flux-rollout (+ its configMapGenerator entry) and a kube_flux_ entry in the
alert-metrics-have-scrape-targets owner table, whose require patterns pin
customResourceState, both metricNamePrefixes, both apiGroups and the roll-counter — so
deleting any of them fails the gate instead of silently re-inerting the alerts.

Two guards, each with a test whose only job is to fail if the clause is removed

and on () (count(kube_flux_gitrepository_info…) > 0) — without it, a missing exporter makes
unless on (revision) <empty> remove nothing and all 16 Kustomizations alert forever. That is
the permanently-firing mode this repo shipped once already (CNPG, 2026-09-12).
unless on (namespace, name) (kube_flux_kustomization_suspended == 1) — a suspended Kustomization
pins an old revision by design and would hold the rollout open, so the "applied" mail would never
arrive. Asserted together with a non-suspended Kustomization that must still alert, so an
unless written with the wrong labels cannot pass by suppressing everything.

Gate

./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' → rc 3 (UNDETERMINED), and that is
reported as-is, not as green.

Green: lint, l0, render-golden, l1, decisions-log, spec-immutability. Coverage 51/51.
l0 includes alert-metrics-have-scrape-targets, alert-rules-unit-tested and
alert-runbooks-resolvable — the new owner entry, the new tests and the new runbook each cleared
their own check.
promtool test rules: SUCCESS, 53 rules, promtool 3.14.0 from the pinned toolbox. 6 scenarios
including both guards and the convergence test that proves the alert empties itself.
Did not run: gitea-settings (no SETTINGS_API_TOKEN — branch protection and runner state are
UNVERIFIED by this run) and L2/L3 conformance (no cluster; vars.CLUSTER_AVAILABLE unset).
Touched requirements R-OBS-01 R-OBS-02 R-OBS-04 R-TEST-01..04 are therefore not certified.

Not verified — read before merging

No mail has been sent. The metrics do not exist until this merges. First check afterwards:
count by (__name__) ({__name__=~"kube_flux.*"}) must return three names. If a status path is
wrong, KSM omits the label, revision!="" drops the series and the alerts go quiet rather than
wrong — the safe direction, but it means silence is the failure signal to watch for.
KSM memory not re-measured. 384Mi limit vs ~111–123Mi working set; this adds two small
informers. 91 OOMKills followed the last CRS addition at a 128Mi limit, so re-measure
max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h]) after it settles.
The notification-controller Provider/Alert half was deliberately cut. Its unique value is
the human-readable commit subject, and notification-controller has no SMTP provider so it must post
into Alertmanager (reachability confirmed: flux-system/allow-egress is egress: [{}], monitoring
has no NetworkPolicy). I could not establish what label set that provider actually emits without
applying it, and the route matches on severity — so the route would have rested on a guess.
Follow-up, in its own wait: false Kustomization.
A pre-existing drift loop is untouched and unexplained. Five Kustomizations have re-applied the
same object every 10 minutes for ~25 days — something outside git is mutating
NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny and CRD/cdis.cdi.kubevirt.io after Flux
applies them. Cause UNVERIFIED (I measured only the counts). Deserves its own branch; the alerts
here are immune to it by construction because they read revisions, not events.

Full gap list in decisions/2026-09-29-flux-deploy-notifications.md.

🤖 Generated with Claude Code

## The defect No notification arrived when Flux began applying a git change, and none when it finished. Measured cause, not inferred: `notification-controller` v1.4.0 is running (`1/1`) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3` — but there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*` rules in `alerts.yaml` only cover *failures*; nothing covered the healthy path. ## Why the obvious fix does not work Neither Flux event can be filtered into a deploy signal. Measured on prod 2026-09-29: | Candidate | Why it fails | |---|---| | `ReconciliationSucceeded` | A **heartbeat**. 16 Kustomizations, 15 on `interval: 10m` → **101 events in one hour**, message `Reconciliation finished in 324.161192ms, next run in 10m0s` — identical in shape whether the apply changed anything or not. | | `Progressing` | **Worse.** Five Kustomizations re-apply the same object on *every* tick — event counts **2505 / 2504 / 2501 / 2495 / 2490**. Anything keyed on it mails 5× every 10 minutes, forever. | | `gotk_*` | Carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40, not one with a revision. | The applied **revision** is the only signal that moves when and only when git moves, and it lives in object status alone. ## What this does 1. **kube-state-metrics** — two resources added to the *existing* `customResourceState` config (the path already serving the Velero Schedule metrics): `kube_flux_kustomization_info{revision}`, `kube_flux_kustomization_suspended`, `kube_flux_gitrepository_info{revision}`. Plus the two mandatory read-only `rbac.extraRules`, and a `podAnnotations` roll-counter — the chart sets **no checksum** over that ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Named `kube_flux_*` not `gotk_*`, same rule as `kube_velero_schedule_*`. Cost: 33 series. 2. **`FluxRolloutInProgress`** (`severity: info`, **no `for:`**) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from `main`, and **clears when all have caught up**. Fired mail = *a deploy started*; **resolved mail = *it completed***. The completion half is `send_resolved`, not a second rule. 3. **`FluxRolloutStuck`** (`severity: warning`, `for: 30m`) — same expression, as the escalation for a rollout that started and never converged. 4. **Alertmanager** — a dedicated `severity = "info"` route + `email-deploy` receiver, subject `[FLUX] applying|applied — …`. Without it a routine deploy arrives `[WARNING]` via the catch-all and becomes indistinguishable from a real fault. `group_by: [alertname]` → one mail per rollout. 5. Runbook `flux-rollout` (+ its `configMapGenerator` entry) and a `kube_flux_` entry in the `alert-metrics-have-scrape-targets` owner table, whose `require` patterns pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so deleting any of them fails the gate instead of silently re-inerting the alerts. ## Two guards, each with a test whose only job is to fail if the clause is removed - `and on () (count(kube_flux_gitrepository_info…) > 0)` — without it, a missing exporter makes `unless on (revision) <empty>` remove **nothing** and all 16 Kustomizations alert forever. That is the permanently-firing mode this repo shipped once already (CNPG, 2026-09-12). - `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended Kustomization pins an old revision by design and would hold the rollout open, so the "applied" mail would never arrive. Asserted together with a non-suspended Kustomization that **must** still alert, so an `unless` written with the wrong labels cannot pass by suppressing everything. ## Gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'` → **rc 3 (UNDETERMINED)**, and that is reported as-is, not as green. - Green: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`, `spec-immutability`. Coverage 51/51. `l0` includes `alert-metrics-have-scrape-targets`, `alert-rules-unit-tested` and `alert-runbooks-resolvable` — the new owner entry, the new tests and the new runbook each cleared their own check. - `promtool test rules`: **SUCCESS**, 53 rules, promtool 3.14.0 from the pinned toolbox. 6 scenarios including both guards and the convergence test that proves the alert empties itself. - **Did not run:** `gitea-settings` (no `SETTINGS_API_TOKEN` — branch protection and runner state are UNVERIFIED by this run) and L2/L3 `conformance` (no cluster; `vars.CLUSTER_AVAILABLE` unset). Touched requirements `R-OBS-01 R-OBS-02 R-OBS-04 R-TEST-01..04` are therefore **not certified**. ## Not verified — read before merging - **No mail has been sent.** The metrics do not exist until this merges. First check afterwards: `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names. If a status path is wrong, KSM omits the label, `revision!=""` drops the series and the alerts go **quiet** rather than wrong — the safe direction, but it means silence is the failure signal to watch for. - **KSM memory not re-measured.** 384Mi limit vs ~111–123Mi working set; this adds two small informers. 91 OOMKills followed the *last* CRS addition at a 128Mi limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])` after it settles. - **The `notification-controller` `Provider`/`Alert` half was deliberately cut.** Its unique value is the human-readable commit subject, and notification-controller has no SMTP provider so it must post into Alertmanager (reachability confirmed: `flux-system/allow-egress` is `egress: [{}]`, `monitoring` has no NetworkPolicy). I could not establish what label set that provider actually emits without applying it, and the route matches on `severity` — so the route would have rested on a guess. Follow-up, in its own `wait: false` Kustomization. - **A pre-existing drift loop is untouched and unexplained.** Five Kustomizations have re-applied the same object every 10 minutes for ~25 days — something outside git is mutating `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and `CRD/cdis.cdi.kubevirt.io` after Flux applies them. Cause UNVERIFIED (I measured only the counts). Deserves its own branch; the alerts here are immune to it by construction because they read revisions, not events. Full gap list in `decisions/2026-09-29-flux-deploy-notifications.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-09-30 04:42:56 +00:00

feat(obs): notify on Flux deploy start and completion

gate / gate (pull_request) Successful in 1m40s

Details

gate / conformance (pull_request) Has been skipped

Details

55b232e2ce

The operator got no mail when Flux began applying a git change and none when
it finished. Measured cause: notification-controller v1.4.0 is running and its
CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the
cluster — nothing in git ever created any.

The obvious fix does not work. Neither Flux event can be filtered into a deploy
signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16
Kustomizations, message byte-identical whether the apply changed anything), and
Progressing is worse (five Kustomizations re-apply the same object on every tick
— event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at
all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one).

The applied revision is the only signal that moves when and only when git moves,
and it lives in object status alone. So export it via the kube-state-metrics
custom-resource-state path that already serves the Velero Schedule metrics —
kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and
ask the question in PromQL:

  FluxRolloutInProgress (severity: info, no `for:`) fires the moment a
  non-suspended Kustomization's applied revision is not the revision the
  GitRepository fetched from main, and CLEARS when all have caught up. Fired
  mail = a deploy started; RESOLVED mail = it completed. The completion half is
  send_resolved, not a second rule.

  FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the
  escalation for a rollout that started and never converged.

Two guards, each with a test whose only job is to fail if the clause is removed:
`and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes
`unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert
forever, the permanently-firing mode this repo shipped once already — and the
suspended-Kustomization exclusion, without which a suspended tier holds the
rollout open and the "applied" mail never arrives.

A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy
notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because
the chart sets no checksum over the custom-resource-state ConfigMap, so a
config-only edit would leave KSM on the old collector set with the new alerts
silently inert.

Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test
that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/
spec-immutability green in the pinned toolbox. NOT verified: no mail has been
sent — the metrics do not exist until this merges. Full gap list, and the
deliberately-cut notification-controller half, in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 added 1 commit 2026-09-30 04:46:41 +00:00

Merge branch 'main' into feat/flux-deploy-notifications

gate / gate (pull_request) Successful in 1m39s

Details

gate / conformance (pull_request) Has been skipped

Details

1ea73544c9

jakman85 approved these changes

2026-09-30 04:52:07 +00:00

jakman85 merged commit 64c72fd394 into main

2026-09-30 04:55:51 +00:00

jakman85 referenced this issue from a commit