Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(obs): notify on Flux deploy start and completion
gate / gate (pull_request) Successful in 1m40s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
The operator got no mail when Flux began applying a git change and none when
it finished. Measured cause: notification-controller v1.4.0 is running and its
CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the
cluster — nothing in git ever created any.

The obvious fix does not work. Neither Flux event can be filtered into a deploy
signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16
Kustomizations, message byte-identical whether the apply changed anything), and
Progressing is worse (five Kustomizations re-apply the same object on every tick
— event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at
all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one).

The applied revision is the only signal that moves when and only when git moves,
and it lives in object status alone. So export it via the kube-state-metrics
custom-resource-state path that already serves the Velero Schedule metrics —
kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and
ask the question in PromQL:

  FluxRolloutInProgress (severity: info, no `for:`) fires the moment a
  non-suspended Kustomization's applied revision is not the revision the
  GitRepository fetched from main, and CLEARS when all have caught up. Fired
  mail = a deploy started; RESOLVED mail = it completed. The completion half is
  send_resolved, not a second rule.

  FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the
  escalation for a rollout that started and never converged.

Two guards, each with a test whose only job is to fail if the clause is removed:
`and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes
`unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert
forever, the permanently-firing mode this repo shipped once already — and the
suspended-Kustomization exclusion, without which a suspended tier holds the
rollout open and the "applied" mail never arrives.

A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy
notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because
the chart sets no checksum over the custom-resource-state ConfigMap, so a
config-only edit would leave KSM on the old collector set with the new alerts
silently inert.

Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test
that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/
spec-immutability green in the pinned toolbox. NOT verified: no mail has been
sent — the metrics do not exist until this merges. Full gap list, and the
deliberately-cut notification-controller half, in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-29 23:40:01 -05:00
1 parent b93594b7c8
commit 55b232e2ce
7 files changed
+789
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh
+25
View File
@@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
"backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
"Velero itself publishes nothing until a backup is attempted",
},
{
"owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
"prefixes": ("kube_flux_",),
"file": KPS,
"require": [
r"customResourceState:\s*\n\s*enabled:\s*true",
r"metricNamePrefix:\s*kube_flux_kustomization",
r"metricNamePrefix:\s*kube_flux_gitrepository",
# The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
# above: without read access to these two API groups KSM publishes the
# custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
# leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
# A config-only edit does not restart KSM (the chart sets no checksum annotation over
# the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
# it and a future change to the flux collector set ships green and never takes effect.
r"homelab\.keithjackson\.dev/crs-config-rollout",
],
"forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
"why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
"revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
"ReconciliationSucceeded on every interval tick whether or not anything changed, so "
"without these metrics nothing can tell a deploy starting from an idle cluster",
},
{
"owner": "kube-prometheus-stack: bundled kube-state-metrics",
"prefixes": ("kube_",),
decisions/2026-09-29-flux-deploy-notifications.md
+144
View File
@@ -0,0 +1,144 @@
---
date: 2026-09-29
branch: feat/flux-deploy-notifications
title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
---
## What changed
The operator was getting **no notification when Flux began applying a git change, and none when it
finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
`FluxResourceSuspended`) — nothing covered the healthy path.
**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
filtered into a deploy signal:
- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
`interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
`platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
`NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
five times every ten minutes, forever.
- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
`gotk_suspend_status` 40, not one with a revision.
The applied **revision** is the only signal that moves when and only when git moves, and it lives in
object status alone. So:
1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
`kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
`kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
`status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
`status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
`podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
a config-only edit would have left KSM running the old collector set with the new metrics absent
and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
`kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
Cost: 33 series.
2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
(`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
revision is not the revision the GitRepository fetched from `main`, and **clears when all have
caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
never converged.
3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
`[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
`[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
`group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
`group_interval: 2m` so completion is not held 4m behind the event.
4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
`conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
deleting any of them fails the gate instead of silently re-inerting the alerts.
Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
and would hold the rollout open, so the "applied" mail would never arrive).
## What I verified
Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
measured:
- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
`alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
`get providers,alerts -A` → **`No resources found`**. This is the defect.
- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
`DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
`flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
(`{kind="Kustomization"}` = 16) — and none carries a revision.
- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
protects_volume_data,info}` = 2 series each.
- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
`.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
`1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
(which is why `nilIsZero: true` is mandatory).
- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
`kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
proves the alert empties itself (the completion mail).
- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
`spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
`alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
and the new runbook all cleared their own checks. Coverage 51/51.
## What I did NOT verify
- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
versions and that each status field is populated on the live objects, but I have not seen KSM emit
`kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
first thing to check after merge:
`count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
after this settles rather than assuming the headroom holds.
- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
`.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
arrives with a poorer subject — cosmetic, but unproven.
- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
deliberately: its unique value over the rules above is the human-readable **commit subject**
(`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
`http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
`severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
(never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
cold-start deadlock this repo already documents.
- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
`NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
`CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
I only measured the event counts. It is a real defect (constant write load on the API server and
etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
alerts here are immune to it by construction: they read revisions, not events.
- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
(`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
that ran is green, and one stage did not run.
platform/monitoring/alerts.yaml
+85
View File
@@ -799,6 +799,91 @@ spec:
summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
# ===================================================================================
# DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to
# send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a
# git change and when it finishes. Read the two together — they share one expression and
# differ only in `for` and severity.
#
# WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a
# deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded
# on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message
# "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply
# changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply
# the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed
# on it mails five times every ten minutes forever. The applied REVISION is the only signal
# that moves when and only when git moves. See the kube_flux_* block in
# kube-prometheus-stack.yaml for how it reaches the TSDB.
#
# THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied
# revision is absent from the set of revisions the GitRepository has fetched — i.e. while it
# is behind main. It empties itself as each Kustomization catches up, which is what makes
# the RESOLVED notification mean "the rollout completed" rather than "the alert timed out".
# Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason,
# and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout.
#
# TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml):
# * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled
# without the ConfigMap, the flux RBAC lost, customResourceState reverted) then
# `unless on (revision) <empty>` removes NOTHING and all 16 Kustomizations alert
# forever, for no reason. That is the permanently-firing failure mode this repo already
# shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT,
# which is the correct direction for a notification — a broken exporter is what
# FluxReconciliationFailure and the platform-health.monitoring-self rules are for.
# * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended
# Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open
# for as long as it stays suspended, so the resolved mail (the "it finished" half of the
# feature) would never arrive. FluxResourceSuspended above is what reports that state.
#
# `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization
# that has never applied anything — or a GitRepository with no artifact yet — publishes the
# series with no `revision` label. Without the matcher those group together under the empty
# string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
# ===================================================================================
- alert: FluxRolloutInProgress
expr: |
(
(
kube_flux_kustomization_info{revision!=""}
unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
)
unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
)
and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
# NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule
# reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the
# operator asked to be told about, so the failure mode to avoid is the opposite: a `for:`
# of even 1m would silently swallow any rollout that converged faster than that (a manual
# `flux reconcile`, a change touching one Kustomization), and a deploy notification that
# only sometimes arrives is worse than no notification at all.
labels:
severity: info # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied"
annotations:
summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
# The escalation: a rollout that STARTED and never finished. Same expression, so it cannot
# disagree with the notification above about what "behind main" means; 30m is comfortably
# past the slowest legitimate convergence (the longest Kustomization interval is 1h for
# cilium, but its dependents are 10m and a wave chain converges well inside 30m).
# severity: warning, so this one takes the normal email-warning path and reads as a fault.
- alert: FluxRolloutStuck
expr: |
(
(
kube_flux_kustomization_info{revision!=""}
unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
)
unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
)
and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
for: 30m
labels:
severity: warning
annotations:
summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision"
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
# =====================================================================================
# Certificates — cert-manager
platform/monitoring/kube-prometheus-stack.yaml
+148
View File
@@ -131,6 +131,36 @@ spec:
# page-worthy — notify fast, remind often
group_wait: 10s
repeat_interval: 1h
# DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires
# the moment the applied revision diverges from main and RESOLVES when every
# Kustomization has caught up — so the FIRED mail is "a deploy started" and the
# RESOLVED mail is "it finished". Both halves are the product, which is why
# send_resolved must stay true on the receiver below.
#
# A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
# routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
# real platform fault, which is how alert mail stops being read at all. `severity: info`
# is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this
# route claims exactly the deploy notices and nothing else. Keep it that way — giving a
# genuine fault severity: info would silently route it here and re-label it a deploy.
- matchers: ['severity = "info"']
receiver: email-deploy
# One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
# Kustomization (16 possible), and grouping on alertname alone collapses them into a
# single mail that lists all of them. The default group_by would have split them by
# namespace as well — harmless today (all 16 are in flux-system) and wrong the moment
# a Kustomization lives elsewhere.
group_by: ["alertname"]
group_wait: 30s
# 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED
# notification waits, and holding "the deploy finished" four minutes behind the
# deploy finishing makes the completion half of this feature feel broken.
group_interval: 2m
# A deploy notice is a statement about one revision, so re-sending it on the 12h
# default would just repeat old news. 24h is long enough never to fire twice for one
# rollout, while still re-nagging if a rollout is genuinely stuck for a whole day
# (FluxRolloutStuck, severity warning, is the real escalation for that case).
repeat_interval: 24h
receivers:
- name: email-warning
email_configs:
@@ -144,6 +174,19 @@ spec:
send_resolved: true
headers:
subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
# Flux deploy notices (the `severity = "info"` route above).
- name: email-deploy
email_configs:
- to: "[email protected]" # alert inbox (operator: change if a different inbox is wanted)
# BOTH transitions are the deliverable here: firing = "Flux started applying a new
# revision", resolved = "every Kustomization reached it". Turning this off would
# delete the completion notification, not merely reduce noise.
send_resolved: true
headers:
# `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview
# without opening the mail, and never the word WARNING for a healthy deploy.
# .Status is "firing" or "resolved" in Alertmanager's notification template data.
subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}'
# --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
grafana:
@@ -272,11 +315,33 @@ spec:
# `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
# restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
# changes the pod spec (new arg + volume), so the initial rollout is automatic.
# FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that
# config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation
# over it — documented in the block above, and the reason the Velero metrics needed a manual
# `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the
# ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set:
# the new metrics never appear and every alert reading them is inert, while the PR is merged
# and green. That is the exact shape of defect this repo keeps getting bitten by, so it is
# closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE
# SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you.
# (Same mechanism as grafana.podAnnotations/oidc-rollout above.)
podAnnotations:
homelab.keithjackson.dev/crs-config-rollout: "1"
rbac:
extraRules:
- apiGroups: ["velero.io"]
resources: ["schedules"]
verbs: ["get", "list", "watch"]
# Flux applied-revision metrics (the kube_flux_* block at the end of `config` below).
# MANDATORY for the same measured reason as velero.io above: without read access KSM
# publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden
# error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently.
- apiGroups: ["kustomize.toolkit.fluxcd.io"]
resources: ["kustomizations"]
verbs: ["get", "list", "watch"]
- apiGroups: ["source.toolkit.fluxcd.io"]
resources: ["gitrepositories"]
verbs: ["get", "list", "watch"]
customResourceState:
enabled: true
config:
@@ -328,6 +393,89 @@ spec:
labelsFromPath:
cron: [spec, schedule]
phase: [status, phase]
# --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) --------------------------
# WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on
# main?", so "Flux started applying a git change" and "Flux is idle" were
# indistinguishable and a deploy could start, stall halfway and never finish with
# nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail:
# * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere —
# gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a
# revision. It can say a Kustomization is unhealthy, never which commit it holds.
# * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits
# ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across
# 16 Kustomizations — with the message "Reconciliation finished in 324ms, next run
# in 10m0s", byte-identical whether the apply changed anything or not.
# * `Progressing` is worse, not better: five Kustomizations re-apply the same object
# on every single tick (2505/2504/2501/2495/2490 on one event object each), so
# anything keyed on it fires five times every ten minutes, forever.
# The applied REVISION is the one signal that moves only when git moves, and it lives
# in these two objects' status and nowhere else:
# GitRepository/flux-system .status.artifact.revision = what main is (poll 1m)
# Kustomization/* .status.lastAppliedRevision = what is applied
# Exporting both makes "applied != desired" a PromQL question, which is what
# FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml).
#
# NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above.
# These are kube-state-metrics series ABOUT Kubernetes objects and must never be
# mistaken for the gotk_* family the Flux controllers themselves export.
#
# VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the
# STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io,
# gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated
# — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that
# same revision as lastAppliedRevision.
#
# Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33.
- groupVersionKind:
group: kustomize.toolkit.fluxcd.io
version: v1
kind: Kustomization
metricNamePrefix: kube_flux_kustomization
labelsFromPath:
name: [metadata, name]
namespace: [metadata, namespace]
metrics:
- name: info
help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)."
each:
type: Info
info:
# KSM OMITS a label whose path is absent (the same behaviour noted for the
# Velero `phase` label above), so a Kustomization that has never applied
# anything publishes this series with NO `revision` label at all rather than
# an empty one. Every rule that reads it therefore matches revision!="" —
# otherwise those would collapse together under the empty string and a
# never-applied Kustomization would compare equal to a converged one.
labelsFromPath:
revision: [status, lastAppliedRevision]
- name: suspended
help: "1 if the Flux Kustomization is suspended (spec.suspend)."
each:
type: Gauge
gauge:
# nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above:
# measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at
# all. Without it this metric would not exist, and the `unless` that stops a
# suspended Kustomization holding a rollout notification open forever would
# silently match nothing.
path: [spec, suspend]
nilIsZero: true
- groupVersionKind:
group: source.toolkit.fluxcd.io
version: v1
kind: GitRepository
metricNamePrefix: kube_flux_gitrepository
labelsFromPath:
name: [metadata, name]
namespace: [metadata, namespace]
metrics:
- name: info
help: "Revision of the artifact a Flux GitRepository has fetched (what main is)."
each:
type: Info
info:
labelsFromPath:
revision: [status, artifact, revision]
# MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
# with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
# roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert
platform/monitoring/tests/flux-rollout.test.yaml
+268
View File
@@ -0,0 +1,268 @@
# promtool rule-unit tests for FluxRolloutInProgress / FluxRolloutStuck — the deploy-notification
# pair in the platform-health.flux group (platform/monitoring/alerts.yaml).
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
# conformance/tests/l0-static/alert-rules-unit-tested)
#
# WHY THESE TESTS CARRY MORE WEIGHT THAN USUAL. This pair is a NOTIFICATION, so both failure
# directions are silent in production and neither is visible on a healthy cluster:
# * fire when it should not -> a permanently-firing "deploy in progress" that mails on every
# repeat_interval and trains the operator to ignore Flux mail. Shipped once already in this
# repo (CNPG, 2026-09-12).
# * fail to fire -> no deploy notification at all, which is the exact complaint this pair was
# written to answer. Indistinguishable from "no deploys happened".
# So every clause is pinned in both directions here, and the two GUARDS in the expression (the
# missing-exporter guard and the suspended-Kustomization exclusion) each get a test whose ONLY
# job is to fail if that clause is deleted.
#
# MODELLING A REVISION CHANGE. `revision` is a LABEL, so a Kustomization moving from revision A to
# revision B is modelled as series A ending and series B beginning — not as one series changing
# value. Prometheus' 5m lookback then means series A is still resolvable for 5m after its last
# sample, so every eval_time below is placed at least 5m clear of a handoff. That is deliberate:
# it keeps these tests about the RULE and not about staleness semantics.
#
# Sample interval 1m; the longest `for` under test is 30m.
rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
# =============================================================================================
# 1. CONVERGED — the steady state this cluster is in almost all of the time.
# Both rules MUST be silent. This is the "not permanently firing" half of the proof, and it
# is the test that fails if `unless on (revision)` is mis-spelled: the left-hand side is
# non-empty here, and only the revision match suppresses it.
# =============================================================================================
- interval: 1m
name: converged (every Kustomization on the GitRepository revision) - silent
input_series:
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
values: '1x40'
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x40'
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x40'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x40'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x40'
alert_rule_test:
- eval_time: 35m
alertname: FluxRolloutInProgress
exp_alerts: []
- eval_time: 35m
alertname: FluxRolloutStuck
exp_alerts: []
# =============================================================================================
# 2. A COMMIT LANDS — the notification this whole change exists to produce.
# GitRepository moves to revision bbbbbbb at t=11m. apps-wave-0 catches up at t=14m;
# `platform` never does.
# t=25m -> FluxRolloutInProgress fires for `platform` ONLY (apps-wave-0 has converged),
# and FluxRolloutStuck is still quiet because its 30m has not elapsed.
# t=50m -> both fire. ("started and never finished" is a real, separate condition.)
# =============================================================================================
- interval: 1m
name: new revision on main - InProgress fires for the lagging Kustomization only
input_series:
# The old artifact revision ends at t=10m, the new one starts at t=11m.
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
values: '1x10'
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
values: '_x10 1x70'
# `platform` stays on the old revision for the whole test.
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x80'
# apps-wave-0 catches up at t=14m.
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x8'
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '_x13 1x67'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x80'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x80'
alert_rule_test:
# The deploy notification, ~14m after the commit was fetched. Exactly ONE alert: a rule that
# ignored the revision match would also list apps-wave-0, which has already converged.
- eval_time: 25m
alertname: FluxRolloutInProgress
exp_alerts:
- exp_labels:
severity: info
job: kube-state-metrics
namespace: flux-system
name: platform
revision: "main@sha1:aaaaaaa"
customresource_group: kustomize.toolkit.fluxcd.io
customresource_kind: Kustomization
customresource_version: v1
exp_annotations:
summary: "Flux is applying a new revision — platform is behind main"
description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
# `for: 30m` on the escalation is real — not yet firing 14m in.
- eval_time: 25m
alertname: FluxRolloutStuck
exp_alerts: []
# ...and it does fire once the rollout has genuinely hung.
- eval_time: 50m
alertname: FluxRolloutStuck
exp_alerts:
- exp_labels:
severity: warning
job: kube-state-metrics
namespace: flux-system
name: platform
revision: "main@sha1:aaaaaaa"
customresource_group: kustomize.toolkit.fluxcd.io
customresource_kind: Kustomization
customresource_version: v1
exp_annotations:
summary: "Flux rollout has not completed: platform stuck on an old revision"
description: "Kustomization flux-system/platform has been behind the flux-system GitRepository revision for 30m (applied main@sha1:aaaaaaa) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
# =============================================================================================
# 3. THE ROLLOUT COMPLETES -> the alert clears, which is what sends the "applied" mail.
# Without this test the completion half of the feature is unproven: an expression that fired
# correctly but never emptied itself would pass test 2 and still never deliver a resolved
# notification.
# =============================================================================================
- interval: 1m
name: rollout completes - InProgress clears (this is the resolved/completion notification)
input_series:
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
values: '1x10'
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
values: '_x10 1x70'
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x20'
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '_x23 1x57'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x80'
alert_rule_test:
# Mid-rollout: firing.
- eval_time: 20m
alertname: FluxRolloutInProgress
exp_alerts:
- exp_labels:
severity: info
job: kube-state-metrics
namespace: flux-system
name: platform
revision: "main@sha1:aaaaaaa"
customresource_group: kustomize.toolkit.fluxcd.io
customresource_kind: Kustomization
customresource_version: v1
exp_annotations:
summary: "Flux is applying a new revision — platform is behind main"
description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
# Converged (5m clear of the old series' last sample at t=20m): silent, and it never had
# 30m of continuous firing, so the escalation must not have fired either.
- eval_time: 32m
alertname: FluxRolloutInProgress
exp_alerts: []
- eval_time: 32m
alertname: FluxRolloutStuck
exp_alerts: []
# =============================================================================================
# 4. GUARD 1 — kube_flux_gitrepository_info ABSENT (KSM rolled without the ConfigMap, the flux
# RBAC was lost, customResourceState reverted). `unless on (revision) <empty set>` removes
# NOTHING, so without the `and on () (count(...) > 0)` guard EVERY Kustomization alerts
# forever. This test exists to go red the moment that guard is deleted.
# Correct behaviour is SILENCE: a broken exporter is FluxReconciliationFailure's job and the
# platform-health.monitoring-self rules', not a deploy notification's.
# =============================================================================================
- interval: 1m
name: GUARD - GitRepository metric absent - silent, NOT 16 alerts forever
input_series:
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x60'
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x60'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x60'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x60'
promql_expr_test:
# Stated explicitly so the reason for the silence is unambiguous in the test output: the
# guard's own count is empty, not merely false.
- expr: 'count(kube_flux_gitrepository_info{name="flux-system", revision!=""})'
eval_time: 45m
exp_samples: []
alert_rule_test:
- eval_time: 45m
alertname: FluxRolloutInProgress
exp_alerts: []
- eval_time: 45m
alertname: FluxRolloutStuck
exp_alerts: []
# =============================================================================================
# 5. GUARD 2 — a SUSPENDED Kustomization pins an old revision by design. It must not hold the
# rollout notification open (or the "applied" mail never arrives), while a non-suspended
# Kustomization on the same old revision MUST still alert.
# Both halves are asserted in one test on purpose: an `unless` written with the wrong labels
# would suppress EVERYTHING and would pass a suppression-only test.
# =============================================================================================
- interval: 1m
name: GUARD - suspended Kustomization excluded, non-suspended one still alerts
input_series:
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
values: '1x40'
# Suspended and behind -> excluded.
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x40'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x40'
# Not suspended and behind -> must alert.
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="conformance",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x40'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="conformance",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x40'
alert_rule_test:
- eval_time: 20m
alertname: FluxRolloutInProgress
exp_alerts:
- exp_labels:
severity: info
job: kube-state-metrics
namespace: flux-system
name: conformance
revision: "main@sha1:aaaaaaa"
customresource_group: kustomize.toolkit.fluxcd.io
customresource_kind: Kustomization
customresource_version: v1
exp_annotations:
summary: "Flux is applying a new revision — conformance is behind main"
description: "Kustomization flux-system/conformance has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
# =============================================================================================
# 6. A Kustomization that has NEVER applied anything publishes no `revision` label at all (KSM
# omits absent paths). `revision!=""` must drop it rather than let it group under the empty
# string and compare equal to the GitRepository revision.
# =============================================================================================
- interval: 1m
name: never-applied Kustomization (no revision label) is not counted as converged or lagging
input_series:
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
values: '1x40'
# No `revision` label at all — this is what KSM publishes for status.lastAppliedRevision="".
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '1x40'
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
values: '0x40'
alert_rule_test:
# Silent: FluxReconciliationFailure (Ready=False for 15m) is what covers a Kustomization
# that cannot apply at all. This pair only ever speaks about revisions it can compare.
- eval_time: 30m
alertname: FluxRolloutInProgress
exp_alerts: []
- eval_time: 30m
alertname: FluxRolloutStuck
exp_alerts: []
runbooks/flux-rollout.md
+118
View File
@@ -0,0 +1,118 @@
# flux-rollout — a git change is rolling out, or has stopped halfway
**Alerts:** `FluxRolloutInProgress` (info, no `for:`) and `FluxRolloutStuck` (warning, 30m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
**Metrics:** `kube_flux_kustomization_info` / `kube_flux_kustomization_suspended` /
`kube_flux_gitrepository_info` — kube-state-metrics custom-resource-state, configured in
`platform/monitoring/kube-prometheus-stack.yaml`
## What this means here
These two are the **deploy notification**, not a fault report. On this cluster merging a PR into
`main` IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event
stream emits `ReconciliationSucceeded` on every 10-minute tick whether or not anything changed (101
such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical.
The pair reads the one signal that moves only when git moves — the **applied revision**:
- `GitRepository/flux-system` `.status.artifact.revision` — what `main` is (polled every 1m)
- `Kustomization/*` `.status.lastAppliedRevision` — what each tier has actually applied
**`FluxRolloutInProgress` fires** the moment any non-suspended Kustomization's applied revision is
not the revision fetched from `main`, and **clears when every one of them has caught up**. So:
| Mail | Subject | Means |
|---|---|---|
| firing | `[FLUX] applying — FluxRolloutInProgress` | a commit started rolling out |
| resolved | `[FLUX] applied — FluxRolloutInProgress` | every Kustomization reached it; rollout complete |
Both are expected, routine, and not a problem. The `severity: info` label routes them to the
dedicated `email-deploy` Alertmanager receiver so they never arrive subject-lined `[WARNING]`.
**`FluxRolloutStuck` is the one that matters.** Same expression, `for: 30m`, `severity: warning`. It
means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail
never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds
every tier below it on "dependency not ready".
## First commands
`kubectl` does not work on the workstation — the toolchain lives only in the pinned container, and
the context must always be named explicitly.
Which tiers are behind, and on what:
```bash
./infra/devcontainer/dev.sh run bash -c '
kubectl --context k3s-prod -n flux-system get gitrepository flux-system \
-o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}"
kubectl --context k3s-prod get kustomizations -A \
-o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"'
```
Any row whose `APPLIED` differs from `WANTED` is a tier this alert is naming. Then ask **why** that
tier is not moving — its Ready condition carries the reason:
```bash
./infra/devcontainer/dev.sh run bash -c '
kubectl --context k3s-prod -n flux-system get kustomization <name> \
-o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"'
```
Confirm the alert is reading live data rather than a stale exporter:
```bash
./infra/devcontainer/dev.sh run bash -c '
POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1)
kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \
/bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"'
```
Three names must come back (`kube_flux_kustomization_info`, `kube_flux_kustomization_suspended`,
`kube_flux_gitrepository_info`). If that query is **empty**, the alerts are inert — go to the last
cause below.
## Likely causes, most likely first
1. **Nothing is wrong — the rollout is simply in flight.** Most Kustomizations reconcile on a 10m
interval and the chain is sequential, so several minutes between the "applying" and "applied"
mails is normal. `cilium` is on a 1h interval and can legitimately be the last to converge.
2. **A gating tier is unhealthy, so everything below it is parked.** Look for `DependencyNotReady`
in the stuck tier's message and walk *up* the chain to the first tier that is not Ready; that one
is the real fault. Its own alert (`FluxReconciliationFailure`) and runbook `flux-reconcile` own it.
3. **A Kustomization is suspended.** A suspended tier is excluded from this alert by design (so it
cannot hold the "applied" mail open forever), but anything that `dependsOn` it still stalls and
*will* show here. `FluxResourceSuspended` / runbook `flux-suspended` covers it.
4. **A health check is timing out.** A `wait: true` tier only records `lastAppliedRevision` once its
objects pass health checks, so a workload that never goes Ready keeps the tier on the old
revision even though the manifests applied fine.
5. **kube-state-metrics is not publishing the `kube_flux_*` family.** Then the rules are inert and
you would get *no* mail at all rather than a stuck one — so this is the cause when the complaint
is silence. The metrics come from `customResourceState` plus a read-only RBAC rule for
`kustomize.toolkit.fluxcd.io` / `source.toolkit.fluxcd.io`; without that RBAC, KSM emits the set
**empty** and logs a forbidden error. Check its logs:
```bash
./infra/devcontainer/dev.sh run bash -c '
kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \
--tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"'
```
Note the chart puts **no checksum annotation** over that ConfigMap, so a change to the config
alone does not restart the pod — that is why `podAnnotations` carries a
`homelab.keithjackson.dev/crs-config-rollout` counter that must be bumped in the same commit.
## When to escalate or stop
- **`FluxRolloutInProgress` alone, cleared within ~15 minutes: do nothing.** That is the feature
working. Do not suspend, reconcile or apply anything in response to it.
- **`FluxRolloutStuck` firing: fix the upstream fault, not this symptom.** This alert names the
tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use
`flux-reconcile`.
- **Never `kubectl apply`, `edit` or `patch` the change through by hand.** Flux reconciles `main`;
a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes
the applied-revision signal these alerts read a lie. `flux reconcile` (an operator action, never
an agent's) is the only nudge that is safe, and it only *hurries* the same pull.
- **Escalate to a revert if a stuck rollout is degrading service.** Reverting the offending commit
on `main` through the normal PR flow is the rollback path; see runbook `update-rollback`.
- **If both alerts are silent during a deploy you know happened,** the notification itself is broken,
not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these
rules is indistinguishable from "no deploys happened", which is the exact defect they were
written to close.
runbooks/kustomization.yaml
+1
View File
@@ -65,6 +65,7 @@ configMapGenerator:
- crafty-admin-credential=crafty-admin-credential.md
- denied-flows=denied-flows.md
- flux-reconcile=flux-reconcile.md
- flux-rollout=flux-rollout.md
- flux-suspended=flux-suspended.md
- image-automation=image-automation.md
- image-policy-stalled=image-policy-stalled.md