feat(obs): notify on Flux deploy start and completion
The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
b93594b7c8
commit
55b232e2ce
7 files changed
+789
No files matched your search
@@ -799,6 +799,91 @@ spec:
|
||||
summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
|
||||
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
|
||||
# ===================================================================================
|
||||
# DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to
|
||||
# send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a
|
||||
# git change and when it finishes. Read the two together — they share one expression and
|
||||
# differ only in `for` and severity.
|
||||
#
|
||||
# WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a
|
||||
# deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded
|
||||
# on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message
|
||||
# "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply
|
||||
# changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply
|
||||
# the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed
|
||||
# on it mails five times every ten minutes forever. The applied REVISION is the only signal
|
||||
# that moves when and only when git moves. See the kube_flux_* block in
|
||||
# kube-prometheus-stack.yaml for how it reaches the TSDB.
|
||||
#
|
||||
# THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied
|
||||
# revision is absent from the set of revisions the GitRepository has fetched — i.e. while it
|
||||
# is behind main. It empties itself as each Kustomization catches up, which is what makes
|
||||
# the RESOLVED notification mean "the rollout completed" rather than "the alert timed out".
|
||||
# Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason,
|
||||
# and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout.
|
||||
#
|
||||
# TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml):
|
||||
# * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled
|
||||
# without the ConfigMap, the flux RBAC lost, customResourceState reverted) then
|
||||
# `unless on (revision) <empty>` removes NOTHING and all 16 Kustomizations alert
|
||||
# forever, for no reason. That is the permanently-firing failure mode this repo already
|
||||
# shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT,
|
||||
# which is the correct direction for a notification — a broken exporter is what
|
||||
# FluxReconciliationFailure and the platform-health.monitoring-self rules are for.
|
||||
# * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended
|
||||
# Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open
|
||||
# for as long as it stays suspended, so the resolved mail (the "it finished" half of the
|
||||
# feature) would never arrive. FluxResourceSuspended above is what reports that state.
|
||||
#
|
||||
# `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization
|
||||
# that has never applied anything — or a GitRepository with no artifact yet — publishes the
|
||||
# series with no `revision` label. Without the matcher those group together under the empty
|
||||
# string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
|
||||
# ===================================================================================
|
||||
- alert: FluxRolloutInProgress
|
||||
expr: |
|
||||
(
|
||||
(
|
||||
kube_flux_kustomization_info{revision!=""}
|
||||
unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
|
||||
)
|
||||
unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
|
||||
)
|
||||
and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
|
||||
# NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule
|
||||
# reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the
|
||||
# operator asked to be told about, so the failure mode to avoid is the opposite: a `for:`
|
||||
# of even 1m would silently swallow any rollout that converged faster than that (a manual
|
||||
# `flux reconcile`, a change touching one Kustomization), and a deploy notification that
|
||||
# only sometimes arrives is worse than no notification at all.
|
||||
labels:
|
||||
severity: info # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied"
|
||||
annotations:
|
||||
summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
|
||||
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
|
||||
# The escalation: a rollout that STARTED and never finished. Same expression, so it cannot
|
||||
# disagree with the notification above about what "behind main" means; 30m is comfortably
|
||||
# past the slowest legitimate convergence (the longest Kustomization interval is 1h for
|
||||
# cilium, but its dependents are 10m and a wave chain converges well inside 30m).
|
||||
# severity: warning, so this one takes the normal email-warning path and reads as a fault.
|
||||
- alert: FluxRolloutStuck
|
||||
expr: |
|
||||
(
|
||||
(
|
||||
kube_flux_kustomization_info{revision!=""}
|
||||
unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
|
||||
)
|
||||
unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
|
||||
)
|
||||
and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision"
|
||||
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
|
||||
|
||||
# =====================================================================================
|
||||
# Certificates — cert-manager
|
||||
|
||||
@@ -131,6 +131,36 @@ spec:
|
||||
# page-worthy — notify fast, remind often
|
||||
group_wait: 10s
|
||||
repeat_interval: 1h
|
||||
# DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires
|
||||
# the moment the applied revision diverges from main and RESOLVES when every
|
||||
# Kustomization has caught up — so the FIRED mail is "a deploy started" and the
|
||||
# RESOLVED mail is "it finished". Both halves are the product, which is why
|
||||
# send_resolved must stay true on the receiver below.
|
||||
#
|
||||
# A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
|
||||
# routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
|
||||
# real platform fault, which is how alert mail stops being read at all. `severity: info`
|
||||
# is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this
|
||||
# route claims exactly the deploy notices and nothing else. Keep it that way — giving a
|
||||
# genuine fault severity: info would silently route it here and re-label it a deploy.
|
||||
- matchers: ['severity = "info"']
|
||||
receiver: email-deploy
|
||||
# One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
|
||||
# Kustomization (16 possible), and grouping on alertname alone collapses them into a
|
||||
# single mail that lists all of them. The default group_by would have split them by
|
||||
# namespace as well — harmless today (all 16 are in flux-system) and wrong the moment
|
||||
# a Kustomization lives elsewhere.
|
||||
group_by: ["alertname"]
|
||||
group_wait: 30s
|
||||
# 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED
|
||||
# notification waits, and holding "the deploy finished" four minutes behind the
|
||||
# deploy finishing makes the completion half of this feature feel broken.
|
||||
group_interval: 2m
|
||||
# A deploy notice is a statement about one revision, so re-sending it on the 12h
|
||||
# default would just repeat old news. 24h is long enough never to fire twice for one
|
||||
# rollout, while still re-nagging if a rollout is genuinely stuck for a whole day
|
||||
# (FluxRolloutStuck, severity warning, is the real escalation for that case).
|
||||
repeat_interval: 24h
|
||||
receivers:
|
||||
- name: email-warning
|
||||
email_configs:
|
||||
@@ -144,6 +174,19 @@ spec:
|
||||
send_resolved: true
|
||||
headers:
|
||||
subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
|
||||
# Flux deploy notices (the `severity = "info"` route above).
|
||||
- name: email-deploy
|
||||
email_configs:
|
||||
- to: "[email protected]" # alert inbox (operator: change if a different inbox is wanted)
|
||||
# BOTH transitions are the deliverable here: firing = "Flux started applying a new
|
||||
# revision", resolved = "every Kustomization reached it". Turning this off would
|
||||
# delete the completion notification, not merely reduce noise.
|
||||
send_resolved: true
|
||||
headers:
|
||||
# `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview
|
||||
# without opening the mail, and never the word WARNING for a healthy deploy.
|
||||
# .Status is "firing" or "resolved" in Alertmanager's notification template data.
|
||||
subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}'
|
||||
|
||||
# --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
|
||||
grafana:
|
||||
@@ -272,11 +315,33 @@ spec:
|
||||
# `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
|
||||
# restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
|
||||
# changes the pod spec (new arg + volume), so the initial rollout is automatic.
|
||||
# FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that
|
||||
# config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation
|
||||
# over it — documented in the block above, and the reason the Velero metrics needed a manual
|
||||
# `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the
|
||||
# ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set:
|
||||
# the new metrics never appear and every alert reading them is inert, while the PR is merged
|
||||
# and green. That is the exact shape of defect this repo keeps getting bitten by, so it is
|
||||
# closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE
|
||||
# SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you.
|
||||
# (Same mechanism as grafana.podAnnotations/oidc-rollout above.)
|
||||
podAnnotations:
|
||||
homelab.keithjackson.dev/crs-config-rollout: "1"
|
||||
rbac:
|
||||
extraRules:
|
||||
- apiGroups: ["velero.io"]
|
||||
resources: ["schedules"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
# Flux applied-revision metrics (the kube_flux_* block at the end of `config` below).
|
||||
# MANDATORY for the same measured reason as velero.io above: without read access KSM
|
||||
# publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden
|
||||
# error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently.
|
||||
- apiGroups: ["kustomize.toolkit.fluxcd.io"]
|
||||
resources: ["kustomizations"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: ["source.toolkit.fluxcd.io"]
|
||||
resources: ["gitrepositories"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
customResourceState:
|
||||
enabled: true
|
||||
config:
|
||||
@@ -328,6 +393,89 @@ spec:
|
||||
labelsFromPath:
|
||||
cron: [spec, schedule]
|
||||
phase: [status, phase]
|
||||
# --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) --------------------------
|
||||
# WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on
|
||||
# main?", so "Flux started applying a git change" and "Flux is idle" were
|
||||
# indistinguishable and a deploy could start, stall halfway and never finish with
|
||||
# nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail:
|
||||
# * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere —
|
||||
# gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a
|
||||
# revision. It can say a Kustomization is unhealthy, never which commit it holds.
|
||||
# * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits
|
||||
# ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across
|
||||
# 16 Kustomizations — with the message "Reconciliation finished in 324ms, next run
|
||||
# in 10m0s", byte-identical whether the apply changed anything or not.
|
||||
# * `Progressing` is worse, not better: five Kustomizations re-apply the same object
|
||||
# on every single tick (2505/2504/2501/2495/2490 on one event object each), so
|
||||
# anything keyed on it fires five times every ten minutes, forever.
|
||||
# The applied REVISION is the one signal that moves only when git moves, and it lives
|
||||
# in these two objects' status and nowhere else:
|
||||
# GitRepository/flux-system .status.artifact.revision = what main is (poll 1m)
|
||||
# Kustomization/* .status.lastAppliedRevision = what is applied
|
||||
# Exporting both makes "applied != desired" a PromQL question, which is what
|
||||
# FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml).
|
||||
#
|
||||
# NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above.
|
||||
# These are kube-state-metrics series ABOUT Kubernetes objects and must never be
|
||||
# mistaken for the gotk_* family the Flux controllers themselves export.
|
||||
#
|
||||
# VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the
|
||||
# STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io,
|
||||
# gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated
|
||||
# — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that
|
||||
# same revision as lastAppliedRevision.
|
||||
#
|
||||
# Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33.
|
||||
- groupVersionKind:
|
||||
group: kustomize.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: Kustomization
|
||||
metricNamePrefix: kube_flux_kustomization
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
namespace: [metadata, namespace]
|
||||
metrics:
|
||||
- name: info
|
||||
help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
# KSM OMITS a label whose path is absent (the same behaviour noted for the
|
||||
# Velero `phase` label above), so a Kustomization that has never applied
|
||||
# anything publishes this series with NO `revision` label at all rather than
|
||||
# an empty one. Every rule that reads it therefore matches revision!="" —
|
||||
# otherwise those would collapse together under the empty string and a
|
||||
# never-applied Kustomization would compare equal to a converged one.
|
||||
labelsFromPath:
|
||||
revision: [status, lastAppliedRevision]
|
||||
- name: suspended
|
||||
help: "1 if the Flux Kustomization is suspended (spec.suspend)."
|
||||
each:
|
||||
type: Gauge
|
||||
gauge:
|
||||
# nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above:
|
||||
# measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at
|
||||
# all. Without it this metric would not exist, and the `unless` that stops a
|
||||
# suspended Kustomization holding a rollout notification open forever would
|
||||
# silently match nothing.
|
||||
path: [spec, suspend]
|
||||
nilIsZero: true
|
||||
- groupVersionKind:
|
||||
group: source.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: GitRepository
|
||||
metricNamePrefix: kube_flux_gitrepository
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
namespace: [metadata, namespace]
|
||||
metrics:
|
||||
- name: info
|
||||
help: "Revision of the artifact a Flux GitRepository has fetched (what main is)."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
revision: [status, artifact, revision]
|
||||
# MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
|
||||
# with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
|
||||
# roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert
|
||||
|
||||
@@ -0,0 +1,268 @@
|
||||
# promtool rule-unit tests for FluxRolloutInProgress / FluxRolloutStuck — the deploy-notification
|
||||
# pair in the platform-health.flux group (platform/monitoring/alerts.yaml).
|
||||
# Run with: platform/monitoring/tests/run.sh (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHY THESE TESTS CARRY MORE WEIGHT THAN USUAL. This pair is a NOTIFICATION, so both failure
|
||||
# directions are silent in production and neither is visible on a healthy cluster:
|
||||
# * fire when it should not -> a permanently-firing "deploy in progress" that mails on every
|
||||
# repeat_interval and trains the operator to ignore Flux mail. Shipped once already in this
|
||||
# repo (CNPG, 2026-09-12).
|
||||
# * fail to fire -> no deploy notification at all, which is the exact complaint this pair was
|
||||
# written to answer. Indistinguishable from "no deploys happened".
|
||||
# So every clause is pinned in both directions here, and the two GUARDS in the expression (the
|
||||
# missing-exporter guard and the suspended-Kustomization exclusion) each get a test whose ONLY
|
||||
# job is to fail if that clause is deleted.
|
||||
#
|
||||
# MODELLING A REVISION CHANGE. `revision` is a LABEL, so a Kustomization moving from revision A to
|
||||
# revision B is modelled as series A ending and series B beginning — not as one series changing
|
||||
# value. Prometheus' 5m lookback then means series A is still resolvable for 5m after its last
|
||||
# sample, so every eval_time below is placed at least 5m clear of a handoff. That is deliberate:
|
||||
# it keeps these tests about the RULE and not about staleness semantics.
|
||||
#
|
||||
# Sample interval 1m; the longest `for` under test is 30m.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. CONVERGED — the steady state this cluster is in almost all of the time.
|
||||
# Both rules MUST be silent. This is the "not permanently firing" half of the proof, and it
|
||||
# is the test that fails if `unless on (revision)` is mis-spelled: the left-hand side is
|
||||
# non-empty here, and only the revision match suppresses it.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: converged (every Kustomization on the GitRepository revision) - silent
|
||||
input_series:
|
||||
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x40'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x40'
|
||||
alert_rule_test:
|
||||
- eval_time: 35m
|
||||
alertname: FluxRolloutInProgress
|
||||
exp_alerts: []
|
||||
- eval_time: 35m
|
||||
alertname: FluxRolloutStuck
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. A COMMIT LANDS — the notification this whole change exists to produce.
|
||||
# GitRepository moves to revision bbbbbbb at t=11m. apps-wave-0 catches up at t=14m;
|
||||
# `platform` never does.
|
||||
# t=25m -> FluxRolloutInProgress fires for `platform` ONLY (apps-wave-0 has converged),
|
||||
# and FluxRolloutStuck is still quiet because its 30m has not elapsed.
|
||||
# t=50m -> both fire. ("started and never finished" is a real, separate condition.)
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: new revision on main - InProgress fires for the lagging Kustomization only
|
||||
input_series:
|
||||
# The old artifact revision ends at t=10m, the new one starts at t=11m.
|
||||
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
|
||||
values: '1x10'
|
||||
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
|
||||
values: '_x10 1x70'
|
||||
# `platform` stays on the old revision for the whole test.
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x80'
|
||||
# apps-wave-0 catches up at t=14m.
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x8'
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '_x13 1x67'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x80'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x80'
|
||||
alert_rule_test:
|
||||
# The deploy notification, ~14m after the commit was fetched. Exactly ONE alert: a rule that
|
||||
# ignored the revision match would also list apps-wave-0, which has already converged.
|
||||
- eval_time: 25m
|
||||
alertname: FluxRolloutInProgress
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: info
|
||||
job: kube-state-metrics
|
||||
namespace: flux-system
|
||||
name: platform
|
||||
revision: "main@sha1:aaaaaaa"
|
||||
customresource_group: kustomize.toolkit.fluxcd.io
|
||||
customresource_kind: Kustomization
|
||||
customresource_version: v1
|
||||
exp_annotations:
|
||||
summary: "Flux is applying a new revision — platform is behind main"
|
||||
description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
|
||||
# `for: 30m` on the escalation is real — not yet firing 14m in.
|
||||
- eval_time: 25m
|
||||
alertname: FluxRolloutStuck
|
||||
exp_alerts: []
|
||||
# ...and it does fire once the rollout has genuinely hung.
|
||||
- eval_time: 50m
|
||||
alertname: FluxRolloutStuck
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
job: kube-state-metrics
|
||||
namespace: flux-system
|
||||
name: platform
|
||||
revision: "main@sha1:aaaaaaa"
|
||||
customresource_group: kustomize.toolkit.fluxcd.io
|
||||
customresource_kind: Kustomization
|
||||
customresource_version: v1
|
||||
exp_annotations:
|
||||
summary: "Flux rollout has not completed: platform stuck on an old revision"
|
||||
description: "Kustomization flux-system/platform has been behind the flux-system GitRepository revision for 30m (applied main@sha1:aaaaaaa) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
|
||||
|
||||
# =============================================================================================
|
||||
# 3. THE ROLLOUT COMPLETES -> the alert clears, which is what sends the "applied" mail.
|
||||
# Without this test the completion half of the feature is unproven: an expression that fired
|
||||
# correctly but never emptied itself would pass test 2 and still never deliver a resolved
|
||||
# notification.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: rollout completes - InProgress clears (this is the resolved/completion notification)
|
||||
input_series:
|
||||
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
|
||||
values: '1x10'
|
||||
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
|
||||
values: '_x10 1x70'
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x20'
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '_x23 1x57'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x80'
|
||||
alert_rule_test:
|
||||
# Mid-rollout: firing.
|
||||
- eval_time: 20m
|
||||
alertname: FluxRolloutInProgress
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: info
|
||||
job: kube-state-metrics
|
||||
namespace: flux-system
|
||||
name: platform
|
||||
revision: "main@sha1:aaaaaaa"
|
||||
customresource_group: kustomize.toolkit.fluxcd.io
|
||||
customresource_kind: Kustomization
|
||||
customresource_version: v1
|
||||
exp_annotations:
|
||||
summary: "Flux is applying a new revision — platform is behind main"
|
||||
description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
|
||||
# Converged (5m clear of the old series' last sample at t=20m): silent, and it never had
|
||||
# 30m of continuous firing, so the escalation must not have fired either.
|
||||
- eval_time: 32m
|
||||
alertname: FluxRolloutInProgress
|
||||
exp_alerts: []
|
||||
- eval_time: 32m
|
||||
alertname: FluxRolloutStuck
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. GUARD 1 — kube_flux_gitrepository_info ABSENT (KSM rolled without the ConfigMap, the flux
|
||||
# RBAC was lost, customResourceState reverted). `unless on (revision) <empty set>` removes
|
||||
# NOTHING, so without the `and on () (count(...) > 0)` guard EVERY Kustomization alerts
|
||||
# forever. This test exists to go red the moment that guard is deleted.
|
||||
# Correct behaviour is SILENCE: a broken exporter is FluxReconciliationFailure's job and the
|
||||
# platform-health.monitoring-self rules', not a deploy notification's.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: GUARD - GitRepository metric absent - silent, NOT 16 alerts forever
|
||||
input_series:
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x60'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x60'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x60'
|
||||
promql_expr_test:
|
||||
# Stated explicitly so the reason for the silence is unambiguous in the test output: the
|
||||
# guard's own count is empty, not merely false.
|
||||
- expr: 'count(kube_flux_gitrepository_info{name="flux-system", revision!=""})'
|
||||
eval_time: 45m
|
||||
exp_samples: []
|
||||
alert_rule_test:
|
||||
- eval_time: 45m
|
||||
alertname: FluxRolloutInProgress
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: FluxRolloutStuck
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 5. GUARD 2 — a SUSPENDED Kustomization pins an old revision by design. It must not hold the
|
||||
# rollout notification open (or the "applied" mail never arrives), while a non-suspended
|
||||
# Kustomization on the same old revision MUST still alert.
|
||||
# Both halves are asserted in one test on purpose: an `unless` written with the wrong labels
|
||||
# would suppress EVERYTHING and would pass a suppression-only test.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: GUARD - suspended Kustomization excluded, non-suspended one still alerts
|
||||
input_series:
|
||||
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
# Suspended and behind -> excluded.
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
# Not suspended and behind -> must alert.
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="conformance",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="conformance",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x40'
|
||||
alert_rule_test:
|
||||
- eval_time: 20m
|
||||
alertname: FluxRolloutInProgress
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: info
|
||||
job: kube-state-metrics
|
||||
namespace: flux-system
|
||||
name: conformance
|
||||
revision: "main@sha1:aaaaaaa"
|
||||
customresource_group: kustomize.toolkit.fluxcd.io
|
||||
customresource_kind: Kustomization
|
||||
customresource_version: v1
|
||||
exp_annotations:
|
||||
summary: "Flux is applying a new revision — conformance is behind main"
|
||||
description: "Kustomization flux-system/conformance has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
|
||||
|
||||
# =============================================================================================
|
||||
# 6. A Kustomization that has NEVER applied anything publishes no `revision` label at all (KSM
|
||||
# omits absent paths). `revision!=""` must drop it rather than let it group under the empty
|
||||
# string and compare equal to the GitRepository revision.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: never-applied Kustomization (no revision label) is not counted as converged or lagging
|
||||
input_series:
|
||||
- series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
# No `revision` label at all — this is what KSM publishes for status.lastAppliedRevision="".
|
||||
- series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '1x40'
|
||||
- series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
|
||||
values: '0x40'
|
||||
alert_rule_test:
|
||||
# Silent: FluxReconciliationFailure (Ready=False for 15m) is what covers a Kustomization
|
||||
# that cannot apply at all. This pair only ever speaks about revisions it can compare.
|
||||
- eval_time: 30m
|
||||
alertname: FluxRolloutInProgress
|
||||
exp_alerts: []
|
||||
- eval_time: 30m
|
||||
alertname: FluxRolloutStuck
|
||||
exp_alerts: []
|
||||
Reference in new issue
Block a user