From a79693d3bd2cb83e48bf6fa3ae1dae0d07dbc0e9 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 08:53:21 -0500 Subject: [PATCH] =?UTF-8?q?feat(monitoring):=20scrape=20the=20Flux=20contr?= =?UTF-8?q?ollers=20=E2=80=94=20five=20gotk=5F*=20alerts=20were=20inert?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit kube-prometheus-stack does not scrape flux-system: the controllers are bootstrapped, expose no Service, and no chart value covers them. So every gotk_* alert rule in platform/monitoring/alerts.yaml could never fire — verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose entire delivery model is Flux had no alerting on reconciliation failing, a Kustomization erroring, or a resource left suspended. Proven live before landing: with this PodMonitor applied, gotk_suspend_status returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended actually fired on a deliberate test suspension — evidence the rule is live, not merely present. Extracted from the wider inert-alert audit on fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the self-contained, verified half; Longhorn and cert-manager metrics are still ABSENT and their alerts remain inert. It also adopts the object into git: it existed on prod only as kubectl-client-side-apply, so a rebuild would have lost it. Co-Authored-By: Claude Opus 5 --- platform-config/kustomization.yaml | 7 ++ platform/monitoring/flux-podmonitor.yaml | 99 ++++++++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 platform/monitoring/flux-podmonitor.yaml diff --git a/platform-config/kustomization.yaml b/platform-config/kustomization.yaml index 6d5cfed..3916e94 100644 --- a/platform-config/kustomization.yaml +++ b/platform-config/kustomization.yaml @@ -48,6 +48,13 @@ resources: # was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the # file header. (2026-09-06, from the CNPG WAL/archiving incident.) - ../platform/monitoring/pvc-capacity-alerts.yaml + # PodMonitor for the six Flux controllers — the ONLY thing that puts `gotk_*` in the TSDB. + # kube-prometheus-stack does not scrape flux-system (the controllers are bootstrapped, expose no + # Service, and no chart value covers them), so until this landed the five gotk_* rules in + # alerts.yaml above were structurally inert: verified on prod 2026-09-13, + # `count({__name__=~"gotk.*"})` returned nothing. A CR of monitoring.coreos.com -> this tier, same + # as alerts.yaml. (2026-09-13, from the inert-alert audit.) + - ../platform/monitoring/flux-podmonitor.yaml # kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the # operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform- # installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain diff --git a/platform/monitoring/flux-podmonitor.yaml b/platform/monitoring/flux-podmonitor.yaml new file mode 100644 index 0000000..f5297ae --- /dev/null +++ b/platform/monitoring/flux-podmonitor.yaml @@ -0,0 +1,99 @@ +--- +# PodMonitor for the Flux controllers — the `gotk_*` metric family (R-OBS-04, docs/10). +# +# WHY THIS FILE EXISTS: platform/monitoring/alerts.yaml ships five rules that query `gotk_*` — +# FluxReconciliationFailure (critical), FluxResourceSuspended, ImageUpdateAutomationFailing, +# ImagePolicyStalled, UpdateHealthGateFailed — and NONE of them could ever fire. +# VERIFIED live on prod 2026-09-13: +# count({__name__=~"gotk.*"}) => (empty) +# count(gotk_reconcile_condition) => (empty) +# count(gotk_suspend_status) => (empty) +# count by (job) (up) => 19 jobs, none in flux-system +# kube-prometheus-stack does NOT scrape flux-system: the Flux controllers are bootstrapped by +# `flux bootstrap` (not by a HelmRelease with a `serviceMonitor` value), they expose no Service at +# all, and nothing in this repo declared a monitor for them. So the whole gotk_* family was absent +# from the TSDB. On a cluster whose ENTIRE delivery model is Flux, that means reconciliation +# stopping, a Kustomization failing, or a resource left `suspend: true` after a debugging session +# was invisible to alerting — the failure mode is silent drift between git and the cluster, which is +# the one thing R-GITOPS-01 promises cannot happen quietly. +# +# WHY A PodMonitor AND NOT A ServiceMonitor: the controllers have no Service. Each controller Pod +# exposes its Prometheus endpoint directly on a container port named `http-prom`. CONFIRMED against +# prod 2026-09-13 — all six controllers (helm-, source-, kustomize-, notification-, +# image-automation-, image-reflector-controller) carry `app: ` and `http-prom=8080`: +# kubectl -n flux-system get pods -o jsonpath='{...ports[*].name}={...containerPort}' +# helm-controller-… http-prom=8080 healthz=9440 +# image-automation-controller-… http-prom=8080 healthz=9440 +# image-reflector-controller-… http-prom=8080 healthz=9440 +# kustomize-controller-… http-prom=8080 healthz=9440 +# notification-controller-… http=9090 http-webhook=9292 http-prom=8080 healthz=9440 +# source-controller-… http=9090 http-prom=8080 healthz=9440 +# This is the shape upstream Flux documents for kube-prometheus-stack (the flux2-monitoring-example +# `podmonitor.yaml`), reproduced here rather than vendored so the selector list is reviewable and +# matches THIS cluster's controller set exactly. +# +# `app` (not app.kubernetes.io/name) is deliberate: that is the label `flux bootstrap` actually +# stamps on the controller Pods, as printed above. A matchLabels on app.kubernetes.io/name would +# select nothing and reproduce the defect this file fixes — a monitor that exists, discovers zero +# targets, and leaves the alerts inert (exactly how the apps/ SLI ServiceMonitors fail today: they +# name a `metrics` port their Service does not have). +# +# WHICH TIER: this is a CR of monitoring.coreos.com, whose CRDs are installed by the +# kube-prometheus-stack HelmRelease in ./platform. It is therefore applied by the `platform-config` +# Flux Kustomization (dependsOn: platform), NOT the controllers pass — identical wiring to +# platform/monitoring/alerts.yaml, platform/kubevirt/servicemonitor.yaml and +# platform/cnpg/alerts.yaml. The file lives here, beside the alerts.yaml rules that consume gotk_*. +# +# No labels are needed for selection: platform/monitoring/kube-prometheus-stack.yaml sets +# podMonitorSelectorNilUsesHelmValues: false, so Prometheus selects every PodMonitor in every +# namespace regardless of labels. `release: kube-prometheus-stack` is set anyway, matching the +# convention of the sibling monitoring CRs in this directory. +# +# Cost: 6 scrape targets, one gotk_* set per controller (reconcile_condition, suspend_status, +# reconcile_duration_seconds + the controller-runtime/Go families) — tens of series per controller. +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: flux-system + namespace: flux-system + labels: + release: kube-prometheus-stack + platform-health: "true" +spec: + namespaceSelector: + matchNames: + - flux-system + selector: + matchExpressions: + # The six controllers `flux bootstrap` installs on this cluster. An explicit list rather than + # `matchLabels: {}`: flux-system also runs short-lived Jobs and (in future) the notification + # receiver, and a blanket selector would add permanently-down targets — the noise the k3s + # control-plane scrape trim in kube-prometheus-stack.yaml exists to avoid. + - key: app + operator: In + values: + - helm-controller + - source-controller + - kustomize-controller + - notification-controller + - image-automation-controller + - image-reflector-controller + podMetricsEndpoints: + - port: http-prom + relabelings: + # Carry the node name onto every gotk_* series. Upstream's documented relabeling; it is what + # lets a "reconciliation is failing" signal be correlated with a node-level fault (the + # prod-1 disk/etcd-starvation pattern in DECISIONS-LOG), where the controller's own labels + # say nothing about where it was running. + # + # Upstream's example spells this with `regex: ^(.*)$` / `replacement: $1`. Both are the + # Prometheus DEFAULTS for action `replace`, and they are omitted here ON PURPOSE: this file + # is rendered by a Flux Kustomization with postBuild.substituteFrom, and a bare `$1` in a + # substituted path is the shape the escape contract in clusters/base/apps-vms.yaml exists to + # reason about. envsubst does leave `$1` alone (it is not a braced expansion and does not + # start with a name character), so it would be safe — but relying on that to carry a value + # that is already the default is a needless hostage to the next toolchain bump. Same + # behaviour, nothing to escape. + - sourceLabels: [__meta_kubernetes_pod_node_name] + targetLabel: node + action: replace