feat(monitoring): scrape the Flux controllers — five gotk_* alerts were inert

kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.

Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.

Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform-config/kustomization.yaml

@@ -48,6 +48,13 @@ resources:
   # was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
   # file header. (2026-09-06, from the CNPG WAL/archiving incident.)
   - ../platform/monitoring/pvc-capacity-alerts.yaml
+  # PodMonitor for the six Flux controllers — the ONLY thing that puts `gotk_*` in the TSDB.
+  # kube-prometheus-stack does not scrape flux-system (the controllers are bootstrapped, expose no
+  # Service, and no chart value covers them), so until this landed the five gotk_* rules in
+  # alerts.yaml above were structurally inert: verified on prod 2026-09-13,
+  # `count({__name__=~"gotk.*"})` returned nothing. A CR of monitoring.coreos.com -> this tier, same
+  # as alerts.yaml. (2026-09-13, from the inert-alert audit.)
+  - ../platform/monitoring/flux-podmonitor.yaml
   # kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
   # operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
   # installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain

platform/monitoring/flux-podmonitor.yaml

@@ -0,0 +1,99 @@
+---
+# PodMonitor for the Flux controllers — the `gotk_*` metric family (R-OBS-04, docs/10).
+#
+# WHY THIS FILE EXISTS: platform/monitoring/alerts.yaml ships five rules that query `gotk_*` —
+# FluxReconciliationFailure (critical), FluxResourceSuspended, ImageUpdateAutomationFailing,
+# ImagePolicyStalled, UpdateHealthGateFailed — and NONE of them could ever fire.
+# VERIFIED live on prod 2026-09-13:
+#     count({__name__=~"gotk.*"})   => (empty)
+#     count(gotk_reconcile_condition) => (empty)
+#     count(gotk_suspend_status)      => (empty)
+#     count by (job) (up)            => 19 jobs, none in flux-system
+# kube-prometheus-stack does NOT scrape flux-system: the Flux controllers are bootstrapped by
+# `flux bootstrap` (not by a HelmRelease with a `serviceMonitor` value), they expose no Service at
+# all, and nothing in this repo declared a monitor for them. So the whole gotk_* family was absent
+# from the TSDB. On a cluster whose ENTIRE delivery model is Flux, that means reconciliation
+# stopping, a Kustomization failing, or a resource left `suspend: true` after a debugging session
+# was invisible to alerting — the failure mode is silent drift between git and the cluster, which is
+# the one thing R-GITOPS-01 promises cannot happen quietly.
+#
+# WHY A PodMonitor AND NOT A ServiceMonitor: the controllers have no Service. Each controller Pod
+# exposes its Prometheus endpoint directly on a container port named `http-prom`. CONFIRMED against
+# prod 2026-09-13 — all six controllers (helm-, source-, kustomize-, notification-,
+# image-automation-, image-reflector-controller) carry `app: <controller-name>` and `http-prom=8080`:
+#     kubectl -n flux-system get pods -o jsonpath='{...ports[*].name}={...containerPort}'
+#       helm-controller-…              http-prom=8080 healthz=9440
+#       image-automation-controller-…  http-prom=8080 healthz=9440
+#       image-reflector-controller-…   http-prom=8080 healthz=9440
+#       kustomize-controller-…         http-prom=8080 healthz=9440
+#       notification-controller-…      http=9090 http-webhook=9292 http-prom=8080 healthz=9440
+#       source-controller-…            http=9090 http-prom=8080 healthz=9440
+# This is the shape upstream Flux documents for kube-prometheus-stack (the flux2-monitoring-example
+# `podmonitor.yaml`), reproduced here rather than vendored so the selector list is reviewable and
+# matches THIS cluster's controller set exactly.
+#
+# `app` (not app.kubernetes.io/name) is deliberate: that is the label `flux bootstrap` actually
+# stamps on the controller Pods, as printed above. A matchLabels on app.kubernetes.io/name would
+# select nothing and reproduce the defect this file fixes — a monitor that exists, discovers zero
+# targets, and leaves the alerts inert (exactly how the apps/ SLI ServiceMonitors fail today: they
+# name a `metrics` port their Service does not have).
+#
+# WHICH TIER: this is a CR of monitoring.coreos.com, whose CRDs are installed by the
+# kube-prometheus-stack HelmRelease in ./platform. It is therefore applied by the `platform-config`
+# Flux Kustomization (dependsOn: platform), NOT the controllers pass — identical wiring to
+# platform/monitoring/alerts.yaml, platform/kubevirt/servicemonitor.yaml and
+# platform/cnpg/alerts.yaml. The file lives here, beside the alerts.yaml rules that consume gotk_*.
+#
+# No labels are needed for selection: platform/monitoring/kube-prometheus-stack.yaml sets
+# podMonitorSelectorNilUsesHelmValues: false, so Prometheus selects every PodMonitor in every
+# namespace regardless of labels. `release: kube-prometheus-stack` is set anyway, matching the
+# convention of the sibling monitoring CRs in this directory.
+#
+# Cost: 6 scrape targets, one gotk_* set per controller (reconcile_condition, suspend_status,
+# reconcile_duration_seconds + the controller-runtime/Go families) — tens of series per controller.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: flux-system
+  namespace: flux-system
+  labels:
+    release: kube-prometheus-stack
+    platform-health: "true"
+spec:
+  namespaceSelector:
+    matchNames:
+      - flux-system
+  selector:
+    matchExpressions:
+      # The six controllers `flux bootstrap` installs on this cluster. An explicit list rather than
+      # `matchLabels: {}`: flux-system also runs short-lived Jobs and (in future) the notification
+      # receiver, and a blanket selector would add permanently-down targets — the noise the k3s
+      # control-plane scrape trim in kube-prometheus-stack.yaml exists to avoid.
+      - key: app
+        operator: In
+        values:
+          - helm-controller
+          - source-controller
+          - kustomize-controller
+          - notification-controller
+          - image-automation-controller
+          - image-reflector-controller
+  podMetricsEndpoints:
+    - port: http-prom
+      relabelings:
+        # Carry the node name onto every gotk_* series. Upstream's documented relabeling; it is what
+        # lets a "reconciliation is failing" signal be correlated with a node-level fault (the
+        # prod-1 disk/etcd-starvation pattern in DECISIONS-LOG), where the controller's own labels
+        # say nothing about where it was running.
+        #
+        # Upstream's example spells this with `regex: ^(.*)$` / `replacement: $1`. Both are the
+        # Prometheus DEFAULTS for action `replace`, and they are omitted here ON PURPOSE: this file
+        # is rendered by a Flux Kustomization with postBuild.substituteFrom, and a bare `$1` in a
+        # substituted path is the shape the escape contract in clusters/base/apps-vms.yaml exists to
+        # reason about. envsubst does leave `$1` alone (it is not a braced expansion and does not
+        # start with a name character), so it would be safe — but relying on that to carry a value
+        # that is already the default is a needless hostage to the next toolchain bump. Same
+        # behaviour, nothing to escape.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node
+          action: replace