jakman85andClaude Opus 5
a79693d3bd
kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.
Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.
Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.
Co-Authored-By: Claude Opus 5 <[email protected]>
100 lines
5.9 KiB
YAML
100 lines
5.9 KiB
YAML
---
|
|
# PodMonitor for the Flux controllers — the `gotk_*` metric family (R-OBS-04, docs/10).
|
|
#
|
|
# WHY THIS FILE EXISTS: platform/monitoring/alerts.yaml ships five rules that query `gotk_*` —
|
|
# FluxReconciliationFailure (critical), FluxResourceSuspended, ImageUpdateAutomationFailing,
|
|
# ImagePolicyStalled, UpdateHealthGateFailed — and NONE of them could ever fire.
|
|
# VERIFIED live on prod 2026-09-13:
|
|
# count({__name__=~"gotk.*"}) => (empty)
|
|
# count(gotk_reconcile_condition) => (empty)
|
|
# count(gotk_suspend_status) => (empty)
|
|
# count by (job) (up) => 19 jobs, none in flux-system
|
|
# kube-prometheus-stack does NOT scrape flux-system: the Flux controllers are bootstrapped by
|
|
# `flux bootstrap` (not by a HelmRelease with a `serviceMonitor` value), they expose no Service at
|
|
# all, and nothing in this repo declared a monitor for them. So the whole gotk_* family was absent
|
|
# from the TSDB. On a cluster whose ENTIRE delivery model is Flux, that means reconciliation
|
|
# stopping, a Kustomization failing, or a resource left `suspend: true` after a debugging session
|
|
# was invisible to alerting — the failure mode is silent drift between git and the cluster, which is
|
|
# the one thing R-GITOPS-01 promises cannot happen quietly.
|
|
#
|
|
# WHY A PodMonitor AND NOT A ServiceMonitor: the controllers have no Service. Each controller Pod
|
|
# exposes its Prometheus endpoint directly on a container port named `http-prom`. CONFIRMED against
|
|
# prod 2026-09-13 — all six controllers (helm-, source-, kustomize-, notification-,
|
|
# image-automation-, image-reflector-controller) carry `app: <controller-name>` and `http-prom=8080`:
|
|
# kubectl -n flux-system get pods -o jsonpath='{...ports[*].name}={...containerPort}'
|
|
# helm-controller-… http-prom=8080 healthz=9440
|
|
# image-automation-controller-… http-prom=8080 healthz=9440
|
|
# image-reflector-controller-… http-prom=8080 healthz=9440
|
|
# kustomize-controller-… http-prom=8080 healthz=9440
|
|
# notification-controller-… http=9090 http-webhook=9292 http-prom=8080 healthz=9440
|
|
# source-controller-… http=9090 http-prom=8080 healthz=9440
|
|
# This is the shape upstream Flux documents for kube-prometheus-stack (the flux2-monitoring-example
|
|
# `podmonitor.yaml`), reproduced here rather than vendored so the selector list is reviewable and
|
|
# matches THIS cluster's controller set exactly.
|
|
#
|
|
# `app` (not app.kubernetes.io/name) is deliberate: that is the label `flux bootstrap` actually
|
|
# stamps on the controller Pods, as printed above. A matchLabels on app.kubernetes.io/name would
|
|
# select nothing and reproduce the defect this file fixes — a monitor that exists, discovers zero
|
|
# targets, and leaves the alerts inert (exactly how the apps/ SLI ServiceMonitors fail today: they
|
|
# name a `metrics` port their Service does not have).
|
|
#
|
|
# WHICH TIER: this is a CR of monitoring.coreos.com, whose CRDs are installed by the
|
|
# kube-prometheus-stack HelmRelease in ./platform. It is therefore applied by the `platform-config`
|
|
# Flux Kustomization (dependsOn: platform), NOT the controllers pass — identical wiring to
|
|
# platform/monitoring/alerts.yaml, platform/kubevirt/servicemonitor.yaml and
|
|
# platform/cnpg/alerts.yaml. The file lives here, beside the alerts.yaml rules that consume gotk_*.
|
|
#
|
|
# No labels are needed for selection: platform/monitoring/kube-prometheus-stack.yaml sets
|
|
# podMonitorSelectorNilUsesHelmValues: false, so Prometheus selects every PodMonitor in every
|
|
# namespace regardless of labels. `release: kube-prometheus-stack` is set anyway, matching the
|
|
# convention of the sibling monitoring CRs in this directory.
|
|
#
|
|
# Cost: 6 scrape targets, one gotk_* set per controller (reconcile_condition, suspend_status,
|
|
# reconcile_duration_seconds + the controller-runtime/Go families) — tens of series per controller.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PodMonitor
|
|
metadata:
|
|
name: flux-system
|
|
namespace: flux-system
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
namespaceSelector:
|
|
matchNames:
|
|
- flux-system
|
|
selector:
|
|
matchExpressions:
|
|
# The six controllers `flux bootstrap` installs on this cluster. An explicit list rather than
|
|
# `matchLabels: {}`: flux-system also runs short-lived Jobs and (in future) the notification
|
|
# receiver, and a blanket selector would add permanently-down targets — the noise the k3s
|
|
# control-plane scrape trim in kube-prometheus-stack.yaml exists to avoid.
|
|
- key: app
|
|
operator: In
|
|
values:
|
|
- helm-controller
|
|
- source-controller
|
|
- kustomize-controller
|
|
- notification-controller
|
|
- image-automation-controller
|
|
- image-reflector-controller
|
|
podMetricsEndpoints:
|
|
- port: http-prom
|
|
relabelings:
|
|
# Carry the node name onto every gotk_* series. Upstream's documented relabeling; it is what
|
|
# lets a "reconciliation is failing" signal be correlated with a node-level fault (the
|
|
# prod-1 disk/etcd-starvation pattern in DECISIONS-LOG), where the controller's own labels
|
|
# say nothing about where it was running.
|
|
#
|
|
# Upstream's example spells this with `regex: ^(.*)$` / `replacement: $1`. Both are the
|
|
# Prometheus DEFAULTS for action `replace`, and they are omitted here ON PURPOSE: this file
|
|
# is rendered by a Flux Kustomization with postBuild.substituteFrom, and a bare `$1` in a
|
|
# substituted path is the shape the escape contract in clusters/base/apps-vms.yaml exists to
|
|
# reason about. envsubst does leave `$1` alone (it is not a braced expansion and does not
|
|
# start with a name character), so it would be safe — but relying on that to carry a value
|
|
# that is already the default is a needless hostage to the next toolchain bump. Same
|
|
# behaviour, nothing to escape.
|
|
- sourceLabels: [__meta_kubernetes_pod_node_name]
|
|
targetLabel: node
|
|
action: replace
|