feat(monitoring): scrape the Flux controllers — five gotk_* alerts were inert
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.

Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.

Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 08:53:21 -05:00
1 parent 16b7fcd677
commit a79693d3bd
2 files changed
+106

No files matched your search

+7
View File
@@ -48,6 +48,13 @@ resources:
# was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
# file header. (2026-09-06, from the CNPG WAL/archiving incident.)
- ../platform/monitoring/pvc-capacity-alerts.yaml
# PodMonitor for the six Flux controllers — the ONLY thing that puts `gotk_*` in the TSDB.
# kube-prometheus-stack does not scrape flux-system (the controllers are bootstrapped, expose no
# Service, and no chart value covers them), so until this landed the five gotk_* rules in
# alerts.yaml above were structurally inert: verified on prod 2026-09-13,
# `count({__name__=~"gotk.*"})` returned nothing. A CR of monitoring.coreos.com -> this tier, same
# as alerts.yaml. (2026-09-13, from the inert-alert audit.)
- ../platform/monitoring/flux-podmonitor.yaml
# kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
# operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
# installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain