feat(monitoring): scrape the Flux controllers — five gotk_* alerts were inert
kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.
Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.
Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
16b7fcd677
commit
a79693d3bd
2 files changed
+106
No files matched your search
@@ -48,6 +48,13 @@ resources:
|
||||
# was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
|
||||
# file header. (2026-09-06, from the CNPG WAL/archiving incident.)
|
||||
- ../platform/monitoring/pvc-capacity-alerts.yaml
|
||||
# PodMonitor for the six Flux controllers — the ONLY thing that puts `gotk_*` in the TSDB.
|
||||
# kube-prometheus-stack does not scrape flux-system (the controllers are bootstrapped, expose no
|
||||
# Service, and no chart value covers them), so until this landed the five gotk_* rules in
|
||||
# alerts.yaml above were structurally inert: verified on prod 2026-09-13,
|
||||
# `count({__name__=~"gotk.*"})` returned nothing. A CR of monitoring.coreos.com -> this tier, same
|
||||
# as alerts.yaml. (2026-09-13, from the inert-alert audit.)
|
||||
- ../platform/monitoring/flux-podmonitor.yaml
|
||||
# kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
|
||||
# operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
|
||||
# installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain
|
||||
|
||||
Reference in new issue
Block a user