feat(velero): scrape the backup controller — three critical alerts could never fire
platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical), VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability wave. None of them could ever have fired. The velero chart defaults `metrics.serviceMonitor.enabled: false`, and prod confirms it live: `kubectl -n velero get servicemonitor` returns no resources, so velero_backup_failure_total and velero_backup_last_successful_timestamp were never scraped. A nightly backup could fail — or silently stop running — for weeks with nothing to say so. That is exactly what R-STATE-03 is for, which makes the scrape part of the backup path rather than an observability extra. Both flags verified against chart 12.1.0: `helm template` emits neither object without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and ServiceMonitor with it — so the chart gates on capability detection AND `enabled`, and helm-controller supplies real capabilities from the API server. No additionalLabels are needed: platform/monitoring already sets serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues to false, so Prometheus selects every monitor regardless of labels. Independent of the R2/bulk-tier fix in the previous commit; droppable on its own if the PM wants to sequence it separately. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
2e259a808f
commit
2d5171e064
1 file changed
+20
@@ -198,6 +198,26 @@ spec:
|
||||
includedNamespaces:
|
||||
- monitoring
|
||||
- unifi
|
||||
# --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
|
||||
# platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
|
||||
# VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,
|
||||
# because the chart defaults `metrics.serviceMonitor.enabled: false` and no ServiceMonitor
|
||||
# existed in ns velero (verified live on prod, 2026-09-12: `get servicemonitor -n velero` →
|
||||
# no resources). The series velero_backup_failure_total / velero_backup_last_successful_timestamp
|
||||
# were never scraped, so a nightly backup could fail — or stop running — indefinitely with no
|
||||
# alert. That is the precise failure mode R-STATE-03 exists to prevent, so the scrape is part of
|
||||
# the backup path, not an observability nice-to-have.
|
||||
# The chart gates these on BOTH `enabled` and capability detection of monitoring.coreos.com/v1
|
||||
# (verified: `helm template` emits neither object without --api-versions, and emits both with
|
||||
# it), so `enabled: true` is required and sufficient — helm-controller supplies real capabilities
|
||||
# from the API server. No additionalLabels needed: platform/monitoring sets
|
||||
# serviceMonitorSelectorNilUsesHelmValues / podMonitorSelectorNilUsesHelmValues false, so
|
||||
# Prometheus selects every ServiceMonitor/PodMonitor regardless of labels.
|
||||
metrics:
|
||||
serviceMonitor:
|
||||
enabled: true # velero server: backup/restore success+failure counters
|
||||
nodeAgentPodMonitor:
|
||||
enabled: true # node-agent: kopia FSB (PodVolumeBackup) progress + failures
|
||||
# Modest home-dev footprint (mirror the other platform components).
|
||||
resources:
|
||||
requests:
|
||||
|
||||
Reference in new issue
Block a user