feat(velero): scrape the backup controller — three critical alerts could never fire

platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical),
VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability
wave. None of them could ever have fired. The velero chart defaults
`metrics.serviceMonitor.enabled: false`, and prod confirms it live:
`kubectl -n velero get servicemonitor` returns no resources, so
velero_backup_failure_total and velero_backup_last_successful_timestamp were
never scraped. A nightly backup could fail — or silently stop running — for
weeks with nothing to say so. That is exactly what R-STATE-03 is for, which
makes the scrape part of the backup path rather than an observability extra.

Both flags verified against chart 12.1.0: `helm template` emits neither object
without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and
ServiceMonitor with it — so the chart gates on capability detection AND
`enabled`, and helm-controller supplies real capabilities from the API server.
No additionalLabels are needed: platform/monitoring already sets
serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues
to false, so Prometheus selects every monitor regardless of labels.

Independent of the R2/bulk-tier fix in the previous commit; droppable on its own
if the PM wants to sequence it separately.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 authored and x committed 2026-09-12 17:18:26 -05:00
1 parent 2e259a808f
commit 2d5171e064
1 file changed
+20
+20
View File
@@ -198,6 +198,26 @@ spec:
includedNamespaces:
- monitoring
- unifi
# --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
# platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
# VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,
# because the chart defaults `metrics.serviceMonitor.enabled: false` and no ServiceMonitor
# existed in ns velero (verified live on prod, 2026-09-12: `get servicemonitor -n velero` →
# no resources). The series velero_backup_failure_total / velero_backup_last_successful_timestamp
# were never scraped, so a nightly backup could fail — or stop running — indefinitely with no
# alert. That is the precise failure mode R-STATE-03 exists to prevent, so the scrape is part of
# the backup path, not an observability nice-to-have.
# The chart gates these on BOTH `enabled` and capability detection of monitoring.coreos.com/v1
# (verified: `helm template` emits neither object without --api-versions, and emits both with
# it), so `enabled: true` is required and sufficient — helm-controller supplies real capabilities
# from the API server. No additionalLabels needed: platform/monitoring sets
# serviceMonitorSelectorNilUsesHelmValues / podMonitorSelectorNilUsesHelmValues false, so
# Prometheus selects every ServiceMonitor/PodMonitor regardless of labels.
metrics:
serviceMonitor:
enabled: true # velero server: backup/restore success+failure counters
nodeAgentPodMonitor:
enabled: true # node-agent: kopia FSB (PodVolumeBackup) progress + failures
# Modest home-dev footprint (mirror the other platform components).
resources:
requests: