From 2d5171e064c8ff2c6ac9fedd73e2c73faf201672 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 17:17:06 -0500 Subject: [PATCH] =?UTF-8?q?feat(velero):=20scrape=20the=20backup=20control?= =?UTF-8?q?ler=20=E2=80=94=20three=20critical=20alerts=20could=20never=20f?= =?UTF-8?q?ire?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical), VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability wave. None of them could ever have fired. The velero chart defaults `metrics.serviceMonitor.enabled: false`, and prod confirms it live: `kubectl -n velero get servicemonitor` returns no resources, so velero_backup_failure_total and velero_backup_last_successful_timestamp were never scraped. A nightly backup could fail — or silently stop running — for weeks with nothing to say so. That is exactly what R-STATE-03 is for, which makes the scrape part of the backup path rather than an observability extra. Both flags verified against chart 12.1.0: `helm template` emits neither object without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and ServiceMonitor with it — so the chart gates on capability detection AND `enabled`, and helm-controller supplies real capabilities from the API server. No additionalLabels are needed: platform/monitoring already sets serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues to false, so Prometheus selects every monitor regardless of labels. Independent of the R2/bulk-tier fix in the previous commit; droppable on its own if the PM wants to sequence it separately. Co-Authored-By: Claude Opus 5 --- platform/velero/helmrelease.yaml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/platform/velero/helmrelease.yaml b/platform/velero/helmrelease.yaml index 975d0d4..27a794b 100644 --- a/platform/velero/helmrelease.yaml +++ b/platform/velero/helmrelease.yaml @@ -198,6 +198,26 @@ spec: includedNamespaces: - monitoring - unifi + # --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------ + # platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical), + # VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire, + # because the chart defaults `metrics.serviceMonitor.enabled: false` and no ServiceMonitor + # existed in ns velero (verified live on prod, 2026-09-12: `get servicemonitor -n velero` → + # no resources). The series velero_backup_failure_total / velero_backup_last_successful_timestamp + # were never scraped, so a nightly backup could fail — or stop running — indefinitely with no + # alert. That is the precise failure mode R-STATE-03 exists to prevent, so the scrape is part of + # the backup path, not an observability nice-to-have. + # The chart gates these on BOTH `enabled` and capability detection of monitoring.coreos.com/v1 + # (verified: `helm template` emits neither object without --api-versions, and emits both with + # it), so `enabled: true` is required and sufficient — helm-controller supplies real capabilities + # from the API server. No additionalLabels needed: platform/monitoring sets + # serviceMonitorSelectorNilUsesHelmValues / podMonitorSelectorNilUsesHelmValues false, so + # Prometheus selects every ServiceMonitor/PodMonitor regardless of labels. + metrics: + serviceMonitor: + enabled: true # velero server: backup/restore success+failure counters + nodeAgentPodMonitor: + enabled: true # node-agent: kopia FSB (PodVolumeBackup) progress + failures # Modest home-dev footprint (mirror the other platform components). resources: requests: