From d99d47494c256d2f375c07328a7e83e6d0844a66 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 18 Sep 2026 23:55:57 -0500 Subject: [PATCH] fix(velero): nightly backup PartiallyFailed on Velero's own maintenance Jobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VeleroNoRecentBackup has been firing CRITICAL on prod over a backup that was actually protecting everything. velero-daily-20260919030041: PartiallyFailed, 3079/3079 items backed up, exactly 2 errors — both PodVolumeBackups for the `plugins` and `scratch` volumes of homepage-default-kopia-maintain-job-1789786837269-qhsvh. Four such PVB failures had accumulated. Velero runs transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository, each with emptyDir `plugins` and `scratch` volumes. Schedule (1) runs defaultVolumesToFsBackup:true — the opt-out model — so it attempts fs-backup on those volumes too. The Job finishes and its pod disappears mid-backup, and the PodVolumeBackup fails. Nothing real was ever missed. But PartiallyFailed does not advance velero_backup_last_successful_timestamp, so the alert that exists to catch a genuine backup outage fires every night over Velero cleaning up after itself. An alert that cries wolf nightly is how a real failure gets ignored — and this one guards R-STATE-03. Fix uses the pattern already established for monitoring+unifi: exclude the namespace's volume DATA from schedule (1), keep its RESOURCES in schedule (2). Safe by inspection — ns velero holds no PVCs, so there is no volume data to lose, while the BSL, credentials and Backup CRs that DR actually needs are still captured. Deliberately narrower than the general rule. The general fix is a volume policy skipping emptyDir by type, which would also cover a transient Job in any other namespace, but it needs Velero's `volumeTypes` policy condition — unproven in this repo on 1.18.1, and a policies.yaml Velero cannot parse breaks EVERY backup. That is a much worse failure than the one being fixed, and all four observed failures were kopia-maintain pods in ns velero. Rationale recorded inline so the next person can widen it on purpose rather than by accident. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 --- platform/velero/helmrelease.yaml | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/platform/velero/helmrelease.yaml b/platform/velero/helmrelease.yaml index 27a794b..2817cd1 100644 --- a/platform/velero/helmrelease.yaml +++ b/platform/velero/helmrelease.yaml @@ -179,6 +179,29 @@ spec: excludedNamespaces: - monitoring # volume DATA excluded here; resources captured by schedule (2) - unifi + # velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs + # transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository + # (`-default-kopia-maintain-job-`), each with `plugins` and `scratch` emptyDir volumes. + # Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job + # completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod: + # velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for + # `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such + # PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but + # PartiallyFailed does not advance velero_backup_last_successful_timestamp, so + # VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A + # backup alert that cries wolf nightly is how a real backup failure gets ignored. + # + # SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here + # to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by + # schedule (2) — which is why `velero` is added there too. + # + # NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping + # emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job + # in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1, + # and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one + # being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only + # mechanisms already proven in this repo. Revisit if the shape shows up elsewhere. + - velero # (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their # PVCs are reconstructable and excluded from data backup per the sizing decision: # - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC @@ -198,6 +221,7 @@ spec: includedNamespaces: - monitoring - unifi + - velero # resources only (BSL/credentials/Backup CRs); see schedule (1) # --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------ # platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical), # VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,