diff --git a/platform/velero/helmrelease.yaml b/platform/velero/helmrelease.yaml index 27a794b..2817cd1 100644 --- a/platform/velero/helmrelease.yaml +++ b/platform/velero/helmrelease.yaml @@ -179,6 +179,29 @@ spec: excludedNamespaces: - monitoring # volume DATA excluded here; resources captured by schedule (2) - unifi + # velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs + # transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository + # (`-default-kopia-maintain-job-`), each with `plugins` and `scratch` emptyDir volumes. + # Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job + # completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod: + # velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for + # `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such + # PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but + # PartiallyFailed does not advance velero_backup_last_successful_timestamp, so + # VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A + # backup alert that cries wolf nightly is how a real backup failure gets ignored. + # + # SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here + # to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by + # schedule (2) — which is why `velero` is added there too. + # + # NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping + # emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job + # in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1, + # and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one + # being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only + # mechanisms already proven in this repo. Revisit if the shape shows up elsewhere. + - velero # (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their # PVCs are reconstructable and excluded from data backup per the sizing decision: # - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC @@ -198,6 +221,7 @@ spec: includedNamespaces: - monitoring - unifi + - velero # resources only (BSL/credentials/Backup CRs); see schedule (1) # --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------ # platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical), # VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,