merge: fix/velero-partialfail-on-own-maintenance-jobs
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

This commit is contained in:
jakman85 committed 2026-09-18 23:55:57 -05:00
commit 05a8094785
1 file changed
+24
+24
View File
@@ -179,6 +179,29 @@ spec:
excludedNamespaces:
- monitoring # volume DATA excluded here; resources captured by schedule (2)
- unifi
# velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs
# transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository
# (`<ns>-default-kopia-maintain-job-<ts>`), each with `plugins` and `scratch` emptyDir volumes.
# Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job
# completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod:
# velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for
# `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such
# PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but
# PartiallyFailed does not advance velero_backup_last_successful_timestamp, so
# VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A
# backup alert that cries wolf nightly is how a real backup failure gets ignored.
#
# SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here
# to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by
# schedule (2) — which is why `velero` is added there too.
#
# NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping
# emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job
# in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1,
# and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one
# being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only
# mechanisms already proven in this repo. Revisit if the shape shows up elsewhere.
- velero
# (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
# PVCs are reconstructable and excluded from data backup per the sizing decision:
# - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
@@ -198,6 +221,7 @@ spec:
includedNamespaces:
- monitoring
- unifi
- velero # resources only (BSL/credentials/Backup CRs); see schedule (1)
# --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
# platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
# VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,