merge: fix/velero-partialfail-on-own-maintenance-jobs
This commit is contained in:
commit
05a8094785
1 file changed
+24
@@ -179,6 +179,29 @@ spec:
|
||||
excludedNamespaces:
|
||||
- monitoring # volume DATA excluded here; resources captured by schedule (2)
|
||||
- unifi
|
||||
# velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs
|
||||
# transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository
|
||||
# (`<ns>-default-kopia-maintain-job-<ts>`), each with `plugins` and `scratch` emptyDir volumes.
|
||||
# Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job
|
||||
# completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod:
|
||||
# velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for
|
||||
# `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such
|
||||
# PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but
|
||||
# PartiallyFailed does not advance velero_backup_last_successful_timestamp, so
|
||||
# VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A
|
||||
# backup alert that cries wolf nightly is how a real backup failure gets ignored.
|
||||
#
|
||||
# SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here
|
||||
# to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by
|
||||
# schedule (2) — which is why `velero` is added there too.
|
||||
#
|
||||
# NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping
|
||||
# emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job
|
||||
# in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1,
|
||||
# and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one
|
||||
# being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only
|
||||
# mechanisms already proven in this repo. Revisit if the shape shows up elsewhere.
|
||||
- velero
|
||||
# (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
|
||||
# PVCs are reconstructable and excluded from data backup per the sizing decision:
|
||||
# - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
|
||||
@@ -198,6 +221,7 @@ spec:
|
||||
includedNamespaces:
|
||||
- monitoring
|
||||
- unifi
|
||||
- velero # resources only (BSL/credentials/Backup CRs); see schedule (1)
|
||||
# --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
|
||||
# platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
|
||||
# VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,
|
||||
|
||||
Reference in new issue
Block a user