fix(velero): nightly backup PartiallyFailed on Velero's own maintenance Jobs

VeleroNoRecentBackup has been firing CRITICAL on prod over a backup that was
actually protecting everything.

velero-daily-20260919030041: PartiallyFailed, 3079/3079 items backed up, exactly
2 errors — both PodVolumeBackups for the `plugins` and `scratch` volumes of
homepage-default-kopia-maintain-job-1789786837269-qhsvh. Four such PVB failures
had accumulated.

Velero runs transient kopia repository-maintenance Jobs in its own namespace, one
per BackupRepository, each with emptyDir `plugins` and `scratch` volumes. Schedule
(1) runs defaultVolumesToFsBackup:true — the opt-out model — so it attempts
fs-backup on those volumes too. The Job finishes and its pod disappears
mid-backup, and the PodVolumeBackup fails.

Nothing real was ever missed. But PartiallyFailed does not advance
velero_backup_last_successful_timestamp, so the alert that exists to catch a
genuine backup outage fires every night over Velero cleaning up after itself. An
alert that cries wolf nightly is how a real failure gets ignored — and this one
guards R-STATE-03.

Fix uses the pattern already established for monitoring+unifi: exclude the
namespace's volume DATA from schedule (1), keep its RESOURCES in schedule (2).
Safe by inspection — ns velero holds no PVCs, so there is no volume data to lose,
while the BSL, credentials and Backup CRs that DR actually needs are still
captured.

Deliberately narrower than the general rule. The general fix is a volume policy
skipping emptyDir by type, which would also cover a transient Job in any other
namespace, but it needs Velero's `volumeTypes` policy condition — unproven in this
repo on 1.18.1, and a policies.yaml Velero cannot parse breaks EVERY backup. That
is a much worse failure than the one being fixed, and all four observed failures
were kopia-maintain pods in ns velero. Rationale recorded inline so the next
person can widen it on purpose rather than by accident.

Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-18 23:55:57 -05:00
1 parent 6f231a4311
commit d99d47494c
1 file changed
+24
+24
View File
@@ -179,6 +179,29 @@ spec:
excludedNamespaces:
- monitoring # volume DATA excluded here; resources captured by schedule (2)
- unifi
# velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs
# transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository
# (`<ns>-default-kopia-maintain-job-<ts>`), each with `plugins` and `scratch` emptyDir volumes.
# Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job
# completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod:
# velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for
# `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such
# PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but
# PartiallyFailed does not advance velero_backup_last_successful_timestamp, so
# VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A
# backup alert that cries wolf nightly is how a real backup failure gets ignored.
#
# SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here
# to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by
# schedule (2) — which is why `velero` is added there too.
#
# NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping
# emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job
# in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1,
# and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one
# being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only
# mechanisms already proven in this repo. Revisit if the shape shows up elsewhere.
- velero
# (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
# PVCs are reconstructable and excluded from data backup per the sizing decision:
# - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
@@ -198,6 +221,7 @@ spec:
includedNamespaces:
- monitoring
- unifi
- velero # resources only (BSL/credentials/Backup CRs); see schedule (1)
# --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
# platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
# VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,