fix(velero): skip emptyDir volumes by type; revert the narrower namespace exclusion

Supersedes 05a8094, which fixed the right bug in the wrong layer.

That commit excluded ns velero's volume data because Velero's own transient kopia
maintenance Jobs (emptyDir `plugins` + `scratch`) were failing their
PodVolumeBackups and turning the nightly Backup PartiallyFailed. It deferred the
general fix — skip emptyDir by type — because Velero's `volumeTypes` policy
condition was unproven here and an unparseable policies.yaml breaks EVERY backup.
It said "revisit if the shape shows up elsewhere."

It showed up elsewhere within the hour. A verification Backup came back
PartiallyFailed with 4 errors, all in ns runbooks: the nginx Deployment's emptyDir
`tmp` + `cache`, failing with

    error to expose PVB: error getting client pod runbooks-...: pods "..." not found

because pushing a runbook edit rolled that Deployment while the backup was
snapshotting it. Cause worth stating plainly: I triggered that rollout myself, so
those four failures were self-inflicted rather than a standing defect — but the
mechanism is not. A PVB exposes the volume through its pod, so ANY rollout landing
in the backup window fails one, and one failed PVB makes the whole Backup
PartiallyFailed. On a GitOps cluster that is routine.

So the property is what matters, not the namespace: an emptyDir is created empty
with its pod and destroyed with it, so backing one up cannot protect anything
while attempting it can break the backup that protects everything else.

The `volumeTypes` risk was retired by testing, not by hoping. The new policy went
into a THROWAWAY ConfigMap first, leaving the live one untouched, and was proven
on prod (velero 1.18.1) with a control:

    control, ns runbooks, NO policy  -> 4 PodVolumeBackups (2x tmp, 2x cache), Completed
    same namespace, WITH the policy  -> 0 PodVolumeBackups, Backup Completed, 0 errors

The control is the point — "zero PVBs" alone is equally consistent with nothing
having been attempted, which is the kind of empty green I have shipped before.

The ns velero exclusion is now reverted as redundant: `plugins` and `scratch` are
emptyDir (verified on a live maintain pod), so the general rule covers them, and
reverting keeps Velero's own resources in schedule (1) where they belong instead
of moving them between schedules for no reason.

Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-19 00:22:21 -05:00
1 parent 6eaaaae334
commit a1cc0a5556
2 files changed
+36 -24

No files matched your search

-24
View File
@@ -179,29 +179,6 @@ spec:
excludedNamespaces:
- monitoring # volume DATA excluded here; resources captured by schedule (2)
- unifi
# velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs
# transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository
# (`<ns>-default-kopia-maintain-job-<ts>`), each with `plugins` and `scratch` emptyDir volumes.
# Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job
# completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod:
# velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for
# `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such
# PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but
# PartiallyFailed does not advance velero_backup_last_successful_timestamp, so
# VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A
# backup alert that cries wolf nightly is how a real backup failure gets ignored.
#
# SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here
# to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by
# schedule (2) — which is why `velero` is added there too.
#
# NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping
# emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job
# in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1,
# and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one
# being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only
# mechanisms already proven in this repo. Revisit if the shape shows up elsewhere.
- velero
# (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
# PVCs are reconstructable and excluded from data backup per the sizing decision:
# - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
@@ -221,7 +198,6 @@ spec:
includedNamespaces:
- monitoring
- unifi
- velero # resources only (BSL/credentials/Backup CRs); see schedule (1)
# --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
# platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
# VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,