jakman85andClaude Opus 5
a1cc0a5556
Supersedes 05a8094, which fixed the right bug in the wrong layer.
That commit excluded ns velero's volume data because Velero's own transient kopia
maintenance Jobs (emptyDir `plugins` + `scratch`) were failing their
PodVolumeBackups and turning the nightly Backup PartiallyFailed. It deferred the
general fix — skip emptyDir by type — because Velero's `volumeTypes` policy
condition was unproven here and an unparseable policies.yaml breaks EVERY backup.
It said "revisit if the shape shows up elsewhere."
It showed up elsewhere within the hour. A verification Backup came back
PartiallyFailed with 4 errors, all in ns runbooks: the nginx Deployment's emptyDir
`tmp` + `cache`, failing with
error to expose PVB: error getting client pod runbooks-...: pods "..." not found
because pushing a runbook edit rolled that Deployment while the backup was
snapshotting it. Cause worth stating plainly: I triggered that rollout myself, so
those four failures were self-inflicted rather than a standing defect — but the
mechanism is not. A PVB exposes the volume through its pod, so ANY rollout landing
in the backup window fails one, and one failed PVB makes the whole Backup
PartiallyFailed. On a GitOps cluster that is routine.
So the property is what matters, not the namespace: an emptyDir is created empty
with its pod and destroyed with it, so backing one up cannot protect anything
while attempting it can break the backup that protects everything else.
The `volumeTypes` risk was retired by testing, not by hoping. The new policy went
into a THROWAWAY ConfigMap first, leaving the live one untouched, and was proven
on prod (velero 1.18.1) with a control:
control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x tmp, 2x cache), Completed
same namespace, WITH the policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
The control is the point — "zero PVBs" alone is equally consistent with nothing
having been attempted, which is the kind of empty green I have shipped before.
The ns velero exclusion is now reverted as redundant: `plugins` and `scratch` are
emptyDir (verified on a live maintain pod), so the general rule covers them, and
reverting keeps Velero's own resources in schedule (1) where they belong instead
of moving them between schedules for no reason.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
87 lines
5.0 KiB
YAML
87 lines
5.0 KiB
YAML
---
|
|
# Velero VOLUME POLICIES — the opt-out escape hatch for the external bulk tier (R-STATE-04).
|
|
#
|
|
# WHY THIS EXISTS. platform/velero/helmrelease.yaml schedule (1) runs
|
|
# `defaultVolumesToFsBackup: true` — the OPT-OUT model, so no volume with real data is ever
|
|
# forgotten. That model collides with the NFS bulk tier prod grew on 2026-09-11:
|
|
#
|
|
# plex-media 9Ti claim -> 192.168.2.102:/mnt/Storage/plex (8.32 TiB measured)
|
|
# sonarr-media 9Ti claim -> the SAME export
|
|
# radarr-media 9Ti claim -> the SAME export
|
|
# nextcloud-legacy-archive 270Gi -> 192.168.2.102:/mnt/Storage/nextcloud
|
|
#
|
|
# Left to opt-out, one nightly run reads that library THREE TIMES over NFS and pushes ~8.6 TiB at
|
|
# Cloudflare R2, every night, held for ${VELERO_RETENTION} (prod: 720h). kopia dedups the STORED
|
|
# bytes, not the reads. That is not DR; it is a self-inflicted outage plus a recurring bill.
|
|
#
|
|
# WHY NOT namespace exclusion. plex / sonarr / radarr / nextcloud each ALSO hold a real Longhorn
|
|
# PVC (plex-config 8Gi, sonarr-config 5Gi, radarr-config 5Gi, nextcloud-data 20Gi) whose loss is
|
|
# unrecoverable. Excluding those namespaces — the shape schedule (2) uses for monitoring+unifi —
|
|
# would silently drop exactly the data that matters. The opt-out has to be per VOLUME.
|
|
#
|
|
# WHY NOT the per-pod `backup.velero.io/backup-volumes-excludes` annotation. It works, but it fails
|
|
# OPEN: the next NFS PV someone mounts is backed up in full until a human remembers to annotate its
|
|
# pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
|
|
# its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
|
|
#
|
|
# RULE 2 — emptyDir (added 2026-09-19). Every emptyDir volume is skipped, because an emptyDir holds
|
|
# no durable state BY DEFINITION: created empty with its pod, destroyed with it. Backing one up
|
|
# cannot protect anything, and ATTEMPTING it actively breaks the backup.
|
|
#
|
|
# HOW IT BREAKS THINGS. A PodVolumeBackup exposes the volume through its pod; if that pod is replaced
|
|
# mid-flight the PVB fails with `error to expose PVB: error getting client pod <pod>: pods "<pod>"
|
|
# not found`, and ONE failed PVB makes the whole Backup PartiallyFailed. PartiallyFailed does not
|
|
# advance velero_backup_last_successful_timestamp, so VeleroNoRecentBackup fires CRITICAL over a
|
|
# backup that protected every real volume. Any Deployment rollout landing in the backup window is
|
|
# enough — on a GitOps cluster that is routine, not an edge case. An alert that cries wolf nightly is
|
|
# how a real backup failure gets ignored, and this one guards R-STATE-03.
|
|
#
|
|
# OBSERVED TWICE IN ONE HOUR in unrelated namespaces, which is what justified a general rule over a
|
|
# targeted exclusion:
|
|
# * ns velero — Velero's OWN transient kopia repository-maintenance Jobs
|
|
# (`<ns>-default-kopia-maintain-job-<ts>`), emptyDir `plugins` + `scratch`. Four
|
|
# failed PVBs had accumulated; velero-daily-20260919030041 was PartiallyFailed,
|
|
# 2 errors, 3079 of 3079 items backed up.
|
|
# * ns runbooks — the nginx Deployment's emptyDir `tmp` + `cache` (a read-only root filesystem needs
|
|
# writable scratch). Four failed PVBs when a runbook edit rolled the Deployment
|
|
# while the backup was running.
|
|
#
|
|
# PROVEN ON PROD, WITH A CONTROL (2026-09-19, velero 1.18.1). `volumeTypes` support was the open
|
|
# question, and a policies.yaml Velero cannot parse breaks EVERY backup — so it was tested against a
|
|
# THROWAWAY ConfigMap first, leaving this one untouched until it passed:
|
|
# control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x `tmp`, 2x `cache`), Completed
|
|
# same namespace, WITH this policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
|
|
# The control is the point: "zero PVBs" on its own is equally consistent with nothing being attempted.
|
|
#
|
|
|
|
# `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
|
|
# narrow it; deliberately not, so a second export is covered too). First matching rule wins.
|
|
#
|
|
# PROVEN, not assumed (prod, 2026-09-12, velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud
|
|
# with defaultVolumesToFsBackup:true and this policy produced exactly ONE PodVolumeBackup —
|
|
# `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume
|
|
# produced NO PodVolumeBackup at all.
|
|
#
|
|
# Referenced by name from schedule (1)'s template (`resourcePolicy: {kind: configmap, name: ...}`).
|
|
# Applied to BOTH environments: dev has no NFS PVs today, so the rule is a no-op there, and it is
|
|
# already correct if dev ever grows one.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: velero-volume-policies
|
|
namespace: velero
|
|
data:
|
|
policies.yaml: |
|
|
version: v1
|
|
volumePolicies:
|
|
# emptyDir first: cheapest condition, and the most common match.
|
|
- conditions:
|
|
volumeTypes:
|
|
- emptyDir
|
|
action:
|
|
type: skip
|
|
- conditions:
|
|
nfs: {}
|
|
action:
|
|
type: skip
|