merge: fix/velero-skip-emptydir-volumes

platform/velero/helmrelease.yaml

@@ -179,29 +179,6 @@ spec:
           excludedNamespaces:
             - monitoring        # volume DATA excluded here; resources captured by schedule (2)
             - unifi
-            # velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs
-            # transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository
-            # (`<ns>-default-kopia-maintain-job-<ts>`), each with `plugins` and `scratch` emptyDir volumes.
-            # Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job
-            # completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod:
-            # velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for
-            # `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such
-            # PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but
-            # PartiallyFailed does not advance velero_backup_last_successful_timestamp, so
-            # VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A
-            # backup alert that cries wolf nightly is how a real backup failure gets ignored.
-            #
-            # SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here
-            # to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by
-            # schedule (2) — which is why `velero` is added there too.
-            #
-            # NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping
-            # emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job
-            # in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1,
-            # and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one
-            # being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only
-            # mechanisms already proven in this repo. Revisit if the shape shows up elsewhere.
-            - velero
       # (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
       #     PVCs are reconstructable and excluded from data backup per the sizing decision:
       #       - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
@@ -221,7 +198,6 @@ spec:
           includedNamespaces:
             - monitoring
             - unifi
-            - velero              # resources only (BSL/credentials/Backup CRs); see schedule (1)
     # --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
     # platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
     # VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,

platform/velero/volume-policies.yaml

@@ -24,6 +24,36 @@
 # pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
 # its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
 #
+# RULE 2 — emptyDir (added 2026-09-19). Every emptyDir volume is skipped, because an emptyDir holds
+# no durable state BY DEFINITION: created empty with its pod, destroyed with it. Backing one up
+# cannot protect anything, and ATTEMPTING it actively breaks the backup.
+#
+# HOW IT BREAKS THINGS. A PodVolumeBackup exposes the volume through its pod; if that pod is replaced
+# mid-flight the PVB fails with `error to expose PVB: error getting client pod <pod>: pods "<pod>"
+# not found`, and ONE failed PVB makes the whole Backup PartiallyFailed. PartiallyFailed does not
+# advance velero_backup_last_successful_timestamp, so VeleroNoRecentBackup fires CRITICAL over a
+# backup that protected every real volume. Any Deployment rollout landing in the backup window is
+# enough — on a GitOps cluster that is routine, not an edge case. An alert that cries wolf nightly is
+# how a real backup failure gets ignored, and this one guards R-STATE-03.
+#
+# OBSERVED TWICE IN ONE HOUR in unrelated namespaces, which is what justified a general rule over a
+# targeted exclusion:
+#   * ns velero   — Velero's OWN transient kopia repository-maintenance Jobs
+#                   (`<ns>-default-kopia-maintain-job-<ts>`), emptyDir `plugins` + `scratch`. Four
+#                   failed PVBs had accumulated; velero-daily-20260919030041 was PartiallyFailed,
+#                   2 errors, 3079 of 3079 items backed up.
+#   * ns runbooks — the nginx Deployment's emptyDir `tmp` + `cache` (a read-only root filesystem needs
+#                   writable scratch). Four failed PVBs when a runbook edit rolled the Deployment
+#                   while the backup was running.
+#
+# PROVEN ON PROD, WITH A CONTROL (2026-09-19, velero 1.18.1). `volumeTypes` support was the open
+# question, and a policies.yaml Velero cannot parse breaks EVERY backup — so it was tested against a
+# THROWAWAY ConfigMap first, leaving this one untouched until it passed:
+#     control, ns runbooks, NO policy  -> 4 PodVolumeBackups (2x `tmp`, 2x `cache`), Completed
+#     same namespace, WITH this policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
+# The control is the point: "zero PVBs" on its own is equally consistent with nothing being attempted.
+#
+
 # `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
 # narrow it; deliberately not, so a second export is covered too). First matching rule wins.
 #
@@ -44,6 +74,12 @@ data:
   policies.yaml: |
     version: v1
     volumePolicies:
+      # emptyDir first: cheapest condition, and the most common match.
+      - conditions:
+          volumeTypes:
+            - emptyDir
+        action:
+          type: skip
       - conditions:
           nfs: {}
         action: