fix(velero): skip emptyDir volumes by type; revert the narrower namespace exclusion

Supersedes 05a8094, which fixed the right bug in the wrong layer.

That commit excluded ns velero's volume data because Velero's own transient kopia
maintenance Jobs (emptyDir `plugins` + `scratch`) were failing their
PodVolumeBackups and turning the nightly Backup PartiallyFailed. It deferred the
general fix — skip emptyDir by type — because Velero's `volumeTypes` policy
condition was unproven here and an unparseable policies.yaml breaks EVERY backup.
It said "revisit if the shape shows up elsewhere."

It showed up elsewhere within the hour. A verification Backup came back
PartiallyFailed with 4 errors, all in ns runbooks: the nginx Deployment's emptyDir
`tmp` + `cache`, failing with

    error to expose PVB: error getting client pod runbooks-...: pods "..." not found

because pushing a runbook edit rolled that Deployment while the backup was
snapshotting it. Cause worth stating plainly: I triggered that rollout myself, so
those four failures were self-inflicted rather than a standing defect — but the
mechanism is not. A PVB exposes the volume through its pod, so ANY rollout landing
in the backup window fails one, and one failed PVB makes the whole Backup
PartiallyFailed. On a GitOps cluster that is routine.

So the property is what matters, not the namespace: an emptyDir is created empty
with its pod and destroyed with it, so backing one up cannot protect anything
while attempting it can break the backup that protects everything else.

The `volumeTypes` risk was retired by testing, not by hoping. The new policy went
into a THROWAWAY ConfigMap first, leaving the live one untouched, and was proven
on prod (velero 1.18.1) with a control:

    control, ns runbooks, NO policy  -> 4 PodVolumeBackups (2x tmp, 2x cache), Completed
    same namespace, WITH the policy  -> 0 PodVolumeBackups, Backup Completed, 0 errors

The control is the point — "zero PVBs" alone is equally consistent with nothing
having been attempted, which is the kind of empty green I have shipped before.

The ns velero exclusion is now reverted as redundant: `plugins` and `scratch` are
emptyDir (verified on a live maintain pod), so the general rule covers them, and
reverting keeps Velero's own resources in schedule (1) where they belong instead
of moving them between schedules for no reason.

Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/velero/helmrelease.yaml

@@ -179,29 +179,6 @@ spec:
           excludedNamespaces:
             - monitoring        # volume DATA excluded here; resources captured by schedule (2)
             - unifi
-            # velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs
-            # transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository
-            # (`<ns>-default-kopia-maintain-job-<ts>`), each with `plugins` and `scratch` emptyDir volumes.
-            # Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job
-            # completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod:
-            # velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for
-            # `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such
-            # PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but
-            # PartiallyFailed does not advance velero_backup_last_successful_timestamp, so
-            # VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A
-            # backup alert that cries wolf nightly is how a real backup failure gets ignored.
-            #
-            # SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here
-            # to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by
-            # schedule (2) — which is why `velero` is added there too.
-            #
-            # NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping
-            # emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job
-            # in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1,
-            # and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one
-            # being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only
-            # mechanisms already proven in this repo. Revisit if the shape shows up elsewhere.
-            - velero
       # (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
       #     PVCs are reconstructable and excluded from data backup per the sizing decision:
       #       - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
@@ -221,7 +198,6 @@ spec:
           includedNamespaces:
             - monitoring
             - unifi
-            - velero              # resources only (BSL/credentials/Backup CRs); see schedule (1)
     # --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
     # platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
     # VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,

platform/velero/volume-policies.yaml

@@ -24,6 +24,36 @@
 # pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
 # its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
 #
+# RULE 2 — emptyDir (added 2026-09-19). Every emptyDir volume is skipped, because an emptyDir holds
+# no durable state BY DEFINITION: created empty with its pod, destroyed with it. Backing one up
+# cannot protect anything, and ATTEMPTING it actively breaks the backup.
+#
+# HOW IT BREAKS THINGS. A PodVolumeBackup exposes the volume through its pod; if that pod is replaced
+# mid-flight the PVB fails with `error to expose PVB: error getting client pod <pod>: pods "<pod>"
+# not found`, and ONE failed PVB makes the whole Backup PartiallyFailed. PartiallyFailed does not
+# advance velero_backup_last_successful_timestamp, so VeleroNoRecentBackup fires CRITICAL over a
+# backup that protected every real volume. Any Deployment rollout landing in the backup window is
+# enough — on a GitOps cluster that is routine, not an edge case. An alert that cries wolf nightly is
+# how a real backup failure gets ignored, and this one guards R-STATE-03.
+#
+# OBSERVED TWICE IN ONE HOUR in unrelated namespaces, which is what justified a general rule over a
+# targeted exclusion:
+#   * ns velero   — Velero's OWN transient kopia repository-maintenance Jobs
+#                   (`<ns>-default-kopia-maintain-job-<ts>`), emptyDir `plugins` + `scratch`. Four
+#                   failed PVBs had accumulated; velero-daily-20260919030041 was PartiallyFailed,
+#                   2 errors, 3079 of 3079 items backed up.
+#   * ns runbooks — the nginx Deployment's emptyDir `tmp` + `cache` (a read-only root filesystem needs
+#                   writable scratch). Four failed PVBs when a runbook edit rolled the Deployment
+#                   while the backup was running.
+#
+# PROVEN ON PROD, WITH A CONTROL (2026-09-19, velero 1.18.1). `volumeTypes` support was the open
+# question, and a policies.yaml Velero cannot parse breaks EVERY backup — so it was tested against a
+# THROWAWAY ConfigMap first, leaving this one untouched until it passed:
+#     control, ns runbooks, NO policy  -> 4 PodVolumeBackups (2x `tmp`, 2x `cache`), Completed
+#     same namespace, WITH this policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
+# The control is the point: "zero PVBs" on its own is equally consistent with nothing being attempted.
+#
+
 # `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
 # narrow it; deliberately not, so a second export is covered too). First matching rule wins.
 #
@@ -44,6 +74,12 @@ data:
   policies.yaml: |
     version: v1
     volumePolicies:
+      # emptyDir first: cheapest condition, and the most common match.
+      - conditions:
+          volumeTypes:
+            - emptyDir
+        action:
+          type: skip
       - conditions:
           nfs: {}
         action: