2 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	a1cc0a5556	fix(velero): skip emptyDir volumes by type; revert the narrower namespace exclusion ... Supersedes 05a8094, which fixed the right bug in the wrong layer. That commit excluded ns velero's volume data because Velero's own transient kopia maintenance Jobs (emptyDir `plugins` + `scratch`) were failing their PodVolumeBackups and turning the nightly Backup PartiallyFailed. It deferred the general fix — skip emptyDir by type — because Velero's `volumeTypes` policy condition was unproven here and an unparseable policies.yaml breaks EVERY backup. It said "revisit if the shape shows up elsewhere." It showed up elsewhere within the hour. A verification Backup came back PartiallyFailed with 4 errors, all in ns runbooks: the nginx Deployment's emptyDir `tmp` + `cache`, failing with error to expose PVB: error getting client pod runbooks-...: pods "..." not found because pushing a runbook edit rolled that Deployment while the backup was snapshotting it. Cause worth stating plainly: I triggered that rollout myself, so those four failures were self-inflicted rather than a standing defect — but the mechanism is not. A PVB exposes the volume through its pod, so ANY rollout landing in the backup window fails one, and one failed PVB makes the whole Backup PartiallyFailed. On a GitOps cluster that is routine. So the property is what matters, not the namespace: an emptyDir is created empty with its pod and destroyed with it, so backing one up cannot protect anything while attempting it can break the backup that protects everything else. The `volumeTypes` risk was retired by testing, not by hoping. The new policy went into a THROWAWAY ConfigMap first, leaving the live one untouched, and was proven on prod (velero 1.18.1) with a control: control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x tmp, 2x cache), Completed same namespace, WITH the policy -> 0 PodVolumeBackups, Backup Completed, 0 errors The control is the point — "zero PVBs" alone is equally consistent with nothing having been attempted, which is the kind of empty green I have shipped before. The ns velero exclusion is now reverted as redundant: `plugins` and `scratch` are emptyDir (verified on a live maintain pod), so the general rule covers them, and reverting keeps Velero's own resources in schedule (1) where they belong instead of moving them between schedules for no reason. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-19 00:22:21 -05:00
jakman85andClaude Opus 5	2e259a808f	fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB ... Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 17:18:26 -05:00