jakman85andClaude Opus 5
a1cc0a5556
Supersedes 05a8094, which fixed the right bug in the wrong layer.
That commit excluded ns velero's volume data because Velero's own transient kopia
maintenance Jobs (emptyDir `plugins` + `scratch`) were failing their
PodVolumeBackups and turning the nightly Backup PartiallyFailed. It deferred the
general fix — skip emptyDir by type — because Velero's `volumeTypes` policy
condition was unproven here and an unparseable policies.yaml breaks EVERY backup.
It said "revisit if the shape shows up elsewhere."
It showed up elsewhere within the hour. A verification Backup came back
PartiallyFailed with 4 errors, all in ns runbooks: the nginx Deployment's emptyDir
`tmp` + `cache`, failing with
error to expose PVB: error getting client pod runbooks-...: pods "..." not found
because pushing a runbook edit rolled that Deployment while the backup was
snapshotting it. Cause worth stating plainly: I triggered that rollout myself, so
those four failures were self-inflicted rather than a standing defect — but the
mechanism is not. A PVB exposes the volume through its pod, so ANY rollout landing
in the backup window fails one, and one failed PVB makes the whole Backup
PartiallyFailed. On a GitOps cluster that is routine.
So the property is what matters, not the namespace: an emptyDir is created empty
with its pod and destroyed with it, so backing one up cannot protect anything
while attempting it can break the backup that protects everything else.
The `volumeTypes` risk was retired by testing, not by hoping. The new policy went
into a THROWAWAY ConfigMap first, leaving the live one untouched, and was proven
on prod (velero 1.18.1) with a control:
control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x tmp, 2x cache), Completed
same namespace, WITH the policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
The control is the point — "zero PVBs" alone is equally consistent with nothing
having been attempted, which is the kind of empty green I have shipped before.
The ns velero exclusion is now reverted as redundant: `plugins` and `scratch` are
emptyDir (verified on a live maintain pod), so the general rule covers them, and
reverting keeps Velero's own resources in schedule (1) where they belong instead
of moving them between schedules for no reason.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/velero
Velero — off-cluster backup and DR of Kubernetes resource state + PVC data.
- Governing ADR: ADR-0003 — Longhorn (+ bulk tier + Velero)
- Spec: docs/04 — Storage & State
- Serves: R-STATE-03
- Owner: WS-STORAGE
What this is (LIVE)
Flux HelmRelease (Velero chart 12.x / Velero 1.18.x) wired to the on-Proxmox MinIO S3 target:
- Endpoint:
http://192.168.1.9:9000(plain HTTP, path-style, regionus-east-1placeholder) - BucketStorageLocation
default: provideraws, bucketvelero,s3ForcePathStyle=true. - AWS plugin (
velero/velero-plugin-for-aws) loaded as an init container. - File System Backup (kopia):
deployNodeAgent=true,uploaderType=kopia,snapshotsEnabled=false— storage-agnostic PVC backup, no CSI snapshot / data-mover. - Credentials:
velero-cloud-credentials— a SOPS Secret (velero-cloud.sops.yaml) holding thecloudINI key, the scoped MinIO key limited to thevelero+cnpgbuckets. Kept in sync with the staging source of truthplatform/backup/minio-credentials.sops.yamland the Proxmox host ansible-vault.
Reconciled by its own non-gating Flux Kustomization clusters/base/backup.yaml
(dependsOn: platform-config, wait:false) — a backup controller must never gate the platform graph.
Backup scope (what's included vs excluded)
Two daily Schedules (cadence ${VELERO_BACKUP_SCHEDULE}, retention ${VELERO_RETENTION}):
| Schedule | Namespaces | Volumes (kopia FSB) |
|---|---|---|
daily |
ALL except monitoring, unifi |
all PVC data (defaultVolumesToFsBackup:true, opt-out) |
daily-resources-only |
monitoring, unifi |
none (defaultVolumesToFsBackup:false) — resources only |
Excluded volume data (reconstructable, per the sizing decision):
monitoring— Prometheus TSDB PVC (ephemeral) + Loki 72h-log PVC. Grafana/Alertmanager areemptyDir(no PVC); Grafana dashboards are GitOps-provisioned.unifi— the ~7.4 GiB KubeVirt VM disk (huge; UniFi is protected by its own config export).
Their Kubernetes resources are still captured (by daily-resources-only) to aid DR.
Included (the irreplaceable ~6 GiB): every other namespace's PVC data via kopia — Nextcloud data,
Gitea, Plex/*arr config, PiHole, and the CNPG Postgres PVCs. (CNPG DBs additionally have native Barman
PITR to the cnpg bucket — see the DB manifests.)
Post-deploy (PM)
velero backup-location get→defaultAvailable.velero backup create --from-schedule daily→ objects land unders3://velero/.- DR drill (R-STATE-03, Phase-4 gate):
velero restore create --from-backup <name>into a scratch namespace and verify PVC data restores via the node-agent.