jakman85andClaude Opus 4.8
178d130dc3
Supersedes the previous kubectl-image repoint. The velero chart's pre-install `velero-upgrade-crds` HOOK job runs docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami removed (Aug 2025); `bitnami/kubectl:1.31` 404s, leaving the hook in Init:ImagePullBackOff so Velero never installs. Repointing the hook to the exact-version rancher/kubectl:v1.31.5 (prior commit) does not work: that image is distroless, and the hook's init runs `/bin/sh -c "cp $(which sh)/... $(which kubectl) /tmp"` — with no /bin/sh it crashes StartError (verified on-cluster). The hook exists ONLY to UPDATE CRDs on chart upgrades — it is NOT how CRDs are first installed. The chart ships all 13 Velero CRDs in its native `crds/` directory, which Helm applies unconditionally on install (before hooks), independent of this flag; the CRDs are already present on-cluster from Helm's crds/ handling. Setting `upgradeCRDs: false` drops the fragile external kubectl-image dependency entirely while CRDs still install. Trade-off: Helm won't manage crds/ after install, so CRD schema upgrades become a manual/Renovate step; acceptable given Velero's CRDs are stable. All other Velero values (BSL -> versitygw s3://velero, kopia FSB node-agent, the two Schedules, the velero-cloud-credentials secret) are unchanged. Co-Authored-By: Claude Opus 4.8 <[email protected]>
platform/velero
Velero — off-cluster backup and DR of Kubernetes resource state + PVC data.
- Governing ADR: ADR-0003 — Longhorn (+ bulk tier + Velero)
- Spec: docs/04 — Storage & State
- Serves: R-STATE-03
- Owner: WS-STORAGE
What this is (LIVE)
Flux HelmRelease (Velero chart 8.x / Velero 1.15.x) wired to the on-Proxmox MinIO S3 target:
- Endpoint:
http://192.168.1.9:9000(plain HTTP, path-style, regionus-east-1placeholder) - BucketStorageLocation
default: provideraws, bucketvelero,s3ForcePathStyle=true. - AWS plugin (
velero/velero-plugin-for-aws) loaded as an init container. - File System Backup (kopia):
deployNodeAgent=true,uploaderType=kopia,snapshotsEnabled=false— storage-agnostic PVC backup, no CSI snapshot / data-mover. - Credentials:
velero-cloud-credentials— a SOPS Secret (velero-cloud.sops.yaml) holding thecloudINI key, the scoped MinIO key limited to thevelero+cnpgbuckets. Kept in sync with the staging source of truthplatform/backup/minio-credentials.sops.yamland the Proxmox host ansible-vault.
Reconciled by its own non-gating Flux Kustomization clusters/base/backup.yaml
(dependsOn: platform-config, wait:false) — a backup controller must never gate the platform graph.
Backup scope (what's included vs excluded)
Two daily Schedules (cadence ${VELERO_BACKUP_SCHEDULE}, retention ${VELERO_RETENTION}):
| Schedule | Namespaces | Volumes (kopia FSB) |
|---|---|---|
daily |
ALL except monitoring, unifi |
all PVC data (defaultVolumesToFsBackup:true, opt-out) |
daily-resources-only |
monitoring, unifi |
none (defaultVolumesToFsBackup:false) — resources only |
Excluded volume data (reconstructable, per the sizing decision):
monitoring— Prometheus TSDB PVC (ephemeral) + Loki 72h-log PVC. Grafana/Alertmanager areemptyDir(no PVC); Grafana dashboards are GitOps-provisioned.unifi— the ~7.4 GiB KubeVirt VM disk (huge; UniFi is protected by its own config export).
Their Kubernetes resources are still captured (by daily-resources-only) to aid DR.
Included (the irreplaceable ~6 GiB): every other namespace's PVC data via kopia — Nextcloud data,
Gitea, Plex/*arr config, PiHole, and the CNPG Postgres PVCs. (CNPG DBs additionally have native Barman
PITR to the cnpg bucket — see the DB manifests.)
Post-deploy (PM)
velero backup-location get→defaultAvailable.velero backup create --from-schedule daily→ objects land unders3://velero/.- DR drill (R-STATE-03, Phase-4 gate):
velero restore create --from-backup <name>into a scratch namespace and verify PVC data restores via the node-agent.