feat(backup): wire Velero + CNPG Barman to the live MinIO S3 target
Author the GitOps side of off-cluster backups against the on-Proxmox MinIO (http://192.168.1.9:9000, path-style; buckets velero + cnpg). Velero (R-STATE-03): - HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container + node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia, snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle. - Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only` captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup). - velero-cloud-credentials SOPS Secret (cloud INI key, ns velero). - Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config, wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp. CNPG native Barman PITR (R-IDP-05 / R-STATE-03): - keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily ScheduledBackup (6-field cron, immediate) each. - Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only. - render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a DB-wave sidecar (+ self-test); wave-0 kustomization regenerated. Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup), SOPS round-trip verified with the cluster age key. gate: green. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
1bee2554f8
commit
d1248e6778
17 files changed
+446
-49
No files matched your search
+41
-11
@@ -1,22 +1,52 @@
|
||||
# platform/velero
|
||||
|
||||
**Velero** — off-cluster backup and DR of volumes + Kubernetes resource state.
|
||||
**Velero** — off-cluster backup and DR of Kubernetes resource state + PVC data.
|
||||
|
||||
- **Governing ADR:** [ADR-0003 — Longhorn (+ bulk tier + Velero)](../../../home-network-k3s/decisions/adr-0003-storage-longhorn.md)
|
||||
- **Spec:** [docs/04 — Storage & State](../../../home-network-k3s/docs/04-storage-and-state.md)
|
||||
- **Serves:** R-STATE-03
|
||||
- **Owner:** WS-STORAGE
|
||||
|
||||
## What this stub is
|
||||
## What this is (LIVE)
|
||||
|
||||
Flux `HelmRelease` with a placeholder off-cluster backup target (S3-compatible/NFS) and scheduled
|
||||
backups whose cadence/retention are overlay knobs (`${VELERO_*}`, R-ENV-02). Backup credentials are
|
||||
referenced **by name** (`velero-backup-credentials`), **SOPS-encrypted** (WS-SEC). Works with Longhorn
|
||||
volume backups.
|
||||
Flux `HelmRelease` (Velero chart 8.x / Velero 1.15.x) wired to the on-Proxmox **MinIO** S3 target:
|
||||
|
||||
## TODO (later phases)
|
||||
- **Endpoint:** `http://192.168.1.9:9000` (plain HTTP, **path-style**, region `us-east-1` placeholder)
|
||||
- **BucketStorageLocation `default`:** provider `aws`, bucket `velero`, `s3ForcePathStyle=true`.
|
||||
- **AWS plugin** (`velero/velero-plugin-for-aws`) loaded as an init container.
|
||||
- **File System Backup (kopia):** `deployNodeAgent=true`, `uploaderType=kopia`, `snapshotsEnabled=false`
|
||||
— storage-agnostic PVC backup, **no** CSI snapshot / data-mover.
|
||||
- **Credentials:** `velero-cloud-credentials` — a SOPS Secret (`velero-cloud.sops.yaml`) holding the
|
||||
`cloud` INI key, the **scoped** MinIO key limited to the `velero`+`cnpg` buckets. Kept in sync with
|
||||
the staging source of truth `platform/backup/minio-credentials.sops.yaml` and the Proxmox host
|
||||
ansible-vault.
|
||||
|
||||
- **OQ-2 (human):** concrete off-cluster target (endpoint, bucket, resilience) is a prerequisite.
|
||||
- **Phase 4 (WS-STORAGE):** wire the target + Longhorn backups; prove a **DR drill** — restore a
|
||||
namespace + volumes from off-cluster backup (R-STATE-03, Phase-4 exit gate).
|
||||
- **DR-critical services** (Gitea, Keycloak, Nextcloud) get off-cluster backup + mirror per docs/05.
|
||||
Reconciled by its **own non-gating** Flux Kustomization `clusters/base/backup.yaml`
|
||||
(`dependsOn: platform-config`, `wait:false`) — a backup controller must never gate the platform graph.
|
||||
|
||||
## Backup scope (what's included vs excluded)
|
||||
|
||||
Two daily `Schedule`s (cadence `${VELERO_BACKUP_SCHEDULE}`, retention `${VELERO_RETENTION}`):
|
||||
|
||||
| Schedule | Namespaces | Volumes (kopia FSB) |
|
||||
|---|---|---|
|
||||
| `daily` | ALL **except** `monitoring`, `unifi` | **all** PVC data (`defaultVolumesToFsBackup:true`, opt-out) |
|
||||
| `daily-resources-only` | `monitoring`, `unifi` | **none** (`defaultVolumesToFsBackup:false`) — resources only |
|
||||
|
||||
**Excluded volume data (reconstructable, per the sizing decision):**
|
||||
- `monitoring` — Prometheus TSDB PVC (ephemeral) + Loki 72h-log PVC. Grafana/Alertmanager are
|
||||
`emptyDir` (no PVC); Grafana dashboards are GitOps-provisioned.
|
||||
- `unifi` — the ~7.4 GiB KubeVirt VM disk (huge; UniFi is protected by its own config export).
|
||||
|
||||
Their **Kubernetes resources** are still captured (by `daily-resources-only`) to aid DR.
|
||||
|
||||
**Included (the irreplaceable ~6 GiB):** every other namespace's PVC data via kopia — Nextcloud data,
|
||||
Gitea, Plex/*arr config, PiHole, and the CNPG Postgres PVCs. (CNPG DBs additionally have native Barman
|
||||
PITR to the `cnpg` bucket — see the DB manifests.)
|
||||
|
||||
## Post-deploy (PM)
|
||||
|
||||
- `velero backup-location get` → `default` Available.
|
||||
- `velero backup create --from-schedule daily` → objects land under `s3://velero/`.
|
||||
- DR drill (R-STATE-03, Phase-4 gate): `velero restore create --from-backup <name>` into a scratch
|
||||
namespace and verify PVC data restores via the node-agent.
|
||||
Reference in new issue
Block a user