# platform/velero

**Velero** — off-cluster backup and DR of Kubernetes resource state + PVC data.

- **Governing ADR:** [ADR-0003 — Longhorn (+ bulk tier + Velero)](../../../home-network-k3s/decisions/adr-0003-storage-longhorn.md)
- **Spec:** [docs/04 — Storage & State](../../../home-network-k3s/docs/04-storage-and-state.md)
- **Serves:** R-STATE-03
- **Owner:** WS-STORAGE

## What this is (LIVE)

Flux `HelmRelease` (Velero chart 12.x / Velero 1.18.x) wired to whatever `${S3_BACKUP_ENDPOINT}` names.
The target is a **cluster-vars knob, not a constant**:

- **Endpoint:** `${S3_BACKUP_ENDPOINT}` — dev has been running against the on-Proxmox **MinIO**
  `http://192.168.1.9:9000`; as of **2026-09-05** the authored target is **Cloudflare R2**
  (`https://<account-id>.r2.cloudflarestorage.com`). The switch-on procedure, the R2 gotchas and the
  honest proven/unproven split all live in **[`infra/BACKUP-R2.md`](../../infra/BACKUP-R2.md)**.
  Always **path-style**; region `${S3_BACKUP_REGION}` (MinIO ignores it, R2 requires exactly `auto`).
- **BucketStorageLocation `default`:** provider `aws`, bucket `velero`, prefix
  `${VELERO_BACKUP_PREFIX}` (dev/prod — both clusters share one R2 endpoint and are isolated by path
  only), `s3ForcePathStyle=true`, **`checksumAlgorithm: ""`**.
  That last one is load-bearing: `velero-plugin-for-aws` v1.14.2 defaults it to `CRC32` and R2 rejects
  AWS checksums with HTTP 400 `InvalidArgument`. Do not remove it, and re-check it on every plugin bump.
- **AWS plugin** (`velero/velero-plugin-for-aws:v1.14.2`) loaded as an init container.
- **File System Backup (kopia):** `deployNodeAgent=true`, `uploaderType=kopia`, `snapshotsEnabled=false`
  — storage-agnostic PVC backup, **no** CSI snapshot / data-mover. This path uploads through kopia's
  **own** S3 client, which does not read `checksumAlgorithm`: metadata compatibility and volume-data
  compatibility with a given object store are two separate questions.
- **Credentials:** `velero-cloud-credentials` — a SOPS Secret (`velero-cloud.sops.yaml`) holding the
  `cloud` INI key, the **scoped** key limited to the `velero`+`cnpg` buckets. Four Secrets across three
  namespaces hold this one key; `platform/backup/set-r2-credentials.sh` writes all four at once with
  `sops set` (no plaintext ever on disk) and proves they agree.
  `platform/backup/minio-credentials.sops.yaml` retains the **pre-migration MinIO** key — that is
  deliberate, it is how the pre-migration backups stay readable.

## Proving the target before trusting it

```bash
bootstrap/standup.sh --kubeconfig <kc> --expect-server <url> \
  [--s3-endpoint <candidate>] s3check
```

`s3check` runs a throwaway in-cluster pod per credential and proves credential load + authentication,
bucket existence and listability under this environment's prefix, and a real
**PUT → GET → byte-compare → DELETE → confirm-gone** round trip — plus whether the R2 checksum trap is
present. It fails closed and loud; it never reports "unknown, carrying on". Pod egress is the only
reading that counts (prod sits on pfSense's WAN, the MinIO inside its LAN).

Reconciled by its **own non-gating** Flux Kustomization `clusters/base/backup.yaml`
(`dependsOn: platform-config`, `wait:false`) — a backup controller must never gate the platform graph.

## Backup scope (what's included vs excluded)

Two daily `Schedule`s (cadence `${VELERO_BACKUP_SCHEDULE}`, retention `${VELERO_RETENTION}`):

| Schedule | Namespaces | Volumes (kopia FSB) |
|---|---|---|
| `daily` | ALL **except** `monitoring`, `unifi` | **all** PVC data (`defaultVolumesToFsBackup:true`, opt-out) |
| `daily-resources-only` | `monitoring`, `unifi` | **none** (`defaultVolumesToFsBackup:false`) — resources only |

**Excluded volume data (reconstructable, per the sizing decision):**
- `monitoring` — Prometheus TSDB PVC (ephemeral) + Loki 72h-log PVC. Grafana/Alertmanager are
  `emptyDir` (no PVC); Grafana dashboards are GitOps-provisioned.
- `unifi` — the ~7.4 GiB KubeVirt VM disk (huge; UniFi is protected by its own config export).

Their **Kubernetes resources** are still captured (by `daily-resources-only`) to aid DR.

**Included (the irreplaceable ~6 GiB):** every other namespace's PVC data via kopia — Nextcloud data,
Gitea, Plex/*arr config, PiHole, and the CNPG Postgres PVCs. (CNPG DBs additionally have native Barman
PITR to the `cnpg` bucket — see the DB manifests.)

## Post-deploy (PM)

- `velero backup-location get` → `default` Available. **Available only means the BSL could be read.**
- `velero backup create --from-schedule daily` → Completed, then read `velero backup logs <name>` and
  look for `PodVolumeBackup` failures specifically (see the kopia note above). Objects land under
  `s3://velero/${VELERO_BACKUP_PREFIX}/`.
- DR drill (R-STATE-03, Phase-4 gate): `velero restore create --from-backup <name>` into a scratch
  namespace and verify PVC data restores via the node-agent.

**R-STATE-03 is PROVEN against MinIO only.** A restore from R2 has never been executed in this repo,
and Velero-against-R2 is unproven here — `infra/BACKUP-R2.md` §5 is the re-proof sequence and §7 is the
honest list of what is and is not established.