home-network-k3s-impl/infra/BACKUP-R2.md

# Off-cluster backup on Cloudflare R2 — operator procedure

**Owner:** WS-BACKUP · **Serves:** R-STATE-03 (DR restore), R-IDP-05 (Keycloak DB), R-ENV-02 (one
environment-agnostic path) · **Status:** authored and staged, **NOT switched on**.

> **Read this first.** Velero-against-R2 and CNPG/barman-against-R2 are **UNPROVEN in this repo**.
> Nothing here claims they work. What *is* proven is in [§7](#7-what-is-proven-and-what-is-not).
> R-STATE-03 is currently PROVEN **against MinIO only** and must be re-proven against R2 before this
> migration can be called done.

## Why R2 at all

The off-cluster target used to be a MinIO on the Proxmox host (`http://192.168.1.9:9000`). Two things
killed it, both recorded on 2026-09-05:

1. **Prod physically cannot reach it.** Prod is on the SOHO network `192.168.2.0/24`, which is
   pfSense's **WAN**; the MinIO sits inside pfSense's LAN `192.168.1.0/24`. Measured *from a pod inside
   prod* — the workstation's own reachability proves nothing about pod egress and has misled this
   project once already — `curl http://192.168.1.9:9000/minio/health/live` timed out after 15 s.
   Reaching it would need a WAN→LAN rule that does not exist.
2. **Dev's "off-cluster" backup was not off-cluster.** That MinIO runs on the *same Proxmox host* as
   dev's VMs, so it shared a failure domain with the thing it was backing up.

R2 needs no firewall change, is genuinely off-site, and is already trusted with the OpenTofu remote
state (`infra/tofu/REMOTE-STATE.md`) — the same account, the same kind of token. It is explicitly an
interim decision: *"that's where backups will go for the time being until a better on-prem backup
solution is created."*

---

## 1. Cloudflare: buckets, token, account id

All of this is in the Cloudflare dashboard; none of it touches the cluster.

1. **R2 → Create bucket** → `velero`. Location hint: whatever is nearest; jurisdiction default.
2. **R2 → Create bucket** → `cnpg`.
   Both names are fixed by the manifests: `bucket: velero` in `platform/velero/helmrelease.yaml`, and
   `cnpg` via `CNPG_BACKUP_PATH` (`s3://cnpg/dev`, `s3://cnpg/prod`). Use exactly those names or change
   the manifests — do not improvise.
3. **R2 → Manage API Tokens → Create API Token**
   - Permission: **Object Read & Write** (not "Admin Read & Write" — the cluster never needs to create
     or delete buckets, and a token that can is a token that can delete your backups).
   - **Specify bucket(s):** `velero` **and** `cnpg`. One token scoped to both — the manifests already
     assume one shared key, and `s3check` proves both buckets with it.
   - Copy the **Access Key ID** and **Secret Access Key**. They are shown **once**. Put them in the
     password manager before you do anything else.
4. **Account id** — 32 hex characters, visible in the dashboard URL and in the R2 S3 endpoint the
   token page shows you.

   **This repo already knows it.** `infra/tofu/backend.r2.hcl` commits
   `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com` for the OpenTofu state bucket,
   with the (correct) note that an account id is not a credential. **If the backup buckets live in the
   same Cloudflare account as the tofu state — they should; there is no reason for a second account —
   then the endpoint is already known and you do not need to look anything up:**

   ```
   https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
   ```

   Confirm it matches the endpoint on the token page. If it does not, you have two accounts and the
   `velero`/`cnpg` buckets are in the other one — use that one's id, and say so in the DECISIONS-LOG.

The endpoint form is always `https://<account-id>.r2.cloudflarestorage.com` — **no bucket in the
hostname** (path-style; see [§6](#6-r2-specific-gotchas)).

---

## 2. Repo values to set

Only **one** non-secret value is missing, and it is not in a secret file:

| File | Key | From | To |
|---|---|---|---|
| `clusters/base/cluster-vars.yaml` | `S3_BACKUP_ENDPOINT` | `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` | `https://<account-id>.r2.cloudflarestorage.com` |

Already authored, **do not change**: `S3_BACKUP_REGION: "auto"`, `VELERO_BACKUP_PREFIX` (dev `dev` /
prod `prod`), `CNPG_BACKUP_PATH` (`s3://cnpg/dev` / `s3://cnpg/prod`), and `checksumAlgorithm: ""` on
the Velero BackupStorageLocation.

The placeholder is deliberately loud, and the guards match it as a **substring** — `bootstrap/standup.sh`
refuses to probe it and refuses to resume `velero` while it is there.

---

## 3. Credentials: four Secrets, one command

The R2 key is read by **four** committed SOPS Secrets, in **two** different shapes. No manifest changes
are needed — only the encrypted values change:

| File | Consumer | Shape |
|---|---|---|
| `platform/velero/velero-cloud.sops.yaml` | Velero (`velero-cloud-credentials`, ns `velero`) | `stringData.cloud` — an **AWS INI file**: `[default]` + `aws_access_key_id=` / `aws_secret_access_key=` |
| `platform/keycloak/keycloak-db-backup-credentials.sops.yaml` | CNPG `keycloak-db` (ns `keycloak`) | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
| `apps/wave-0/nextcloud-db-backup-credentials.sops.yaml` | CNPG `nextcloud-db` | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
| `apps/wave-0/semaphore-db-backup-credentials.sops.yaml` | CNPG `semaphore-db` | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |

A Secret is namespace-scoped and a Pod reads only its own namespace's, which is why there are four
copies of one key rather than one.

### The one command

```bash
./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh
```

It prompts for the two values (never echoed, never in shell history, never written to a file), writes
them into all four Secrets with `sops set` — which encrypts in place, so **no decrypted copy of any
Secret is ever created** — then decrypts each file back and proves all four now hold the same key by
comparing 32-bit fingerprints. It never prints a credential.

There is deliberately **no plaintext template to fill in and encrypt**. The
`cp x.example x.yaml && $EDITOR && sops -e -i` pattern leaves a plaintext Kubernetes Secret sitting in
the working tree, one `git add -A` away from being in git forever; `sops set` removes that window
entirely. If you would rather edit by hand, `sops platform/velero/velero-cloud.sops.yaml` decrypts to
`$EDITOR` and re-encrypts on save — also with no plaintext on disk — but then you must repeat it four
times and keep them identical yourself.

To check the four files agree without changing anything:

```bash
./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh --check
```

### What is NOT changed

`platform/backup/minio-credentials.sops.yaml` keeps the **MinIO** key. That is not an oversight — it
is what you need to read the pre-migration backups ([§8](#8-going-back-to-minio)). Its
`.yaml.example` sibling documents the MinIO shape and stays accurate for the MinIO target.

Review the diff before committing: you should see **only** `ENC[...]` blobs and `lastmodified`/`mac`
change in four files. If a key name or a `metadata.name` moved, stop.

---

## 4. Prove the target BEFORE merging

```bash
K=infra/ansible/.cache/dev.kubeconfig        # or prod.kubeconfig
./infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig $K --expect-server https://192.168.2.50:6443 \
  --s3-endpoint https://<account-id>.r2.cloudflarestorage.com s3check
```

`--s3-endpoint` probes a **candidate** endpoint instead of the live `cluster-vars` value, so the target
is proven before it is authored. Run it against **both** clusters — prod's egress is the whole reason
this migration exists, and it is a different network position from dev's.

`s3check` runs a throwaway pod per credential and, for each, proves:

1. the committed credential **loads and authenticates**, in the exact shape its consumer reads it
   (Velero's INI file mounted from the `cloud` key; CNPG's discrete `ACCESS_KEY_ID`/`ACCESS_SECRET_KEY`);
2. the bucket **exists and is listable** under this environment's own prefix (`ListObjectsV2`, the
   operation Velero's BSL sync and Barman's WAL scan both perform);
3. **PUT → GET → byte-compare (sha256 + length) → DELETE → confirm-gone** of one throwaway object at
   `<prefix>/.s3check/<pod>.probe` — a genuine write, not a HEAD. A HEAD passes with a read-only token
   and with R2 rejecting every write;
4. whether the **R2 checksum trap** ([§6](#6-r2-specific-gotchas)) is present, probed on purpose by
   repeating the PUT with checksums *enabled*, and named explicitly either way.

It **fails closed**: a pod that cannot be created, an empty log, a missing verdict line, a short
assertion count, an OK-line count that disagrees with the claimed count, or any single failed
assertion all exit non-zero. There is no "reachability unknown, carrying on". Each failure names its
own cause and fix — wrong key, unscoped token, missing bucket, no pod egress, checksum rejection.

It writes and deletes exactly one object per bucket and leaves no pod behind. It applies nothing and
does not enable `velero`.

---

## 5. The verification sequence — and it ends in a RESTORE

**A backup that has never been restored is not a backup.** R-STATE-03 is PROVEN against MinIO; that
proof does not transfer to a different object store with a different S3 implementation.

1. **`s3check` on dev, then on prod.** Both must pass. (§4)
2. Merge the endpoint + the four re-encrypted Secrets. Let Flux reconcile.
3. **Resume `velero`** — `bootstrap/standup.sh … resume` does it once the endpoint is no longer a
   placeholder, or `flux resume kustomization velero`.
4. `velero backup-location get` → `default` **Available**. *Available only means the BSL could be
   read;* it is not a backup.
5. `velero backup create --from-schedule daily` → wait for **Completed**, then
   `velero backup logs <name>`. Watch specifically for `PodVolumeBackup` failures: the PVC-data path
   is kopia, a **different S3 client** from the one the BSL uses (see §7), so metadata can succeed
   while volume data fails.
6. **Restore.** `velero restore create --from-backup <name> --include-namespaces <scratch-ns>` into a
   *scratch* namespace, and verify the PVC contents actually arrived — not just that the Restore says
   Completed. This is the R-STATE-03 re-proof.
7. **CNPG PITR.** Confirm WAL is arriving:
   `kubectl -n keycloak get cluster keycloak-db -o jsonpath='{.status.conditions}'` (look for
   `ContinuousArchiving`), and that objects exist under `s3://cnpg/<env>/keycloak-db/`. Then do the
   restore that matters: a `Cluster` with `bootstrap.recovery` from the barman object store into a
   scratch namespace, and check the recovered database actually contains data. The existing
   `conformance/tests/l2-behavioral/keycloak-db-dr` and `l3-fault-injection/dr-restore` tests are the
   harness for this; they must be re-run **against R2**, and their previous PROVEN state records MinIO,
   not R2.
8. Only after 6 and 7 pass may this migration be described as done, and the ledger updated. Until
   then the honest status is "R2 accepts objects; recovery from R2 is untested".

---

## 6. R2-specific gotchas

**`region: auto`.** R2 accepts nothing else, and S3 clients require *some* region. Authored as
`S3_BACKUP_REGION: "auto"`. This repo already learned it — `infra/tofu/backend.tf` carries
`region = "auto"  # R2 ignores region, but the S3 backend requires a value.`

**`checksumAlgorithm: ""` — do not delete this line.** `velero-plugin-for-aws`'s own
`backupstoragelocation.md` for **v1.14.2** (the pinned version) documents:

> `checksumAlgorithm` — The checksum algorithm to use for uploading objects to S3. Supported values
> "CRC32", "CRC32C", "SHA1", "SHA256". If the value is set as empty string "", no checksum will be
> calculated and attached to the request headers. **Optional (defaults to "CRC32")**

So *omitting* the key means CRC32 **is** sent: `x-amz-sdk-checksum-algorithm` +
`x-amz-checksum-crc32`. This repo has already measured that R2 rejects AWS checksums — that is exactly
why `infra/tofu/backend.tf` needs `skip_s3_checksum = true  # R2 rejects AWS trailing (streaming)
checksums.` for the state bucket. The plugin's own compatibility table lists `checksumAlgorithm=""` as
the remedy for the same failure on Ceph S3, Backblaze B2 and IBM COS. If backups start failing with
**HTTP 400 `InvalidArgument`** after a plugin bump, look here first. `s3check` probes for this on
purpose and names it.

**Path-style.** `s3ForcePathStyle: "true"` is authored and correct for R2 as well as MinIO. Never put
the bucket in the hostname.

**No bucket auto-creation.** Velero does not create buckets and the token deliberately cannot. A
missing bucket is a `NoSuchBucket` at runtime; `s3check` catches it first.

**Token scope is per-bucket.** A token scoped only to `velero` authenticates fine and then 403s on
`cnpg` — the two consumers fail independently and quietly. `s3check` proves both.

**Both clusters share one endpoint.** Isolation is by path only (`VELERO_BACKUP_PREFIX`,
`CNPG_BACKUP_PATH`), because R2 is one environment-independent URL. Those two knobs are load-bearing:
without them prod would adopt and TTL-delete dev's Velero backups, and both clusters' identically-named
CNPG Clusters would interleave their WAL into one Barman stream, destroying PITR for both.

---

## 7. What is proven, and what is not

**Proven (2026-09-05, WS-BACKUP, against the live dev cluster and its MinIO at
`http://192.168.1.9:9000`):**

- the rewritten `s3check` itself: 7 assertions per credential path, both credential shapes, from
  in-cluster pods — credential load, bucket list, PUT/GET/byte-compare/DELETE/confirm-gone, no object
  and no pod left behind;
- its failure paths: the `TBD` placeholder is refused; an unreachable endpoint (`:9099`) produces a
  named no-egress failure and exit 1, not a pass;
- MinIO accepts a checksum-bearing PUT, i.e. the trap is a *non-*MinIO problem, and
  `checksumAlgorithm: ""` is harmless there.

**Established from the pinned versions (documentation and image inspection, not measured against R2):**

- `velero-plugin-for-aws` **v1.14.2** pairs with Velero **v1.18.x** (upstream compatibility table) and
  defaults `checksumAlgorithm` to `CRC32` — so the authored `""` is load-bearing, not decoration.
- That plugin's "Non-AWS S3 compatible provider known issues with plugin v1.14.x" table **does not
  mention Cloudflare R2 at all** — neither as working nor as broken. There is no upstream statement to
  lean on in either direction.
- **The PVC-data path is a different S3 client.** `checksumAlgorithm` is a BackupStorageLocation
  *object-store plugin* setting. Velero's File System Backup uploads through the unified repository's
  kopia S3 backend (`pkg/repository/udmrepo/kopialib/backend/s3.go` in Velero v1.18.0), which
  references only endpoint/region-style options and has **no checksum knob at all**. So R2
  compatibility for backup *metadata* and for *volume data* are two separate questions, and only step
  5 of §5 answers the second.
- **CNPG/barman is likely clear of the checksum trap, by version.** Inspected inside the live
  `ghcr.io/cloudnative-pg/postgresql:16.4` image on dev: `barman-cloud-wal-archive 3.11.1`,
  `botocore 1.35.60`, `boto3 1.35.60`. That botocore predates the 1.36 line in which botocore began
  attaching CRC32 request checksums by default (the change that introduced
  `AWS_REQUEST_CHECKSUM_CALCULATION`), so this image should not send them.
  **Forward risk:** barman-cloud exposes **no** `checksumAlgorithm` equivalent. If the CNPG image is
  ever bumped to one carrying botocore ≥ 1.36, WAL archiving to R2 could start failing with 400
  `InvalidArgument` and there is no config knob to turn it off — only `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`
  in the Pod environment, which CNPG does not currently set. Treat a CNPG image bump as a change that
  must be re-verified against R2.

**Not proven, at all:**

- that Velero can write a usable backup to R2 (metadata or kopia volume data);
- that CNPG/barman can archive WAL to R2 and that a `bootstrap.recovery` from R2 succeeds;
- that a restore from R2 yields correct data. R-STATE-03's PROVEN state is **MinIO-only**.

---

## 8. Going back to MinIO

The MinIO copy is **not deleted** and stays restorable. To read pre-migration backups:

1. `clusters/base/cluster-vars.yaml` → `S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"`
2. `VELERO_BACKUP_PREFIX` → `""` (the MinIO backups are at the **bucket root**, not under `dev/`) and
   `CNPG_BACKUP_PATH` → `s3://cnpg` (Barman objects are at `s3://cnpg/<clusterName>/`).
3. Restore the MinIO credential into the four Secrets. `platform/backup/minio-credentials.sops.yaml`
   still holds it: `sops -d --extract '["stringData"]["ACCESS_KEY_ID"]'` it and feed the two values to
   `platform/backup/set-r2-credentials.sh` (the script cares about the *shape*, not the provider).
4. `S3_BACKUP_REGION` may stay `auto` — MinIO ignores the region.
5. **Dev only.** Prod cannot reach that MinIO at all (§Why R2); pointing prod back is not an option.

Reverting for a restore and then returning to R2 costs another PITR reset. Prefer restoring from MinIO
into a *scratch* namespace on dev with a second, temporary BackupStorageLocation over flipping the
whole cluster's endpoint back and forth.

---

## 9. Merging this resets dev's PITR window

Say it out loud before merging, because it is not recoverable by retrying:

- Dev's R2 prefix starts **empty**. Velero writes a **fresh kopia repository** — the first backup is a
  full upload of every included PVC, and no earlier restore point exists in R2.
- Every CNPG Cluster takes **new base backups** (`ScheduledBackup … immediate: true`), and the 30-day
  Barman retention window (`retentionPolicy: "30d"`) rebuilds **from zero**. For roughly a month, dev's
  point-in-time recovery reaches back only as far as the merge.
- The MinIO copy still holds the old window (§8), but reaching it means reverting the endpoint — so it
  is a deliberate excursion, not a fallback that happens automatically.

**Do not merge this while dev's proven MinIO restore is the only recovery path you are relying on.**
The same applies to prod the day it first backs up anywhere.