0 Fork 0

Files

T

jakman85andClaude Opus 5 8f9b2e4270 feat(backup): s3check proves a real round trip — and revives a guard I had killed

*** THE BUG, AND IT WAS MINE. *** The placeholder guards were `case "$s3" in
""|TBD-*)` — a PREFIX glob. Two commits earlier (32d6766) I changed the
placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`, which does NOT start with
`TBD-`. Verified: fnmatch('https://TBD-R2-...', 'TBD-*') is False. All three
guards silently stopped matching, so **step_resume would have resumed velero on
prod straight onto a nonexistent hostname** — velero installs, reports healthy,
and no backup exists. That is precisely the failure the guard was written to
prevent, disarmed by a value change in a DIFFERENT file with nothing tying the
two together. Replaced with substring-matching s3_is_placeholder in preflight,
s3check and resume.

s3check rewritten from a reachability ping into a real proof. Per credential, from
a throwaway in-cluster pod (pod egress is the only egress that matters — the
workstation sits on a different network side and a host-side reading has already
misled this project once):
  1. the credential LOADS AND AUTHENTICATES in the exact shape its consumer reads
     — Velero's INI mounted from the `cloud` key, CNPG's discrete env keys;
  2. the bucket EXISTS and is LISTABLE under this environment's own prefix, via
     ListObjectsV2 — the op Velero's BSL sync and Barman's WAL scan actually use.
     ListBuckets deliberately avoided: a bucket-scoped R2 token may not grant it;
  3. PUT -> GET -> sha256+length compare -> DELETE -> confirm-gone. A HEAD would
     pass with a read-only token and with every write being rejected;
  4. the R2 checksum trap probed ON PURPOSE by repeating the PUT with checksums
     enabled, and named either way.
Buckets and prefixes are DERIVED (velero's from the authored HelmRelease via yq,
cnpg's parsed from CNPG_BACKUP_PATH), not restated.

Fails closed, with no "unknown, carrying on" arm left: no pod, unfinished pod,
empty log, missing/duplicate/malformed verdict, any failed assertion, a claimed
PASS with failures>0, an assertion count below the caller-computed minimum, an
OK-line count disagreeing with the claimed count, or a PASS from a pod that ended
Failed. Failures are classified by cause — no egress, NoSuchBucket,
InvalidAccessKeyId, AccessDenied (token not scoped to BOTH buckets),
InvalidArgument -> the checksum trap.

`amazon/aws-cli` pinned rather than `minio/mc`: mc lacks sed/awk (which already
broke one probe) AND cannot send checksum headers, so it cannot test for the trap.

TIER_ORDER now includes platform-config-cdi, between platform-config and the leaf
tiers. Without it `resume` returned success without ever waiting on the tier that
guarantees the CDI StorageProfiles — and apps-vms depends on them: unifi
provisions on longhorn-vm, where an unapplied profile means CDI infers RWO instead
of the authored RWX, permanently non-live-migratable.

CREDENTIALS: one key, four committed Secrets, two shapes — and NO manifest change
is needed, only encrypted values. platform/backup/set-r2-credentials.sh writes all
four with `sops set`, which encrypts IN PLACE, so no decrypted Secret ever exists
in the working tree to be committed by accident — strictly better than staging a
plaintext template. Prompts with `read -rs`, rejects anything outside the S3
credential charset (a paste that swallowed a newline authenticates nowhere), never
prints a value, and verifies by decrypting back and comparing fingerprints.
`--check` is read-only. The MinIO key is RETAINED — it is what reads the
pre-migration backups.

THE ACCOUNT ID IS ALREADY IN THE REPO: infra/tofu/backend.r2.hcl commits
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com for the tofu
state bucket. If the backup buckets go in the same account, only the two token
values are needed. The placeholder is KEPT anyway — fail-closed beats convenience,
and the guard above just demonstrated why.
Corroborating the checksum fix independently: infra/tofu/backend.tf already carries
`region = "auto"` and `skip_s3_checksum = true # R2 rejects AWS trailing
(streaming) checksums`. This repo had learned both R2 facts for tofu state already.

VERSION FINDINGS, documented not measured against R2: plugin v1.14.2 documents
checksumAlgorithm defaulting to CRC32, so the authored "" is load-bearing;
Velero's kopia backend has NO checksum knob, so BSL metadata and PVC-data
compatibility are SEPARATE questions; and in the live CNPG 16.4 image,
boto3/botocore 1.35.60 is below the 1.36 line that turned on default request
checksums — so CNPG should be clear today, but barman-cloud exposes no escape
hatch, making a CNPG image bump an R2 re-verification event.

VERIFIED BY ME, independently of the agent: the dead-guard glob (fnmatch False);
the placeholder now refused on prod before any pod is created; a full round trip
against dev's MinIO — 7/7 assertions on BOTH credential paths, byte-identical,
pod Succeeded, nothing left behind, checksum trap correctly reported absent; all
four Secrets and the committed account id present. Full CI gate PASS.

STILL UNPROVEN AND SAID SO IN THE DOC: Velero-to-R2 (metadata or kopia data),
CNPG-barman-to-R2, and any restore from R2. R-STATE-03 is PROVEN MinIO-ONLY and
its ledger row predates this move. infra/BACKUP-R2.md §5 is the re-proof sequence
and it ends in a RESTORE, not an upload.

Refs R-STATE-03, R-IDP-05, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-05 13:24:26 -05:00

5.1 KiB

Raw Blame History

platform/velero

Velero — off-cluster backup and DR of Kubernetes resource state + PVC data.

Governing ADR: ADR-0003 — Longhorn (+ bulk tier + Velero)
Spec: docs/04 — Storage & State
Serves: R-STATE-03
Owner: WS-STORAGE

What this is (LIVE)

Flux HelmRelease (Velero chart 12.x / Velero 1.18.x) wired to whatever ${S3_BACKUP_ENDPOINT} names. The target is a cluster-vars knob, not a constant:

Endpoint: ${S3_BACKUP_ENDPOINT} — dev has been running against the on-Proxmox MinIO http://192.168.1.9:9000; as of 2026-09-05 the authored target is Cloudflare R2 (https://<account-id>.r2.cloudflarestorage.com). The switch-on procedure, the R2 gotchas and the honest proven/unproven split all live in infra/BACKUP-R2.md. Always path-style; region ${S3_BACKUP_REGION} (MinIO ignores it, R2 requires exactly auto).
BucketStorageLocation default: provider aws, bucket velero, prefix ${VELERO_BACKUP_PREFIX} (dev/prod — both clusters share one R2 endpoint and are isolated by path only), s3ForcePathStyle=true, checksumAlgorithm: "". That last one is load-bearing: velero-plugin-for-aws v1.14.2 defaults it to CRC32 and R2 rejects AWS checksums with HTTP 400 InvalidArgument. Do not remove it, and re-check it on every plugin bump.
AWS plugin (velero/velero-plugin-for-aws:v1.14.2) loaded as an init container.
File System Backup (kopia): deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false — storage-agnostic PVC backup, no CSI snapshot / data-mover. This path uploads through kopia's own S3 client, which does not read checksumAlgorithm: metadata compatibility and volume-data compatibility with a given object store are two separate questions.
Credentials: velero-cloud-credentials — a SOPS Secret (velero-cloud.sops.yaml) holding the cloud INI key, the scoped key limited to the velero+cnpg buckets. Four Secrets across three namespaces hold this one key; platform/backup/set-r2-credentials.sh writes all four at once with sops set (no plaintext ever on disk) and proves they agree. platform/backup/minio-credentials.sops.yaml retains the pre-migration MinIO key — that is deliberate, it is how the pre-migration backups stay readable.

Proving the target before trusting it

bootstrap/standup.sh --kubeconfig <kc> --expect-server <url> \
  [--s3-endpoint <candidate>] s3check

s3check runs a throwaway in-cluster pod per credential and proves credential load + authentication, bucket existence and listability under this environment's prefix, and a real PUT → GET → byte-compare → DELETE → confirm-gone round trip — plus whether the R2 checksum trap is present. It fails closed and loud; it never reports "unknown, carrying on". Pod egress is the only reading that counts (prod sits on pfSense's WAN, the MinIO inside its LAN).

Reconciled by its own non-gating Flux Kustomization clusters/base/backup.yaml (dependsOn: platform-config, wait:false) — a backup controller must never gate the platform graph.

Backup scope (what's included vs excluded)

Two daily Schedules (cadence ${VELERO_BACKUP_SCHEDULE}, retention ${VELERO_RETENTION}):

Schedule	Namespaces	Volumes (kopia FSB)
`daily`	ALL except `monitoring`, `unifi`	all PVC data (`defaultVolumesToFsBackup:true`, opt-out)
`daily-resources-only`	`monitoring`, `unifi`	none (`defaultVolumesToFsBackup:false`) — resources only

Excluded volume data (reconstructable, per the sizing decision):

monitoring — Prometheus TSDB PVC (ephemeral) + Loki 72h-log PVC. Grafana/Alertmanager are emptyDir (no PVC); Grafana dashboards are GitOps-provisioned.
unifi — the ~7.4 GiB KubeVirt VM disk (huge; UniFi is protected by its own config export).

Their Kubernetes resources are still captured (by daily-resources-only) to aid DR.

Included (the irreplaceable ~6 GiB): every other namespace's PVC data via kopia — Nextcloud data, Gitea, Plex/*arr config, PiHole, and the CNPG Postgres PVCs. (CNPG DBs additionally have native Barman PITR to the cnpg bucket — see the DB manifests.)

Post-deploy (PM)

velero backup-location get → default Available. Available only means the BSL could be read.
velero backup create --from-schedule daily → Completed, then read velero backup logs <name> and look for PodVolumeBackup failures specifically (see the kopia note above). Objects land under s3://velero/${VELERO_BACKUP_PREFIX}/.
DR drill (R-STATE-03, Phase-4 gate): velero restore create --from-backup <name> into a scratch namespace and verify PVC data restores via the node-agent.

R-STATE-03 is PROVEN against MinIO only. A restore from R2 has never been executed in this repo, and Velero-against-R2 is unproven here — infra/BACKUP-R2.md §5 is the re-proof sequence and §7 is the honest list of what is and is not established.

5.1 KiB Raw Blame History