*** THE BUG, AND IT WAS MINE. *** The placeholder guards were `case "$s3" in
""|TBD-*)` — a PREFIX glob. Two commits earlier (32d6766) I changed the
placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`, which does NOT start with
`TBD-`. Verified: fnmatch('https://TBD-R2-...', 'TBD-*') is False. All three
guards silently stopped matching, so **step_resume would have resumed velero on
prod straight onto a nonexistent hostname** — velero installs, reports healthy,
and no backup exists. That is precisely the failure the guard was written to
prevent, disarmed by a value change in a DIFFERENT file with nothing tying the
two together. Replaced with substring-matching s3_is_placeholder in preflight,
s3check and resume.
s3check rewritten from a reachability ping into a real proof. Per credential, from
a throwaway in-cluster pod (pod egress is the only egress that matters — the
workstation sits on a different network side and a host-side reading has already
misled this project once):
1. the credential LOADS AND AUTHENTICATES in the exact shape its consumer reads
— Velero's INI mounted from the `cloud` key, CNPG's discrete env keys;
2. the bucket EXISTS and is LISTABLE under this environment's own prefix, via
ListObjectsV2 — the op Velero's BSL sync and Barman's WAL scan actually use.
ListBuckets deliberately avoided: a bucket-scoped R2 token may not grant it;
3. PUT -> GET -> sha256+length compare -> DELETE -> confirm-gone. A HEAD would
pass with a read-only token and with every write being rejected;
4. the R2 checksum trap probed ON PURPOSE by repeating the PUT with checksums
enabled, and named either way.
Buckets and prefixes are DERIVED (velero's from the authored HelmRelease via yq,
cnpg's parsed from CNPG_BACKUP_PATH), not restated.
Fails closed, with no "unknown, carrying on" arm left: no pod, unfinished pod,
empty log, missing/duplicate/malformed verdict, any failed assertion, a claimed
PASS with failures>0, an assertion count below the caller-computed minimum, an
OK-line count disagreeing with the claimed count, or a PASS from a pod that ended
Failed. Failures are classified by cause — no egress, NoSuchBucket,
InvalidAccessKeyId, AccessDenied (token not scoped to BOTH buckets),
InvalidArgument -> the checksum trap.
`amazon/aws-cli` pinned rather than `minio/mc`: mc lacks sed/awk (which already
broke one probe) AND cannot send checksum headers, so it cannot test for the trap.
TIER_ORDER now includes platform-config-cdi, between platform-config and the leaf
tiers. Without it `resume` returned success without ever waiting on the tier that
guarantees the CDI StorageProfiles — and apps-vms depends on them: unifi
provisions on longhorn-vm, where an unapplied profile means CDI infers RWO instead
of the authored RWX, permanently non-live-migratable.
CREDENTIALS: one key, four committed Secrets, two shapes — and NO manifest change
is needed, only encrypted values. platform/backup/set-r2-credentials.sh writes all
four with `sops set`, which encrypts IN PLACE, so no decrypted Secret ever exists
in the working tree to be committed by accident — strictly better than staging a
plaintext template. Prompts with `read -rs`, rejects anything outside the S3
credential charset (a paste that swallowed a newline authenticates nowhere), never
prints a value, and verifies by decrypting back and comparing fingerprints.
`--check` is read-only. The MinIO key is RETAINED — it is what reads the
pre-migration backups.
THE ACCOUNT ID IS ALREADY IN THE REPO: infra/tofu/backend.r2.hcl commits
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com for the tofu
state bucket. If the backup buckets go in the same account, only the two token
values are needed. The placeholder is KEPT anyway — fail-closed beats convenience,
and the guard above just demonstrated why.
Corroborating the checksum fix independently: infra/tofu/backend.tf already carries
`region = "auto"` and `skip_s3_checksum = true # R2 rejects AWS trailing
(streaming) checksums`. This repo had learned both R2 facts for tofu state already.
VERSION FINDINGS, documented not measured against R2: plugin v1.14.2 documents
checksumAlgorithm defaulting to CRC32, so the authored "" is load-bearing;
Velero's kopia backend has NO checksum knob, so BSL metadata and PVC-data
compatibility are SEPARATE questions; and in the live CNPG 16.4 image,
boto3/botocore 1.35.60 is below the 1.36 line that turned on default request
checksums — so CNPG should be clear today, but barman-cloud exposes no escape
hatch, making a CNPG image bump an R2 re-verification event.
VERIFIED BY ME, independently of the agent: the dead-guard glob (fnmatch False);
the placeholder now refused on prod before any pod is created; a full round trip
against dev's MinIO — 7/7 assertions on BOTH credential paths, byte-identical,
pod Succeeded, nothing left behind, checksum trap correctly reported absent; all
four Secrets and the committed account id present. Full CI gate PASS.
STILL UNPROVEN AND SAID SO IN THE DOC: Velero-to-R2 (metadata or kopia data),
CNPG-barman-to-R2, and any restore from R2. R-STATE-03 is PROVEN MinIO-ONLY and
its ledger row predates this move. infra/BACKUP-R2.md §5 is the re-proof sequence
and it ends in a RESTORE, not an upload.
Refs R-STATE-03, R-IDP-05, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5