jakman85andClaude Opus 5
b80e8eef82
Operator ruling: "the S3 backup location is the same as Dev." Prod's
S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder
is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would
inherit the same value silently, and a shared backup target must be visible in
the overlay that owns it.
SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS
was per-environment, and enabling prod against dev's MinIO would have been the
2026-09-04 external-dns incident again, in a worse place:
VELERO — bucket `velero`, NO prefix, and both clusters run schedules named
`daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL
into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would
have adopted dev's backups and then deleted them on schedule.
CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending
serverName, which defaults to the CLUSTER NAME. Both environments have
keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive
WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one
Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is
PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not.
Two knobs, authored once, environments differing only by scoped overrides
(R-ENV-02):
VELERO_BACKUP_PREFIX dev "" | prod "prod"
CNPG_BACKUP_PATH dev "s3://cnpg" | prod "s3://cnpg/prod"
THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live
at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very
backups its proven restore depends on — the same reasoning that keeps dev's
EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve.
*** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. ***
Measured from a throwaway pod INSIDE the prod cluster (the workstation proves
nothing about pod egress, and a workstation-side reading has already misled this
project once): curl http://192.168.1.9:9000/minio/health/live → connection timed
out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO.
OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN
regardless — the token IS defined so strict substitution succeeds — while every
write fails at RUNTIME. A green apps-wave-0 does not mean backups work.
New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster
pod and fails closed, with `--s3-endpoint <url>` to test a CANDIDATE before
authoring it into cluster-vars. That is how the unreachability above was
established rather than assumed.
VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions,
and neither new knob appears in the dead-knob advisory list, so both are really
consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's
prefix carry the tokens; dev resolves to its existing layout, prod to the nested
paths.
DELIVERY: prod reconciles the bootstrap Gitea's main at c8c1d29. This branch also
carries the DNS inversion, which must NOT reach prod. Delivering needs a
c8c1d29-based branch with only these commits — no urgency while the routing is
still closed.
Refs R-STATE-03, R-ENV-02, R-IDP-05.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/velero
Velero — off-cluster backup and DR of Kubernetes resource state + PVC data.
- Governing ADR: ADR-0003 — Longhorn (+ bulk tier + Velero)
- Spec: docs/04 — Storage & State
- Serves: R-STATE-03
- Owner: WS-STORAGE
What this is (LIVE)
Flux HelmRelease (Velero chart 12.x / Velero 1.18.x) wired to the on-Proxmox MinIO S3 target:
- Endpoint:
http://192.168.1.9:9000(plain HTTP, path-style, regionus-east-1placeholder) - BucketStorageLocation
default: provideraws, bucketvelero,s3ForcePathStyle=true. - AWS plugin (
velero/velero-plugin-for-aws) loaded as an init container. - File System Backup (kopia):
deployNodeAgent=true,uploaderType=kopia,snapshotsEnabled=false— storage-agnostic PVC backup, no CSI snapshot / data-mover. - Credentials:
velero-cloud-credentials— a SOPS Secret (velero-cloud.sops.yaml) holding thecloudINI key, the scoped MinIO key limited to thevelero+cnpgbuckets. Kept in sync with the staging source of truthplatform/backup/minio-credentials.sops.yamland the Proxmox host ansible-vault.
Reconciled by its own non-gating Flux Kustomization clusters/base/backup.yaml
(dependsOn: platform-config, wait:false) — a backup controller must never gate the platform graph.
Backup scope (what's included vs excluded)
Two daily Schedules (cadence ${VELERO_BACKUP_SCHEDULE}, retention ${VELERO_RETENTION}):
| Schedule | Namespaces | Volumes (kopia FSB) |
|---|---|---|
daily |
ALL except monitoring, unifi |
all PVC data (defaultVolumesToFsBackup:true, opt-out) |
daily-resources-only |
monitoring, unifi |
none (defaultVolumesToFsBackup:false) — resources only |
Excluded volume data (reconstructable, per the sizing decision):
monitoring— Prometheus TSDB PVC (ephemeral) + Loki 72h-log PVC. Grafana/Alertmanager areemptyDir(no PVC); Grafana dashboards are GitOps-provisioned.unifi— the ~7.4 GiB KubeVirt VM disk (huge; UniFi is protected by its own config export).
Their Kubernetes resources are still captured (by daily-resources-only) to aid DR.
Included (the irreplaceable ~6 GiB): every other namespace's PVC data via kopia — Nextcloud data,
Gitea, Plex/*arr config, PiHole, and the CNPG Postgres PVCs. (CNPG DBs additionally have native Barman
PITR to the cnpg bucket — see the DB manifests.)
Post-deploy (PM)
velero backup-location get→defaultAvailable.velero backup create --from-schedule daily→ objects land unders3://velero/.- DR drill (R-STATE-03, Phase-4 gate):
velero restore create --from-backup <name>into a scratch namespace and verify PVC data restores via the node-agent.