From b80e8eef823c3f40f6fb271933e8ab3891b100c6 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 5 Sep 2026 10:59:50 -0500 Subject: [PATCH] =?UTF-8?q?feat(backup):=20prod=20shares=20dev's=20MinIO?= =?UTF-8?q?=20=E2=80=94=20plus=20the=20path=20tenancy=20that=20makes=20it?= =?UTF-8?q?=20survivable?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator ruling: "the S3 backup location is the same as Dev." Prod's S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would inherit the same value silently, and a shared backup target must be visible in the overlay that owns it. SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS was per-environment, and enabling prod against dev's MinIO would have been the 2026-09-04 external-dns incident again, in a worse place: VELERO — bucket `velero`, NO prefix, and both clusters run schedules named `daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would have adopted dev's backups and then deleted them on schedule. CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the CLUSTER NAME. Both environments have keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not. Two knobs, authored once, environments differing only by scoped overrides (R-ENV-02): VELERO_BACKUP_PREFIX dev "" | prod "prod" CNPG_BACKUP_PATH dev "s3://cnpg" | prod "s3://cnpg/prod" THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very backups its proven restore depends on — the same reasoning that keeps dev's EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve. *** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. *** Measured from a throwaway pod INSIDE the prod cluster (the workstation proves nothing about pod egress, and a workstation-side reading has already misled this project once): curl http://192.168.1.9:9000/minio/health/live → connection timed out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO. OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN regardless — the token IS defined so strict substitution succeeds — while every write fails at RUNTIME. A green apps-wave-0 does not mean backups work. New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster pod and fails closed, with `--s3-endpoint ` to test a CANDIDATE before authoring it into cluster-vars. That is how the unreachability above was established rather than assumed. VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions, and neither new knob appears in the dead-knob advisory list, so both are really consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's prefix carry the tokens; dev resolves to its existing layout, prod to the nested paths. DELIVERY: prod reconciles the bootstrap Gitea's main at c8c1d29. This branch also carries the DNS inversion, which must NOT reach prod. Delivering needs a c8c1d29-based branch with only these commits — no urgency while the routing is still closed. Refs R-STATE-03, R-ENV-02, R-IDP-05. Co-Authored-By: Claude Opus 5 --- apps/wave-0/nextcloud-db.yaml | 6 ++- apps/wave-0/semaphore-db.yaml | 6 ++- bootstrap/standup.sh | 57 +++++++++++++++++++++++-- clusters/base/cluster-vars.yaml | 22 ++++++++++ clusters/prod/kustomization.yaml | 36 ++++++++++++++-- infra/PROD-STANDUP.md | 52 ++++++++++++++++++---- platform/keycloak/postgres-cluster.yaml | 6 ++- platform/velero/helmrelease.yaml | 6 +++ 8 files changed, 172 insertions(+), 19 deletions(-) diff --git a/apps/wave-0/nextcloud-db.yaml b/apps/wave-0/nextcloud-db.yaml index accab0f..d0a189c 100644 --- a/apps/wave-0/nextcloud-db.yaml +++ b/apps/wave-0/nextcloud-db.yaml @@ -79,7 +79,11 @@ spec: backup: retentionPolicy: "30d" barmanObjectStore: - destinationPath: s3://cnpg + # PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER + # NAME — and both environments have a `nextcloud-db`. Sharing one MinIO with a shared path + # would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and + # destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests. + destinationPath: ${CNPG_BACKUP_PATH} endpointURL: ${S3_BACKUP_ENDPOINT} s3Credentials: accessKeyId: diff --git a/apps/wave-0/semaphore-db.yaml b/apps/wave-0/semaphore-db.yaml index 7c8c3a5..aedb536 100644 --- a/apps/wave-0/semaphore-db.yaml +++ b/apps/wave-0/semaphore-db.yaml @@ -80,7 +80,11 @@ spec: backup: retentionPolicy: "30d" barmanObjectStore: - destinationPath: s3://cnpg + # PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER + # NAME — and both environments have a `semaphore-db`. Sharing one MinIO with a shared path + # would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and + # destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests. + destinationPath: ${CNPG_BACKUP_PATH} endpointURL: ${S3_BACKUP_ENDPOINT} s3Credentials: accessKeyId: diff --git a/bootstrap/standup.sh b/bootstrap/standup.sh index 4cd858f..66f3eff 100755 --- a/bootstrap/standup.sh +++ b/bootstrap/standup.sh @@ -18,7 +18,7 @@ # USAGE # bootstrap/standup.sh --kubeconfig --expect-server [--confirm-destructive] # -# steps: preflight crds unwedge externaldns resume verify (or: all) +# steps: preflight crds unwedge externaldns s3check resume verify (or: all) # # --expect-server is REQUIRED and is compared against the live API server before ANY command runs. # Pointing this at the wrong cluster is the one mistake with unbounded blast radius: dev is live and @@ -29,13 +29,14 @@ set -euo pipefail # ---------------------------------------------------------------------------- args -KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0 +KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0 S3_ENDPOINT_OVERRIDE="" STEPS=() while [ $# -gt 0 ]; do case "$1" in --kubeconfig) KUBECONFIG_ARG="${2:?--kubeconfig needs a value}"; shift 2 ;; --expect-server) EXPECT_SERVER="${2:?--expect-server needs a value}"; shift 2 ;; --confirm-destructive) CONFIRM_DESTRUCTIVE=1; shift ;; + --s3-endpoint) S3_ENDPOINT_OVERRIDE="${2:?--s3-endpoint needs a value}"; shift 2 ;; -h|--help) sed -n '2,30p' "$0"; exit 0 ;; -*) echo "ERROR: unknown flag '$1'" >&2; exit 2 ;; *) STEPS+=("$1"); shift ;; @@ -44,7 +45,7 @@ done [ -n "${KUBECONFIG_ARG}" ] || { echo "ERROR: --kubeconfig is required." >&2; exit 2; } [ -n "${EXPECT_SERVER}" ] || { echo "ERROR: --expect-server is required (safety guard)." >&2; exit 2; } -[ "${#STEPS[@]}" -gt 0 ] || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns resume verify all" >&2; exit 2; } +[ "${#STEPS[@]}" -gt 0 ] || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns s3check resume verify all" >&2; exit 2; } script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/.." && pwd))" @@ -391,6 +392,53 @@ step_externaldns() { Any change to a name outside it, or to a TXT entry owned by another cluster, is a REGRESSION." } +step_s3check() { + say "S3 BACKUP ENDPOINT — reachability FROM INSIDE THE CLUSTER" + # Velero runs in-cluster, so the only reachability that counts is the pods' own. Probing from the + # workstation proves nothing either way: it sits on a different side of pfSense than the pod + # network, and a healthy result there has already misled this project once. + local ep live; live="$(cvar S3_BACKUP_ENDPOINT)" + info "live S3_BACKUP_ENDPOINT = ${live:-}" + if [ -n "${S3_ENDPOINT_OVERRIDE}" ]; then + # Probe a CANDIDATE endpoint before adopting it. The whole point is to find out whether a proposed + # backup target is reachable from this cluster's pods BEFORE authoring it into cluster-vars — + # otherwise the first evidence is silently-missing backups. + ep="${S3_ENDPOINT_OVERRIDE}" + info "probing CANDIDATE endpoint from --s3-endpoint: ${ep}" + else + ep="${live}" + fi + case "${ep}" in + ""|TBD-*) die "no usable endpoint to probe (live='${live}'). Pass --s3-endpoint to test a candidate." ;; + esac + + local ns=velero pod=s3check-$$ + kubectl get ns "${ns}" >/dev/null 2>&1 || ns=default + info "probing from a throwaway pod in namespace '${ns}'" + # --restart=Never + explicit delete in a trap: a probe must never leave a pod behind on a cluster. + trap 'kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found --wait=false >/dev/null 2>&1 || true' RETURN + if kubectl -n "${ns}" run "${pod}" --restart=Never --image=curlimages/curl:8.11.1 --command --quiet \ + -- curl -sS -o /dev/null -w '%{http_code}' --max-time 15 "${ep}/minio/health/live" >/dev/null 2>&1; then + local i=0 phase="" + while [ "$i" -lt 90 ]; do + phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)" + case "${phase}" in Succeeded|Failed) break ;; esac + sleep 5; i=$((i+5)) + done + local out; out="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)" + info "pod phase=${phase:-unknown} response=${out:-}" + case "${out}" in + 200|2??) info "REACHABLE: ${ep} answered from inside the cluster." ;; + *) warn "NOT REACHABLE from inside the cluster (phase=${phase}, body='${out}'). + Velero would install cleanly and then fail every backup at RUNTIME — a tier that looks healthy + while no backup exists. Fix routing, or give this environment its own endpoint, before + resuming velero." ;; + esac + else + warn "could not create the probe pod (admission policy? image pull?). Reachability UNKNOWN." + fi +} + step_resume() { say "RESUME TIERS (dependency order)" local n s3; s3="$(cvar S3_BACKUP_ENDPOINT)" @@ -467,13 +515,14 @@ step_verify() { for s in "${STEPS[@]}"; do case "$s" in preflight) step_preflight ;; + s3check) step_s3check ;; crds) step_crds ;; unwedge) step_unwedge ;; externaldns) step_externaldns ;; resume) step_resume ;; verify) step_verify ;; all) step_preflight; step_crds; step_unwedge; step_externaldns; step_resume; step_verify ;; - *) die "unknown step '$s' (preflight crds unwedge externaldns resume verify all)" ;; + *) die "unknown step '$s' (preflight crds unwedge externaldns s3check resume verify all)" ;; esac done diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index 2f6c4cd..c8b8f37 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -212,6 +212,28 @@ data: VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ VELERO_RETENTION: "168h" # dev short | prod long + # BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each + # other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev. + # + # Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before: + # + # VELERO — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and + # `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into + # itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would + # therefore adopt dev's backups and delete them on schedule. + # CNPG — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the + # CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`, + # so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers + # interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and + # dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two. + # + # THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE, and must stay that way: dev's existing + # backups live at the bucket root / at `s3://cnpg`. Giving dev a prefix would ORPHAN the very + # backups its proven restore depends on — the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays + # `default`. Prod, which has no history to preserve, takes the namespaced paths. + VELERO_BACKUP_PREFIX: "" # dev: bucket root (existing layout) | prod: "prod" + CNPG_BACKUP_PATH: "s3://cnpg" # dev: bucket root (existing layout) | prod: "s3://cnpg/prod" + # --- placement --- PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles PLATFORM_NODE_SELECTOR_VALUE: "linux" diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 21fb94a..ccae77d 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -383,10 +383,38 @@ patches: - op: replace path: /data/CRAFTY_L4_VIP value: "192.168.2.22" - # Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the - # network — a prod backup written there is not off-cluster DR and would mix two environments' - # PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port). + # ------------------------------------------------------------------------------------------ + # OFF-CLUSTER DR TARGET — operator ruling 2026-09-05: "the S3 backup location is the same as + # Dev." Prod therefore points at the on-Proxmox MinIO, and the TBD placeholder is retired. + # Stated EXPLICITLY rather than omitting the op: an omitted op would inherit the same value + # silently, and a shared backup target is exactly the kind of decision that must be visible + # in the overlay that owns it. + # ------------------------------------------------------------------------------------------ + # *** NOT YET REACHABLE — velero and the CNPG backups CANNOT be enabled on prod today. *** + # Measured 2026-09-05 from a throwaway pod INSIDE the prod cluster (not from the workstation, + # which proves nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` + # → connection timed out after 15s. pfSense does not route prod's pod network to the dev LAN's + # MinIO. Until that routing exists, Velero and CNPG would install cleanly and fail every + # backup at RUNTIME — a tier that reports healthy while no backup exists. + # Re-test with: bootstrap/standup.sh ... --s3-endpoint http://192.168.1.9:9000 s3check + # + # SHARING THE ENDPOINT IS SAFE ONLY BECAUSE OF THE TWO PATH KNOBS BELOW. Nothing about the + # paths was per-environment before this change: Velero used one unprefixed bucket, and every + # CNPG Cluster used `s3://cnpg` with serverName defaulting to the CLUSTER NAME — and both + # environments have keycloak-db / nextcloud-db / semaphore-db. Enabling prod against dev's + # MinIO without these knobs would (a) let prod adopt and TTL-delete dev's Velero backups and + # (b) interleave two Postgres servers' WAL into one Barman stream, CORRUPTING dev's chain and + # destroying the PITR that R-STATE-03 proved. See clusters/base/cluster-vars.yaml. - op: replace path: /data/S3_BACKUP_ENDPOINT - value: "TBD-PROD-S3-BACKUP-ENDPOINT" + value: "http://192.168.1.9:9000" + # Prod's Velero objects nest under `prod/`; dev keeps the bucket root where its backups already + # are (renaming dev's path would orphan the backups its proven restore depends on). + - op: replace + path: /data/VELERO_BACKUP_PREFIX + value: "prod" + # Prod's Barman objects nest under s3://cnpg/prod//; dev keeps s3://cnpg//. + - op: replace + path: /data/CNPG_BACKUP_PATH + value: "s3://cnpg/prod" # TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist. diff --git a/infra/PROD-STANDUP.md b/infra/PROD-STANDUP.md index 3f3e066..ba76121 100644 --- a/infra/PROD-STANDUP.md +++ b/infra/PROD-STANDUP.md @@ -261,16 +261,52 @@ subtle because the *apex* appears to work: dev's `*.home.keithjackson.dev` wildc ## Blockers -### 1. `S3_BACKUP_ENDPOINT` is an intentional placeholder — operator decision required +### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage** -Prod sets `TBD-PROD-S3-BACKUP-ENDPOINT` on purpose: base defaults are the **dev** values, so an -omitted override would make prod silently inherit dev's address (`http://192.168.1.9:9000`), which -prod cannot reach anyway. A loud placeholder beats a silent wrong answer. +**Operator ruling 2026-09-05: "the S3 backup location is the same as Dev."** Prod's +`S3_BACKUP_ENDPOINT` is now `http://192.168.1.9:9000`, and the `TBD-PROD-*` placeholder is retired. -It is consumed by **`velero` and `apps-wave-0`**. `velero` is skipped by C5. **`apps-wave-0` still -applies** — the token *is* defined, so strict substitution succeeds — but anything writing to that -endpoint fails at **runtime**, not at apply time. **A green `apps-wave-0` does not mean backups -work.** Until a real prod endpoint is set, treat prod as having **no** backup coverage. +**It is not reachable from prod.** Measured from a throwaway pod **inside the prod cluster** — not +from the workstation, which proves nothing about pod egress: + +``` +curl http://192.168.1.9:9000/minio/health/live → connection timed out after 15s +``` + +pfSense does not route prod's pod network to the dev LAN's MinIO. **Operator action required.** +Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so. + +Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the +token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at +**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup +coverage.** + +#### Sharing one MinIO is only safe because of two new knobs + +Nothing about the backup *paths* was per-environment before 2026-09-05. Enabling prod against dev's +MinIO without these would have been the external-dns incident again, in a worse place: + +| | before | after | +|---|---|---| +| Velero | `bucket: velero`, **no prefix**, both clusters run schedules named `daily` | `prefix: ${VELERO_BACKUP_PREFIX}` — dev `""`, prod `prod` | +| CNPG | `destinationPath: s3://cnpg`, serverName defaults to the **Cluster name** | `${CNPG_BACKUP_PATH}` — dev `s3://cnpg`, prod `s3://cnpg/prod` | + +- **Velero** syncs backups it finds in its BSL into itself as `Backup` CRs and **garbage-collects them + when their TTL expires**. Prod would have adopted dev's backups and deleted them on schedule. +- **CNPG is the worse one.** Both environments have `keycloak-db`, `nextcloud-db` and `semaphore-db`, + so prod's Postgres would archive WAL into the *exact prefix* dev's does. Two servers interleaving + WAL into one Barman stream **corrupts the chain and destroys PITR** — and dev's restore is + **proven** (R-STATE-03). + +**The base values are dev's current layout on purpose and must stay that way.** Dev's existing +backups live at the bucket root and at `s3://cnpg`; giving dev a prefix would **orphan the very +backups its proven restore depends on** — the same reasoning that keeps dev's +`EXTERNALDNS_TXT_OWNER_ID` at `default`. Prod has no history to preserve, so it takes the nested paths. + +> **Delivery note.** Prod reconciles the bootstrap Gitea's `main` at `c8c1d29`. This change sits on a +> later branch that also carries the DNS inversion, which must **not** reach prod (it would move prod +> to the apex and collide with dev). Delivering it needs a `c8c1d29`-based branch carrying only these +> commits. No urgency: velero cannot be enabled until the routing above is fixed. ### 2. Prod is on `prod.home.keithjackson.dev`, not the apex diff --git a/platform/keycloak/postgres-cluster.yaml b/platform/keycloak/postgres-cluster.yaml index 4a71b21..639e5df 100644 --- a/platform/keycloak/postgres-cluster.yaml +++ b/platform/keycloak/postgres-cluster.yaml @@ -45,7 +45,11 @@ spec: backup: retentionPolicy: "30d" # keep 30 days of base backups + WAL (PITR window); prunes older barmanObjectStore: - destinationPath: s3://cnpg + # PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER + # NAME — and both environments have a `keycloak-db`. Sharing one MinIO with a shared path + # would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and + # destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests. + destinationPath: ${CNPG_BACKUP_PATH} endpointURL: ${S3_BACKUP_ENDPOINT} s3Credentials: accessKeyId: diff --git a/platform/velero/helmrelease.yaml b/platform/velero/helmrelease.yaml index b1fb799..477b5a6 100644 --- a/platform/velero/helmrelease.yaml +++ b/platform/velero/helmrelease.yaml @@ -73,6 +73,12 @@ spec: - name: default provider: aws bucket: velero + # PER-ENVIRONMENT PREFIX (cluster-vars). Empty on dev = the bucket root it already uses; + # `prod` on prod. Without it both clusters share one BackupStorageLocation — and Velero + # SYNCS backups it finds there into itself as Backup CRs and GARBAGE-COLLECTS them when + # their TTL expires, so prod would adopt and then delete dev's backups. Both clusters also + # run schedules named `daily`/`daily-resources-only`, so even the object keys would collide. + prefix: ${VELERO_BACKUP_PREFIX} default: true config: region: us-east-1 # placeholder; MinIO ignores it (S3 clients require one)