feat(backup): prod shares dev's MinIO — plus the path tenancy that makes it survivable

Operator ruling: "the S3 backup location is the same as Dev." Prod's
S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder
is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would
inherit the same value silently, and a shared backup target must be visible in
the overlay that owns it.

SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS
was per-environment, and enabling prod against dev's MinIO would have been the
2026-09-04 external-dns incident again, in a worse place:

  VELERO — bucket `velero`, NO prefix, and both clusters run schedules named
    `daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL
    into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would
    have adopted dev's backups and then deleted them on schedule.
  CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending
    serverName, which defaults to the CLUSTER NAME. Both environments have
    keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive
    WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one
    Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is
    PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not.

Two knobs, authored once, environments differing only by scoped overrides
(R-ENV-02):
  VELERO_BACKUP_PREFIX  dev ""          | prod "prod"
  CNPG_BACKUP_PATH      dev "s3://cnpg" | prod "s3://cnpg/prod"

THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live
at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very
backups its proven restore depends on — the same reasoning that keeps dev's
EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve.

*** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. ***
Measured from a throwaway pod INSIDE the prod cluster (the workstation proves
nothing about pod egress, and a workstation-side reading has already misled this
project once): curl http://192.168.1.9:9000/minio/health/live → connection timed
out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO.
OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN
regardless — the token IS defined so strict substitution succeeds — while every
write fails at RUNTIME. A green apps-wave-0 does not mean backups work.

New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster
pod and fails closed, with `--s3-endpoint <url>` to test a CANDIDATE before
authoring it into cluster-vars. That is how the unreachability above was
established rather than assumed.

VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions,
and neither new knob appears in the dead-knob advisory list, so both are really
consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's
prefix carry the tokens; dev resolves to its existing layout, prod to the nested
paths.

DELIVERY: prod reconciles the bootstrap Gitea's main at c8c1d29. This branch also
carries the DNS inversion, which must NOT reach prod. Delivering needs a
c8c1d29-based branch with only these commits — no urgency while the routing is
still closed.

Refs R-STATE-03, R-ENV-02, R-IDP-05.

Co-Authored-By: Claude Opus 5 <[email protected]>

bootstrap/standup.sh

@@ -18,7 +18,7 @@
 # USAGE
 #   bootstrap/standup.sh --kubeconfig <path> --expect-server <url> [--confirm-destructive] <step...>
 #
-#   steps:  preflight  crds  unwedge  externaldns  resume  verify   (or: all)
+#   steps:  preflight  crds  unwedge  externaldns  s3check  resume  verify   (or: all)
 #
 #   --expect-server is REQUIRED and is compared against the live API server before ANY command runs.
 #   Pointing this at the wrong cluster is the one mistake with unbounded blast radius: dev is live and
@@ -29,13 +29,14 @@
 set -euo pipefail
 
 # ---------------------------------------------------------------------------- args
-KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0
+KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0 S3_ENDPOINT_OVERRIDE=""
 STEPS=()
 while [ $# -gt 0 ]; do
   case "$1" in
     --kubeconfig)          KUBECONFIG_ARG="${2:?--kubeconfig needs a value}"; shift 2 ;;
     --expect-server)       EXPECT_SERVER="${2:?--expect-server needs a value}"; shift 2 ;;
     --confirm-destructive) CONFIRM_DESTRUCTIVE=1; shift ;;
+    --s3-endpoint)         S3_ENDPOINT_OVERRIDE="${2:?--s3-endpoint needs a value}"; shift 2 ;;
     -h|--help)             sed -n '2,30p' "$0"; exit 0 ;;
     -*)                    echo "ERROR: unknown flag '$1'" >&2; exit 2 ;;
     *)                     STEPS+=("$1"); shift ;;
@@ -44,7 +45,7 @@ done
 
 [ -n "${KUBECONFIG_ARG}" ] || { echo "ERROR: --kubeconfig is required." >&2; exit 2; }
 [ -n "${EXPECT_SERVER}" ]  || { echo "ERROR: --expect-server is required (safety guard)." >&2; exit 2; }
-[ "${#STEPS[@]}" -gt 0 ]   || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns resume verify all" >&2; exit 2; }
+[ "${#STEPS[@]}" -gt 0 ]   || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns s3check resume verify all" >&2; exit 2; }
 
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/.." && pwd))"
@@ -391,6 +392,53 @@ step_externaldns() {
        Any change to a name outside it, or to a TXT entry owned by another cluster, is a REGRESSION."
 }
 
+step_s3check() {
+  say "S3 BACKUP ENDPOINT — reachability FROM INSIDE THE CLUSTER"
+  # Velero runs in-cluster, so the only reachability that counts is the pods' own. Probing from the
+  # workstation proves nothing either way: it sits on a different side of pfSense than the pod
+  # network, and a healthy result there has already misled this project once.
+  local ep live; live="$(cvar S3_BACKUP_ENDPOINT)"
+  info "live S3_BACKUP_ENDPOINT = ${live:-<unset>}"
+  if [ -n "${S3_ENDPOINT_OVERRIDE}" ]; then
+    # Probe a CANDIDATE endpoint before adopting it. The whole point is to find out whether a proposed
+    # backup target is reachable from this cluster's pods BEFORE authoring it into cluster-vars —
+    # otherwise the first evidence is silently-missing backups.
+    ep="${S3_ENDPOINT_OVERRIDE}"
+    info "probing CANDIDATE endpoint from --s3-endpoint: ${ep}"
+  else
+    ep="${live}"
+  fi
+  case "${ep}" in
+    ""|TBD-*) die "no usable endpoint to probe (live='${live}'). Pass --s3-endpoint <url> to test a candidate." ;;
+  esac
+
+  local ns=velero pod=s3check-$$
+  kubectl get ns "${ns}" >/dev/null 2>&1 || ns=default
+  info "probing from a throwaway pod in namespace '${ns}'"
+  # --restart=Never + explicit delete in a trap: a probe must never leave a pod behind on a cluster.
+  trap 'kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found --wait=false >/dev/null 2>&1 || true' RETURN
+  if kubectl -n "${ns}" run "${pod}" --restart=Never --image=curlimages/curl:8.11.1 --command --quiet \
+       -- curl -sS -o /dev/null -w '%{http_code}' --max-time 15 "${ep}/minio/health/live" >/dev/null 2>&1; then
+    local i=0 phase=""
+    while [ "$i" -lt 90 ]; do
+      phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+      case "${phase}" in Succeeded|Failed) break ;; esac
+      sleep 5; i=$((i+5))
+    done
+    local out; out="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
+    info "pod phase=${phase:-unknown} response=${out:-<none>}"
+    case "${out}" in
+      200|2??) info "REACHABLE: ${ep} answered from inside the cluster." ;;
+      *) warn "NOT REACHABLE from inside the cluster (phase=${phase}, body='${out}').
+       Velero would install cleanly and then fail every backup at RUNTIME — a tier that looks healthy
+       while no backup exists. Fix routing, or give this environment its own endpoint, before
+       resuming velero." ;;
+    esac
+  else
+    warn "could not create the probe pod (admission policy? image pull?). Reachability UNKNOWN."
+  fi
+}
+
 step_resume() {
   say "RESUME TIERS (dependency order)"
   local n s3; s3="$(cvar S3_BACKUP_ENDPOINT)"
@@ -467,13 +515,14 @@ step_verify() {
 for s in "${STEPS[@]}"; do
   case "$s" in
     preflight)   step_preflight ;;
+    s3check)     step_s3check ;;
     crds)        step_crds ;;
     unwedge)     step_unwedge ;;
     externaldns) step_externaldns ;;
     resume)      step_resume ;;
     verify)      step_verify ;;
     all)         step_preflight; step_crds; step_unwedge; step_externaldns; step_resume; step_verify ;;
-    *)           die "unknown step '$s' (preflight crds unwedge externaldns resume verify all)" ;;
+    *)           die "unknown step '$s' (preflight crds unwedge externaldns s3check resume verify all)" ;;
   esac
 done