feat(backup): prod shares dev's MinIO — plus the path tenancy that makes it survivable

Operator ruling: "the S3 backup location is the same as Dev." Prod's S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would inherit the same value silently, and a shared backup target must be visible in the overlay that owns it. SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS was per-environment, and enabling prod against dev's MinIO would have been the 2026-09-04 external-dns incident again, in a worse place: VELERO — bucket `velero`, NO prefix, and both clusters run schedules named `daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would have adopted dev's backups and then deleted them on schedule. CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the CLUSTER NAME. Both environments have keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not. Two knobs, authored once, environments differing only by scoped overrides (R-ENV-02): VELERO_BACKUP_PREFIX dev "" | prod "prod" CNPG_BACKUP_PATH dev "s3://cnpg" | prod "s3://cnpg/prod" THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very backups its proven restore depends on — the same reasoning that keeps dev's EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve. *** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. *** Measured from a throwaway pod INSIDE the prod cluster (the workstation proves nothing about pod egress, and a workstation-side reading has already misled this project once): curl http://192.168.1.9:9000/minio/health/live → connection timed out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO. OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN regardless — the token IS defined so strict substitution succeeds — while every write fails at RUNTIME. A green apps-wave-0 does not mean backups work. New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster pod and fails closed, with `--s3-endpoint <url>` to test a CANDIDATE before authoring it into cluster-vars. That is how the unreachability above was established rather than assumed. VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions, and neither new knob appears in the dead-knob advisory list, so both are really consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's prefix carry the tokens; dev resolves to its existing layout, prod to the nested paths. DELIVERY: prod reconciles the bootstrap Gitea's main at c8c1d29. This branch also carries the DNS inversion, which must NOT reach prod. Delivering needs a c8c1d29-based branch with only these commits — no urgency while the routing is still closed. Refs R-STATE-03, R-ENV-02, R-IDP-05. Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 10:59:50 -05:00
1 parent e6bcecc78c
commit b80e8eef82
@@ -261,16 +261,52 @@ subtle because the *apex* appears to work: dev's `*.home.keithjackson.dev` wildc

 ## Blockers

-### 1. `S3_BACKUP_ENDPOINT` is an intentional placeholder — operator decision required
+### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**

-Prod sets `TBD-PROD-S3-BACKUP-ENDPOINT` on purpose: base defaults are the **dev** values, so an
-omitted override would make prod silently inherit dev's address (`http://192.168.1.9:9000`), which
-prod cannot reach anyway. A loud placeholder beats a silent wrong answer.
+**Operator ruling 2026-09-05: "the S3 backup location is the same as Dev."** Prod's
+`S3_BACKUP_ENDPOINT` is now `http://192.168.1.9:9000`, and the `TBD-PROD-*` placeholder is retired.

-It is consumed by **`velero` and `apps-wave-0`**. `velero` is skipped by C5. **`apps-wave-0` still
-applies** — the token *is* defined, so strict substitution succeeds — but anything writing to that
-endpoint fails at **runtime**, not at apply time. **A green `apps-wave-0` does not mean backups
-work.** Until a real prod endpoint is set, treat prod as having **no** backup coverage.
+**It is not reachable from prod.** Measured from a throwaway pod **inside the prod cluster** — not
+from the workstation, which proves nothing about pod egress:
+
+```
+curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s
+```
+
+pfSense does not route prod's pod network to the dev LAN's MinIO. **Operator action required.**
+Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.
+
+Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
+token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
+**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
+coverage.**
+
+#### Sharing one MinIO is only safe because of two new knobs
+
+Nothing about the backup *paths* was per-environment before 2026-09-05. Enabling prod against dev's
+MinIO without these would have been the external-dns incident again, in a worse place:
+
+| | before | after |
+|---|---|---|
+| Velero | `bucket: velero`, **no prefix**, both clusters run schedules named `daily` | `prefix: ${VELERO_BACKUP_PREFIX}` — dev `""`, prod `prod` |
+| CNPG | `destinationPath: s3://cnpg`, serverName defaults to the **Cluster name** | `${CNPG_BACKUP_PATH}` — dev `s3://cnpg`, prod `s3://cnpg/prod` |
+
+- **Velero** syncs backups it finds in its BSL into itself as `Backup` CRs and **garbage-collects them
+  when their TTL expires**. Prod would have adopted dev's backups and deleted them on schedule.
+- **CNPG is the worse one.** Both environments have `keycloak-db`, `nextcloud-db` and `semaphore-db`,
+  so prod's Postgres would archive WAL into the *exact prefix* dev's does. Two servers interleaving
+  WAL into one Barman stream **corrupts the chain and destroys PITR** — and dev's restore is
+  **proven** (R-STATE-03).
+
+**The base values are dev's current layout on purpose and must stay that way.** Dev's existing
+backups live at the bucket root and at `s3://cnpg`; giving dev a prefix would **orphan the very
+backups its proven restore depends on** — the same reasoning that keeps dev's
+`EXTERNALDNS_TXT_OWNER_ID` at `default`. Prod has no history to preserve, so it takes the nested paths.
+
+> **Delivery note.** Prod reconciles the bootstrap Gitea's `main` at `c8c1d29`. This change sits on a
+> later branch that also carries the DNS inversion, which must **not** reach prod (it would move prod
+> to the apex and collide with dev). Delivering it needs a `c8c1d29`-based branch carrying only these
+> commits. No urgency: velero cannot be enabled until the routing above is fixed.

 ### 2. Prod is on `prod.home.keithjackson.dev`, not the apex