Files
home-network-k3s-impl/clusters/prod/kustomization.yaml
T
jakman85andClaude Opus 5 b80e8eef82 feat(backup): prod shares dev's MinIO — plus the path tenancy that makes it survivable
Operator ruling: "the S3 backup location is the same as Dev." Prod's
S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder
is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would
inherit the same value silently, and a shared backup target must be visible in
the overlay that owns it.

SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS
was per-environment, and enabling prod against dev's MinIO would have been the
2026-09-04 external-dns incident again, in a worse place:

  VELERO — bucket `velero`, NO prefix, and both clusters run schedules named
    `daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL
    into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would
    have adopted dev's backups and then deleted them on schedule.
  CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending
    serverName, which defaults to the CLUSTER NAME. Both environments have
    keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive
    WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one
    Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is
    PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not.

Two knobs, authored once, environments differing only by scoped overrides
(R-ENV-02):
  VELERO_BACKUP_PREFIX  dev ""          | prod "prod"
  CNPG_BACKUP_PATH      dev "s3://cnpg" | prod "s3://cnpg/prod"

THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live
at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very
backups its proven restore depends on — the same reasoning that keeps dev's
EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve.

*** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. ***
Measured from a throwaway pod INSIDE the prod cluster (the workstation proves
nothing about pod egress, and a workstation-side reading has already misled this
project once): curl http://192.168.1.9:9000/minio/health/live → connection timed
out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO.
OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN
regardless — the token IS defined so strict substitution succeeds — while every
write fails at RUNTIME. A green apps-wave-0 does not mean backups work.

New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster
pod and fails closed, with `--s3-endpoint <url>` to test a CANDIDATE before
authoring it into cluster-vars. That is how the unreachability above was
established rather than assumed.

VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions,
and neither new knob appears in the dead-knob advisory list, so both are really
consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's
prefix carry the tokens; dev resolves to its existing layout, prod to the nested
paths.

DELIVERY: prod reconciles the bootstrap Gitea's main at c8c1d29. This branch also
carries the DNS inversion, which must NOT reach prod. Delivering needs a
c8c1d29-based branch with only these commits — no urgency while the routing is
still closed.

Refs R-STATE-03, R-ENV-02, R-IDP-05.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 10:59:50 -05:00

421 lines
25 KiB
YAML

---
# clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod
# (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production").
#
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
#
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
patches:
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: platform
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-0
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-1
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps-wave-2
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: oauth2-proxy
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: headlamp
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: crafty
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: velero
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
#
# ############################################################################################
# ### ###
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
# ### ###
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
# ### ###
# ############################################################################################
#
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
# REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
# which changes WHY the blocker holds but not THAT it holds:
# 1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
# pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
# inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
# BEFORE dev is retired, or prod loses its source of truth.
# NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
# gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
# the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
# cutover below a one-line change once prod actually runs Gitea.
# 2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
# easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
# ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
# prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
# listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
# to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
# remains is purely reason 1 — the Gitea is not there YET.
#
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
# its Phase-6 cutover (docs/05 "Source-of-truth").
#
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
#
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
# THAT FIRST: a private repo here fails exactly that way.
#
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
# through prod's own Gateway VIP with a publicly-trusted cert.
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
# 2. change the url below to
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
# clusters/dev/kustomization.yaml, is what "done" looks like);
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
# exactly as dev retained its bootstrap Gitea.
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
#
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
- patch: |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-system
namespace: flux-system
spec:
# TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
# above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
ref:
branch: main
- target:
kind: ConfigMap
name: cluster-vars
patch: |-
- op: replace
path: /data/CLUSTER_ENV
value: "prod"
# trusted, rate-limited real certs in prod — ADR-0006.
- op: replace
path: /data/ACME_CLUSTER_ISSUER
value: "letsencrypt-prod"
# ==========================================================================================
# PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
# ==========================================================================================
# OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
# (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
#
# dev -> dev.home.keithjackson.dev (dev carries the label; clusters/dev)
# prod -> home.keithjackson.dev (THE APEX — this knob)
#
# There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
# name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
# Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
#
# BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
# HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
# EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
# consequences, one of which already happened:
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
# have raced each other's `_acme-challenge` TXT records on that one zone.
#
# *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
# Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
# therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
# tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
# (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
# (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
# (verified against Cloudflare directly — a pfSense override is NOT evidence about public
# DNS; that mistake was made on 2026-09-04), AND
# (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
# that same name set while dev's still exists counts against LE's duplicate-certificate
# limit (5 per week per identical name set).
# Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
# the 2026-09-04 incident with the roles reversed.
#
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
# trusts a different Keycloak than the one prod actually runs.
- op: replace
path: /data/BASE_DOMAIN
value: "home.keithjackson.dev"
- op: replace
path: /data/EXTERNALDNS_DOMAIN_FILTER
# MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
# like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
# F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
# zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
# record DNS Name", so prod publishes NOTHING. An agent narrowed this to
# prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
# Isolation from dev does NOT come from this filter — it comes from
# (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
# (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
value: "keithjackson.dev"
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
# its own and, under policy: sync, deleted the records behind them.
#
# PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
# NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
# the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
# prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
# never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
# invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
# simply fails to publish until those records are retired, which is the correct behaviour.
# dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
# non-colliding sentinel. All three differ.
- op: replace
path: /data/EXTERNALDNS_TXT_OWNER_ID
value: "prod"
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
#
# [parity] MUST equal node_baseline_longhorn_data_path in
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
# path on the OS filesystem instead of the data partition and /var fills up.
- op: replace
path: /data/LONGHORN_DATA_PATH
value: "/data/longhorn"
# longer retention / larger volumes on prod NVMe.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
value: "30d"
- op: replace
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
value: "50Gi"
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
# size cap before 30d if series volume is high, keeping the disk from filling either way.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
value: "45GiB"
- op: replace
path: /data/MONITORING_LOKI_PVC_SIZE
value: "50Gi"
# HA control-plane addons in prod.
- op: replace
path: /data/KEYCLOAK_REPLICAS
value: "2"
- op: replace
path: /data/OAUTH2_PROXY_REPLICAS
value: "2"
- op: replace
path: /data/VELERO_RETENTION
value: "720h"
# ==========================================================================================
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
# ==========================================================================================
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
#
# ------------------------------------------------------------------------------------------
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
# ------------------------------------------------------------------------------------------
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
#
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
# LB-IPAM must never be able to hand it to a Service.
- op: replace
path: /data/GATEWAY_VIP
value: "192.168.2.20"
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
- op: replace
path: /data/CILIUM_LB_POOL_START
value: "192.168.2.20"
- op: replace
path: /data/CILIUM_LB_POOL_STOP
value: "192.168.2.30"
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
# Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
# platform/cilium/helmrelease.yaml. Not yet decided.
- op: replace
path: /data/K3S_API_HOST
value: "192.168.2.10"
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
- op: replace
path: /data/UNIFI_L4_VIP
value: "192.168.2.21"
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.2.22"
# ------------------------------------------------------------------------------------------
# OFF-CLUSTER DR TARGET — operator ruling 2026-09-05: "the S3 backup location is the same as
# Dev." Prod therefore points at the on-Proxmox MinIO, and the TBD placeholder is retired.
# Stated EXPLICITLY rather than omitting the op: an omitted op would inherit the same value
# silently, and a shared backup target is exactly the kind of decision that must be visible
# in the overlay that owns it.
# ------------------------------------------------------------------------------------------
# *** NOT YET REACHABLE — velero and the CNPG backups CANNOT be enabled on prod today. ***
# Measured 2026-09-05 from a throwaway pod INSIDE the prod cluster (not from the workstation,
# which proves nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live`
# → connection timed out after 15s. pfSense does not route prod's pod network to the dev LAN's
# MinIO. Until that routing exists, Velero and CNPG would install cleanly and fail every
# backup at RUNTIME — a tier that reports healthy while no backup exists.
# Re-test with: bootstrap/standup.sh ... --s3-endpoint http://192.168.1.9:9000 s3check
#
# SHARING THE ENDPOINT IS SAFE ONLY BECAUSE OF THE TWO PATH KNOBS BELOW. Nothing about the
# paths was per-environment before this change: Velero used one unprefixed bucket, and every
# CNPG Cluster used `s3://cnpg` with serverName defaulting to the CLUSTER NAME — and both
# environments have keycloak-db / nextcloud-db / semaphore-db. Enabling prod against dev's
# MinIO without these knobs would (a) let prod adopt and TTL-delete dev's Velero backups and
# (b) interleave two Postgres servers' WAL into one Barman stream, CORRUPTING dev's chain and
# destroying the PITR that R-STATE-03 proved. See clusters/base/cluster-vars.yaml.
- op: replace
path: /data/S3_BACKUP_ENDPOINT
value: "http://192.168.1.9:9000"
# Prod's Velero objects nest under `prod/`; dev keeps the bucket root where its backups already
# are (renaming dev's path would orphan the backups its proven restore depends on).
- op: replace
path: /data/VELERO_BACKUP_PREFIX
value: "prod"
# Prod's Barman objects nest under s3://cnpg/prod/<cluster>/; dev keeps s3://cnpg/<cluster>/.
- op: replace
path: /data/CNPG_BACKUP_PATH
value: "s3://cnpg/prod"
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.