From 2590bc4317ee3c223e6cb42750fe6f28c009693e Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 23:47:39 -0500 Subject: [PATCH] =?UTF-8?q?feat(env):=20BULK=5FNFS=5FSERVER=20=E2=80=94=20?= =?UTF-8?q?the=20one=20knob=20prod=20cannot=20answer=20yet,=20and=20why=20?= =?UTF-8?q?it=20is=20a=20placeholder?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one interface (vmbr0 192.168.1.9/24, on pfSense's LAN). dev 192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so base's default is the dev value per this file's existing convention, and dev restates it. prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention. Two findings recorded in the prod overlay because they change what the operator has to do: * THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this workstation (192.168.2.132) through the existing port-forward: the storage host logged the connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address (`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision. * ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol — no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing else. NFSv3 would have needed four ports, three of them on ephemeral numbers. *** AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. *** Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts — MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here, so the guard has to be procedural: set the value before prod reconciles a revision carrying the bulk volumes. Dev renders a working value and is where these manifests are meant to be proven. Related finding, NOT fixed here because the file does not exist on main: the static guard this convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent from main (it lives on unmerged prod branches), so conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover this value automatically, since the detector's SEED vocabulary already matches "TBD". Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9; `kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0. Co-Authored-By: Claude Opus 5 --- clusters/base/cluster-vars.yaml | 22 ++++++++++++++++++ clusters/dev/kustomization.yaml | 9 +++++++ clusters/prod/kustomization.yaml | 40 ++++++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+) diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index 2762d67..5df3c1f 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -160,6 +160,28 @@ data: UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi) CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports + # --- Bulk (external NFS) storage host (docs/04 bulk tier) --------------------------------------- + # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media + # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive + # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's + # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is + # declared in the ServiceDefinition, because the path is identical in every environment and only + # the ADDRESS is not. + # + # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense. + # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN. + # dev nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one + # NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in + # /etc/exports), so this path is proven, not theoretical. Base default = the dev value, + # per the convention of the LAN block above. + # prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today; + # 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening + # the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give + # DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route). + # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the + # blast radius of shipping it unset, which is NOT the usual "fails to apply". + BULK_NFS_SERVER: "192.168.1.9" + # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style. # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db, # nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port. diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index cb2cb1a..0e38564 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -225,6 +225,15 @@ patches: - op: replace path: /data/CRAFTY_L4_VIP value: "192.168.1.55" + # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes + # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes). + # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention. + # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED + # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=` + # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses. + - op: replace + path: /data/BULK_NFS_SERVER + value: "192.168.1.9" # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG # barmanObjectStore and Velero's backupStorageLocation write to. # diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 78c5da0..697297f 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -392,6 +392,46 @@ patches: - op: replace path: /data/CRAFTY_L4_VIP value: "192.168.2.22" + # ------------------------------------------------------------------------------------------ + # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. *** + # ------------------------------------------------------------------------------------------ + # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr, + # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud). + # + # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface, + # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN. + # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102; + # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate + # outcomes give DIFFERENT values here: + # port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102" + # a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9" + # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess + # ever became reachable — mounts something unintended. So it stays a placeholder. + # + # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and + # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4 + # over v3 was in part this decision's cost: v3 would have needed four ports and three of them + # on ephemeral numbers. + # + # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes. + # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward: + # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN + # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via + # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node + # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md. + # + # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. *** + # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this + # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The + # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server + # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in + # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at + # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles + # a revision containing the bulk volumes; the dev overlay renders a working value, which is + # where these manifests are meant to be proven first. + - op: replace + path: /data/BULK_NFS_SERVER + value: "TBD-PROD-BULK-NFS-SERVER" # ========================================================================================== # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07). # ==========================================================================================