feat(env): BULK_NFS_SERVER — the one knob prod cannot answer yet, and why it is a placeholder

The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the
two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one
interface (vmbr0 192.168.1.9/24, on pfSense's LAN).

  dev  192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS
       export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so
       base's default is the dev value per this file's existing convention, and dev restates it.
  prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006
       answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path
       is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here:
       192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds
       TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention.

Two findings recorded in the prod overlay because they change what the operator has to do:

  * THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this
    workstation (192.168.2.132) through the existing port-forward: the storage host logged the
    connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address
    (`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1).
    pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either
    way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision.
  * ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol —
    no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing
    else. NFSv3 would have needed four ports, three of them on ephemeral numbers.

*** AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. ***
Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's
spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the
PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts —
MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr
and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here,
so the guard has to be procedural: set the value before prod reconciles a revision carrying the
bulk volumes. Dev renders a working value and is where these manifests are meant to be proven.

Related finding, NOT fixed here because the file does not exist on main: the static guard this
convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent
from main (it lives on unmerged prod branches), so
conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate
currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an
arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover
this value automatically, since the detector's SEED vocabulary already matches "TBD".

Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9;
`kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 23:47:39 -05:00
1 parent cec550045b
commit 2590bc4317
3 files changed
+71

No files matched your search

+9
View File
@@ -225,6 +225,15 @@ patches:
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.1.55"
# Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
# (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
# Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
# Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
# end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
# change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
- op: replace
path: /data/BULK_NFS_SERVER
value: "192.168.1.9"
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
# barmanObjectStore and Velero's backupStorageLocation write to.
#