Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(env): BULK_NFS_SERVER — the one knob prod cannot answer yet, and why it is a placeholder

Browse Source 
The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the
two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one
interface (vmbr0 192.168.1.9/24, on pfSense's LAN).

  dev  192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS
       export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so
       base's default is the dev value per this file's existing convention, and dev restates it.
  prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006
       answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path
       is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here:
       192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds
       TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention.

Two findings recorded in the prod overlay because they change what the operator has to do:

  * THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this
    workstation (192.168.2.132) through the existing port-forward: the storage host logged the
    connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address
    (`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1).
    pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either
    way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision.
  * ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol —
    no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing
    else. NFSv3 would have needed four ports, three of them on ephemeral numbers.

*** AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. ***
Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's
spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the
PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts —
MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr
and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here,
so the guard has to be procedural: set the value before prod reconciles a revision carrying the
bulk volumes. Dev renders a working value and is where these manifests are meant to be proven.

Related finding, NOT fixed here because the file does not exist on main: the static guard this
convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent
from main (it lives on unmerged prod branches), so
conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate
currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an
arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover
this value automatically, since the detector's SEED vocabulary already matches "TBD".

Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9;
`kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 23:47:39 -05:00
1 parent cec550045b
commit 2590bc4317
3 files changed
+71
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

clusters/prod/kustomization.yaml
+40
View File
@@ -392,6 +392,46 @@ patches:
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.2.22"
# ------------------------------------------------------------------------------------------
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
# ------------------------------------------------------------------------------------------
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
#
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
# outcomes give DIFFERENT values here:
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
# ever became reachable — mounts something unintended. So it stays a placeholder.
#
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
# on ephemeral numbers.
#
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
#
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
# where these manifests are meant to be proven first.
- op: replace
path: /data/BULK_NFS_SERVER
value: "TBD-PROD-BULK-NFS-SERVER"
# ==========================================================================================
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
# ==========================================================================================