fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
gate / gate (pull_request) Successful in 1m53s
gate / conformance (pull_request) Has been skipped

Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on
192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob
load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through.
Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do.

BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked
inert (dev is decommissioned); it carries the real address only so nothing in
the tree renders a host that no longer exists.

THE OPERATIONAL POINT, and why this is not just a value change: a bound
PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07:

  spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is
  immutable after creation

so Flux's apply FAILS rather than updating, and the pods keep their old, dead
mount. The five PVs must be deleted and recreated -- safe, because Retain never
touches data and render.py emits a pre-set claimRef so each PV rebinds its own
PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR
action; this commit does not perform it.

Pairs with the hazard already on the page: spec.nfs.server is unvalidated at
create (any string binds) and immutable at update. A wrong value is accepted
silently and then cannot be corrected in place.

docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured
exports, the service settings that decide whether uid 1000 survives the wire,
the three stale ZFS sharenfs properties that had to be cleared, and the ACL
hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor
propagates to the whole future library). Section 7 rewritten now that the
placeholder is gone, section 9 retargeted from dev to prod, section 10 new.
Sections 2/3/4/6 are unchanged by the move.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-07 01:15:04 -05:00
1 parent ad37acdf53
commit 0747849837
7 files changed
+369 -188

No files matched your search

+17 -15
View File
@@ -234,26 +234,28 @@ data:
WG_CLIENT_CIDR: "10.90.0.0/24" # remote VPN clients; .1 = server. NOT a LAN range
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
# library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
# THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
# (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
# (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
# `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
# declared in the ServiceDefinition, because the path is identical in every environment and only
# the ADDRESS is not.
#
# WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
# The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
# dev nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
# NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
# /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
# per the convention of the LAN block above.
# prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
# 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
# the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
# DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
# So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
# blast radius of shipping it unset, which is NOT the usual "fails to apply".
BULK_NFS_SERVER: "192.168.1.9"
# IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
# OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
# vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
# only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
# TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
# values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
# 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
# this stays a knob only because the renderer reads it per-environment.
#
# *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
# immutable — measured by server dry-run 2026-10-07:
# spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
# so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
# deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
BULK_NFS_SERVER: "192.168.2.30"
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,