fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
gate / gate (pull_request) Successful in 1m53s
gate / conformance (pull_request) Has been skipped

Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on
192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob
load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through.
Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do.

BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked
inert (dev is decommissioned); it carries the real address only so nothing in
the tree renders a host that no longer exists.

THE OPERATIONAL POINT, and why this is not just a value change: a bound
PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07:

  spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is
  immutable after creation

so Flux's apply FAILS rather than updating, and the pods keep their old, dead
mount. The five PVs must be deleted and recreated -- safe, because Retain never
touches data and render.py emits a pre-set claimRef so each PV rebinds its own
PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR
action; this commit does not perform it.

Pairs with the hazard already on the page: spec.nfs.server is unvalidated at
create (any string binds) and immutable at update. A wrong value is accepted
silently and then cannot be corrected in place.

docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured
exports, the service settings that decide whether uid 1000 survives the wire,
the three stale ZFS sharenfs properties that had to be cleared, and the ACL
hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor
propagates to the whole future library). Section 7 rewritten now that the
placeholder is gone, section 9 retargeted from dev to prod, section 10 new.
Sections 2/3/4/6 are unchanged by the move.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-07 01:15:04 -05:00
1 parent ad37acdf53
commit 0747849837
7 files changed
+369 -188

No files matched your search

@@ -199,7 +199,7 @@ data:
"id": 15,
"type": "timeseries",
"title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
"description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
"description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
{"color": "green", "value": null}, {"color": "red", "value": 0.01}