jakman85andClaude Opus 5
0747849837
Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on 192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through. Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do. BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked inert (dev is decommissioned); it carries the real address only so nothing in the tree renders a host that no longer exists. THE OPERATIONAL POINT, and why this is not just a value change: a bound PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation so Flux's apply FAILS rather than updating, and the pods keep their old, dead mount. The five PVs must be deleted and recreated -- safe, because Retain never touches data and render.py emits a pre-set claimRef so each PV rebinds its own PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR action; this commit does not perform it. Pairs with the hazard already on the page: spec.nfs.server is unvalidated at create (any string binds) and immutable at update. A wrong value is accepted silently and then cannot be corrected in place. docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured exports, the service settings that decide whether uid 1000 survives the wire, the three stale ZFS sharenfs properties that had to be cleared, and the ACL hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor propagates to the whole future library). Section 7 rewritten now that the placeholder is gone, section 9 retargeted from dev to prod, section 10 new. Sections 2/3/4/6 are unchanged by the move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
5.6 KiB
5.6 KiB
date, branch, title
| date | branch | title |
|---|---|---|
| 2026-10-07 | fix/bulk-nfs-truenas | Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled |
What changed
Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
Edition at 192.168.2.30, which sits on prod's own 192.168.2.0/24. BULK_NFS_SERVER moves from
192.168.2.102 (the pfSense DNAT prod used to mount through) to 192.168.2.30.
clusters/prod/kustomization.yaml—BULK_NFS_SERVER→192.168.2.30. The comment block that justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are replaced with the measured export configuration and the recreate requirement below.clusters/base/cluster-vars.yaml— base default192.168.1.9→192.168.2.30. That host no longer exists.clusters/dev/kustomization.yaml— same; marked explicitly inert, since dev is decommissioned and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.docs/BULK-NFS-STORAGE.md— §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFSsharenfsproperties that had to be cleared, and the ACL hazard); §7 rewritten now that the placeholder is gone; §9 retargeted from dev to prod; new §10 documents the PV delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.catalog/qbittorrent.yaml,platform/monitoring/node-pressure-dashboard.yaml— two comment/prose references to192.168.2.102as the live export address.
The headline operational fact: repointing this knob is NOT a plain reconcile. A bound PV's NFS source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action; it is not performed by this PR.
What I verified
On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:
zfs list—Storage→/mnt/Storage,Storage/plex→/mnt/Storage/plex,Storage/nextcloud→/mnt/Storage/nextcloud. The PV paths resolve unchanged.- Ownership intact:
/mnt/Storage/plex,tv/,downloads/,movies/,downloads/complete/,nextcloud/nextcloud/alldrwxr-xr-x 1000:3000.movies/1171 entries andtv/107, against 1165 / 107 measured on Proxmox. acltype=nfsv4(SCALE's built-in default,SOURCE: default),aclinherit=passthrough, and no non-trivial ACLs — mode bits still govern.- Three stale
sharenfsdataset properties (plex,nextcloud,docker-volumes) carried over from Proxmox; cleared withzfs inherit, re-checked empty. exportfs -v: both shares to192.168.2.10/.11/.12withsec=sys,secure,root_squash,no_all_squash,rwon plex androon nextcloud./proc/fs/nfsd/versions→+3 +4 +4.1 +4.2, so the PVs'nfsvers=4.1negotiates.- A defect was found and fixed during setup: the plex share listed
192.168.2.13— an address no node holds — and so omitted192.168.2.12(k3s-prod-3). Corrected and re-verified.
From the cluster:
- Node InternalIPs are
192.168.2.10/.11/.12(kubectl get nodes), confirming.13was wrong. - All five NFS PVs currently name
192.168.2.102. 192.168.2.30:2049and:111accept connections;192.168.2.102:2049does not.spec.nfsimmutability, by server-side dry-run — this is the finding that shapes the change:spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation.- The pods report
Readywhile their mounts are dead: astat /mediaanswers from attribute cache, but areaddirblocks indefinitely (the PVs mounthard). Hence "scale to zero first" in §10.
Gate: see the PR; run in the pinned toolbox from this worktree.
What I did NOT verify
- Nothing has mounted 192.168.2.30. No pod has touched the new server — the PVs still point at the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000 write, the hardlink/EXDEV check, Plex's read-only posture) is outstanding, not passing. The equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
- The §10 sequence is reasoned, not executed. Retain + the renderer's pre-set
claimRefare read from the manifests and fromcatalog/render.py; I have not watched a PVC rebind after a PV delete in this cluster. root_squashis a behaviour change from the Proxmox export, which carriedno_root_squash. The argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their Longhorn/config; both subPath directories already exist at 0755) is reasoned from the images and the measured permissions. It has not been exercised. If a mount or subPath resolution fails with EACCES, Maprootroot:rootreproduces the old export and is the first thing to try.- Whether NFSv3 can be disabled. Enabled Protocols is
NFSv3, NFSv4; the PVs only need v4.1, so narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the cutover is proven, not during it. - Whether pfSense / the legacy stack is actually retired. I measured only that
192.168.2.102:2049stopped answering. The other legacy consumers of these datasets (192.168.1.6,10.123.1.10,192.168.1.18) were dropped from the new exports; if any is still live, it has lost access. Not checked. - L2/L3 conformance.
vars.CLUSTER_AVAILABLEis unset, so CI proves the L0/L1 offline subset only. Nothing in the gate exercises an NFS mount.