--- date: 2026-10-07 branch: fix/bulk-nfs-truenas title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled --- ## What changed Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from `192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`. - `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are replaced with the measured export configuration and the recreate requirement below. - `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no longer exists. - `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one. - `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV delete-and-recreate. §2/§3/§4/§6 are unchanged by the move. - `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose references to `192.168.2.102` as the live export address. **The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action; it is **not** performed by this PR. ## What I verified **On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:** - `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`, `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged. - Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`, `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries and `tv/` 107, against 1165 / 107 measured on Proxmox. - `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no non-trivial ACLs — mode bits still govern. - Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from Proxmox; cleared with `zfs inherit`, re-checked empty. - `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`, `rw` on plex and `ro` on nextcloud. - `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates. - **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified. **From the cluster:** - Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong. - All five NFS PVs currently name `192.168.2.102`. - `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not. - **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change: `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`. - The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache, but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10. **Gate:** see the PR; run in the pinned toolbox from this worktree. ## What I did NOT verify - **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000 write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The equivalents passed against the old host on 2026-09-12; that is not evidence about this one. - **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete in this cluster. - **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and the measured permissions. It has not been exercised. If a mount or subPath resolution fails with EACCES, Maproot `root:root` reproduces the old export and is the first thing to try. - **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the cutover is proven, not during it. - **Whether pfSense / the legacy stack is actually retired.** I measured only that `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still live, it has lost access. Not checked. - **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset only. Nothing in the gate exercises an NFS mount.