jakman85andClaude Opus 5
0747849837
Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on 192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through. Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do. BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked inert (dev is decommissioned); it carries the real address only so nothing in the tree renders a host that no longer exists. THE OPERATIONAL POINT, and why this is not just a value change: a bound PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation so Flux's apply FAILS rather than updating, and the pods keep their old, dead mount. The five PVs must be deleted and recreated -- safe, because Retain never touches data and render.py emits a pre-set claimRef so each PV rebinds its own PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR action; this commit does not perform it. Pairs with the hazard already on the page: spec.nfs.server is unvalidated at create (any string binds) and immutable at update. A wrong value is accepted silently and then cannot be corrected in place. docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured exports, the service settings that decide whether uid 1000 survives the wire, the three stale ZFS sharenfs properties that had to be cleared, and the ACL hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor propagates to the whole future library). Section 7 rewritten now that the placeholder is gone, section 9 retargeted from dev to prod, section 10 new. Sections 2/3/4/6 are unchanged by the move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
86 lines
5.6 KiB
Markdown
86 lines
5.6 KiB
Markdown
---
|
|
date: 2026-10-07
|
|
branch: fix/bulk-nfs-truenas
|
|
title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
|
|
---
|
|
|
|
## What changed
|
|
|
|
Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
|
|
Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
|
|
`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
|
|
|
|
- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
|
|
justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
|
|
replaced with the measured export configuration and the recreate requirement below.
|
|
- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
|
|
longer exists.
|
|
- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
|
|
and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
|
|
- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
|
|
`sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
|
|
placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
|
|
delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
|
|
- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
|
|
references to `192.168.2.102` as the live export address.
|
|
|
|
**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
|
|
source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
|
|
mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
|
|
it is **not** performed by this PR.
|
|
|
|
## What I verified
|
|
|
|
**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
|
|
|
|
- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
|
|
`Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
|
|
- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
|
|
`downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
|
|
and `tv/` 107, against 1165 / 107 measured on Proxmox.
|
|
- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
|
|
non-trivial ACLs — mode bits still govern.
|
|
- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
|
|
Proxmox; cleared with `zfs inherit`, re-checked empty.
|
|
- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
|
|
`rw` on plex and `ro` on nextcloud.
|
|
- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
|
|
- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
|
|
node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
|
|
|
|
**From the cluster:**
|
|
|
|
- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
|
|
- All five NFS PVs currently name `192.168.2.102`.
|
|
- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
|
|
- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
|
|
`spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
|
|
- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
|
|
but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
|
|
|
|
**Gate:** see the PR; run in the pinned toolbox from this worktree.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
|
|
the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
|
|
write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
|
|
equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
|
|
- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
|
|
from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
|
|
in this cluster.
|
|
- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
|
|
argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
|
|
Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
|
|
the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
|
|
EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
|
|
- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
|
|
narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
|
|
cutover is proven, not during it.
|
|
- **Whether pfSense / the legacy stack is actually retired.** I measured only that
|
|
`192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
|
|
(`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
|
|
live, it has lost access. Not checked.
|
|
- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
|
|
only. Nothing in the gate exercises an NFS mount.
|