fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled #67

2026-10-07T06:17:36Z

gitea-admin commented

2026-10-07 06:17:36 +00:00

Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community Edition at 192.168.2.30, which sits on prod's own 192.168.2.0/24. BULK_NFS_SERVER moves from 192.168.2.102 (the pfSense DNAT prod mounted through) to 192.168.2.30.

This retires the repo's one load-bearing placeholder: the correct value used to depend on an undecided operator choice about crossing pfSense, and there is no crossing left to arrange.

The operational point — this is not a plain reconcile

A bound PersistentVolume's NFS source is immutable. Server-side dry-run, 2026-10-07:

spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation

So Flux's apply fails rather than updating, and the pods keep their old, dead mount. The five PVs (plex-media, sonarr-media, radarr-media, qbittorrent-downloads, nextcloud-legacy-archive) must be deleted and recreated.

That is safe — Retain never touches data and catalog/render.py emits a pre-set claimRef, so each recreated PV rebinds its own PVC — but it is an operator action, sequenced in docs/BULK-NFS-STORAGE.md §10, and this PR does not perform it. Merging this alone leaves the cluster exactly as it is now.

Note the pairing with the hazard already on that page: spec.nfs.server is unvalidated at create (any string binds) and immutable at update. A wrong value is accepted silently and then cannot be corrected in place.

Verified on the NAS (2026-10-07, by the operator)

zfs list — PV paths /mnt/Storage/plex and /mnt/Storage/nextcloud resolve unchanged.
Ownership intact: the tree is drwxr-xr-x 1000:3000 throughout; movies/ 1171 entries, tv/ 107 (vs 1165/107 on Proxmox).
acltype=nfsv4 (SCALE's default), aclinherit=passthrough, no non-trivial ACLs — mode bits still govern.
Three stale sharenfs dataset properties carried over from Proxmox; cleared with zfs inherit.
exportfs -v — both shares to 192.168.2.10/.11/.12 with sec=sys,secure,root_squash,no_all_squash; rw on plex, ro on nextcloud.
/proc/fs/nfsd/versions → +3 +4 +4.1 +4.2, so the PVs' nfsvers=4.1 negotiates.
A defect was caught during setup: the plex share listed 192.168.2.13, an address no node holds, and so omitted 192.168.2.12 (k3s-prod-3). Nothing was failing at the time — it would have surfaced on the first reschedule onto that node. Corrected and re-verified.

Verified from the cluster

Node InternalIPs are 192.168.2.10/.11/.12, confirming .13 was wrong.
192.168.2.30:2049 and :111 accept connections; 192.168.2.102:2049 does not.
The media pods report Ready while their mounts are dead — stat /media answers from attribute cache, readdir blocks indefinitely (the PVs mount hard). Hence "scale to zero first" in §10.

NOT verified

Nothing has mounted 192.168.2.30. Every §9 probe — numeric-id passthrough, the uid-1000 write, the hardlink/EXDEV check, Plex's read-only posture — is outstanding, not passing. The §10 sequence is reasoned from the manifests, not executed. root_squash is a deliberate tightening over the Proxmox export's no_root_squash and has not been exercised. Full list in the handoff entry.

Gate

rc 3, with gitea-settings as the only stage that did not run — it needs SETTINGS_API_TOKEN, which exists only in CI, so this is the best achievable local result. All other stages green: spec-immutability, lint, l0, render-golden, l1, decisions-log, approval-policy. Touched requirements: R-ENV-02 R-GITOPS-01 R-GITOPS-02 R-OBS-01 R-OBS-02.

clusters/** is a sensitive path, so this needs a non-author review or an /approve-override posted as a PR review of type Comment.

🤖 Generated with Claude Code

Both ZFS pools were physically moved off the Proxmox host onto a standalone **TrueNAS Community Edition at 192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from `192.168.2.102` (the pfSense DNAT prod mounted through) to `192.168.2.30`. This retires the repo's one load-bearing placeholder: the correct value used to depend on an undecided operator choice about crossing pfSense, and there is no crossing left to arrange. ## The operational point — this is not a plain reconcile **A bound PersistentVolume's NFS source is immutable.** Server-side dry-run, 2026-10-07: ``` spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation ``` So Flux's apply **fails** rather than updating, and the pods keep their old, dead mount. The five PVs (`plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) must be deleted and recreated. That is safe — `Retain` never touches data and `catalog/render.py` emits a pre-set `claimRef`, so each recreated PV rebinds its own PVC — but it is an **operator action, sequenced in `docs/BULK-NFS-STORAGE.md` §10, and this PR does not perform it.** Merging this alone leaves the cluster exactly as it is now. Note the pairing with the hazard already on that page: `spec.nfs.server` is **unvalidated at create** (any string binds) and **immutable at update**. A wrong value is accepted silently and then cannot be corrected in place. ## Verified on the NAS (2026-10-07, by the operator) - `zfs list` — PV paths `/mnt/Storage/plex` and `/mnt/Storage/nextcloud` resolve unchanged. - Ownership intact: the tree is `drwxr-xr-x 1000:3000` throughout; `movies/` 1171 entries, `tv/` 107 (vs 1165/107 on Proxmox). - `acltype=nfsv4` (SCALE's default), `aclinherit=passthrough`, **no non-trivial ACLs** — mode bits still govern. - Three stale `sharenfs` dataset properties carried over from Proxmox; cleared with `zfs inherit`. - `exportfs -v` — both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`; `rw` on plex, `ro` on nextcloud. - `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates. - **A defect was caught during setup:** the plex share listed `192.168.2.13`, an address no node holds, and so omitted `192.168.2.12` (k3s-prod-3). Nothing was failing at the time — it would have surfaced on the first reschedule onto that node. Corrected and re-verified. ## Verified from the cluster - Node InternalIPs are `192.168.2.10/.11/.12`, confirming `.13` was wrong. - `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not. - The media pods report `Ready` while their mounts are dead — `stat /media` answers from attribute cache, `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10. ## NOT verified **Nothing has mounted 192.168.2.30.** Every §9 probe — numeric-id passthrough, the uid-1000 write, the hardlink/EXDEV check, Plex's read-only posture — is **outstanding, not passing**. The §10 sequence is reasoned from the manifests, not executed. `root_squash` is a deliberate tightening over the Proxmox export's `no_root_squash` and has not been exercised. Full list in the handoff entry. ## Gate `rc 3`, with **`gitea-settings` as the only stage that did not run** — it needs `SETTINGS_API_TOKEN`, which exists only in CI, so this is the best achievable local result. All other stages green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`, `decisions-log`, `approval-policy`. Touched requirements: `R-ENV-02 R-GITOPS-01 R-GITOPS-02 R-OBS-01 R-OBS-02`. `clusters/**` is a sensitive path, so this needs a non-author review or an `/approve-override` posted as a PR **review** of type Comment. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-07 06:17:36 +00:00

fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled

gate / gate (pull_request) Successful in 1m53s

Details

gate / conformance (pull_request) Has been skipped

Details

0747849837

Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on
192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob
load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through.
Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do.

BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked
inert (dev is decommissioned); it carries the real address only so nothing in
the tree renders a host that no longer exists.

THE OPERATIONAL POINT, and why this is not just a value change: a bound
PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07:

  spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is
  immutable after creation

so Flux's apply FAILS rather than updating, and the pods keep their old, dead
mount. The five PVs must be deleted and recreated -- safe, because Retain never
touches data and render.py emits a pre-set claimRef so each PV rebinds its own
PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR
action; this commit does not perform it.

Pairs with the hazard already on the page: spec.nfs.server is unvalidated at
create (any string binds) and immutable at update. A wrong value is accepted
silently and then cannot be corrected in place.

docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured
exports, the service settings that decide whether uid 1000 survives the wire,
the three stale ZFS sharenfs properties that had to be cleared, and the ACL
hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor
propagates to the whole future library). Section 7 rewritten now that the
placeholder is gone, section 9 retargeted from dev to prod, section 10 new.
Sections 2/3/4/6 are unchanged by the move.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes

2026-10-07 06:24:11 +00:00

jakman85 merged commit e89df78210 into main

2026-10-07 06:26:45 +00:00

jakman85 referenced this issue from a commit

2026-10-07 06:26:46 +00:00

Merge pull request 'fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled' (#67) from fix/bulk-nfs-truenas into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#67