fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on 192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through. Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do. BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked inert (dev is decommissioned); it carries the real address only so nothing in the tree renders a host that no longer exists. THE OPERATIONAL POINT, and why this is not just a value change: a bound PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation so Flux's apply FAILS rather than updating, and the pods keep their old, dead mount. The five PVs must be deleted and recreated -- safe, because Retain never touches data and render.py emits a pre-set claimRef so each PV rebinds its own PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR action; this commit does not perform it. Pairs with the hazard already on the page: spec.nfs.server is unvalidated at create (any string binds) and immutable at update. A wrong value is accepted silently and then cannot be corrected in place. docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured exports, the service settings that decide whether uid 1000 survives the wire, the three stale ZFS sharenfs properties that had to be cleared, and the ACL hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor propagates to the whole future library). Section 7 rewritten now that the placeholder is gone, section 9 retargeted from dev to prod, section 10 new. Sections 2/3/4/6 are unchanged by the move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
ad37acdf53
commit
0747849837
7 files changed
+369
-188
No files matched your search
@@ -540,60 +540,48 @@ patches:
|
||||
path: /data/LAN_CIDR
|
||||
value: "192.168.2.0/24"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
|
||||
# BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
|
||||
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
|
||||
# radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
|
||||
#
|
||||
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
|
||||
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
|
||||
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
|
||||
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
|
||||
# outcomes give DIFFERENT values here:
|
||||
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
|
||||
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
|
||||
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
|
||||
# ever became reachable — mounts something unintended. So it stays a placeholder.
|
||||
# THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
|
||||
# The datasets lived on a Proxmox host with a single interface on pfSense's LAN
|
||||
# (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
|
||||
# value here depended on an undecided operator choice, and guessing it would have authored a
|
||||
# cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
|
||||
# 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
|
||||
# Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
|
||||
#
|
||||
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
|
||||
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
|
||||
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
|
||||
# on ephemeral numbers.
|
||||
#
|
||||
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
|
||||
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
|
||||
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
|
||||
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
|
||||
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
|
||||
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
|
||||
#
|
||||
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
|
||||
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
|
||||
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
|
||||
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
|
||||
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
|
||||
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
|
||||
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
|
||||
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
|
||||
# where these manifests are meant to be proven first.
|
||||
# ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
|
||||
# and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
|
||||
# `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
|
||||
- op: replace
|
||||
path: /data/BULK_NFS_SERVER
|
||||
# 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
|
||||
# host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
|
||||
# reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
|
||||
# keeps the base value 192.168.1.9.
|
||||
# VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
|
||||
# /mnt/Storage/plex 192.168.2.10/.11/.12 rw,sec=sys,secure,root_squash,no_all_squash
|
||||
# /mnt/Storage/nextcloud 192.168.2.10/.11/.12 ro,sec=sys,secure,root_squash,no_all_squash
|
||||
#
|
||||
# PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
|
||||
# - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
|
||||
# - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
|
||||
# carries no_root_squash, so a root write would have passed and proved nothing, while
|
||||
# LSIO images default to PUID 911 and would have gone Ready then failed every import
|
||||
# with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
|
||||
# - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
|
||||
# real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
|
||||
# pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
|
||||
# so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
|
||||
value: "192.168.2.102"
|
||||
# no_all_squash is the flag the entire permission model rests on: client uids pass through
|
||||
# untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
|
||||
# keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
|
||||
# no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
|
||||
# the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
|
||||
# /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
|
||||
# 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
|
||||
#
|
||||
# The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
|
||||
# The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
|
||||
# can land on any node, so the server sees the same three addresses for reader and writers
|
||||
# alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
|
||||
#
|
||||
# *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
|
||||
# spec.nfs is immutable — measured by server dry-run 2026-10-07:
|
||||
# spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
|
||||
# so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
|
||||
# operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
|
||||
# claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
|
||||
value: "192.168.2.30"
|
||||
# ==========================================================================================
|
||||
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
|
||||
# ==========================================================================================
|
||||
|
||||
Reference in new issue
Block a user