jakman85andClaude Opus 5
87c3d83dff
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]>
19 KiB
Bulk NFS storage — the media library and the legacy Nextcloud archive
The operator-facing companion to the class: bulk storage entries in catalog/{plex,sonarr,radarr,nextcloud}.yaml.
Everything here was measured on the storage host on 2026-09-07, over read-only SSH; nothing is
assumed. It resolves the long-standing OQ-2 stub (platform/longhorn/storageclasses.yaml) for the
case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
provisioning unimplemented, because nothing needs it.
1. The storage host, and why prod cannot reach it
The bulk data lives on the Proxmox host's ZFS pool. That host has exactly one interface:
vmbr0 192.168.1.9/24 default via 192.168.1.1 (pfSense LAN)
| nodes | reaches 192.168.1.9? | |
|---|---|---|
| dev | 192.168.1.40 / .41 / .42 | Yes — same L2 segment, and it already holds an NFS export from this host (/mnt/Storage/k8s-volumes, in /etc/exports) |
| prod | 192.168.2.10 / .11 / .12 | No — 192.168.2.0/24 is pfSense's WAN. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
Opening the storage path for prod is a pending operator decision. Its two candidate outcomes give
different values for the BULK_NFS_SERVER cluster-var:
| outcome | BULK_NFS_SERVER for prod |
|---|---|
port-forward 192.168.2.102:2049 → 192.168.1.9:2049 |
192.168.2.102 |
| a WAN→LAN route to 192.168.1.0/24 | 192.168.1.9 |
Until it lands, clusters/prod holds TBD-PROD-BULK-NFS-SERVER. See §7 for why that placeholder
behaves differently from the repo's other ones.
It only needs ONE port
The PVs mount nfsvers=4.1. NFSv4 carries mount and lock in-protocol, so the only thing that
must be reachable is tcp/2049 — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
server advertises +3 +4 +4.1 +4.2 (/proc/fs/nfsd/versions), so v4.1 is available and is the
conservative floor.
The export's client list is the SAME under both outcomes — measured, not assumed
pfSense DNATs the existing port-forward WITHOUT source NAT. Connecting from this workstation
(192.168.2.132) to 192.168.2.102:22, the storage host logged the peer as 192.168.2.132:
SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
ss -tn -> 192.168.1.9:22 192.168.2.132:39768
The original WAN-side source address survives the translation (return traffic finds its way back via
the host's default route, 192.168.1.1). So under either outcome the NFS server will see prod's
real node addresses — 192.168.2.10/.11/.12 — and the sharenfs change in §5 is the same either
way. It does not have to wait for the routing decision.
2. The layout, as found
Storage/plex → /mnt/Storage/plex — 8.33 T used, 6.91 T avail
/mnt/Storage/plex/ drwxr-xr-x 1000:3000
├── downloads/ drwxr-xr-x 1000:1000
│ ├── complete/ drwxr-xr-x 1000:3000
│ │ ├── radarr/ drwxr-xr-x 1000:3000
│ │ └── tv-sonarr/ drwxr-xr-x 1000:3000
│ ├── incomplete/ drwxrwxr-x 1000:3000
│ ├── downloads -> /shared/downloads (symlink OUTSIDE the export — see §8)
│ └── <in-flight release dirs> drwxr-xr-x 1000:1000
├── movies/ 1165 entries drwxr-xr-x 1000:3000
├── tv/ 107 entries drwxr-xr-x 1000:3000
└── photos/ drwxr-xr-x 1000:3000
The single most important fact about this layout: downloads/ is a SIBLING of tv/ and movies/
inside the same dataset. Sonarr's import source and its library are on one filesystem on the
server, and so are Radarr's. That is what makes hardlinked / atomically-renamed imports possible at
all — and §4 is about not throwing it away on the client side.
Note the download client's completed folders are already named radarr and tv-sonarr, i.e. this
tree has been driven by an arr stack for a long time. The k3s arrs are joining an existing layout,
not defining one.
Storage/nextcloud → /mnt/Storage/nextcloud — 270 G used
/mnt/Storage/nextcloud/ drwxr-xr-x root:root
└── nextcloud/ drwxr-xr-x 1000:3000
├── config/ 103 M drwxr-xr-x 1000:3000
├── config_old/ 3.4 G drwxr-xr-x 1000:3000
└── config_backup/ 267 G drwxr-xr-x 1000:3000
There is no data/ directory. 99% of the dataset is config_backup. This is the old, non-k3s
Nextcloud's configuration plus a backup blob — an archive, not a live user-data tree. That is
why it is mounted read-only; see §6.
3. What each app mounts, and why
| app | export | mountPath | subPath | mode | access mode |
|---|---|---|---|---|---|
| plex | /mnt/Storage/plex |
/media |
— | ro | ReadOnlyMany |
| sonarr | /mnt/Storage/plex |
/media |
— | rw | ReadWriteMany |
| radarr | /mnt/Storage/plex |
/media |
— | rw | ReadWriteMany |
| nextcloud | /mnt/Storage/nextcloud |
/mnt/legacy-nextcloud |
nextcloud |
ro | ReadOnlyMany |
In-app paths that follow: Plex libraries at /media/movies, /media/tv, /media/photos; Sonarr
Root Folder /media/tv; Radarr Root Folder /media/movies; both importing from
/media/downloads/complete/{tv-sonarr,radarr}.
All three media consumers use the identical mountPath on purpose. When qbittorrent is eventually given the same export (it is not in this change), the arrs and the download client will already agree on absolute paths, which is the second precondition for hardlinked imports — otherwise every import needs a Remote Path Mapping and quietly falls back to copying when one is missing.
/media and not /data, and that is not cosmetic. The arrs' Longhorn config PVC renders at the
renderer default /data/config. A media mount at /data would be mounted over it and mask the
config volume entirely. See §8 for the separate latent bug that discovery exposed.
4. Why the arrs mount the export ROOT, not two subPaths
This is the design decision most likely to be "simplified" later by someone who has not read this.
Sonarr and Radarr import a finished download into the library by hardlinking it (preserving the seeding copy) or by an atomic rename. Both are same-filesystem operations, and to the kernel, "same filesystem" means "same mount."
The tempting narrower shape is two subPath volumes — subPath: tv and subPath: downloads. Those
are two separate mounts of the same export. link() across them returns EXDEV no matter that
the server knows perfectly well they are one dataset. Sonarr does not fail loudly at that: it
degrades to copy-then-delete. Every import then rewrites the entire file over NFS and destroys the
seeding copy. The symptom is "imports are slow and my ratio is gone", weeks later.
The smallest subtree containing both tv/ and downloads/ is the export root. So the root mount
is the minimal mount that preserves hardlinks: least privilege bounded below by a hard requirement,
not a shortcut.
The cost is stated rather than hidden: Sonarr can also write /media/movies and Radarr /media/tv.
Narrowing that would cost hardlinks, which is the worse trade.
5. The export change the operator must apply — STRICTLY ADDITIVE
Do not apply this from the repo. It is an operator step on the storage host, recorded here so the exact value is reviewable. The existing consumers must keep working: 192.168.1.6, 10.123.1.10 and 192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the live non-k3s prod stack.
Current values (zfs get sharenfs Storage):
Storage/plex [email protected]:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
Storage/nextcloud [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
Storage/plex — append the six k3s node addresses to the existing rw= list
zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
Every pre-existing host is preserved verbatim and no option changes; only clients are added.
Storage/nextcloud — add the k3s nodes to a ro= list, not to rw=
zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
This is the one place read-only can be enforced server-side, and it is worth taking: only Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a misconfigured client cannot then damage the archive.
The media tree cannot get the same treatment. Plex is read-only and the arrs are read-write, but
all three may be scheduled on any node, so the server sees the same source addresses for all of
them and cannot distinguish the reader from the writers. Plex's read-only posture is therefore a
client-side guarantee (spec.nfs.readOnly: true on plex-media, plus the volumeMount flag) — an
honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
dedicated node pool for Plex, which is a much larger change.
Per-node addresses rather than @192.168.2.0/24 follows the precedent already on this host
(/etc/exports lists the dev nodes individually, as /32). A /24 would work and would survive
adding a node; it also grants the whole subnet. Per-node is the tighter default.
Verifying, without changing anything
exportfs -v # the new client lines appear, existing ones unchanged
zfs get sharenfs Storage/plex Storage/nextcloud
6. uid / gid — the failure this design exists to avoid
The export uses sec=sys and no_root_squash, so the client's numeric uid is passed through
verbatim and the server's on-disk ownership is the entire access story. The media tree is
0755 1000:3000 throughout (some directories 1000:1000), so only uid 1000 can write it.
fsGroup cannot fix this, and reaching for it is how this gets misdiagnosed. The in-tree NFS
volume plugin performs no ownership management, so podSecurityContext.fsGroup is a no-op on
an NFS volume. There is no recursive chown, no g+rwX, nothing. The container's own uid has to be
right.
| app | container uid | writes? | outcome |
|---|---|---|---|
| plex | 1000 (PLEX_UID/PLEX_GID, already set) |
no | reads fine — the tree is 0755, so world-readable anyway |
| sonarr | 1000 via PUID (added) |
yes | matches the owner ⇒ writes fine |
| radarr | 1000 via PUID (added) |
yes | matches the owner ⇒ writes fine |
| nextcloud | 33 (www-data) |
no | reads and traverses 0755 fine |
Without the added PUID/PGID, sonarr and radarr would have failed exactly the way this tier
always fails: the mount succeeds, the pod is Ready, and every import fails EACCES. LinuxServer
images default to uid/gid 911, which owns nothing on that tree.
PGID: 3000 is chosen over 1000 so files these write are group-owned by the dominant group already
on the tree, keeping them readable to the live non-k3s consumers.
podSecurityContext.fsGroup: 3000 is also added to sonarr and radarr — for the Longhorn config
PVC, not for the NFS mount. fsGroup does work on a Longhorn filesystem, and it makes /data/config
writable by the now-uid-1000 process without depending on the image re-chowning it after PUID
changed.
Nextcloud needs no uid work at all, which is one more reason its mount is read-only.
7. The prod placeholder does NOT fail at apply time — read this before merging
Every other TBD-PROD-* in clusters/prod/kustomization.yaml is rejected loudly by CRD validation
(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. This one is not.
A PersistentVolume's spec.nfs.server is an unvalidated string. The API server accepts
TBD-PROD-BULK-NFS-SERVER, the PV is created, and the PVC binds to it — binding is pure
control-plane and never touches the server. The failure appears only when a pod mounts:
MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
so plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.
Consequently: set BULK_NFS_SERVER in clusters/prod/kustomization.yaml BEFORE prod reconciles a
revision that carries these bulk volumes. Dev renders a working value and is where these manifests
are meant to be proven first.
The static guard this convention leans on is missing from main — but the gate now refuses the value anyway
bootstrap/standup.sh — which carries the *_is_placeholder guard family — does not exist on
main (verified: git ls-tree main bootstrap/ lists only README.md, flux/, gitea/, sops/;
the file lives on unmerged prod branches, and there is no deletion commit for it anywhere — it never
landed rather than having been removed).
UPDATED 2026-09-13. This section used to end "so
conformance/tests/l0-static/placeholder-guard-coupling/check.sh exits 3 SKIPPED, and nothing in
the gate currently refuses a placeholder-shaped overlay value." That is no longer true. That check
was restructured (see DECISIONS-LOG, branch fix/placeholder-guard-coupling-unskipped): it never
exits 3, and its HALF A screens every environment overlay's effective cluster-vars value against
an independent placeholder detector with no guard required. A BULK_NFS_SERVER of
TBD-PROD-BULK-NFS-SERVER in clusters/prod/ is now a merge-gate FAIL — verified by deliberately
authoring exactly that value and watching the check fail, which is the one hazard on this page that
CRD validation cannot catch. HALF A also refuses a LOUD base sentinel that an overlay inherits by
omitting its patch op, which is the other way this knob can silently ship wrong.
The guard is now discovered by content, not by path, so when standup.sh (or its predicate in a
sourceable library) lands, the coupling half activates with no edit to the test — and it then covers
this knob provided the guard gains an arm for BULK_NFS_SERVER and a consumer that calls it.
Without the arm, HALF A is what holds the line, by design.
8. Loose ends and things deliberately not done
/data/configis very likely the wrong home for the arrs' config PVC — a pre-existing latent bug, not fixed here. LinuxServer's arr images run with-data=/config, so the app writes/config, which is the container's ephemeral layer, while the Longhorn PVC mounted at/data/configsits unused. This is the exact failurecatalog/plex.yaml'smountPathcomment records having already been bitten by ("the PVC sat unused and Plex wrote to the container's EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it means those apps start from an empty config volume, so it belongs in its own change with the operator present. This change does not make it worse — that is why the media mount is/media.- qbittorrent has no bulk mount. Today the arrs' import source is written by the legacy
non-k3s download client, which is coherent while the legacy stack stays live. Giving the k3s
qbittorrent the same export at the same
/mediapath is the natural follow-up, and the paths were chosen so it needs no Remote Path Mapping. /mnt/Storage/plex/downloads/downloadsis a symlink to/shared/downloads, which is outside the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log it while scanningdownloads/.nfsvers=4.2was not chosen, though the server advertises it. Nothing in these workloads uses what 4.2 adds (server-side copy, sparse-fileSEEK/ALLOCATE), and 4.1 is the version the existingbulkStorageClass stub already named.nconnectwas not set. It would likely help throughput on the media tree, but it is a tuning change that should be measured, not guessed, and it changes client behaviour under server failure.- Nothing was verified against a live cluster. No
kubectlcontext is usable from this workstation (browser-OIDC only), and this change is manifest-authoring under an explicit do-not-apply constraint. Every claim about the host is measured; every claim about binding and mounting is reasoned from the manifests. - The bulk-tier requirement ID is deliberately not typed on any non-
.mdline of this change.ci/touched-requirements.shattributes requirements from R-IDs appearing in added lines, and the in-cluster conformance test for the bulk tier is still a gap, so typing it would select a requirement the gate cannot satisfy — the hazardcatalog/qbittorrent.yamlalready warns about. Scaffolding that test needs a cluster and is the correct follow-up.
9. Validating on dev, once the export is extended
Dev can prove all of this before prod's network path exists. Read-only checks first:
infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
Expect four PVs — plex-media, sonarr-media, radarr-media, nextcloud-legacy-archive (three
of them naming the same export, one per namespace, which is the cross-namespace model) — each
Bound to the same-named PVC, with RECLAIM POLICY Retain and an empty STORAGECLASS.
Then the two things that actually matter, from inside a pod:
# (a) the writer can write — this is the uid check, and it is the one that fails in practice
infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'
# (c) the reader really is read-only
infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
sh -c 'touch /media/.probe 2>&1 | head -1' # expect "Read-only file system"
(b) printing two identical inode numbers is the proof that §4's argument holds in practice. If it
prints EXDEV, something has split the single mount back into two.