# Bulk NFS storage — the media library and the legacy Nextcloud archive The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`. **Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 — was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk provisioning unimplemented, because nothing needs it. --- ## 1. The storage host The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same `192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT and no routing decision. **This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**. tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.** ### It only needs ONE port The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor. **NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs, just no volume. ## 2. The layout, as found ### `Storage/plex` → `/mnt/Storage/plex` — 8.33 T used, 6.91 T avail ``` /mnt/Storage/plex/ drwxr-xr-x 1000:3000 ├── downloads/ drwxr-xr-x 1000:1000 │ ├── complete/ drwxr-xr-x 1000:3000 │ │ ├── radarr/ drwxr-xr-x 1000:3000 │ │ └── tv-sonarr/ drwxr-xr-x 1000:3000 │ ├── incomplete/ drwxrwxr-x 1000:3000 │ ├── downloads -> /shared/downloads (symlink OUTSIDE the export — see §8) │ └── drwxr-xr-x 1000:1000 ├── movies/ 1165 entries drwxr-xr-x 1000:3000 ├── tv/ 107 entries drwxr-xr-x 1000:3000 └── photos/ drwxr-xr-x 1000:3000 ``` **The single most important fact about this layout: `downloads/` is a SIBLING of `tv/` and `movies/` inside the same dataset.** Sonarr's import source and its library are on one filesystem on the server, and so are Radarr's. That is what makes hardlinked / atomically-renamed imports possible at all — and §4 is about not throwing it away on the client side. Note the download client's completed folders are already named `radarr` and `tv-sonarr`, i.e. this tree has been driven by an arr stack for a long time. The k3s arrs are joining an existing layout, not defining one. ### `Storage/nextcloud` → `/mnt/Storage/nextcloud` — 270 G used ``` /mnt/Storage/nextcloud/ drwxr-xr-x root:root └── nextcloud/ drwxr-xr-x 1000:3000 ├── config/ 103 M drwxr-xr-x 1000:3000 ├── config_old/ 3.4 G drwxr-xr-x 1000:3000 └── config_backup/ 267 G drwxr-xr-x 1000:3000 ``` **There is no `data/` directory.** 99% of the dataset is `config_backup`. This is the old, non-k3s Nextcloud's configuration plus a backup blob — an **archive**, not a live user-data tree. That is why it is mounted read-only; see §6. --- ## 3. What each app mounts, and why | app | export | mountPath | subPath | mode | access mode | |---|---|---|---|---|---| | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany | | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany | | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany | | qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany | | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany | In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from `/media/downloads/complete/{tv-sonarr,radarr}`. **The three library consumers use the identical mountPath on purpose**, and qbittorrent has since been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute paths, which is the second precondition for hardlinked imports — otherwise every import needs a Remote Path Mapping and quietly falls back to copying when one is missing. **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the config volume entirely. See §8 for the separate latent bug that discovery exposed. --- ## 4. Why the arrs mount the export ROOT, not two subPaths This is the design decision most likely to be "simplified" later by someone who has not read this. Sonarr and Radarr import a finished download into the library by **hardlinking** it (preserving the seeding copy) or by an **atomic rename**. Both are same-filesystem operations, and **to the kernel, "same filesystem" means "same mount."** The tempting narrower shape is two subPath volumes — `subPath: tv` and `subPath: downloads`. Those are **two separate mounts** of the same export. `link()` across them returns **EXDEV** no matter that the server knows perfectly well they are one dataset. Sonarr does not fail loudly at that: it degrades to copy-then-delete. Every import then rewrites the entire file over NFS and destroys the seeding copy. The symptom is "imports are slow and my ratio is gone", weeks later. The smallest subtree containing both `tv/` and `downloads/` **is the export root**. So the root mount is the *minimal* mount that preserves hardlinks: least privilege bounded below by a hard requirement, not a shortcut. The cost is stated rather than hidden: Sonarr can also write `/media/movies` and Radarr `/media/tv`. Narrowing that would cost hardlinks, which is the worse trade. --- ## 5. The exports, as configured on TrueNAS — and the settings that decide permissions **Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after the move. ### The exports ``` /mnt/Storage/plex 192.168.2.10 192.168.2.11 192.168.2.12 (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash) /mnt/Storage/nextcloud 192.168.2.10 192.168.2.11 192.168.2.12 (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash) ``` Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24` would work and would survive adding a node; it also grants the whole subnet. **All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules there, as `access denied by server while mounting`, with the cause weeks stale. One was caught exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time. ### The option flags ARE the permission contract | flag | why it matters here | |---|---| | **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. | | **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. | | **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. | | **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. | | **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. | **Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but all four can land on any node, so the server sees the same three source addresses for all of them and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex. ### Service settings that are NOT obvious from the export list - **Enabled Protocols must include NFSv4** (above). - **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model. - **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the client's supplied group list* and looks that uid's groups up in TrueNAS's own user database instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group — but it is a free way to lose group semantics. - **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses. - **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`, not `localhost`. There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it. ### The ZFS `sharenfs` properties must be CLEARED, not edited The pools arrived with the old host's exports still set as dataset properties: ``` Storage/plex rw=@192.168.1.6:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash Storage/nextcloud rw=@192.168.1.6:@192.168.1.5,ro=@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash Storage/docker-volumes rw=@192.168.1.6:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash ``` TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did not set — the failure mode is two differently-configured exports of one path, with no indication which is in force. Cleared 2026-10-07: ```sh zfs inherit sharenfs Storage/plex zfs inherit sharenfs Storage/nextcloud zfs inherit sharenfs Storage/docker-volumes ``` `Storage` is `off`, so each resolves to `off`. ### Do NOT touch the dataset's permissions or ACLs The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it irreversibly across 8.33 T. `Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and `aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755 1000:3000` governs exactly as it did under Proxmox. - **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted. There is no undo. - `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by everything created under it afterwards** — the blast radius is the whole future library, not one directory. - Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`. `xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would not affect existing files. ### Verifying, without changing anything ```sh cat /proc/fs/nfsd/versions # +4.1 must be present exportfs -v # the two shares, three nodes each, flags as tabulated above showmount -e 192.168.2.30 # NOT localhost -- the service is bound to one address zfs get -r -o name,property,value,source sharenfs Storage # every row off or - ``` ## 6. uid / gid — the failure this design exists to avoid The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries `no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that matters is 1000, not 0.) The media tree is `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**. **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS volume plugin performs **no ownership management**, so `podSecurityContext.fsGroup` is a **no-op on an NFS volume**. There is no recursive chown, no `g+rwX`, nothing. The container's own uid has to be right. | app | container uid | writes? | outcome | |---|---|---|---| | plex | 1000 (`PLEX_UID`/`PLEX_GID`, already set) | no | reads fine — the tree is 0755, so world-readable anyway | | sonarr | **1000 via `PUID`** (added) | yes | matches the owner ⇒ writes fine | | radarr | **1000 via `PUID`** (added) | yes | matches the owner ⇒ writes fine | | nextcloud | 33 (`www-data`) | no | reads and traverses 0755 fine | **Without the added `PUID`/`PGID`, sonarr and radarr would have failed exactly the way this tier always fails: the mount succeeds, the pod is Ready, and every import fails EACCES.** LinuxServer images default to uid/gid **911**, which owns nothing on that tree. `PGID: 3000` is chosen over 1000 so files these write are group-owned by the dominant group already on the tree, keeping them readable to the live non-k3s consumers. `podSecurityContext.fsGroup: 3000` is also added to sonarr and radarr — **for the Longhorn config PVC, not for the NFS mount.** fsGroup does work on a Longhorn filesystem, and it makes `/data/config` writable by the now-uid-1000 process without depending on the image re-chowning it after `PUID` changed. Nextcloud needs no uid work at all, which is one more reason its mount is read-only. --- ## 7. Why this knob is unvalidated at create AND immutable at update **The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The hazard that made it load-bearing is worth keeping on the page, because it is the half that explains why §10 exists. Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation (Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.** A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted `TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure control-plane and never touches the server. The failure appeared only when a pod mounted: ``` MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER ``` **The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a wrong value is accepted silently at create, and then cannot be corrected in place. Together they are why repointing this knob is a delete-and-recreate rather than a reconcile. ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway `bootstrap/standup.sh` — which carries the `*_is_placeholder` guard family — **does not exist on `main`** (verified: `git ls-tree main bootstrap/` lists only `README.md`, `flux/`, `gitea/`, `sops/`; the file lives on unmerged prod branches, and there is no deletion commit for it anywhere — it never landed rather than having been removed). **UPDATED 2026-09-13.** This section used to end "so `conformance/tests/l0-static/placeholder-guard-coupling/check.sh` exits **3 SKIPPED**, and nothing in the gate currently refuses a placeholder-shaped overlay value." That is **no longer true.** That check was restructured (see DECISIONS-LOG, branch `fix/placeholder-guard-coupling-unskipped`): it never exits 3, and its HALF A screens **every environment overlay's effective cluster-vars value** against an independent placeholder detector with no guard required. A `BULK_NFS_SERVER` of `TBD-PROD-BULK-NFS-SERVER` in `clusters/prod/` is now a **merge-gate FAIL** — verified by deliberately authoring exactly that value and watching the check fail, which is the one hazard on this page that CRD validation cannot catch. HALF A also refuses a LOUD base sentinel that an overlay inherits by *omitting* its patch op, which is the other way this knob can silently ship wrong. The guard is now **discovered by content**, not by path, so when `standup.sh` (or its predicate in a sourceable library) lands, the coupling half activates with no edit to the test — and it then covers this knob **provided** the guard gains an arm for `BULK_NFS_SERVER` and a consumer that calls it. Without the arm, HALF A is what holds the line, by design. --- ## 8. Loose ends and things deliberately not done 1. **`/data/config` is very likely the wrong home for the arrs' config PVC — a pre-existing latent bug, not fixed here.** LinuxServer's arr images run with `-data=/config`, so the app writes `/config`, which is the container's **ephemeral layer**, while the Longhorn PVC mounted at `/data/config` sits unused. This is the exact failure `catalog/plex.yaml`'s `mountPath` comment records having already been bitten by ("the PVC sat unused and Plex wrote to the container's EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it means those apps start from an empty config volume, so it belongs in its own change with the operator present. **This change does not make it worse** — that is why the media mount is `/media`. 2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the import source is written by the k3s download client rather than the legacy non-k3s one. See `catalog/qbittorrent.yaml`. 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log it while scanning `downloads/`. 4. **`nfsvers=4.2` was not chosen**, though the server advertises it. Nothing in these workloads uses what 4.2 adds (server-side copy, sparse-file `SEEK`/`ALLOCATE`), and 4.1 is the version the existing `bulk` StorageClass stub already named. 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning change that should be measured, not guessed, and it changes client behaviour under server failure. 6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the outstanding verification, not a formality. The pre-move equivalents did pass against the old host on 2026-09-12. 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.** `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the in-cluster conformance test for the bulk tier is still a gap, so typing it would select a requirement the gate cannot satisfy — the hazard `catalog/qbittorrent.yaml` already warns about. Scaffolding that test needs a cluster and is the correct follow-up. --- ## 9. Validating, after the PVs are recreated These run against **prod**; dev is decommissioned. They are the outstanding verification for the move — nothing below has been run against 192.168.2.30. Read-only checks first: ```sh ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\ NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase' ``` Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and **`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not happen; Flux will NOT have updated it in place. Then the four things that actually matter, from inside a pod: ```sh # (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3' # expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not # passed through, and every import will fail EACCES while the pod still reports Ready. # (b) the writer can write -- the uid check, and the one that fails in practice kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \ sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK' # (c) hardlinks work BETWEEN the library and the download path -- the root-mount check kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \ sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link; echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK' # (d) the reader really is read-only kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1' # expect "Read-only file system" ``` `(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it prints `EXDEV`, something has split the single mount back into two. Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`** is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership — stop and check `ls -ldn` on the NAS before changing anything else. --- ## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs **A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on 2026-10-07: ``` $ kubectl patch pv sonarr-media --type=json --dry-run=server \ -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]' The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation ``` Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly and does not work either — Flux's apply errors on the five PVs and the pods keep their previous, now-dead mount. This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV rebinds its own PVC with no manual step. **Operator sequence** (mutating `kubectl` is an operator action, not an agent action): ```sh # 1. Stop the consumers. This also releases the hung hard-mounts against the old server: # the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops # answering, because the PVs mount `hard`. for ns in plex sonarr radarr qbittorrent nextcloud; do kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0 done # 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased. kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \ qbittorrent-downloads nextcloud-legacy-archive # 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them. # Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing. # 4. Bring the consumers back, then run every probe in §9. ``` The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.