From da8a0eac0485fe78513a1d634abecf3436723558 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 8 Sep 2026 00:03:34 -0500 Subject: [PATCH] =?UTF-8?q?test(l0):=20R-STATE-04=20gets=20a=20real=20chec?= =?UTF-8?q?k=20=E2=80=94=20bulk=20storage=20provably=20is=20not=20the=20re?= =?UTF-8?q?plicated=20tier?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate BLOCKED this branch, correctly, and the block was the interesting part. Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml, which say in as many words that naming the requirement would block the merge. That made sense while the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean shipping the requirement and claiming nothing. So: the check. WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments. That is the class of failure being closed: a bulk volume on the replicated tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold. (1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner (derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot), and the class is explicitly EMPTY. (2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the same service's replicated volume. The requirement is about two magnitudes of data; a "bulk" volume sized like a config PVC means the tier was named, not applied. The 10x floor is CALIBRATED against the real corpus, not guessed: the media library is ~460x its config volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate volume and teach the next reader to weaken the check. My first draft used 100x and did exactly that — the corpus caught it. (3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached. (4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft` — a soft mount hands a server hiccup to the app as a short read, which for a media library is silent corruption; and v4 is what keeps the firewall surface at one port. (6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer sees that in a diff. (7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so dropping one still leaves the others; and every bulk mode is a `...Many`. (8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay supplies it — the seam that lets one authored definition serve two clusters that reach this host by different addresses. (9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather than passing trivially, and a floor on the assertion count keeps a pass from degenerating. It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is already its own gate stage. PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete; `hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into two subPath mounts (assertion 6); and the prod overlay's knob deleted. Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy. Found by the same perturbation pass. Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical. BASE_REF=main make -C ci gate -> MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 --- catalog/render.py | 14 + ci/l0.sh | 38 ++ .../chainsaw-test.yaml | 99 ++++ .../check.sh | 480 ++++++++++++++++++ 4 files changed, 631 insertions(+) create mode 100644 conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh diff --git a/catalog/render.py b/catalog/render.py index e75141d..9fd868b 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -284,6 +284,20 @@ def r_bulk_pvs(s: dict) -> list: if vol.get("class") != "bulk": continue name = f"{s['name']}-{vol['name']}" + # `export` and `size` are both structurally required of a bulk entry, and the JSON + # Schema only enforces the first. A PersistentVolume with no `capacity` is rejected by + # the API server, so a missing size must fail HERE with a sentence a human can act on + # rather than as a bare KeyError three frames down. + for field, why in ( + ("export", "the absolute NFS export path this volume attaches"), + ("size", "the PV capacity (advisory, but the API requires the field)"), + ): + if not vol.get(field): + raise ValueError( + f"{s['name']}: storage entry {vol['name']!r} has class 'bulk' but no " + f"'{field}' — {why}. A bulk volume attaches storage that already exists " + f"outside the cluster, so it cannot be rendered without it." + ) nfs = {"server": BULK_NFS_SERVER, "path": vol["export"]} if vol.get("readOnly"): nfs["readOnly"] = True diff --git a/ci/l0.sh b/ci/l0.sh index 7101823..0c2bb28 100755 --- a/ci/l0.sh +++ b/ci/l0.sh @@ -175,6 +175,44 @@ else esac fi +# ---- Stage 3e: bulk tier distinct from the replicated tier (UNCONDITIONAL) ---- +# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only +# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four +# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated +# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data +# three mini-PCs cannot hold. The check asserts the tier separation, that a bulk volume is +# statically pre-bound from BOTH ends (so nothing can dynamically provision an EMPTY +# stand-in that mounts and Binds perfectly while the real data is never attached), Retain, +# hard/NFSv4 mount options, the read-only coupling between PV and volumeMount, that no +# workload mounts one export twice (which breaks Sonarr/Radarr hardlinks with EXDEV), and +# that the server is a per-environment knob every overlay supplies. +# +# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c/3d: Stage 3b scopes the conformance +# run to the TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's path map +# keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most +# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one +# export into two mounts — could attribute no requirement at all and be gated by nothing. +# The invariant spans catalog/, apps/ and clusters/, so it runs on EVERY gate invocation. +BULK_TIER_CHECK="${BULK_TIER_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh}" +log "Stage: bulk tier distinct from replicated tier (${BULK_TIER_CHECK})" +if [ ! -f "$BULK_TIER_CHECK" ]; then + skip "bulk tier check not present at '${BULK_TIER_CHECK}' — UNWIRED, not green" + summary "- bulk-tier: **SKIPPED** (check missing)" + if [ "$rc" -eq 0 ]; then rc=3; fi +else + bulk_rc=0 + bash "$BULK_TIER_CHECK" || bulk_rc=$? + case "$bulk_rc" in + 0) ok "Bulk tier distinctness + static binding passed" + summary "- bulk-tier: **passed**" ;; + 3) skip "bulk tier check SKIPPED (catalog/ or apps/ absent) — not a pass" + summary "- bulk-tier: **SKIPPED** (nothing to audit)" + if [ "$rc" -eq 0 ]; then rc=3; fi ;; + *) err "Bulk tier distinctness + static binding FAILED" + summary "- bulk-tier: **FAILED**"; rc=1 ;; + esac +fi + case "$rc" in 0) ok "L0 stage complete" ;; 3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;; diff --git a/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml b/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml new file mode 100644 index 0000000..c589489 --- /dev/null +++ b/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml @@ -0,0 +1,99 @@ +# ============================================================================= +# L0 STATIC — the bulk tier's capacity strategy is DISTINCT from the replicated tier's +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# THE BUG CLASS THIS CLOSES. The `bulk` StorageClass existed from Phase 0 as a stub: +# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME`, TODO(OQ-2). +# Four ServiceDefinitions carried a comment promising to attach their media once OQ-2 was +# resolved, and the ONLY thing keeping a multi-terabyte library off the 3-replica +# `longhorn` class was those four comments. A bulk volume that lands on the replicated +# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data +# three mini-PCs cannot hold. R-STATE-04 is precisely that the two tiers are distinct, so +# the distinctness is asserted rather than reviewed. +# +# It also pins the four decisions that make the STATIC form of this tier safe, each with a +# plausible "simplification" that breaks something silently: `Retain` (bulk data pre-exists +# the cluster and has consumers outside it, so anything else makes `delete pvc` a data-loss +# operation); NO dynamic provisioning (a dynamic bulk class hands out a FRESH EMPTY +# directory that mounts and Binds perfectly while the real 8 TiB is never attached — the +# most deceptive failure available here); `hard` not `soft` (a soft mount hands a server +# hiccup to the app as a short read, which for a media library is silent corruption); and +# NFSv4 (mount and lock in-protocol ⇒ ONE reachable port, tcp/2049, which is what the +# operator must open in pfSense for prod to reach this server at all). +# +# AND the EXDEV trap, which is not about storage. Sonarr/Radarr import by hardlinking or +# atomically renaming, both same-filesystem operations, and to the kernel "same filesystem" +# means SAME MOUNT. Splitting one export into two subPath mounts looks TIGHTER and breaks +# link() with EXDEV; the arrs then degrade to copy-then-delete, rewriting every import over +# NFS and destroying the seeding copy. It surfaces weeks later as "imports are slow and my +# ratio is gone". A reviewer cannot see that; this check can. +# +# GIVEN the repository as authored, +# WHEN every `class: bulk` storage entry in catalog/ is compared against the objects the +# COMMITTED apps/ goldens carry for it (what Flux reconciles — deliberately not the +# renderer, which would only prove it agrees with itself), +# THEN (1) no bulk volume is on a Longhorn-provisioned class, and its class is explicitly +# EMPTY so nothing can dynamically provision an empty stand-in; +# (2) its capacity is declared and is at least an order of magnitude above the same +# service's replicated volume — the requirement stated as arithmetic; +# (3) PV and PVC are pre-bound from BOTH ends (claimRef + volumeName) and agree; +# (4) the PV reclaim policy is Retain; +# (5) mountOptions carry `hard` and exactly one `nfsvers=4.x`, and never `soft`; +# (6) no workload mounts one export more than once (the EXDEV trap); +# (7) a readOnly entry renders nfs.readOnly AND a readOnly volumeMount AND +# ReadOnlyMany; a writable one renders neither and is ReadWriteMany; every bulk +# access mode is a `...Many`, since one export serves several namespaces and +# several nodes; +# (8) the PV's nfs.server is a BARE cluster-vars token (never a literal — dev and +# prod reach this host by different addresses), that knob has a base default, and +# EVERY overlay supplies it; +# AND (9) the check is LOAD-BEARING (ci/l0.sh runs it unconditionally, ci/gate.sh blocks +# on the l0 stage) and NON-VACUOUS (an empty bulk corpus FAILS rather than +# passing trivially, and a floor on the assertion count keeps a pass from +# degenerating into a no-op). +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + +# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the +# SAME script. +# +# NOT proven — the runtime half, which needs a cluster: that the server is reachable, that +# the export admits these clients, that the PVC binds, that a write succeeds, or that a +# hardlink between the library and the download path actually works. The uid analysis is +# reasoning (docs/BULK-NFS-STORAGE.md §6) and the in-cluster probes that would settle it +# are written down in §9 of that document, unrun. Nor can this prove prod's address is +# CORRECT — only that it is a supplied knob; while it is a placeholder, prod's pods will +# not mount, and catching that statically needs the placeholder guard, which is absent from +# main (§7). Read-only on the media tree is a CLIENT-side guarantee only: the export serves +# one reader and two writers from the same node addresses, so the server cannot tell them +# apart (§5). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: bulk-tier-distinct-from-replicated + labels: + conformance.home.k3s/R-STATE-04: "true" + annotations: + conformance.home.k3s/requirements: "R-STATE-04" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Bulk media storage is provably NOT on the 3-replica Longhorn tier, is sized for a + different order of magnitude, cannot be silently replaced by a dynamically + provisioned empty volume, is never reclaimable by the cluster, mounts hard over + NFSv4, and is never split into two mounts of one export — so hardlinked imports keep + working. Its server is a per-environment knob, not a literal, because the two + clusters reach the same storage host by different addresses. +spec: + steps: + - name: assert-bulk-tier-is-distinct-and-statically-bound + description: >- + Read every `class: bulk` storage entry from catalog/, pair it with the + PersistentVolume, PersistentVolumeClaim and volumeMount the committed apps/ goldens + carry for it, and assert the tier separation, the static pre-binding, the reclaim + and mount-option decisions, the read-only coupling, the one-export-one-mount + invariant, and that the server address is supplied by every environment overlay. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh b/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh new file mode 100755 index 0000000..da8dcc5 --- /dev/null +++ b/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh @@ -0,0 +1,480 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — bulk-tier-distinct-from-replicated +# Serves: R-STATE-04 ("Bulk media storage has a capacity strategy distinct from the +# replicated tier." — spec docs/04-storage-and-state.md) +# +# WHY THIS EXISTS — THE BULK TIER WAS A STUB FOR SIX ROADMAP PHASES, AND STUBS ROT QUIETLY +# ---------------------------------------------------------------------------------------- +# `platform/longhorn/storageclasses.yaml` declared a `bulk` StorageClass from Phase 0 with +# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME` and a TODO(OQ-2). +# Four ServiceDefinitions carried a comment promising to attach their media "once OQ-2 is +# resolved", and each of them said, correctly, that adding `class: bulk` before then would +# leave the PVC unbound and the pod Pending. So the tier existed as a NAME with nothing +# behind it, and the only thing preventing a multi-terabyte media library from being +# authored onto the 3-replica `longhorn` class was four hand-written comments. +# +# That is the failure this check makes impossible to reintroduce. R-STATE-04 is not "bulk +# storage exists" — it is that bulk storage's capacity strategy is DISTINCT from the +# replicated tier's. Three mini-PCs cannot hold 3x of 8.33 TiB. A bulk volume that lands on +# `longhorn` does not fail loudly: Longhorn accepts the PVC and starts replicating, and the +# tier fills up. Distinctness is therefore a property worth asserting statically, on every +# gate run, rather than trusting to review. +# +# It also pins the four decisions that make the STATIC form of this tier safe, each of +# which has a plausible-looking "simplification" that silently breaks something: +# +# * RECLAIM POLICY. Bulk data pre-exists the cluster, is larger than the cluster, and has +# consumers OUTSIDE it. `Retain` is the whole reason a PV is used here instead of an +# inline volume. `Delete` on a dynamically-provisioned bulk class would make `kubectl +# delete pvc` a data-loss operation. +# * NO DYNAMIC PROVISIONING. A dynamic bulk class hands out FRESH EMPTY directories. The +# pod mounts successfully, the PVC is Bound, everything is green — and the 8 TiB of +# existing data was never attached. This is the single most deceptive failure available +# in this tier, so `storageClassName` must be explicitly empty and binding must be +# pinned from BOTH ends (claimRef + volumeName). +# * `hard`, NOT `soft`. A soft NFS mount converts a server hiccup into a short read or +# EIO handed to the application. For a media library that is silent corruption. `hard` +# hangs instead — recoverable, where truncated is not. +# * NFSv4. v4 carries mount and lock in-protocol, so the tier needs ONE reachable port +# (tcp/2049) instead of v3's rpcbind/111 + mountd + statd on ephemeral numbers. That is +# a FIREWALL fact as much as a storage one: prod reaches this server through pfSense, +# and the number of ports the operator must open is a consequence of this field. +# +# AND ONE INVARIANT THAT IS NOT ABOUT STORAGE AT ALL — the EXDEV trap (assertion 6). +# Sonarr and Radarr import by hardlinking or atomically renaming a finished download into +# the library. Both are same-filesystem operations, and to the kernel "same filesystem" +# means SAME MOUNT. Mounting one export twice (say `subPath: tv` and `subPath: downloads`) +# looks TIGHTER — less of the tree exposed per mount — and breaks `link()` with EXDEV no +# matter that the server knows they are one dataset. The arrs do not fail loudly on that: +# they degrade to copy-then-delete, rewriting every imported file over NFS and destroying +# the seeding copy. The symptom appears weeks later as "imports are slow and my ratio is +# gone". A reviewer cannot see it, so this check does: one export, at most one mount, per +# workload. +# +# GIVEN the repository as authored, +# WHEN every `class: bulk` storage entry in catalog/ is compared against the manifests +# render.py produces for it (read from the committed apps/ goldens, i.e. what Flux +# actually reconciles — not from the renderer, which would only agree with itself), +# THEN (1) DISTINCTNESS: no bulk volume is on the replicated tier. Its PV/PVC +# storageClassName is explicitly empty — never `longhorn`, never any class the +# repo declares with the Longhorn provisioner — and its PV's source is an +# EXTERNAL one (nfs), not a Longhorn volume; +# (2) CAPACITY STRATEGY: every bulk PV declares a capacity, and for each service the +# bulk volume is at least an ORDER OF MAGNITUDE larger than that service's own +# replicated volume. This is the requirement stated as arithmetic: the two tiers +# exist because the data has two magnitudes, so a "bulk" volume sized like a +# config PVC means the strategy was named and not actually applied; +# (3) NOT DYNAMICALLY PROVISIONED, AND PRE-BOUND BOTH WAYS: every bulk PVC names its +# PV by volumeName, every bulk PV names its PVC by claimRef (namespace+name), and +# the two agree — so no other claim can win a race, and nothing can provision an +# empty replacement; +# (4) RECLAIM: every bulk PV is `Retain`; +# (5) MOUNT SAFETY: every bulk PV declares mountOptions carrying `hard` and an +# `nfsvers=4.x`, and carrying NEITHER `soft` NOR an `nfsvers=3`; +# (6) ONE EXPORT, ONE MOUNT PER WORKLOAD (the EXDEV trap above); +# (7) READ-ONLY IS REAL AND CONSISTENT: a bulk entry declared readOnly renders BOTH +# `spec.nfs.readOnly: true` on the PV AND `readOnly: true` on the volumeMount, +# and its access mode is ReadOnlyMany; a writable one renders NEITHER and is +# ReadWriteMany. Every bulk access mode is a `...Many` — one export is consumed +# by several namespaces and several nodes, which ReadWriteOnce cannot express; +# (8) AUTHORED ONCE, PROMOTED (the R-ENV-02 seam this tier depends on): every bulk +# PV's `nfs.server` is the BARE cluster-vars token, never a literal address; that +# knob has a base default; and EVERY environment overlay supplies it — so the two +# clusters, which reach this storage by different addresses, cannot be served by +# one hardcoded value; +# AND (9) LOAD-BEARING + NON-VACUOUS: ci/l0.sh runs this check unconditionally and +# ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT +# empty — a repo with zero bulk volumes fails here rather than passing trivially, +# because a check that is satisfiable by deleting the feature proves nothing. +# +# WHAT THIS DOES *NOT* PROVE (honest gaps) +# - Nothing about RUNTIME. Not that the server is reachable, not that the export permits +# these clients, not that the PVC binds, not that a write succeeds. The uid analysis in +# docs/BULK-NFS-STORAGE.md §6 is reasoning, and the write/hardlink probes in §9 are the +# in-cluster proof — they need a cluster and are not run here. +# - It cannot prove the prod address is CORRECT, only that it is a supplied knob. While +# prod's value is a placeholder, prod's pods will not mount (docs/BULK-NFS-STORAGE.md +# §7); catching that needs the placeholder guard, which is absent from main. +# - Read-only for the media tree is a CLIENT-side guarantee, and this check can only +# assert the client side. The export serves a reader and two writers from the same node +# addresses, so the server cannot distinguish them (docs/BULK-NFS-STORAGE.md §5). +# - Assertion (2)'s magnitude ratio is a heuristic with a deliberately generous floor. It +# catches "bulk volume sized like a config volume", not a subtly wrong capacity — and a +# static NFS PV's capacity is advisory anyway, since the ZFS pool is the real limit. +# - Assertion (6) reasons over ONE workload's mounts. It cannot see two different +# services hardlinking between each other's mounts, which is not a thing that happens. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" + +if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ]; then + echo "check SKIPPED (bulk-tier-distinct-from-replicated): catalog/ or apps/ absent — nothing to audit" >&2 + exit 3 +fi + +"${PYTHON}" - "${ROOT}" <<'PY' +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def bail(msg): + """A failure that makes the rest of the check meaningless: report and stop.""" + print(f"CHECK FAIL (bulk-tier-distinct-from-replicated): {msg}", file=sys.stderr) + raise SystemExit(1) + + +def qty_to_bytes(q): + """Kubernetes quantity -> bytes. Only the binary suffixes the schema permits.""" + m = re.fullmatch(r"([0-9]+)(Mi|Gi|Ti)", str(q).strip()) + if not m: + return None + return int(m.group(1)) * {"Mi": 2 ** 20, "Gi": 2 ** 30, "Ti": 2 ** 40}[m.group(2)] + + +# ===================================================================================== +# The replicated tier, named by the repo rather than by this file. Any StorageClass the +# repo declares with the Longhorn provisioner is "the replicated tier" for assertion (1); +# deriving it means adding a new Longhorn class cannot create a blind spot here. +# ===================================================================================== +REPLICATED_CLASSES: set[str] = set() +LONGHORN_DIR = ROOT / "platform" / "longhorn" +if LONGHORN_DIR.is_dir(): + for f in sorted(LONGHORN_DIR.rglob("*.yaml")): + for doc in yaml.safe_load_all(f.read_text()): + if isinstance(doc, dict) and doc.get("kind") == "StorageClass": + if "longhorn" in str(doc.get("provisioner", "")): + REPLICATED_CLASSES.add(doc.get("metadata", {}).get("name", "")) +# `longhorn` is the name every ServiceDefinition uses for the replicated tier, so it is in +# the set even if platform/longhorn/ is not present in this checkout. +REPLICATED_CLASSES.add("longhorn") +REPLICATED_CLASSES.discard("") + +# ===================================================================================== +# The bulk corpus: every `class: bulk` storage entry in every ServiceDefinition, paired +# with the objects the COMMITTED goldens carry for it. Reading the goldens (not calling +# the renderer) is deliberate: the goldens are what Flux reconciles, and render==golden is +# a separate gate stage. Asking the renderer would only prove it agrees with itself. +# ===================================================================================== +sds: dict[str, dict] = {} +for f in sorted((ROOT / "catalog").glob("*.yaml")): + doc = yaml.safe_load(f.read_text()) + if isinstance(doc, dict) and doc.get("name"): + sds[doc["name"]] = doc + +if not sds: + bail("no ServiceDefinitions under catalog/ — cannot audit the bulk tier") + +# Every rendered app-tier document, indexed by (kind, name). +rendered: dict[tuple, dict] = {} +for f in sorted((ROOT / "apps").rglob("*.yaml")): + if f.name == "kustomization.yaml" or ".sops." in f.name: + continue + try: + docs = list(yaml.safe_load_all(f.read_text())) + except yaml.YAMLError: + continue + for doc in docs: + if isinstance(doc, dict) and doc.get("kind") and doc.get("metadata", {}).get("name"): + rendered[(doc["kind"], doc["metadata"]["name"])] = doc + +# (svc, vol_entry) for every bulk entry, and the replicated sizes alongside for (2). +corpus: list[tuple[str, dict]] = [] +replicated_bytes: dict[str, list[int]] = {} +for svc, sd in sds.items(): + for vol in sd.get("storage") or []: + cls = vol.get("class", "longhorn") + if cls == "bulk": + corpus.append((svc, vol)) + else: + b = qty_to_bytes(vol.get("size")) + if b: + replicated_bytes.setdefault(svc, []).append(b) + +# ------------------------------------------------------------------- (9) NON-VACUOUS +# A corpus of zero would make every loop below trivially satisfied. The bulk tier is +# DELIVERED; a change that removes the last bulk volume must fail here and be argued for, +# not sail through on an empty iteration. +check(len(corpus) > 0, + "NO `class: bulk` storage entry exists in catalog/ — this check would then pass " + "vacuously, which is exactly the posture that let the bulk tier sit as a REPLACE-ME " + "stub for six roadmap phases. If the tier is genuinely being retired, retire this " + "check in the same change and say so; do not let it pass on an empty corpus") +if not corpus: + bail("empty bulk corpus — refusing to report a vacuous pass (see assertion 9)") + +inventory: list[str] = [] + +for svc, vol in corpus: + vname = vol.get("name", "") + label = f"{svc}/{vname}" + obj = f"{svc}-{vname}" + ro = bool(vol.get("readOnly")) + + # `export` is what the schema requires of a bulk entry; without it there is nothing to + # attach and the rest of the assertions have no subject. + export = vol.get("export") + check(bool(export) and str(export).startswith("/"), + f"{label}: bulk entry has no absolute `export` path — a bulk volume attaches an " + f"EXISTING dataset, so it must say which one (schema requires it when class is bulk)") + + pv = rendered.get(("PersistentVolume", obj)) + pvc = rendered.get(("PersistentVolumeClaim", obj)) + check(pv is not None, + f"{label}: no PersistentVolume '{obj}' in the committed apps/ goldens. A bulk " + f"entry that renders no PV has NO external storage behind it — the pod would get " + f"an unbound PVC, or (worse, with a dynamic class) a fresh EMPTY directory") + check(pvc is not None, + f"{label}: no PersistentVolumeClaim '{obj}' in the committed apps/ goldens") + if pv is None or pvc is None: + continue + + pvs, pvcs = pv.get("spec", {}) or {}, pvc.get("spec", {}) or {} + + # ------------------------------------------------------ (1) DISTINCT FROM REPLICATED + for what, spec in (("PV", pvs), ("PVC", pvcs)): + sc = spec.get("storageClassName") + check(sc not in REPLICATED_CLASSES, + f"{label}: bulk {what} storageClassName is '{sc}', which the repo declares " + f"with the Longhorn provisioner. That puts BULK data on the REPLICATED tier " + f"— 3x of a multi-terabyte library across three mini-PCs. Longhorn will not " + f"refuse it; it will accept the PVC and fill the tier (R-STATE-04)") + check(sc == "", + f"{label}: bulk {what} storageClassName is {sc!r}, not the explicitly EMPTY " + f"string. Empty is what disables dynamic provisioning AND the default-class " + f"annotation. Any named class risks a provisioner handing out a fresh EMPTY " + f"directory that mounts and binds cleanly while the real data is never attached") + check("nfs" in pvs, + f"{label}: bulk PV declares no external `nfs` source — the bulk tier's whole " + f"premise is that durability lives OUTSIDE the cluster (docs/04)") + + # ------------------------------------------------------------ (4) RECLAIM POLICY + check(pvs.get("persistentVolumeReclaimPolicy") == "Retain", + f"{label}: bulk PV reclaim policy is " + f"{pvs.get('persistentVolumeReclaimPolicy')!r}, not Retain. This data pre-exists " + f"the cluster and has consumers outside it; anything but Retain makes `kubectl " + f"delete pvc` a data-loss operation") + + # ------------------------------------------------- (3) PRE-BOUND FROM BOTH ENDS + check(pvcs.get("volumeName") == obj, + f"{label}: bulk PVC does not name its PV via volumeName (got " + f"{pvcs.get('volumeName')!r}, expected {obj!r}) — without it the claim can be " + f"satisfied by some OTHER volume of matching size and mode") + cref = pvs.get("claimRef") or {} + check(cref.get("name") == obj and cref.get("namespace") == svc, + f"{label}: bulk PV claimRef is {cref!r}, expected namespace={svc!r} name={obj!r} " + f"— pre-binding must be pinned from BOTH ends or another PVC can win the race") + + # ------------------------------------------------------------- (5) MOUNT SAFETY + mopts = [str(o) for o in (pvs.get("mountOptions") or [])] + check(bool(mopts), + f"{label}: bulk PV declares no mountOptions — then `hard` vs `soft` and the NFS " + f"version are whatever the client happens to default to, and both are decisions " + f"with consequences (silent short reads; how many firewall ports must open)") + check("hard" in mopts, + f"{label}: bulk PV mountOptions do not include `hard` ({mopts}). A soft mount " + f"hands a server hiccup to the application as a short read or EIO — silent " + f"corruption for a media library. `hard` hangs instead, which is recoverable") + check("soft" not in mopts, + f"{label}: bulk PV mountOptions include `soft` ({mopts}) — see above; this is the " + f"one option that turns an outage into damaged data") + vers = [o for o in mopts if o.startswith("nfsvers=")] + check(len(vers) == 1 and re.fullmatch(r"nfsvers=4(\.[0-9]+)?", vers[0]), + f"{label}: bulk PV must pin exactly one NFSv4 `nfsvers=` option (got {vers}). " + f"NFSv4 carries mount and lock in-protocol, so the tier needs ONE reachable port " + f"(tcp/2049); v3 also needs rpcbind/111, mountd and statd, which changes what the " + f"operator has to open in pfSense for prod to reach this server at all") + + # ---------------------------------------------- (7) READ-ONLY IS REAL + CONSISTENT + want_mode = "ReadOnlyMany" if ro else "ReadWriteMany" + for what, spec in (("PV", pvs), ("PVC", pvcs)): + modes = spec.get("accessModes") or [] + check(modes == [want_mode], + f"{label}: bulk {what} accessModes {modes} != ['{want_mode}']. Every bulk " + f"volume is a `...Many` (one export, several namespaces, several nodes — " + f"ReadWriteOnce cannot express that), and WHICH `Many` is how the read/write " + f"posture stays legible in `kubectl get pv`") + nfs_ro = bool((pvs.get("nfs") or {}).get("readOnly")) + check(nfs_ro == ro, + f"{label}: ServiceDefinition readOnly={ro} but the PV's nfs.readOnly={nfs_ro}. " + f"The PV field is what makes the kernel mount `ro`; a mismatch means the declared " + f"posture is not the one the kernel enforces") + + # The volumeMount half, read out of the rendered Deployment. + dep = rendered.get(("Deployment", svc)) + check(dep is not None, f"{label}: no rendered Deployment '{svc}' to check the mount against") + if dep is not None: + containers = (((dep.get("spec") or {}).get("template") or {}).get("spec") or {}).get("containers") or [] + # The app container is containers[0] by the renderer's contract (a forward-auth + # sidecar is appended AFTER it). + mounts = [m for m in (containers[0].get("volumeMounts") or []) if m.get("name") == vname] if containers else [] + check(len(mounts) == 1, + f"{label}: the app container has {len(mounts)} volumeMount(s) named {vname!r}, " + f"expected exactly 1 — a bulk volume that is declared and never mounted is a " + f"PV/PVC pair doing nothing, and two mounts of one volume is assertion (6)") + if len(mounts) == 1: + m = mounts[0] + check(bool(m.get("readOnly")) == ro, + f"{label}: volumeMount readOnly={bool(m.get('readOnly'))} but the " + f"ServiceDefinition says readOnly={ro} — the PV and the mount must agree, " + f"so that dropping one still leaves the other") + check(m.get("subPath", vol.get("subPath")) == vol.get("subPath"), + f"{label}: volumeMount subPath {m.get('subPath')!r} does not match the " + f"declared subPath {vol.get('subPath')!r}") + + # ------------------------------------------ (8) AUTHORED ONCE, SUPPLIED PER ENV + server = (pvs.get("nfs") or {}).get("server") + m = re.fullmatch(r"\$\{([A-Z0-9_]+)\}", str(server or "")) + check(m is not None, + f"{label}: bulk PV nfs.server is {server!r} — it must be a BARE cluster-vars " + f"token, not a literal address. dev and prod reach this storage host by DIFFERENT " + f"addresses (dev is on its LAN; prod is across pfSense), so a literal is correct " + f"in at most one environment and silently wrong in the other (R-ENV-02)") + + cap = (pvs.get("capacity") or {}).get("storage") + req = ((pvcs.get("resources") or {}).get("requests") or {}).get("storage") + check(cap is not None and qty_to_bytes(cap) is not None, + f"{label}: bulk PV declares no parseable capacity ({cap!r}) — the API requires it, " + f"and it is what makes `kubectl get pv` report something true about this tier") + check(cap == req, + f"{label}: PV capacity {cap!r} != PVC request {req!r}; a request above the " + f"capacity never binds") + + inventory.append( + f" {label:<28} export={export} {'ro' if ro else 'rw'} {want_mode:<14} " + f"cap={cap} server={server} opts={','.join(mopts)}" + ) + + # ---------------------------------- (2) CAPACITY STRATEGY: two tiers, two magnitudes + cap_b = qty_to_bytes(cap) + reps = replicated_bytes.get(svc) or [] + if cap_b and reps: + biggest = max(reps) + # ONE ORDER OF MAGNITUDE. Calibrated against the real corpus rather than guessed: + # the media library is ~460x its service's config volume, and the Nextcloud archive + # ~13x its 20Gi Longhorn volume — so a floor above ~13x would fire on a legitimate + # bulk volume, and this check would be teaching people to weaken it. 10x cannot be + # met by accident and still catches the regression that matters: a "bulk" volume + # sized like a config PVC, i.e. the strategy named but never applied. + check(cap_b >= 10 * biggest, + f"{label}: bulk capacity {cap} is less than 10x this service's largest " + f"replicated volume ({biggest // 2 ** 30}Gi). R-STATE-04 exists because the " + f"two tiers hold data of two different MAGNITUDES — the replicated tier keeps " + f"the small config/DB and the bulk tier keeps what cannot be replicated 3x. A " + f"bulk volume sized like a config volume means the tier was named, not applied") + +# ------------------------------------------------- (6) ONE EXPORT, AT MOST ONE MOUNT +# The EXDEV trap. Grouped per SERVICE, because the hazard is two mounts inside ONE pod. +by_service: dict[str, list[dict]] = {} +for svc, vol in corpus: + by_service.setdefault(svc, []).append(vol) +for svc, vols in sorted(by_service.items()): + seen: dict[str, str] = {} + for vol in vols: + exp = vol.get("export") + prev = seen.get(exp) + check(prev is None, + f"{svc}: mounts the export {exp} TWICE (volumes {prev!r} and " + f"{vol.get('name')!r}). Two mounts of one export are two FILESYSTEMS to the " + f"kernel, so link() between them fails EXDEV even though the server knows " + f"they are one dataset. Sonarr/Radarr do not fail loudly on that — they " + f"degrade to copy-then-delete, rewriting every imported file over NFS and " + f"destroying the seeding copy. Mount the common ancestor ONCE instead " + f"(docs/BULK-NFS-STORAGE.md §4)") + seen[exp] = vol.get("name") + +# ------------------------------------- (8b) the knob has a base default AND every overlay +tokens = set() +for _svc, vol in corpus: + pv = rendered.get(("PersistentVolume", f"{_svc}-{vol.get('name')}")) + if pv: + mm = re.fullmatch(r"\$\{([A-Z0-9_]+)\}", str(((pv.get("spec") or {}).get("nfs") or {}).get("server") or "")) + if mm: + tokens.add(mm.group(1)) + +cv = ROOT / "clusters" / "base" / "cluster-vars.yaml" +base_keys: set[str] = set() +if cv.is_file(): + doc = yaml.safe_load(cv.read_text()) or {} + base_keys = set((doc.get("data") or {}).keys()) + +overlays = sorted( + d for d in (ROOT / "clusters").iterdir() + if cv.is_file() and d.is_dir() and d.name != "base" and (d / "kustomization.yaml").is_file() +) if (ROOT / "clusters").is_dir() else [] + +for tok in sorted(tokens): + check(tok in base_keys, + f"bulk PVs reference ${{{tok}}} but clusters/base/cluster-vars.yaml declares no " + f"such knob. Under StrictPostBuildSubstitutions that fails the whole tier; without " + f"it the token ships into the cluster as a literal server name") + for ov in overlays: + text = (ov / "kustomization.yaml").read_text() + check(f"/data/{tok}" in text, + f"clusters/{ov.name} does not supply ${{{tok}}}. The base default is the DEV " + f"value by this repo's convention, so an overlay that omits it silently " + f"inherits an address on the WRONG SIDE of pfSense — a mount that either " + f"never connects or connects to something unintended (R-ENV-02)") + +# --------------------------------------------------------------- (9) LOAD-BEARING +L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" +GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" +check("bulk-tier-distinct-from-replicated/check.sh" in L0, + "ci/l0.sh does not run bulk-tier-distinct-from-replicated/check.sh. Stage 3b scopes " + "the conformance run to TOUCHED requirements, and ci/touched-requirements.sh maps " + "neither apps/* nor platform/longhorn/* to the bulk requirement — so a change that " + "moves a bulk volume onto the replicated tier could attribute no requirement at all " + "and be gated by nothing. It must be invoked directly and unconditionally") +check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), + "ci/gate.sh does not set the merge block when the l0 stage fails") + +# ------------------------------------------------------------------------- report +print("Bulk tier inventory (from catalog/ + the committed apps/ goldens):") +for line in inventory: + print(line) +print(f"\nreplicated tier StorageClasses (derived from platform/longhorn/): " + f"{', '.join(sorted(REPLICATED_CLASSES))}") +print(f"bulk volumes: {len(corpus)} distinct exports: " + f"{len({v.get('export') for _s, v in corpus})} " + f"server knob(s): {', '.join(sorted(tokens)) or 'NONE'}") + +# A floor on the assertion count, for the same reason the corpus may not be empty: a pass +# must not be able to degenerate into "nothing was actually asked". +MIN_CHECKS = 40 +check(CHECKS >= MIN_CHECKS, + f"only {CHECKS} assertions ran (floor {MIN_CHECKS}) — a pass this thin is not " + f"evidence; the corpus or the extraction has degraded") + +print() +if FAILURES: + print(f"CHECK FAIL (bulk-tier-distinct-from-replicated): {len(FAILURES)} finding(s)", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) + +print(f"check OK: bulk-tier-distinct-from-replicated — {CHECKS} assertions (R-STATE-04)") +PY