test(l0): R-STATE-04 gets a real check — bulk storage provably is not the replicated tier

The gate BLOCKED this branch, correctly, and the block was the interesting part.

Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.

WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.

  (1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
      (derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
      and the class is explicitly EMPTY.
  (2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
      same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
      volume sized like a config PVC means the tier was named, not applied. The 10x floor is
      CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
      volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
      volume and teach the next reader to weaken the check. My first draft used 100x and did
      exactly that — the corpus caught it.
  (3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
      failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
      pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
  (4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
      — a soft mount hands a server hiccup to the app as a short read, which for a media library
      is silent corruption; and v4 is what keeps the firewall surface at one port.
  (6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
      one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
      Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
      seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
      sees that in a diff.
  (7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
      dropping one still leaves the others; and every bulk mode is a `...Many`.
  (8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
      supplies it — the seam that lets one authored definition serve two clusters that reach this
      host by different addresses.
  (9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
      reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
      apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
      this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
      than passing trivially, and a floor on the assertion count keeps a pass from degenerating.

It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.

PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.

Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.

Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-08 00:03:34 -05:00
1 parent 82aff4d053
commit da8a0eac04
4 files changed
+631

No files matched your search

+38
View File
@@ -175,6 +175,44 @@ else
esac
fi
# ---- Stage 3e: bulk tier distinct from the replicated tier (UNCONDITIONAL) ----
# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
# three mini-PCs cannot hold. The check asserts the tier separation, that a bulk volume is
# statically pre-bound from BOTH ends (so nothing can dynamically provision an EMPTY
# stand-in that mounts and Binds perfectly while the real data is never attached), Retain,
# hard/NFSv4 mount options, the read-only coupling between PV and volumeMount, that no
# workload mounts one export twice (which breaks Sonarr/Radarr hardlinks with EXDEV), and
# that the server is a per-environment knob every overlay supplies.
#
# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c/3d: Stage 3b scopes the conformance
# run to the TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's path map
# keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
# export into two mounts — could attribute no requirement at all and be gated by nothing.
# The invariant spans catalog/, apps/ and clusters/, so it runs on EVERY gate invocation.
BULK_TIER_CHECK="${BULK_TIER_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh}"
log "Stage: bulk tier distinct from replicated tier (${BULK_TIER_CHECK})"
if [ ! -f "$BULK_TIER_CHECK" ]; then
skip "bulk tier check not present at '${BULK_TIER_CHECK}' — UNWIRED, not green"
summary "- bulk-tier: **SKIPPED** (check missing)"
if [ "$rc" -eq 0 ]; then rc=3; fi
else
bulk_rc=0
bash "$BULK_TIER_CHECK" || bulk_rc=$?
case "$bulk_rc" in
0) ok "Bulk tier distinctness + static binding passed"
summary "- bulk-tier: **passed**" ;;
3) skip "bulk tier check SKIPPED (catalog/ or apps/ absent) — not a pass"
summary "- bulk-tier: **SKIPPED** (nothing to audit)"
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
*) err "Bulk tier distinctness + static binding FAILED"
summary "- bulk-tier: **FAILED**"; rc=1 ;;
esac
fi
case "$rc" in
0) ok "L0 stage complete" ;;
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;