jakman85andClaude Opus 5
da8a0eac04
The gate BLOCKED this branch, correctly, and the block was the interesting part.
Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.
WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.
(1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
(derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
and the class is explicitly EMPTY.
(2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
volume sized like a config PVC means the tier was named, not applied. The 10x floor is
CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
volume and teach the next reader to weaken the check. My first draft used 100x and did
exactly that — the corpus caught it.
(3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
(4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
— a soft mount hands a server hiccup to the app as a short read, which for a media library
is silent corruption; and v4 is what keeps the firewall surface at one port.
(6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
sees that in a diff.
(7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
dropping one still leaves the others; and every bulk mode is a `...Many`.
(8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
supplies it — the seam that lets one authored definition serve two clusters that reach this
host by different addresses.
(9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
than passing trivially, and a floor on the assertion count keeps a pass from degenerating.
It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.
PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.
Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.
Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
100 lines
6.4 KiB
YAML
100 lines
6.4 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the bulk tier's capacity strategy is DISTINCT from the replicated tier's
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# THE BUG CLASS THIS CLOSES. The `bulk` StorageClass existed from Phase 0 as a stub:
|
|
# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME`, TODO(OQ-2).
|
|
# Four ServiceDefinitions carried a comment promising to attach their media once OQ-2 was
|
|
# resolved, and the ONLY thing keeping a multi-terabyte library off the 3-replica
|
|
# `longhorn` class was those four comments. A bulk volume that lands on the replicated
|
|
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
|
|
# three mini-PCs cannot hold. R-STATE-04 is precisely that the two tiers are distinct, so
|
|
# the distinctness is asserted rather than reviewed.
|
|
#
|
|
# It also pins the four decisions that make the STATIC form of this tier safe, each with a
|
|
# plausible "simplification" that breaks something silently: `Retain` (bulk data pre-exists
|
|
# the cluster and has consumers outside it, so anything else makes `delete pvc` a data-loss
|
|
# operation); NO dynamic provisioning (a dynamic bulk class hands out a FRESH EMPTY
|
|
# directory that mounts and Binds perfectly while the real 8 TiB is never attached — the
|
|
# most deceptive failure available here); `hard` not `soft` (a soft mount hands a server
|
|
# hiccup to the app as a short read, which for a media library is silent corruption); and
|
|
# NFSv4 (mount and lock in-protocol ⇒ ONE reachable port, tcp/2049, which is what the
|
|
# operator must open in pfSense for prod to reach this server at all).
|
|
#
|
|
# AND the EXDEV trap, which is not about storage. Sonarr/Radarr import by hardlinking or
|
|
# atomically renaming, both same-filesystem operations, and to the kernel "same filesystem"
|
|
# means SAME MOUNT. Splitting one export into two subPath mounts looks TIGHTER and breaks
|
|
# link() with EXDEV; the arrs then degrade to copy-then-delete, rewriting every import over
|
|
# NFS and destroying the seeding copy. It surfaces weeks later as "imports are slow and my
|
|
# ratio is gone". A reviewer cannot see that; this check can.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN every `class: bulk` storage entry in catalog/ is compared against the objects the
|
|
# COMMITTED apps/ goldens carry for it (what Flux reconciles — deliberately not the
|
|
# renderer, which would only prove it agrees with itself),
|
|
# THEN (1) no bulk volume is on a Longhorn-provisioned class, and its class is explicitly
|
|
# EMPTY so nothing can dynamically provision an empty stand-in;
|
|
# (2) its capacity is declared and is at least an order of magnitude above the same
|
|
# service's replicated volume — the requirement stated as arithmetic;
|
|
# (3) PV and PVC are pre-bound from BOTH ends (claimRef + volumeName) and agree;
|
|
# (4) the PV reclaim policy is Retain;
|
|
# (5) mountOptions carry `hard` and exactly one `nfsvers=4.x`, and never `soft`;
|
|
# (6) no workload mounts one export more than once (the EXDEV trap);
|
|
# (7) a readOnly entry renders nfs.readOnly AND a readOnly volumeMount AND
|
|
# ReadOnlyMany; a writable one renders neither and is ReadWriteMany; every bulk
|
|
# access mode is a `...Many`, since one export serves several namespaces and
|
|
# several nodes;
|
|
# (8) the PV's nfs.server is a BARE cluster-vars token (never a literal — dev and
|
|
# prod reach this host by different addresses), that knob has a base default, and
|
|
# EVERY overlay supplies it;
|
|
# AND (9) the check is LOAD-BEARING (ci/l0.sh runs it unconditionally, ci/gate.sh blocks
|
|
# on the l0 stage) and NON-VACUOUS (an empty bulk corpus FAILS rather than
|
|
# passing trivially, and a floor on the assertion count keeps a pass from
|
|
# degenerating into a no-op).
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
|
# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
|
|
# SAME script.
|
|
#
|
|
# NOT proven — the runtime half, which needs a cluster: that the server is reachable, that
|
|
# the export admits these clients, that the PVC binds, that a write succeeds, or that a
|
|
# hardlink between the library and the download path actually works. The uid analysis is
|
|
# reasoning (docs/BULK-NFS-STORAGE.md §6) and the in-cluster probes that would settle it
|
|
# are written down in §9 of that document, unrun. Nor can this prove prod's address is
|
|
# CORRECT — only that it is a supplied knob; while it is a placeholder, prod's pods will
|
|
# not mount, and catching that statically needs the placeholder guard, which is absent from
|
|
# main (§7). Read-only on the media tree is a CLIENT-side guarantee only: the export serves
|
|
# one reader and two writers from the same node addresses, so the server cannot tell them
|
|
# apart (§5).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: bulk-tier-distinct-from-replicated
|
|
labels:
|
|
conformance.home.k3s/R-STATE-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-STATE-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Bulk media storage is provably NOT on the 3-replica Longhorn tier, is sized for a
|
|
different order of magnitude, cannot be silently replaced by a dynamically
|
|
provisioned empty volume, is never reclaimable by the cluster, mounts hard over
|
|
NFSv4, and is never split into two mounts of one export — so hardlinked imports keep
|
|
working. Its server is a per-environment knob, not a literal, because the two
|
|
clusters reach the same storage host by different addresses.
|
|
spec:
|
|
steps:
|
|
- name: assert-bulk-tier-is-distinct-and-statically-bound
|
|
description: >-
|
|
Read every `class: bulk` storage entry from catalog/, pair it with the
|
|
PersistentVolume, PersistentVolumeClaim and volumeMount the committed apps/ goldens
|
|
carry for it, and assert the tier separation, the static pre-binding, the reclaim
|
|
and mount-option decisions, the read-only coupling, the one-export-one-mount
|
|
invariant, and that the server address is supplied by every environment overlay.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|