jakman85andClaude Opus 5
5fa278f61c
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
115 lines
7.4 KiB
YAML
115 lines
7.4 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the bulk tier's capacity strategy is DISTINCT from the replicated tier's
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# THE BUG CLASS THIS CLOSES. The `bulk` StorageClass existed from Phase 0 as a stub:
|
|
# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME`, TODO(OQ-2).
|
|
# Four ServiceDefinitions carried a comment promising to attach their media once OQ-2 was
|
|
# resolved, and the ONLY thing keeping a multi-terabyte library off the 3-replica
|
|
# `longhorn` class was those four comments. A bulk volume that lands on the replicated
|
|
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
|
|
# three mini-PCs cannot hold. R-STATE-04 is precisely that the two tiers are distinct, so
|
|
# the distinctness is asserted rather than reviewed.
|
|
#
|
|
# It also pins the four decisions that make the STATIC form of this tier safe, each with a
|
|
# plausible "simplification" that breaks something silently: `Retain` (bulk data pre-exists
|
|
# the cluster and has consumers outside it, so anything else makes `delete pvc` a data-loss
|
|
# operation); NO dynamic provisioning (a dynamic bulk class hands out a FRESH EMPTY
|
|
# directory that mounts and Binds perfectly while the real 8 TiB is never attached — the
|
|
# most deceptive failure available here); `hard` not `soft` (a soft mount hands a server
|
|
# hiccup to the app as a short read, which for a media library is silent corruption); and
|
|
# NFSv4 (mount and lock in-protocol ⇒ ONE reachable port, tcp/2049, which is what the
|
|
# operator must open in pfSense for prod to reach this server at all).
|
|
#
|
|
# AND the EXDEV trap, which is not about storage. Sonarr/Radarr import by hardlinking or
|
|
# atomically renaming, both same-filesystem operations, and to the kernel "same filesystem"
|
|
# means SAME MOUNT. Splitting one export into two subPath mounts looks TIGHTER and breaks
|
|
# link() with EXDEV; the arrs then degrade to copy-then-delete, rewriting every import over
|
|
# NFS and destroying the seeding copy. It surfaces weeks later as "imports are slow and my
|
|
# ratio is gone". A reviewer cannot see that; this check can.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN every `class: bulk` storage entry in catalog/ is compared against the objects the
|
|
# COMMITTED apps/ goldens carry for it (what Flux reconciles — deliberately not the
|
|
# renderer, which would only prove it agrees with itself),
|
|
# THEN (1) no bulk volume is on a Longhorn-provisioned class, and its class is explicitly
|
|
# EMPTY so nothing can dynamically provision an empty stand-in;
|
|
# (2) its capacity is declared and is at least an order of magnitude above the same
|
|
# service's replicated volume — the requirement stated as arithmetic;
|
|
# (3) PV and PVC are pre-bound from BOTH ends (claimRef + volumeName) and agree;
|
|
# (4) the PV reclaim policy is Retain;
|
|
# (5) mountOptions carry `hard` and exactly one `nfsvers=4.x`, and never `soft`;
|
|
# (6) no workload mounts one export more than once (the EXDEV trap);
|
|
# (7) a readOnly entry renders nfs.readOnly AND a readOnly volumeMount AND
|
|
# ReadOnlyMany; a writable one renders neither and is ReadWriteMany; every bulk
|
|
# access mode is a `...Many`, since one export serves several namespaces and
|
|
# several nodes;
|
|
# (8) the PV's nfs.server is a BARE cluster-vars token (never a literal — dev and
|
|
# prod reach this host by different addresses), that knob has a base default, and
|
|
# EVERY overlay supplies it;
|
|
# AND (9) the check is LOAD-BEARING (ci/l0.sh runs it unconditionally, ci/gate.sh blocks
|
|
# on the l0 stage) and NON-VACUOUS (an empty bulk corpus FAILS rather than
|
|
# passing trivially, and a floor on the assertion count keeps a pass from
|
|
# degenerating into a no-op).
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
|
# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
|
|
# SAME script.
|
|
#
|
|
# NOT proven — the runtime half, which needs a cluster: that the server is reachable, that
|
|
# the export admits these clients, that the PVC binds, that a write succeeds, or that a
|
|
# hardlink between the library and the download path actually works. The uid analysis is
|
|
# reasoning (docs/BULK-NFS-STORAGE.md §6) and the in-cluster probes that would settle it
|
|
# are written down in §9 of that document, unrun. Nor can this prove prod's address is
|
|
# CORRECT — only that it is a supplied knob; while it is a placeholder, prod's pods will
|
|
# not mount, and catching that statically needs the placeholder guard, which is absent from
|
|
# main (§7). Read-only on the media tree is a CLIENT-side guarantee only: the export serves
|
|
# one reader and two writers from the same node addresses, so the server cannot tell them
|
|
# apart (§5).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: bulk-tier-distinct-from-replicated
|
|
labels:
|
|
conformance.home.k3s/R-STATE-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-STATE-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e
|
|
# (UNCONDITIONAL)", which this annotation replaces:
|
|
#
|
|
# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
|
|
# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
|
|
# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
|
|
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
|
|
# three mini-PCs cannot hold.
|
|
#
|
|
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys
|
|
# NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
|
|
# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
|
|
# export into two mounts — could attribute no requirement at all and be gated by nothing.
|
|
# The invariant spans catalog/, apps/ and clusters/.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Bulk media storage is provably NOT on the 3-replica Longhorn tier, is sized for a
|
|
different order of magnitude, cannot be silently replaced by a dynamically
|
|
provisioned empty volume, is never reclaimable by the cluster, mounts hard over
|
|
NFSv4, and is never split into two mounts of one export — so hardlinked imports keep
|
|
working. Its server is a per-environment knob, not a literal, because the two
|
|
clusters reach the same storage host by different addresses.
|
|
spec:
|
|
steps:
|
|
- name: assert-bulk-tier-is-distinct-and-statically-bound
|
|
description: >-
|
|
Read every `class: bulk` storage entry from catalog/, pair it with the
|
|
PersistentVolume, PersistentVolumeClaim and volumeMount the committed apps/ goldens
|
|
carry for it, and assert the tier separation, the static pre-binding, the reclaim
|
|
and mount-option decisions, the read-only coupling, the one-export-one-mount
|
|
invariant, and that the server address is supplied by every environment overlay.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|