d8b25a67feeea46f3e1311d4e8fb15aedaaad8e8
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
5fa278f61c |
fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
da8a0eac04 |
test(l0): R-STATE-04 gets a real check — bulk storage provably is not the replicated tier
The gate BLOCKED this branch, correctly, and the block was the interesting part.
Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.
WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.
(1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
(derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
and the class is explicitly EMPTY.
(2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
volume sized like a config PVC means the tier was named, not applied. The 10x floor is
CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
volume and teach the next reader to weaken the check. My first draft used 100x and did
exactly that — the corpus caught it.
(3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
(4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
— a soft mount hands a server hiccup to the app as a short read, which for a media library
is silent corruption; and v4 is what keeps the firewall surface at one port.
(6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
sees that in a diff.
(7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
dropping one still leaves the others; and every bulk mode is a `...Many`.
(8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
supplies it — the seam that lets one authored definition serve two clusters that reach this
host by different addresses.
(9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
than passing trivially, and a floor on the assertion count keeps a pass from degenerating.
It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.
PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.
Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.
Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
|