jakman85andClaude Opus 5
5fa278f61c
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
84 lines
5.2 KiB
YAML
84 lines
5.2 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC:
|
|
# created, baselined, cluster-joined, and managed as code"); the four stages
|
|
# are enumerated in docs/02 §"Node lifecycle stages".
|
|
#
|
|
# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live
|
|
# Node object cannot answer it — a node built by OpenTofu and one built by hand in a
|
|
# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with
|
|
# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four
|
|
# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the
|
|
# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection
|
|
# to any node), not by reading files: a chain that does not resolve fails the check.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test.
|
|
# The dev group names are read from the playbooks' own default host patterns, the node
|
|
# names are generated from the IaC module's prefix/count variables, and the two sets are
|
|
# compared against each other — so the assertion survives a rename or a resize.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the provisioning chain is parsed and (where offline execution is possible) driven,
|
|
# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted
|
|
# provider — a cloud provider fails), from a for_each map rather than a count index,
|
|
# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the
|
|
# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies
|
|
# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid
|
|
# declaratively; one k3s role founds the cluster with --cluster-init and joins the
|
|
# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at
|
|
# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux)
|
|
# resolves and targets every dev node at every stage.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
|
|
# where present — ansible-playbook in no-connection modes; no cluster, no network);
|
|
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
|
|
# wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or
|
|
# hypervisor access, neither available offline.
|
|
# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is
|
|
# parsed, not evaluated; a network-allowed CI stage would close this.
|
|
# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check.
|
|
# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by
|
|
# tests/l3-fault-injection/node-loss (R-STATE-02).
|
|
# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the
|
|
# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code",
|
|
# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the
|
|
# divergence rather than hiding it.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: dev-nodes-provisioned-as-code
|
|
labels:
|
|
conformance.home.k3s/R-ENV-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-ENV-01"
|
|
conformance.home.k3s/layer: "L0"
|
|
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
|
|
# such (2026-09-12). Its entire input surface is infra/ (OpenTofu + the Ansible roles that
|
|
# provision the dev nodes); it reads nothing else. The gap was that
|
|
# ci/touched-requirements.sh keyed infra/* to NOTHING, so an infra/ edit attributed no
|
|
# requirement and this check was filtered out. The right fix for a path-scoped invariant is
|
|
# a path-map entry, not an unconditional run: `infra/*) echo "R-ENV-01"` was added there in
|
|
# the same change. Keep the two in sync — if this check ever grows an assertion outside
|
|
# infra/, either widen that map entry or give this test a whole-repo scope.
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the dev provisioning chain is parsed and driven
|
|
offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined
|
|
and handed to Flux by one code path that covers all of them — with no node outside
|
|
it and no stage missing.
|
|
spec:
|
|
steps:
|
|
- name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|