jakman85andClaude Opus 5
23c335257b
ci/touched-requirements.sh's coarse path map had no arm for apps/*. A change adding
or editing ANY workload on the cluster — 49 tracked manifests across apps/wave-0,
-1, -2 and apps/vms — attributed zero requirements, so the merge gate selected only
the whole-repo-scoped checks and none of the requirement-scoped L2 behaviour tests.
The tests that enforce default-deny, egress class and auth mode did not run on
exactly the changes that can violate them. Kyverno admission and the L0 egress-path
check are real backstops, but admission is post-merge and the gate exists to catch
it first.
platform/homepage and platform/headlamp were unmapped too — the C2 UI. That gap was
already written down in changed_diff()'s rule-3 comment ("the ONLY route to
R-C2-01/02 was rule 3", i.e. editing the C2 test itself) and had sat there since.
Twelve arms added: apps/vms/* (ordered BEFORE apps/*, because `case` takes the first
match), apps/*, platform/homepage|headlamp, platform/kubevirt, platform/gateway,
platform/coredns, platform/crafty, platform/external-snapshotter|backup, runbooks/*,
platform-config/*, platform/kustomization.yaml and .sops.yaml.
platform/kustomization.yaml earns its own arm: it is the assembly list, so one
deleted line disables an entire platform component while every file it points at
stays perfectly valid on disk. That is the most consequential single-line edit in
the repo and it attributed nothing.
Measured effect for an apps/ change, via --print-selection:
before: 0 requirement-scoped tests
after: 13 (3 L1, 8 L2, 2 L3)
Coverage stays 51/51, so the gate's "attributed IDs must be covered" assertion does
not false-block.
Ten new assertions in ci/tests/touched-requirements.test.sh (26 total, all passing),
including the ORDER property — apps/vms/* must win over apps/* — because arm order
is a real failure mode of `case`, not merely a question of both arms existing. Each
fixture cites no R-ID of its own, so any ID in the output came from the PATH and not
the content scan.
A correction on numbers I reported earlier in this session: I claimed "192 unmapped
files", then "182". Both were inflated by my own audit script rather than the repo.
Its regex used `[^)]*`, which matches across a newline, so the first arm parsed as
`case "$1" in\n catalog/*` and never matched anything; I also mis-parsed
alternation arms like `.gitea/*|ci/*`, which made ci/ look unmapped when it is
mapped. True remainder after this change is 18 files, all prose or captured
artifacts (docs/plans/, root READMEs, .standup-dns/*.txt).
Gated: offline l0,l1,meta --include-scope whole-repo, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
234 lines
13 KiB
Bash
Executable File
234 lines
13 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/tests/touched-requirements.test.sh — regression tests for the content scan in
|
|
# ci/touched-requirements.sh, focused on rule 3: R-IDs cited only in a whole-line
|
|
# YAML/shell `#` comment are PROSE (a reference), not a declaration, and must NOT be
|
|
# attributed — while R-IDs on real code/spec lines still ARE.
|
|
#
|
|
# The tests point the real script at a THROWAWAY git repo via the REPO_ROOT override
|
|
# (ci/lib.sh honours a pre-set REPO_ROOT); the script itself is run from its true
|
|
# location so it still sources the real lib.sh. No network, no cluster.
|
|
#
|
|
# Run: ci/tests/touched-requirements.test.sh (or `make -C ci test`)
|
|
set -euo pipefail
|
|
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
|
|
|
PASS=0 FAIL=0
|
|
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
|
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
|
|
|
# assert_absent <needle> <haystack> <msg>
|
|
assert_absent() {
|
|
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))";
|
|
else ok "$3"; fi
|
|
}
|
|
# assert_present <needle> <haystack> <msg>
|
|
assert_present() {
|
|
if grep -qxF "$1" <<<"$2"; then ok "$3";
|
|
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
|
|
}
|
|
|
|
TMP="$(mktemp -d)"
|
|
trap 'rm -rf "$TMP"' EXIT
|
|
|
|
git -C "$TMP" init -q -b main
|
|
git -C "$TMP" config user.email [email protected]
|
|
git -C "$TMP" config user.name t
|
|
git -C "$TMP" config commit.gpgsign false # throwaway repo: never sign (ignore host gpg config)
|
|
mkdir -p "$TMP/notes" # a path OUTSIDE path_requirements() so we isolate the content scan
|
|
echo "baseline" > "$TMP/notes/base.txt"
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
|
|
|
|
git -C "$TMP" checkout -q -b feature
|
|
|
|
# A YAML file: an R-ID in a whole-line comment (must be IGNORED) and an R-ID on a real
|
|
# value line (must be DETECTED). Also an indented comment, to prove leading space is fine.
|
|
cat > "$TMP/notes/example.yaml" <<'YAML'
|
|
# This header explains R-DOC-01 and must NOT be attributed.
|
|
kind: Thing
|
|
metadata:
|
|
name: implements-R-REAL-01
|
|
spec:
|
|
note: value # trailing inline comment mentioning R-INLINE-01 (conservatively still seen)
|
|
# deeply indented comment about R-DOC-02 — also NOT attributed
|
|
YAML
|
|
|
|
# A shell file: a `#` comment R-ID (IGNORED) plus a real command line R-ID (DETECTED).
|
|
cat > "$TMP/notes/example.sh" <<'SH'
|
|
#!/usr/bin/env bash
|
|
# doc: this script relates to R-SHDOC-01 (comment only — ignored)
|
|
echo "runs check for R-SHREAL-01"
|
|
SH
|
|
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm feature
|
|
|
|
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set:\n%s\n---\n' "$OUT"
|
|
|
|
# Comment-only citations are dropped (rule 3):
|
|
assert_absent "R-DOC-01" "$OUT" "whole-line YAML comment R-ID is NOT attributed"
|
|
assert_absent "R-DOC-02" "$OUT" "indented YAML comment R-ID is NOT attributed"
|
|
assert_absent "R-SHDOC-01" "$OUT" "shell '#' comment R-ID is NOT attributed"
|
|
# Real code/spec lines still count:
|
|
assert_present "R-REAL-01" "$OUT" "R-ID on a real YAML value line IS attributed"
|
|
assert_present "R-SHREAL-01" "$OUT" "R-ID on a real shell command line IS attributed"
|
|
# Conservative: a trailing inline comment on a REAL line is still seen (only whole-line
|
|
# comments are dropped) — documents the intentional boundary of the fix.
|
|
assert_present "R-INLINE-01" "$OUT" "inline comment on a real line is (conservatively) still seen"
|
|
|
|
# Removed comment lines are also excluded, while removed real lines still count. The file
|
|
# must EXIST at the base, so we diff the removal against the commit that added it.
|
|
git -C "$TMP" checkout -q -b feature2 main
|
|
cat > "$TMP/notes/todelete.yaml" <<'YAML'
|
|
# will delete a comment citing R-DELDOC-01 and a real line citing R-DELREAL-01
|
|
kind: Thing
|
|
name: implements-R-DELREAL-01
|
|
YAML
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm add
|
|
ADD_SHA="$(git -C "$TMP" rev-parse HEAD)"
|
|
git -C "$TMP" rm -q "$TMP/notes/todelete.yaml" && git -C "$TMP" commit -qm remove
|
|
OUT2="$(REPO_ROOT="$TMP" BASE_REF="$ADD_SHA" "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (removal):\n%s\n---\n' "$OUT2"
|
|
assert_absent "R-DELDOC-01" "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
|
|
assert_present "R-DELREAL-01" "$OUT2" "REMOVED real-line R-ID IS attributed"
|
|
|
|
# ---------------------------------------------------------------------------------------
|
|
# PATH MAP — bootstrap/* (added 2026-09-13)
|
|
#
|
|
# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
|
|
# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
|
|
# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
|
|
# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
|
|
# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
|
|
# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
|
|
# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
|
|
# guard by content rather than by path, so it activates on exactly that change. Removing
|
|
# the entry would let the change introducing a guard be gated by nothing, which is the
|
|
# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
|
|
# silent loss of selection.
|
|
#
|
|
# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
|
|
# map precisely so the other cases isolate the scan, and here the file content cites no
|
|
# R-ID at all, so any R-ID in the output came from the PATH.
|
|
git -C "$TMP" checkout -q -b feature3 main
|
|
mkdir -p "$TMP/bootstrap/flux"
|
|
cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
|
|
#!/usr/bin/env bash
|
|
# a bootstrap-path change that cites no requirement id of its own
|
|
echo "flux bootstrap"
|
|
SH
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
|
|
OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
|
|
assert_present "R-STATE-03" "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
|
|
assert_present "R-GITOPS-01" "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
|
|
|
|
# And the neighbouring entry the same test leans on for its other half: clusters/* must
|
|
# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
|
|
# authored-values sweep over clusters/<env> effective cluster-vars).
|
|
git -C "$TMP" checkout -q -b feature4 main
|
|
mkdir -p "$TMP/clusters/base"
|
|
printf 'kind: ConfigMap\nmetadata:\n name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
|
|
OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
|
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
|
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
|
|
|
# ---------------------------------------------------------------------------------------
|
|
# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
|
|
#
|
|
# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
|
|
# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
|
|
# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
|
|
# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
|
|
# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
|
|
# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
|
|
#
|
|
# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
|
|
# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
|
|
#
|
|
# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
|
|
# from the PATH and not from the content scan.
|
|
|
|
# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
|
|
# under test here, not just the presence of both arms.
|
|
git -C "$TMP" checkout -q -b feature-apps-vms main
|
|
mkdir -p "$TMP/apps/vms/demo"
|
|
printf 'kind: VirtualMachine\nmetadata:\n name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
|
|
OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
|
|
assert_present "R-PLAT-03" "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
|
|
assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
|
|
assert_absent "R-PLAT-02" "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
|
|
|
|
# A plain ServiceDefinition under apps/ gets the container-workload set.
|
|
git -C "$TMP" checkout -q -b feature-apps main
|
|
mkdir -p "$TMP/apps/wave-1"
|
|
printf 'kind: ServiceDefinition\nmetadata:\n name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
|
|
OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
|
|
assert_present "R-PLAT-02" "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
|
|
assert_present "R-NET-01" "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
|
|
assert_present "R-NET-06" "$OUT_APPS" "apps/* attributes the egress-class requirements"
|
|
assert_present "R-IDP-02" "$OUT_APPS" "apps/* attributes the auth-mode requirements"
|
|
|
|
# homepage / headlamp ARE the command-centre surface.
|
|
git -C "$TMP" checkout -q -b feature-c2 main
|
|
mkdir -p "$TMP/platform/headlamp"
|
|
printf 'kind: Deployment\nmetadata:\n name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
|
|
OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
|
|
assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
|
|
assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
|
|
|
|
# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
|
|
git -C "$TMP" checkout -q -b feature-assembly main
|
|
mkdir -p "$TMP/platform"
|
|
printf 'resources:\n - cilium\n - longhorn\n' > "$TMP/platform/kustomization.yaml"
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
|
|
OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
|
|
assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
|
|
|
|
# ---------------------------------------------------------------------------------------
|
|
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
|
|
#
|
|
# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
|
|
# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
|
|
# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
|
|
# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
|
|
# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
|
|
# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
|
|
#
|
|
# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
|
|
# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
|
|
# somehow matched only the digit form would still fail here.
|
|
git -C "$TMP" checkout -q -b feature5 main
|
|
cat > "$TMP/notes/c2.yaml" <<'YAML'
|
|
kind: Test
|
|
metadata:
|
|
labels:
|
|
conformance.home.k3s/R-C2-01: "true"
|
|
conformance.home.k3s/R-C2-02: "true"
|
|
conformance.home.k3s/R-OBS-04: "true"
|
|
YAML
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
|
|
OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
|
|
assert_present "R-C2-01" "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
|
|
assert_present "R-C2-02" "$OUT5" "the second such ID is attributed too (R-C2-02)"
|
|
assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
|
|
# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
|
|
# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
|
|
# noise (over-broad selection, never a false green, but noisy and misleading).
|
|
assert_absent "R-1-01" "$OUT5" "a purely numeric workstream segment is not invented from thin air"
|
|
|
|
echo
|
|
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
|
|
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|