jakman85andClaude Opus 5
87c3d83dff
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]>
142 lines
7.0 KiB
Bash
Executable File
142 lines
7.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/tests/touched-requirements.test.sh — regression tests for the content scan in
|
|
# ci/touched-requirements.sh, focused on rule 3: R-IDs cited only in a whole-line
|
|
# YAML/shell `#` comment are PROSE (a reference), not a declaration, and must NOT be
|
|
# attributed — while R-IDs on real code/spec lines still ARE.
|
|
#
|
|
# The tests point the real script at a THROWAWAY git repo via the REPO_ROOT override
|
|
# (ci/lib.sh honours a pre-set REPO_ROOT); the script itself is run from its true
|
|
# location so it still sources the real lib.sh. No network, no cluster.
|
|
#
|
|
# Run: ci/tests/touched-requirements.test.sh (or `make -C ci test`)
|
|
set -euo pipefail
|
|
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
|
|
|
PASS=0 FAIL=0
|
|
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
|
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
|
|
|
# assert_absent <needle> <haystack> <msg>
|
|
assert_absent() {
|
|
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))";
|
|
else ok "$3"; fi
|
|
}
|
|
# assert_present <needle> <haystack> <msg>
|
|
assert_present() {
|
|
if grep -qxF "$1" <<<"$2"; then ok "$3";
|
|
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
|
|
}
|
|
|
|
TMP="$(mktemp -d)"
|
|
trap 'rm -rf "$TMP"' EXIT
|
|
|
|
git -C "$TMP" init -q -b main
|
|
git -C "$TMP" config user.email [email protected]
|
|
git -C "$TMP" config user.name t
|
|
git -C "$TMP" config commit.gpgsign false # throwaway repo: never sign (ignore host gpg config)
|
|
mkdir -p "$TMP/notes" # a path OUTSIDE path_requirements() so we isolate the content scan
|
|
echo "baseline" > "$TMP/notes/base.txt"
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
|
|
|
|
git -C "$TMP" checkout -q -b feature
|
|
|
|
# A YAML file: an R-ID in a whole-line comment (must be IGNORED) and an R-ID on a real
|
|
# value line (must be DETECTED). Also an indented comment, to prove leading space is fine.
|
|
cat > "$TMP/notes/example.yaml" <<'YAML'
|
|
# This header explains R-DOC-01 and must NOT be attributed.
|
|
kind: Thing
|
|
metadata:
|
|
name: implements-R-REAL-01
|
|
spec:
|
|
note: value # trailing inline comment mentioning R-INLINE-01 (conservatively still seen)
|
|
# deeply indented comment about R-DOC-02 — also NOT attributed
|
|
YAML
|
|
|
|
# A shell file: a `#` comment R-ID (IGNORED) plus a real command line R-ID (DETECTED).
|
|
cat > "$TMP/notes/example.sh" <<'SH'
|
|
#!/usr/bin/env bash
|
|
# doc: this script relates to R-SHDOC-01 (comment only — ignored)
|
|
echo "runs check for R-SHREAL-01"
|
|
SH
|
|
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm feature
|
|
|
|
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set:\n%s\n---\n' "$OUT"
|
|
|
|
# Comment-only citations are dropped (rule 3):
|
|
assert_absent "R-DOC-01" "$OUT" "whole-line YAML comment R-ID is NOT attributed"
|
|
assert_absent "R-DOC-02" "$OUT" "indented YAML comment R-ID is NOT attributed"
|
|
assert_absent "R-SHDOC-01" "$OUT" "shell '#' comment R-ID is NOT attributed"
|
|
# Real code/spec lines still count:
|
|
assert_present "R-REAL-01" "$OUT" "R-ID on a real YAML value line IS attributed"
|
|
assert_present "R-SHREAL-01" "$OUT" "R-ID on a real shell command line IS attributed"
|
|
# Conservative: a trailing inline comment on a REAL line is still seen (only whole-line
|
|
# comments are dropped) — documents the intentional boundary of the fix.
|
|
assert_present "R-INLINE-01" "$OUT" "inline comment on a real line is (conservatively) still seen"
|
|
|
|
# Removed comment lines are also excluded, while removed real lines still count. The file
|
|
# must EXIST at the base, so we diff the removal against the commit that added it.
|
|
git -C "$TMP" checkout -q -b feature2 main
|
|
cat > "$TMP/notes/todelete.yaml" <<'YAML'
|
|
# will delete a comment citing R-DELDOC-01 and a real line citing R-DELREAL-01
|
|
kind: Thing
|
|
name: implements-R-DELREAL-01
|
|
YAML
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm add
|
|
ADD_SHA="$(git -C "$TMP" rev-parse HEAD)"
|
|
git -C "$TMP" rm -q "$TMP/notes/todelete.yaml" && git -C "$TMP" commit -qm remove
|
|
OUT2="$(REPO_ROOT="$TMP" BASE_REF="$ADD_SHA" "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (removal):\n%s\n---\n' "$OUT2"
|
|
assert_absent "R-DELDOC-01" "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
|
|
assert_present "R-DELREAL-01" "$OUT2" "REMOVED real-line R-ID IS attributed"
|
|
|
|
# ---------------------------------------------------------------------------------------
|
|
# PATH MAP — bootstrap/* (added 2026-09-13)
|
|
#
|
|
# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
|
|
# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
|
|
# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
|
|
# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
|
|
# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
|
|
# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
|
|
# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
|
|
# guard by content rather than by path, so it activates on exactly that change. Removing
|
|
# the entry would let the change introducing a guard be gated by nothing, which is the
|
|
# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
|
|
# silent loss of selection.
|
|
#
|
|
# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
|
|
# map precisely so the other cases isolate the scan, and here the file content cites no
|
|
# R-ID at all, so any R-ID in the output came from the PATH.
|
|
git -C "$TMP" checkout -q -b feature3 main
|
|
mkdir -p "$TMP/bootstrap/flux"
|
|
cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
|
|
#!/usr/bin/env bash
|
|
# a bootstrap-path change that cites no requirement id of its own
|
|
echo "flux bootstrap"
|
|
SH
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
|
|
OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
|
|
assert_present "R-STATE-03" "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
|
|
assert_present "R-GITOPS-01" "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
|
|
|
|
# And the neighbouring entry the same test leans on for its other half: clusters/* must
|
|
# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
|
|
# authored-values sweep over clusters/<env> effective cluster-vars).
|
|
git -C "$TMP" checkout -q -b feature4 main
|
|
mkdir -p "$TMP/clusters/base"
|
|
printf 'kind: ConfigMap\nmetadata:\n name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
|
|
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
|
|
OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
|
printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
|
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
|
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
|
|
|
echo
|
|
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
|
|
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|