jakman85andClaude Opus 5
5fa278f61c
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
145 lines
8.2 KiB
Bash
Executable File
145 lines
8.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/l0.sh — Stage 2+3: ServiceDefinition schema validation + Kyverno L0 policy gate.
|
|
# Both run in CI WITHOUT a cluster (docs/14 §"Where each layer runs") — live now.
|
|
#
|
|
# Stage 2 schema : catalog/*.yaml validated against the ServiceDefinition
|
|
# schema (owner: T-0.3, schemas/).
|
|
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
|
# present, no plaintext secrets, required labels/egress-class,
|
|
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
|
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
|
# a per-resource policy cannot express (secret hygiene across every
|
|
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
|
# Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
|
|
# test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
|
|
# replaces what used to be a hand-written "UNCONDITIONAL" stage per
|
|
# whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
|
|
# bulk-tier): three near-identical 12-line rationales, three hardcoded
|
|
# check.sh paths and three copies of the same rc 0/3/* case block, which
|
|
# did not scale and — worse — failed SILENTLY, because whoever added the
|
|
# next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
|
|
# test declares its own scope and the runner enforces it. The reasoning and
|
|
# the incidents those stages recorded were moved INTO the tests they
|
|
# describe (see each chainsaw-test.yaml's `scope` annotation) and the
|
|
# general mechanism is documented in conformance/lib/common.sh.
|
|
#
|
|
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
|
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
|
# rather than a false pass — the gate then knows the seam is unwired, not green.
|
|
set -euo pipefail
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/lib.sh"
|
|
|
|
rc=0
|
|
|
|
# ---- Stage 2: ServiceDefinition schema validation ---------------------------
|
|
log "Stage: ServiceDefinition schema validation (catalog/ against schemas/)"
|
|
if ! entrypoint_ready "$SCHEMA_VALIDATE_CMD"; then
|
|
skip "schema validator not present at '${SCHEMA_VALIDATE_CMD}' (T-0.3 / schemas/ not delivered yet) — UNWIRED, not green"
|
|
summary "- schema: **SKIPPED** (validator entrypoint \`${SCHEMA_VALIDATE_CMD}\` missing — T-0.3)"
|
|
else
|
|
if [ -d "$CATALOG_DIR" ] && compgen -G "${CATALOG_DIR}/*.y*ml" >/dev/null 2>&1; then
|
|
if "$SCHEMA_VALIDATE_CMD" "${CATALOG_DIR}"/*.y*ml; then
|
|
ok "schema validation passed"
|
|
summary "- schema: **passed**"
|
|
else
|
|
err "schema validation failed"; summary "- schema: **FAILED**"; rc=1
|
|
fi
|
|
else
|
|
log "no ServiceDefinitions in ${CATALOG_DIR} yet — validating validator is callable only"
|
|
"$SCHEMA_VALIDATE_CMD" --version >/dev/null 2>&1 || true
|
|
ok "schema validator present; no catalog entries to validate"
|
|
summary "- schema: **passed** (no catalog entries yet)"
|
|
fi
|
|
fi
|
|
|
|
# ---- Stage 3: Kyverno L0 policy gate ----------------------------------------
|
|
log "Stage: Kyverno L0 policy gate (kyverno test ${KYVERNO_POLICY_DIR})"
|
|
# `kyverno test <dir>` discovers kyverno-test.yaml files recursively.
|
|
if ! have kyverno; then
|
|
skip "kyverno CLI not installed (pin v${KYVERNO_VERSION}) — cannot run L0 gate"
|
|
summary "- kyverno-l0: **SKIPPED** (kyverno CLI not installed)"
|
|
# Missing tool is an environment gap, not a code pass -> exit 3 unless already failing.
|
|
[ "$rc" -eq 0 ] && rc=3
|
|
elif [ ! -d "$KYVERNO_POLICY_DIR" ] || ! compgen -G "${KYVERNO_POLICY_DIR}/**/kyverno-test.yaml" >/dev/null 2>&1 \
|
|
&& ! find "$KYVERNO_POLICY_DIR" -name 'kyverno-test.yaml' 2>/dev/null | grep -q .; then
|
|
skip "no kyverno-test.yaml under '${KYVERNO_POLICY_DIR}' (T-0.5 / platform/kyverno not delivered yet) — UNWIRED, not green"
|
|
summary "- kyverno-l0: **SKIPPED** (no policy tests yet — T-0.5)"
|
|
else
|
|
# The ClusterPolicies carry a Flux postBuild token in validationFailureAction
|
|
# (${KYVERNO_VALIDATION_ACTION}, substituted in-cluster from
|
|
# clusters/base/cluster-vars.yaml). The offline `kyverno test` CLI does NOT run
|
|
# Flux substitution, so the literal token is an invalid enum value and those
|
|
# policies fail to load (no result rows -> "Not found"). Render a throwaway copy
|
|
# with the base/prod value (Enforce — the strict posture the gate must prove)
|
|
# before testing. This does not change what the policies enforce in-cluster.
|
|
: "${KYVERNO_VALIDATION_ACTION:=Enforce}"
|
|
l0_render="$(mktemp -d)"
|
|
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
|
|
find "$l0_render" -type f -name '*.yaml' -print0 \
|
|
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
|
|
if kyverno test "$l0_render"; then
|
|
ok "Kyverno L0 policy tests passed"
|
|
summary "- kyverno-l0: **passed**"
|
|
else
|
|
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
|
|
fi
|
|
rm -rf "$l0_render"
|
|
fi
|
|
|
|
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
|
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
|
# than what Kyverno alone can express over single resources: whole-repo invariants
|
|
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
|
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
|
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
|
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
|
#
|
|
# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
|
|
# --------------------------------------------------------------
|
|
# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
|
|
# whose inputs live under the subtree its requirements map to. It is WRONG for one that
|
|
# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
|
|
# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
|
|
# most likely to break the invariant attributes no requirement at all, so requirement-scoped
|
|
# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
|
|
# in every L0 test that DECLARES that scope, so those run on every gate invocation.
|
|
#
|
|
# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
|
|
# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
|
|
# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
|
|
# the gate's own rc 3 below, never to rc 0.
|
|
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
|
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
|
|
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
|
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
|
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi
|
|
else
|
|
load_touched_requirements
|
|
l0_static_rc=0
|
|
"$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
|
|
${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
|
|
case "$l0_static_rc" in
|
|
0) ok "L0 static conformance checks passed"
|
|
summary "- l0-static: **passed**" ;;
|
|
3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
|
|
summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
|
*) err "L0 static conformance checks failed"
|
|
summary "- l0-static: **FAILED**"
|
|
rc=1 ;;
|
|
esac
|
|
# Per-check step-summary lines, so the gate output still NAMES what ran — the generated
|
|
# replacement for the one hardcoded `summary` line each deleted stage carried.
|
|
emit_offline_summary
|
|
fi
|
|
|
|
case "$rc" in
|
|
0) ok "L0 stage complete" ;;
|
|
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
|
*) err "L0 stage failed" ;;
|
|
esac
|
|
exit "$rc"
|