jakman85andClaude Opus 5
5fa278f61c
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
115 lines
7.8 KiB
YAML
115 lines
7.8 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
|
|
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
|
|
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
|
|
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
|
|
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
|
|
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
|
|
# one-time checks", with the acceptance gate "The five auditability proofs above can
|
|
# be produced on demand."
|
|
#
|
|
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
|
|
# -----------------------------------------------------------------------------------
|
|
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
|
|
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
|
|
# failure modes are asserted instead, each by DRIVING the real machinery:
|
|
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
|
|
# the spec's own words require (fault injection / a DR drill => L3), taken from
|
|
# lib/status.py's own verdict so this check cannot disagree with the status report;
|
|
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
|
|
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
|
|
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
|
|
# attribute — a proof that passes but records nothing leaves the operator with a green
|
|
# terminal and no evidence;
|
|
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
|
|
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
|
|
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
|
|
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
|
|
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
|
|
# ci/ or run.sh ran before this test existed.
|
|
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
|
|
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
|
|
# so a sixth proof added to the spec fails here instead of being silently unverified.
|
|
#
|
|
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
|
|
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
|
|
# anyone needs it.
|
|
#
|
|
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
|
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
|
|
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
|
|
# and two synthetic forgeries are presented to them,
|
|
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
|
|
# documented one-command invocation selects and asserts them, each is capturable as a
|
|
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
|
|
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
|
|
# it directly, and this Chainsaw Test wraps the SAME script.
|
|
#
|
|
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
|
|
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
|
|
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
|
|
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
|
|
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
|
|
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
|
|
# command, capturable, and that the chain refuses forgeries — not that isolation holds
|
|
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
|
|
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
|
|
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
|
|
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
|
|
# would need signed history.
|
|
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
|
|
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
|
|
# not re-asserted here.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: auditability-proofs-producible
|
|
labels:
|
|
conformance.home.k3s/R-SEC-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-SEC-03"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE. This is a META-invariant over the ENTIRE conformance suite: the five
|
|
# spec-named proofs must each still have an EXECUTABLE test at the right layer, with a
|
|
# pass sentinel, resolvable by the documented command. Every one of its inputs lives under
|
|
# conformance/, which ci/touched-requirements.sh keys to R-TEST-01..04 — never to R-SEC-03.
|
|
# So downgrading conformance/tests/l3-fault-injection/dr-restore to a scaffold attributes
|
|
# R-STATE-03/R-TEST-*, and requirement-scoped selection drops the one check that notices
|
|
# the audit evidence just stopped being producible.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree and the harness's own tooling, when status.py, the requirement
|
|
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
|
|
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
|
|
of docs/07's five auditability proofs has an executable test at the layer the spec
|
|
requires, is selectable and asserted by the documented one-command invocation, is
|
|
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
|
|
evidence.
|
|
spec:
|
|
steps:
|
|
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|
|
# ---------------------------------------------------------------------------------------------
|
|
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
|
|
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
|
|
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
|
|
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
|
|
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
|
|
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
|
|
# the wrong trade (R-TEST-06).
|