0 Fork 0

Files

T

jakman85andClaude Opus 5 6e77150ae1 fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)

Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.

=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.

Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.

Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.

=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.

R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.

R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.

R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.

Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.

Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.

Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-08-31 23:26:49 -05:00

5.5 KiB

Raw Blame History

Conformance tests — layout & requirement-tagging convention

Every test is a Chainsaw Test in its own directory:

tests/
  l0-static/           L0 — whole-repo invariants over the tracked tree (no cluster)
  l1-rendering/        L1 — ServiceDefinition -> rendered resources match golden
  l2-behavioral/       L2 — real runtime behavior (probes against a cluster)
  l3-fault-injection/  L3 — break something; assert fail-closed / no-data-loss
  meta/                harness self-checks (R-TEST-*), cluster-independent
    <test-name>/chainsaw-test.yaml

The layer a test belongs to is its subtree and its .../layer annotation; the coverage generator cross-checks the two.

L0 static (`l0-static/`) — what belongs here, and what does not

docs/14's L0 row is "Pre-apply, in CI · manifests/ServiceDefinitions obey invariants". Kyverno (platform/kyverno/) covers the invariants expressible over a single resource. This subtree covers the L0 invariants that are properties of the repository as a whole and that no per-resource policy can state — e.g. no plaintext secret exists anywhere in the tracked tree, no tracked RoleBinding grants a workload identity a privileged role, the spec-immutability gate actually blocks a modified spec.

Each l0-static/<name>/ dir carries a check.sh: a self-contained offline assertion script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:

by conformance/run.sh --layer l0 (and as the offline leg of a full run), via lib/run_l1_offline.py, producing a REAL pass/fail JUnit entry — never a skip; and
by the merge gate's ci/l0.sh stage, so these assertions run on every PR; and
by the wrapping Chainsaw Test, which runs the SAME check.sh, so an in-cluster run executes byte-identical assertions.

Two authoring rules specific to this layer:

State the gap. A statically-provable half of a requirement is still only half. Every check.sh header ends with an explicit "what this does NOT prove" list naming the cluster-layer remainder. Covering a requirement at L0 does not finish it.
Exemptions must be self-policing. Where a check must tolerate something (a deliberately-plaintext policy fixture, an .example template), the exemption is conditioned on a property that would break if the exemption were abused — e.g. the plaintext Secret fixture is exempt only while a kyverno-test.yaml asserts the policy rejects it. Never a bare path allowlist.

Requirement-tagging convention (authoritative)

A test declares the requirement(s) it verifies three ways; the label is the source of truth, the annotations must agree with it (the coverage generator fails on drift):

metadata:
  name: default-deny
  labels:
    # ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
    conformance.home.k3s/R-NET-01: "true"
  annotations:
    # Human-readable canonical list — MUST equal the label set.
    conformance.home.k3s/requirements: "R-NET-01"
    # Declared layer (L0|L1|L2|L3|META) — must match the subtree.
    conformance.home.k3s/layer: "L2"
    # true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
    conformance.home.k3s/safety-critical: "false"
    # Given/When/Then intent, for the coverage report and reviewers.
    conformance.home.k3s/intent: "Given ... When ... Then ..."

Why a label per requirement (not a comma list): Kubernetes label values can't hold a comma-separated list, and chainsaw test --selector selects on label keys/values. One conformance.home.k3s/<R-ID>=true label per requirement makes both subset-selection and coverage extraction exact and robust.

How this drives the tooling

Subset runs (merge gate): run.sh --requirements R-NET-01,R-NET-06 resolves the R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once.
Coverage (lib/coverage.py): reads these labels from every chainsaw-test.yaml, maps each registry requirement -> its verifying test(s), and flags gaps + any label/annotation mismatch + tags not in the registry.
JUnit: results (real or skipped-scaffold) carry the R-IDs in classname + properties, so the JUnit itself stays traceable (R-TEST-04).

Authoring rules (R-TEST-01/02/06)

Behavior, not existence. Assert what the system does (a probe cannot connect), never that an object merely exists (a policy can exist and not enforce).
Given/When/Then structure with given-*/when-*/then-* step names.
Hermetic & self-cleaning. Rely on Chainsaw's per-test namespace + auto-teardown; put any out-of-namespace fixture cleanup in a finally block. A leaked fixture is a failure.
Bounded, deterministic waits. Use assert/error polling within the config timeouts; never sleep-and-hope.
Safety-critical => fault-injection. For a safety-critical requirement, actually break the thing and assert fail-closed / no-data-loss, with finally restoring state.
Never weaken a test to go green (R-TEST-06). If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement.

Scaffolds in this tree carry TODO(T-x.y) markers where a fixture/assertion lands as a later roadmap phase delivers the capability under test. lib/status.py treats such a marker — in the chainsaw-test.yaml, in render-check.sh, or in check.sh — as outranking any conformance.home.k3s/executable: "true" annotation, so a placeholder body can never be reported as an executable test.

5.5 KiB Raw Blame History