jakman85andClaude Opus 5
6e77150ae1
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
102 lines
5.5 KiB
Markdown
102 lines
5.5 KiB
Markdown
# Conformance tests — layout & requirement-tagging convention
|
|
|
|
Every test is a **Chainsaw `Test`** in its own directory:
|
|
|
|
```
|
|
tests/
|
|
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
|
|
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
|
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
|
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
|
meta/ harness self-checks (R-TEST-*), cluster-independent
|
|
<test-name>/chainsaw-test.yaml
|
|
```
|
|
|
|
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
|
coverage generator cross-checks the two.
|
|
|
|
### L0 static (`l0-static/`) — what belongs here, and what does not
|
|
|
|
docs/14's L0 row is *"Pre-apply, in CI · manifests/ServiceDefinitions obey invariants"*.
|
|
Kyverno (`platform/kyverno/`) covers the invariants expressible over a **single
|
|
resource**. This subtree covers the L0 invariants that are **properties of the repository
|
|
as a whole** and that no per-resource policy can state — e.g. *no plaintext secret exists
|
|
anywhere in the tracked tree*, *no tracked RoleBinding grants a workload identity a
|
|
privileged role*, *the spec-immutability gate actually blocks a modified spec*.
|
|
|
|
Each `l0-static/<name>/` dir carries a **`check.sh`**: a self-contained offline assertion
|
|
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
|
|
|
|
- by `conformance/run.sh --layer l0` (and as the offline leg of a full run), via
|
|
`lib/run_l1_offline.py`, producing a REAL pass/fail JUnit entry — never a skip; and
|
|
- by the merge gate's **`ci/l0.sh`** stage, so these assertions run on every PR; and
|
|
- by the wrapping Chainsaw `Test`, which runs the SAME `check.sh`, so an in-cluster run
|
|
executes byte-identical assertions.
|
|
|
|
Two authoring rules specific to this layer:
|
|
|
|
1. **State the gap.** A statically-provable half of a requirement is still only half.
|
|
Every `check.sh` header ends with an explicit *"what this does NOT prove"* list naming
|
|
the cluster-layer remainder. Covering a requirement at L0 does not finish it.
|
|
2. **Exemptions must be self-policing.** Where a check must tolerate something (a
|
|
deliberately-plaintext policy fixture, an `.example` template), the exemption is
|
|
conditioned on a property that would break if the exemption were abused — e.g. the
|
|
plaintext Secret fixture is exempt *only while* a `kyverno-test.yaml` asserts the
|
|
policy rejects it. Never a bare path allowlist.
|
|
|
|
## Requirement-tagging convention (authoritative)
|
|
|
|
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
|
of truth, the annotations must agree with it (the coverage generator fails on drift):
|
|
|
|
```yaml
|
|
metadata:
|
|
name: default-deny
|
|
labels:
|
|
# ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
|
|
conformance.home.k3s/R-NET-01: "true"
|
|
annotations:
|
|
# Human-readable canonical list — MUST equal the label set.
|
|
conformance.home.k3s/requirements: "R-NET-01"
|
|
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
|
|
conformance.home.k3s/layer: "L2"
|
|
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
|
conformance.home.k3s/safety-critical: "false"
|
|
# Given/When/Then intent, for the coverage report and reviewers.
|
|
conformance.home.k3s/intent: "Given ... When ... Then ..."
|
|
```
|
|
|
|
Why a label **per requirement** (not a comma list): Kubernetes label *values* can't hold
|
|
a comma-separated list, and `chainsaw test --selector` selects on label **keys/values**.
|
|
One `conformance.home.k3s/<R-ID>=true` label per requirement makes both subset-selection
|
|
and coverage extraction exact and robust.
|
|
|
|
### How this drives the tooling
|
|
- **Subset runs (merge gate):** `run.sh --requirements R-NET-01,R-NET-06` resolves the
|
|
R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once.
|
|
- **Coverage (`lib/coverage.py`):** reads these labels from every `chainsaw-test.yaml`,
|
|
maps each registry requirement -> its verifying test(s), and flags gaps + any
|
|
label/annotation mismatch + tags not in the registry.
|
|
- **JUnit:** results (real or skipped-scaffold) carry the R-IDs in `classname` +
|
|
properties, so the JUnit itself stays traceable (R-TEST-04).
|
|
|
|
## Authoring rules (R-TEST-01/02/06)
|
|
- **Behavior, not existence.** Assert what the system *does* (a probe cannot connect),
|
|
never that an object merely exists (a policy can exist and not enforce).
|
|
- **Given/When/Then** structure with `given-*/when-*/then-*` step names.
|
|
- **Hermetic & self-cleaning.** Rely on Chainsaw's per-test namespace + auto-teardown;
|
|
put any out-of-namespace fixture cleanup in a `finally` block. A leaked fixture is a
|
|
failure.
|
|
- **Bounded, deterministic waits.** Use assert/error polling within the config timeouts;
|
|
never `sleep`-and-hope.
|
|
- **Safety-critical => fault-injection.** For a safety-critical requirement, actually
|
|
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
|
|
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
|
|
the PM — do not relax the assertion or the requirement.
|
|
|
|
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
|
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
|
|
marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
|
|
outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
|
|
can never be reported as an executable test.
|