T-0.4 Chainsaw conformance harness + coverage + behavior scaffolds (R-TEST-01..06)

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 00:08:32 -05:00
1 parent a778d923b6
commit 9d8e03892f
@@ -1,3 +1,111 @@
-# conformance
+# conformance/ — the home-k3s conformance harness (WS-TEST)

-_Placeholder — populated in Phase 0. See ../home-network-k3s (read-only spec)._
+This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
+on a demo, a screenshot, or "it reconciled": every requirement is proven by an
+**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
+verifies (spec [`docs/14`](../../home-network-k3s/docs/14-testing-and-conformance.md),
+[ADR-0010](../../home-network-k3s/decisions/adr-0010-conformance-testing.md),
+[ADR-0011](../../home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md)).
+
+Tooling is fixed by ADR-0011: **Kyverno** for L0 (the CI policy gate + in-cluster
+admission enforcer — owned by task T-0.5) and **Chainsaw** for **L1 rendering / L2
+behavioral / L3 fault-injection** (this harness).
+
+## The single command
+
+```bash
+./run.sh                              # full suite (skips L1/L2/L3 if no cluster)
+./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
+./run.sh --layer l2                   # one layer: l1 | l2 | l3 | meta | all
+./run.sh --coverage-only              # regenerate the coverage report (no cluster)
+./run.sh --lint                       # structural-validate test defs (no cluster)
+./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
+./run.sh --require-all                # phase-exit gate: every requirement must be covered
+```
+
+**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
+- `--strict-coverage` — **hygiene** (always-on, WS-TEST owns it): fails only on
+  tag/annotation drift or a tag absent from the registry. Uncovered future-phase
+  requirements are *visible gaps in the report*, not a harness failure.
+- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
+  those touched requirements are covered (a listed safety-critical one must have an L3 test).
+- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
+- Documented out-of-scope gaps (e.g. `R-STATE-02` node-loss, owned by WS-STORAGE) are
+  exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
+  `DOCUMENTED_GAPS` so the exemption is explicit and reviewable.
+
+`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
+the *same* script with the same config, so results are identical.
+
+### Artifacts (always emitted, under `_artifacts/`, git-ignored)
+| Path | What | Consumer |
+|---|---|---|
+| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
+| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
+| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
+
+## Reproducibility contract (R-TEST-02 — what the harness guarantees)
+- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
+  artifacts above — locally and in CI.
+- **Hermetic.** Each Chainsaw test provisions its own namespace + fixtures/probes and tears
+  them down (`config/chainsaw-config.yaml` auto-creates/auto-deletes the namespace; tests
+  add `finally` for anything out-of-namespace). It never relies on pre-existing state.
+- **Deterministic.** Tool versions pinned in [`versions.env`](versions.env); eventual
+  consistency handled by **bounded polling with explicit timeouts** (config), never sleeps
+  or wall-clock/ordering assumptions.
+- **Isolated & parallel-safe.** Unique generated namespaces + per-requirement labels; tests
+  run concurrently without cross-talk.
+- **Machine-readable & requirement-tagged.** JUnit + coverage, every result carrying its
+  `R-*` tags (convention in [`tests/README.md`](tests/README.md)).
+- **Self-cleaning.** Fixtures removed on pass **or** fail; a leaked namespace/fixture is
+  itself a failure (`skipDelete: false`, enforced delete/cleanup timeouts).
+- **Versioned with the code.** Tests live next to what they verify and change only when the
+  human-owned spec they encode changes — **never** weakened to go green (R-TEST-06).
+
+## Layout
+```
+run.sh                      single-command runner (the entrypoint)
+versions.env                pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
+config/chainsaw-config.yaml  shared execution contract (timeouts, ns isolation, JUnit)
+requirements/registry.yaml   mirror of EVERY spec R-* ID (coverage enumerates from here)
+lib/
+  common.sh                  tagging convention + test discovery/selection helpers
+  coverage.py                requirement-coverage generator (stdlib only)
+  emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
+tests/
+  l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
+incluster/
+  chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)
+```
+
+## How CI invokes it
+CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
+
+1. **No-cluster stage (every PR, fast):** `./run.sh --lint` and `./run.sh --coverage-only
+   --strict-coverage --requirements <IDs the change touches>`. Validates every test parses
+   + is tagged, regenerates coverage, **fails the PR** on tag/annotation drift (hygiene)
+   and on any *touched* requirement lacking a test (merge gate). (L0 Kyverno CLI runs
+   alongside — task T-0.5.)
+2. **Cluster stage (L2/L3):** CI does **not** hold a kubeconfig. It **triggers the
+   in-cluster Chainsaw Job** ([`incluster/`](incluster/README.md)), which runs `./run.sh`
+   under the scoped `conformance-runner` ServiceAccount and writes JUnit + coverage to a
+   results volume CI collects.
+   - **Merge gate:** the Job runs `--requirements <IDs the change touches>`; that subset
+     runs green **and** those requirements are covered to merge (R-TEST-03).
+   - **Phase-exit gate:** the Job runs `--layer all --strict-coverage`; add `--require-all`
+     only at the milestone where the full requirement set is expected covered. The phase
+     suite must be green before the phase's exit criteria are met.
+
+## Phase 0 status (no cluster yet)
+No cluster exists, so L1/L2/L3 **cannot execute**. The behavior tests are **valid
+scaffolds** (steps + assertions authored, tagged, `TODO(T-x.y)` where a later phase
+delivers the fixture). In this phase `run.sh`:
+- always generates the **coverage report** (cluster-independent), and
+- emits a **skipped** JUnit listing every scaffold (so the gate is green and the artifact
+  shape is real), and
+- the **`meta/harness-selfcheck`** checks (reproducible output, JUnit+coverage emitted,
+  everything tagged, zero safety-critical gaps) are what `run.sh --strict-coverage`
+  actually enforces today.
+
+As later phases deliver capabilities, each scaffold's `TODO`s are filled and the test flips
+from *skipped* to *executing green* — landing coverage without re-architecting the harness.