home-network-k3s-impl/conformance

conformance/ — the home-k3s conformance harness (WS-TEST)

This subtree is the definition of "done" for the whole platform. Nothing is accepted on a demo, a screenshot, or "it reconciled": every requirement is proven by an automated, reproducible, behavior-based test tagged with the requirement ID(s) it verifies (spec docs/14, ADR-0010, ADR-0011).

Tooling is fixed by ADR-0011: Kyverno for L0 (the CI policy gate + in-cluster admission enforcer — owned by task T-0.5) and Chainsaw for L1 rendering / L2 behavioral / L3 fault-injection (this harness).

The single command

./run.sh                              # full suite (skips L1/L2/L3 if no cluster)
./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
./run.sh --layer l2                   # one layer: l1 | l2 | l3 | meta | all
./run.sh --coverage-only              # regenerate the coverage report (no cluster)
./run.sh --status                     # derived requirement-STATUS report (no cluster)
./run.sh --lint                       # structural-validate test defs (no cluster)
./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
./run.sh --require-all                # phase-exit gate: every requirement must be covered

Gating is scoped, not "all 51 always" (R-TEST-03). Three distinct checks:

• --strict-coverage — hygiene (always-on, WS-TEST owns it): fails only on tag/annotation drift or a tag absent from the registry. Uncovered future-phase requirements are visible gaps in the report, not a harness failure.
• --requirements R-… — merge gate: selects the tagged tests to run and asserts those touched requirements are covered (a listed safety-critical one must have an L3 test).
• --require-all — phase-exit gate: every registry requirement must be covered.
• Documented out-of-scope gaps (e.g. R-ENV-02 dev→prod promote and R-GITOPS-01 state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map) are exempted from the completeness gates via --allow-gap, wired in run.sh as DOCUMENTED_GAPS so the exemption is explicit and reviewable. (R-STATE-02 node-loss used to be here; it now has a real L3 test — tests/l3-fault-injection/node-loss — and is covered, not exempted.)

Coverage vs status — two different questions

coverage.md answers "does a requirement-tagged test EXIST for R-x?" (R-TEST-04). status.md (./run.sh --status, generator lib/status.py) answers the operator's question — "what is the real state of each requirement given what is actually in the repo?" — deriving three orthogonal axes per requirement:

1. implementation — is there a tracked artifact naming the requirement?
2. reconciliation — is that artifact reachable by walking resources: from clusters/<env>/kustomization.yaml and expanding every Flux Kustomization's spec.path? This is what separates authored from actually reconciled: a commented-out resources: entry (e.g. platform/image-automation) is correctly reported as in-git-but-not-on-any-cluster.
3. verification — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts both as "covered"; status does not. TODO markers in a body always outrank an executable: "true" annotation, so ambiguity never upgrades a test.

It deliberately refuses to derive proof-of-pass. Nothing tracked in git records a test result — _artifacts/ and results/ are git-ignored and DECISIONS-LOG.md prose is not a signal — so every requirement's proof state is reported as not-derivable rather than defaulting to green. A status report that overstates completion is worse than none (agents/definition-of-done.md). Closing that gap needs a committed, machine-readable run ledger; until then the strongest claim the report makes is wired and testable. --strict-coverage also applies status's hygiene gate (registry↔spec drift, annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold). Exit codes follow ci/lib.sh: 0 pass / 1 fail / 3 skipped — a skip is never green.

run.sh is the only entrypoint. Local runs and the in-cluster CI Chainsaw Job call the same script with the same config, so results are identical.

Artifacts (always emitted, under _artifacts/, git-ignored)

Path	What	Consumer
_artifacts/report.junit.xml	JUnit — Chainsaw results, or a skipped-scaffold shim when no cluster	CI gate
_artifacts/coverage/coverage.md	Human requirement-coverage report	PR reviewers / logs
_artifacts/coverage/coverage.json	Machine coverage: {requirement: [tests]}, gaps, safety gaps	CI gate
_artifacts/status/status.md	Human requirement-STATUS report derived from repo state	operator / PM
_artifacts/status/status.json	Machine status: implementation / reconciliation / verification per requirement	tooling

Reproducibility contract (R-TEST-02 — what the harness guarantees)

• Single command. ./run.sh (or a tagged/layer subset) runs everything and emits the artifacts above — locally and in CI.
• Hermetic. Each Chainsaw test provisions its own namespace + fixtures/probes and tears them down (config/chainsaw-config.yaml auto-creates/auto-deletes the namespace; tests add finally for anything out-of-namespace). It never relies on pre-existing state.
• Deterministic. Tool versions pinned in versions.env; eventual consistency handled by bounded polling with explicit timeouts (config), never sleeps or wall-clock/ordering assumptions.
• Isolated & parallel-safe. Unique generated namespaces + per-requirement labels; tests run concurrently without cross-talk.
• Machine-readable & requirement-tagged. JUnit + coverage, every result carrying its R-* tags (convention in tests/README.md).
• Self-cleaning. Fixtures removed on pass or fail; a leaked namespace/fixture is itself a failure (skipDelete: false, enforced delete/cleanup timeouts).
• Versioned with the code. Tests live next to what they verify and change only when the human-owned spec they encode changes — never weakened to go green (R-TEST-06).

Layout

run.sh                      single-command runner (the entrypoint)
versions.env                pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
config/chainsaw-config.yaml  shared execution contract (timeouts, ns isolation, JUnit)
requirements/registry.yaml   mirror of EVERY spec R-* ID (coverage enumerates from here)
lib/
  common.sh                  tagging convention + test discovery/selection helpers
  coverage.py                requirement-coverage generator (stdlib only)
  status.py                  requirement-STATUS generator (stdlib only; --selftest)
  emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
tests/
  l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
incluster/
  chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)

How CI invokes it

CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls this same run.sh:

1. No-cluster stage (every PR, fast): ./run.sh --lint and ./run.sh --coverage-only --strict-coverage --requirements <IDs the change touches>. Validates every test parses
   • is tagged, regenerates coverage, fails the PR on tag/annotation drift (hygiene) and on any touched requirement lacking a test (merge gate). (L0 Kyverno CLI runs alongside — task T-0.5.)
2. Cluster stage (L2/L3): CI does not hold a kubeconfig. It triggers the in-cluster Chainsaw Job (incluster/), which runs ./run.sh under the scoped conformance-runner ServiceAccount and writes JUnit + coverage to a results volume CI collects.
   • Merge gate: the Job runs --requirements <IDs the change touches>; that subset runs green and those requirements are covered to merge (R-TEST-03).
   • Phase-exit gate: the Job runs --layer all --strict-coverage; add --require-all only at the milestone where the full requirement set is expected covered. The phase suite must be green before the phase's exit criteria are met.

Phase 0 status (no cluster yet)

No cluster exists, so L1/L2/L3 cannot execute. The behavior tests are valid scaffolds (steps + assertions authored, tagged, TODO(T-x.y) where a later phase delivers the fixture). In this phase run.sh:

• always generates the coverage report (cluster-independent), and
• emits a skipped JUnit listing every scaffold (so the gate is green and the artifact shape is real), and
• the meta/harness-selfcheck checks (reproducible output, JUnit+coverage emitted, everything tagged, zero safety-critical gaps) are what run.sh --strict-coverage actually enforces today.

As later phases deliver capabilities, each scaffold's TODOs are filled and the test flips from skipped to executing green — landing coverage without re-architecting the harness.