home-network-k3s-impl/conformance

conformance/ — the home-k3s conformance harness (WS-TEST)

This subtree is the definition of "done" for the whole platform. Nothing is accepted on a demo, a screenshot, or "it reconciled": every requirement is proven by an automated, reproducible, behavior-based test tagged with the requirement ID(s) it verifies (spec docs/14, ADR-0010, ADR-0011).

Tooling is fixed by ADR-0011: Kyverno for L0 (the CI policy gate + in-cluster admission enforcer — owned by task T-0.5) and Chainsaw for L1 rendering / L2 behavioral / L3 fault-injection (this harness).

The single command

./run.sh                              # full suite (skips L1/L2/L3 if no cluster)
./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
./run.sh --layer l2                   # one layer: l1 | l2 | l3 | meta | all
./run.sh --coverage-only              # regenerate the coverage report (no cluster)
./run.sh --status                     # derived requirement-STATUS report (no cluster)
./run.sh --lint                       # structural-validate test defs (no cluster)
./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
./run.sh --require-all                # phase-exit gate: every requirement must be covered
./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
./run.sh --layer l0 --include-scope whole-repo   # union: the tagged subset PLUS every test
                                      #   declaring `conformance.home.k3s/scope: whole-repo`
./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing

Gating is scoped, not "all 51 always" (R-TEST-03). Three distinct checks:

• --strict-coverage — hygiene (always-on, WS-TEST owns it): fails only on tag/annotation drift or a tag absent from the registry. Uncovered future-phase requirements are visible gaps in the report, not a harness failure.

• --requirements R-… — merge gate: selects the tagged tests to run and asserts those touched requirements are covered (a listed safety-critical one must have an L3 test).

• --require-all — phase-exit gate: every registry requirement must be covered.

• --include-scope whole-repo — the escape hatch scoping needs, because scoping has one blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees the coarse path→requirement map keys to nothing (apps/, infra/, most of platform/). For those, the change most likely to break the invariant attributes no requirement at all, so requirement-scoped selection filters the check out of exactly that change. Such a test declares its scope:

  metadata:
    annotations:
      conformance.home.k3s/scope: "whole-repo"
  

  and --include-scope whole-repo unions every declaring test into the selection. The CI stages (ci/l0.sh, ci/l1.sh, ci/conformance.sh) all pass it, so adding the annotation is the only step — there is nothing to remember and nothing to hand-write. Two consequences, both deliberate: a declaring test always runs, and a SKIP of a declaring test is UNWIRED, not green (the runner exits 3, which ci/l0.sh turns into the gate's own rc 3 — never a pass). tests/l0-static/whole-repo-invariants-wired proves the mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a hand-written "UNCONDITIONAL" stage per check in ci/l0.sh; see lib/common.sh for the full rationale and the incidents behind it.

• Documented out-of-scope gaps (e.g. R-ENV-02 dev→prod promote and R-GITOPS-01 state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map) are exempted from the completeness gates via --allow-gap, wired in run.sh as DOCUMENTED_GAPS so the exemption is explicit and reviewable. (R-STATE-02 node-loss used to be here; it now has a real L3 test — tests/l3-fault-injection/node-loss — and is covered, not exempted.)

Coverage vs status — two different questions

coverage.md answers "does a requirement-tagged test EXIST for R-x?" (R-TEST-04). status.md (./run.sh --status, generator lib/status.py) answers the operator's question — "what is the real state of each requirement given what is actually in the repo?" — deriving three orthogonal axes per requirement:

1. implementation — is there a tracked artifact naming the requirement?
2. reconciliation — is that artifact reachable by walking resources: from clusters/<env>/kustomization.yaml and expanding every Flux Kustomization's spec.path? This is what separates authored from actually reconciled: a commented-out resources: entry (e.g. platform/image-automation) is correctly reported as in-git-but-not-on-any-cluster.
3. verification — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts both as "covered"; status does not. TODO markers in a body always outrank an executable: "true" annotation, so ambiguity never upgrades a test.

It deliberately refuses to derive proof-of-pass. Nothing tracked in git records a test result — _artifacts/ and results/ are git-ignored and DECISIONS-LOG.md prose is not a signal — so every requirement's proof state is reported as not-derivable rather than defaulting to green. A status report that overstates completion is worse than none (agents/definition-of-done.md). Closing that gap needs a committed, machine-readable run ledger; until then the strongest claim the report makes is wired and testable. --strict-coverage also applies status's hygiene gate (registry↔spec drift, annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold). Exit codes follow ci/lib.sh: 0 pass / 1 fail / 3 skipped — a skip is never green.

run.sh is the only entrypoint. Local runs and the in-cluster CI Chainsaw Job call the same script with the same config, so results are identical.

Artifacts (always emitted, under _artifacts/, git-ignored)

Path	What	Consumer
_artifacts/report.junit.xml	JUnit — Chainsaw results, or a skipped-scaffold shim when no cluster	CI gate
_artifacts/coverage/coverage.md	Human requirement-coverage report	PR reviewers / logs
_artifacts/coverage/coverage.json	Machine coverage: {requirement: [tests]}, gaps, safety gaps	CI gate
_artifacts/status/status.md	Human requirement-STATUS report derived from repo state	operator / PM
_artifacts/status/status.json	Machine status: implementation / reconciliation / verification per requirement	tooling

Reproducibility contract (R-TEST-02 — what the harness guarantees)

• Single command. ./run.sh (or a tagged/layer subset) runs everything and emits the artifacts above — locally and in CI.
• Hermetic. Each Chainsaw test provisions its own namespace + fixtures/probes and tears them down (config/chainsaw-config.yaml auto-creates/auto-deletes the namespace; tests add finally for anything out-of-namespace). It never relies on pre-existing state.
• Deterministic. Tool versions pinned in versions.env; eventual consistency handled by bounded polling with explicit timeouts (config), never sleeps or wall-clock/ordering assumptions.
• Isolated & parallel-safe. Unique generated namespaces + per-requirement labels; tests run concurrently without cross-talk.
• Machine-readable & requirement-tagged. JUnit + coverage, every result carrying its R-* tags (convention in tests/README.md).
• Self-cleaning. Fixtures removed on pass or fail; a leaked namespace/fixture is itself a failure (skipDelete: false, enforced delete/cleanup timeouts).
• Versioned with the code. Tests live next to what they verify and change only when the human-owned spec they encode changes — never weakened to go green (R-TEST-06).

Layout

run.sh                      single-command runner (the entrypoint)
versions.env                pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
config/chainsaw-config.yaml  shared execution contract (timeouts, ns isolation, JUnit)
requirements/registry.yaml   mirror of EVERY spec R-* ID (coverage enumerates from here)
lib/
  common.sh                  tagging convention + test discovery/selection helpers
  coverage.py                requirement-coverage generator (stdlib only)
  status.py                  requirement-STATUS generator (stdlib only; --selftest)
  record_run.py              run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
  emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
runs/
  ledger.jsonl               TRACKED append-only proof-of-pass ledger (one row per
                             testcase per run) — the only source status.py trusts
  <UTC>-<short-sha>/         TRACKED per-run evidence: run.log + report.junit.xml
tests/
  l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
incluster/
  chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)

Proof of pass — the run ledger (runs/)

--status used to report PROVEN: 0 for all 51 requirements, and said so honestly: proof-of-pass was not recoverable from repo state. _artifacts/ is gitignored and the root .gitignore kills *.junit.xml globally, so no run result ever reached git; the only record of a green run was DECISIONS-LOG.md prose. runs/ is the fix lib/status.py asked for — a committed, machine-readable run ledger.

./run.sh --record                     # runs normally, and captures the evidence
# -> conformance/runs/<UTC>-<short-sha>/run.log            (tee'd, carries the sentinels)
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml   (that run's JUnit)
# -> conformance/runs/ledger.jsonl                         (one appended row per testcase)
python3 lib/record_run.py --run-dir runs/<id> [--dry-run]  # or record an existing run dir
python3 lib/record_run.py --selftest                       # deterministic, no cluster

Recording is explicit, never implicit. Plain ./run.sh writes no ledger row: a run that silently records itself is how a bad run becomes a permanent false proof. Commit the run dir and the ledger — an uncommitted ledger proves nothing.

Why a committed JUnit alone would be WORSE than nothing

The L2/L3 tests are runnable and self-skipping: when a precondition is missing (no longhorn StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token and exit 0, and their check: accepts either outcome —

(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
 || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true

So a self-skip exits 0 and Chainsaw records passed exactly as a real run does, and Chainsaw's JUnit carries no stdout — its <testcase> elements hold only <properties>. A committed JUnit on its own would therefore record DR_RESTORE_SKIPPED_OK as proof of R-STATE-03: fake green, and worse than no ledger because it would look authoritative.

The sentinel token in run.log is the only thing that separates proven from skipped. record_run.py derives each test's tokens from its own chainsaw-test.yaml (every contains($stdout, 'TOKEN'); *_SKIP*_OK is a skip token, the last non-skip token is the verdict token), so the table below is never hand-maintained:

Test	pass token (proof)	skip token (never proof)
l3/node-loss	NODE_LOSS_VOLUME_AVAILABLE_OK	NODE_LOSS_SKIPPED_OK
l3/dr-restore	DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK (+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK)	DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK
l3/data-survival	DATA_SURVIVED_OK	DATA_SURVIVAL_SKIPPED_OK
l3/vm-data-survival	VM_DATA_SURVIVED_OK	VM_DATA_SURVIVAL_SKIPPED_OK
l3/hands-off-update	HANDS_OFF_UPDATE_OK	HANDS_OFF_UPDATE_SKIPPED_OK
l2/vm-lifecycle	VM_LIFECYCLE_OK	VM_LIFECYCLE_SKIPPED_OK
l2/vm-default-deny	VM_DEFAULT_DENY_OK	VM_DEFAULT_DENY_SKIPPED_OK

A row reaches proof: "proven" on one path: JUnit passed, every declared pass token present in that test's log lines, and no skip token anywhere in them. A skip token beats a pass token (worst wins). Anything else is skipped, failed, junit-skipped, unknown-no-sentinel-declared or unknown-sentinel-absent — all non-green. L0/L1 need no sentinel because lib/run_l1_offline.py already renders their skips as JUnit <skipped/>, never as a green pass.

Staleness — a proof is a claim about a tree, not a badge

--status honours a proven row only while the run's git_sha is a descendant of the last commit touching that requirement's implementing manifests or its verifying test. Otherwise the row degrades to stale-proof, which is not green. A run captured from a dirty working tree is recorded as <sha>-dirty, which no ancestry check can satisfy — so it never counts. Re-run with --record to refresh.

A requirement reads proven only when every verifying test does; one skipped, failed or unrecorded test drags the whole requirement down. With an empty ledger --status behaves exactly as before: every requirement not-derivable, headline PROVEN 0.

How CI invokes it

CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls this same run.sh:

1. No-cluster stage (every PR, fast): ./run.sh --lint and ./run.sh --coverage-only --strict-coverage --requirements <IDs the change touches>. Validates every test parses
   • is tagged, regenerates coverage, fails the PR on tag/annotation drift (hygiene) and on any touched requirement lacking a test (merge gate). (L0 Kyverno CLI runs alongside — task T-0.5.)
2. Cluster stage (L2/L3): CI does not hold a kubeconfig. It triggers the in-cluster Chainsaw Job (incluster/), which runs ./run.sh under the scoped conformance-runner ServiceAccount and writes JUnit + coverage to a results volume CI collects.
   • Merge gate: the Job runs --requirements <IDs the change touches>; that subset runs green and those requirements are covered to merge (R-TEST-03).
   • Phase-exit gate: the Job runs --layer all --strict-coverage; add --require-all only at the milestone where the full requirement set is expected covered. The phase suite must be green before the phase's exit criteria are met.

Phase 0 status (no cluster yet)

No cluster exists, so L1/L2/L3 cannot execute. The behavior tests are valid scaffolds (steps + assertions authored, tagged, TODO(T-x.y) where a later phase delivers the fixture). In this phase run.sh:

• always generates the coverage report (cluster-independent), and
• emits a skipped JUnit listing every scaffold (so the gate is green and the artifact shape is real), and
• the meta/harness-selfcheck checks (reproducible output, JUnit+coverage emitted, everything tagged, zero safety-critical gaps) are what run.sh --strict-coverage actually enforces today.

As later phases deliver capabilities, each scaffold's TODOs are filled and the test flips from skipped to executing green — landing coverage without re-architecting the harness.