home-network-k3s-impl/conformance

conformance/ — the home-k3s conformance harness (WS-TEST)

STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below. This document speaks as though no cluster exists and the platform is scaffolding. A three-node k3s cluster has existed for weeks and Flux has been reconciling against it. Gitea is live, the Actions runner incluster-dind is online, and main is a protected branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as describing a live gap rather than an expected stage — most importantly, L2/L3 conformance still does not run in CI (vars.CLUSTER_AVAILABLE is unset), which is a real coverage hole, not a phase.

This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on documentation rather than measurement, and main was unprotected with zero runners registered the whole time. Query the live API before asserting platform state.

This subtree is the definition of "done" for the whole platform. Nothing is accepted on a demo, a screenshot, or "it reconciled": every requirement is proven by an automated, reproducible, behavior-based test tagged with the requirement ID(s) it verifies (spec docs/14, ADR-0010, ADR-0011).

Tooling is fixed by ADR-0011: Kyverno for L0 (the CI policy gate + in-cluster admission enforcer — owned by task T-0.5) and Chainsaw for L1 rendering / L2 behavioral / L3 fault-injection (this harness).

The single command

./run.sh                              # full suite (skips L1/L2/L3 if no cluster)
./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
./run.sh --layer l2                   # one layer: l1 | l2 | l3 | meta | all
./run.sh --coverage-only              # regenerate the coverage report (no cluster)
./run.sh --status                     # derived requirement-STATUS report (no cluster)
./run.sh --lint                       # structural-validate test defs (no cluster)
./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
./run.sh --require-all                # phase-exit gate: every requirement must be covered
./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
./run.sh --layer l0 --include-scope whole-repo   # union: the tagged subset PLUS every test
                                      #   declaring `conformance.home.k3s/scope: whole-repo`
./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing

Gating is scoped, not "all 51 always" (R-TEST-03). Three distinct checks:

• --strict-coverage — hygiene (always-on, WS-TEST owns it): fails only on tag/annotation drift or a tag absent from the registry. Uncovered future-phase requirements are visible gaps in the report, not a harness failure.

• --requirements R-… — merge gate: selects the tagged tests to run and asserts those touched requirements are covered (a listed safety-critical one must have an L3 test).

• --require-all — phase-exit gate: every registry requirement must be covered.

• --include-scope whole-repo — the escape hatch scoping needs, because scoping has one blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees the coarse path→requirement map keys to nothing (apps/, infra/, most of platform/). For those, the change most likely to break the invariant attributes no requirement at all, so requirement-scoped selection filters the check out of exactly that change. Such a test declares its scope:

  metadata:
    annotations:
      conformance.home.k3s/scope: "whole-repo"
  

  and --include-scope whole-repo unions every declaring test into the selection. The CI stages (ci/l0.sh, ci/l1.sh, ci/conformance.sh) all pass it, so adding the annotation is the only step — there is nothing to remember and nothing to hand-write. Two consequences, both deliberate: a declaring test always runs, and a SKIP of a declaring test is UNWIRED, not green (the runner exits 3, which ci/l0.sh turns into the gate's own rc 3 — never a pass). tests/l0-static/whole-repo-invariants-wired proves the mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a hand-written "UNCONDITIONAL" stage per check in ci/l0.sh; see lib/common.sh for the full rationale and the incidents behind it.

• Documented out-of-scope gaps (e.g. R-ENV-02 dev→prod promote and R-GITOPS-01 state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map) are exempted from the completeness gates via --allow-gap, wired in run.sh as DOCUMENTED_GAPS so the exemption is explicit and reviewable. (R-STATE-02 node-loss used to be here; it now has a real L3 test — tests/l3-fault-injection/node-loss — and is covered, not exempted.)

Coverage vs status — two different questions

coverage.md answers "does a requirement-tagged test EXIST for R-x?" (R-TEST-04). status.md (./run.sh --status, generator lib/status.py) answers the operator's question — "what is the real state of each requirement given what is actually in the repo?" — deriving three orthogonal axes per requirement:

1. implementation — is there a tracked artifact naming the requirement?
2. reconciliation — is that artifact reachable by walking resources: from clusters/<env>/kustomization.yaml and expanding every Flux Kustomization's spec.path? This is what separates authored from actually reconciled: a commented-out resources: entry (e.g. platform/image-automation) is correctly reported as in-git-but-not-on-any-cluster.
3. verification — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts both as "covered"; status does not. TODO markers in a body always outrank an executable: "true" annotation, so ambiguity never upgrades a test.

It deliberately refuses to derive proof-of-pass. Nothing tracked in git records a test result — _artifacts/ and results/ are git-ignored and DECISIONS-LOG.md prose is not a signal — so every requirement's proof state is reported as not-derivable rather than defaulting to green. A status report that overstates completion is worse than none (agents/definition-of-done.md). Closing that gap needs a committed, machine-readable run ledger; until then the strongest claim the report makes is wired and testable. --strict-coverage also applies status's hygiene gate (registry↔spec drift, annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold). Exit codes follow ci/lib.sh: 0 pass / 1 fail / 3 skipped — a skip is never green.

run.sh is the only entrypoint. Local runs and the in-cluster CI Chainsaw Job call the same script with the same config, so results are identical.

Artifacts (always emitted, under _artifacts/, git-ignored)

Path	What	Consumer
_artifacts/report.junit.xml	JUnit — Chainsaw results, or a skipped-scaffold shim when no cluster	CI gate
_artifacts/coverage/coverage.md	Human requirement-coverage report	PR reviewers / logs
_artifacts/coverage/coverage.json	Machine coverage: {requirement: [tests]}, gaps, safety gaps	CI gate
_artifacts/status/status.md	Human requirement-STATUS report derived from repo state	operator / PM
_artifacts/status/status.json	Machine status: implementation / reconciliation / verification per requirement	tooling

Reproducibility contract (R-TEST-02 — what the harness guarantees)

• Single command. ./run.sh (or a tagged/layer subset) runs everything and emits the artifacts above — locally and in CI.
• Hermetic. Each Chainsaw test provisions its own namespace + fixtures/probes and tears them down (config/chainsaw-config.yaml auto-creates/auto-deletes the namespace; tests add finally for anything out-of-namespace). It never relies on pre-existing state.
• Deterministic. Tool versions pinned in versions.env; eventual consistency handled by bounded polling with explicit timeouts (config), never sleeps or wall-clock/ordering assumptions.
• Isolated & parallel-safe. Unique generated namespaces + per-requirement labels; tests run concurrently without cross-talk.
• Machine-readable & requirement-tagged. JUnit + coverage, every result carrying its R-* tags (convention in tests/README.md).
• Self-cleaning. Fixtures removed on pass or fail; a leaked namespace/fixture is itself a failure (skipDelete: false, enforced delete/cleanup timeouts).
• Versioned with the code. Tests live next to what they verify and change only when the human-owned spec they encode changes — never weakened to go green (R-TEST-06).

Layout

run.sh                      single-command runner (the entrypoint)
versions.env                pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
config/chainsaw-config.yaml  shared execution contract (timeouts, ns isolation, JUnit)
requirements/registry.yaml   mirror of EVERY spec R-* ID (coverage enumerates from here)
lib/
  common.sh                  tagging convention + test discovery/selection helpers
  coverage.py                requirement-coverage generator (stdlib only)
  status.py                  requirement-STATUS generator (stdlib only; --selftest)
  record_run.py              run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
  emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
runs/
  ledger.jsonl               TRACKED append-only proof-of-pass ledger (one row per
                             testcase per run) — the only source status.py trusts
  <UTC>-<short-sha>/         TRACKED per-run evidence: run.log + report.junit.xml
tests/
  l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
incluster/
  chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)

Proof of pass — the run ledger (runs/)

--status used to report PROVEN: 0 for all 51 requirements, and said so honestly: proof-of-pass was not recoverable from repo state. _artifacts/ is gitignored and the root .gitignore kills *.junit.xml globally, so no run result ever reached git; the only record of a green run was DECISIONS-LOG.md prose. runs/ is the fix lib/status.py asked for — a committed, machine-readable run ledger.

./run.sh --record                     # runs normally, and captures the evidence
# -> conformance/runs/<UTC>-<short-sha>/run.log            (tee'd, carries the sentinels)
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml   (that run's JUnit)
# -> conformance/runs/ledger.jsonl                         (one appended row per testcase)
python3 lib/record_run.py --run-dir runs/<id> [--dry-run]  # or record an existing run dir
python3 lib/record_run.py --selftest                       # deterministic, no cluster

Recording is explicit, never implicit. Plain ./run.sh writes no ledger row: a run that silently records itself is how a bad run becomes a permanent false proof. Commit the run dir and the ledger — an uncommitted ledger proves nothing.

Why a committed JUnit alone would be WORSE than nothing

The L2/L3 tests are runnable and self-skipping: when a precondition is missing (no longhorn StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token and exit 0, and their check: accepts either outcome —

(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
 || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true

So a self-skip exits 0 and Chainsaw records passed exactly as a real run does, and Chainsaw's JUnit carries no stdout — its <testcase> elements hold only <properties>. A committed JUnit on its own would therefore record DR_RESTORE_SKIPPED_OK as proof of R-STATE-03: fake green, and worse than no ledger because it would look authoritative.

The sentinel token in run.log is the only thing that separates proven from skipped. record_run.py derives each test's tokens from its own chainsaw-test.yaml (every contains($stdout, 'TOKEN'); *_SKIP*_OK is a skip token, the last non-skip token is the verdict token), so the table below is never hand-maintained:

Test	pass token (proof)	skip token (never proof)
l3/node-loss	NODE_LOSS_VOLUME_AVAILABLE_OK	NODE_LOSS_SKIPPED_OK
l3/dr-restore	DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK (+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK)	DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK
l3/data-survival	DATA_SURVIVED_OK	DATA_SURVIVAL_SKIPPED_OK
l3/vm-data-survival	VM_DATA_SURVIVED_OK	VM_DATA_SURVIVAL_SKIPPED_OK
l3/hands-off-update	HANDS_OFF_UPDATE_OK	HANDS_OFF_UPDATE_SKIPPED_OK
l2/vm-lifecycle	VM_LIFECYCLE_OK	VM_LIFECYCLE_SKIPPED_OK
l2/vm-default-deny	VM_DEFAULT_DENY_OK	VM_DEFAULT_DENY_SKIPPED_OK

A row reaches proof: "proven" on one path: JUnit passed, every declared pass token present in that test's log lines, and no skip token anywhere in them. A skip token beats a pass token (worst wins). Anything else is skipped, failed, junit-skipped, unknown-no-sentinel-declared or unknown-sentinel-absent — all non-green. L0/L1 need no sentinel because lib/run_l1_offline.py already renders their skips as JUnit <skipped/>, never as a green pass.

Staleness — a proof is a claim about a tree, not a badge

--status honours a proven row only while the run's git_sha is a descendant of the last commit touching that requirement's implementing manifests or its verifying test. Otherwise the row degrades to stale-proof, which is not green. A run captured from a dirty working tree is recorded as <sha>-dirty, which no ancestry check can satisfy — so it never counts. Re-run with --record to refresh.

A requirement reads proven only when every verifying test does; one skipped, failed or unrecorded test drags the whole requirement down. With an empty ledger --status behaves exactly as before: every requirement not-derivable, headline PROVEN 0.

How CI invokes it

CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls this same run.sh:

1. No-cluster stage (every PR, fast): ./run.sh --lint and ./run.sh --coverage-only --strict-coverage --requirements <IDs the change touches>. Validates every test parses
   • is tagged, regenerates coverage, fails the PR on tag/annotation drift (hygiene) and on any touched requirement lacking a test (merge gate). (L0 Kyverno CLI runs alongside — task T-0.5.)
2. Cluster stage (L2/L3): CI does not hold a kubeconfig. It triggers the in-cluster Chainsaw Job (incluster/), which runs ./run.sh under the scoped conformance-runner ServiceAccount and writes JUnit + coverage to a results volume CI collects.
   • Merge gate: the Job runs --requirements <IDs the change touches>; that subset runs green and those requirements are covered to merge (R-TEST-03).
   • Phase-exit gate: the Job runs --layer all --strict-coverage; add --require-all only at the milestone where the full requirement set is expected covered. The phase suite must be green before the phase's exit criteria are met.

Phase 0 status (no cluster yet)

No cluster exists, so L1/L2/L3 cannot execute. The behavior tests are valid scaffolds (steps + assertions authored, tagged, TODO(T-x.y) where a later phase delivers the fixture). In this phase run.sh:

• always generates the coverage report (cluster-independent), and
• emits a skipped JUnit listing every scaffold (so the gate is green and the artifact shape is real), and
• the meta/harness-selfcheck checks (reproducible output, JUnit+coverage emitted, everything tagged, zero safety-critical gaps) are what run.sh --strict-coverage actually enforces today.

As later phases deliver capabilities, each scaffold's TODOs are filled and the test flips from skipped to executing green — landing coverage without re-architecting the harness.