home-network-k3s-impl/conformance/README.md

# conformance/ — the home-k3s conformance harness (WS-TEST)

> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**


This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
verifies (spec [`docs/14`](../../home-network-k3s/docs/14-testing-and-conformance.md),
[ADR-0010](../../home-network-k3s/decisions/adr-0010-conformance-testing.md),
[ADR-0011](../../home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md)).

Tooling is fixed by ADR-0011: **Kyverno** for L0 (the CI policy gate + in-cluster
admission enforcer — owned by task T-0.5) and **Chainsaw** for **L1 rendering / L2
behavioral / L3 fault-injection** (this harness).

## The single command

```bash
./run.sh                              # full suite (skips L1/L2/L3 if no cluster)
./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
./run.sh --layer l2                   # one layer: l1 | l2 | l3 | meta | all
./run.sh --coverage-only              # regenerate the coverage report (no cluster)
./run.sh --status                     # derived requirement-STATUS report (no cluster)
./run.sh --lint                       # structural-validate test defs (no cluster)
./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
./run.sh --require-all                # phase-exit gate: every requirement must be covered
./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
./run.sh --layer l0 --include-scope whole-repo   # union: the tagged subset PLUS every test
                                      #   declaring `conformance.home.k3s/scope: whole-repo`
./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
```

**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
- `--strict-coverage` — **hygiene** (always-on, WS-TEST owns it): fails only on
  tag/annotation drift or a tag absent from the registry. Uncovered future-phase
  requirements are *visible gaps in the report*, not a harness failure.
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
  those touched requirements are covered (a listed safety-critical one must have an L3 test).
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
  blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
  tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
  the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
  `platform/`). For those, *the change most likely to break the invariant attributes no
  requirement at all*, so requirement-scoped selection filters the check out of exactly that
  change. Such a test **declares** its scope:

  ```yaml
  metadata:
    annotations:
      conformance.home.k3s/scope: "whole-repo"
  ```

  and `--include-scope whole-repo` unions every declaring test into the selection. The CI
  stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
  annotation is the only step** — there is nothing to remember and nothing to hand-write.
  Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
  declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
  gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
  mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
  hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
  full rationale and the incidents behind it.
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
  state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
  are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
  `DOCUMENTED_GAPS` so the exemption is explicit and reviewable. (`R-STATE-02` node-loss
  used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
  is covered, not exempted.)

### Coverage vs status — two different questions

`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
question — **"what is the real state of each requirement given what is actually in the
repo?"** — deriving three orthogonal axes per requirement:

1. **implementation** — is there a tracked artifact naming the requirement?
2. **reconciliation** — is that artifact reachable by walking `resources:` from
   `clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
   `spec.path`? This is what separates **authored** from **actually reconciled**: a
   commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
   reported as in-git-but-not-on-any-cluster.
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
   both as "covered"; status does not. TODO markers in a body always outrank an
   `executable: "true"` annotation, so ambiguity never upgrades a test.

**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
signal — so every requirement's proof state is reported as `not-derivable` rather than
defaulting to green. A status report that overstates completion is worse than none
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
run ledger**; until then the strongest claim the report makes is *wired and testable*.
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.

`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
the *same* script with the same config, so results are identical.

### Artifacts (always emitted, under `_artifacts/`, git-ignored)
| Path | What | Consumer |
|---|---|---|
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |

## Reproducibility contract (R-TEST-02 — what the harness guarantees)
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
  artifacts above — locally and in CI.
- **Hermetic.** Each Chainsaw test provisions its own namespace + fixtures/probes and tears
  them down (`config/chainsaw-config.yaml` auto-creates/auto-deletes the namespace; tests
  add `finally` for anything out-of-namespace). It never relies on pre-existing state.
- **Deterministic.** Tool versions pinned in [`versions.env`](versions.env); eventual
  consistency handled by **bounded polling with explicit timeouts** (config), never sleeps
  or wall-clock/ordering assumptions.
- **Isolated & parallel-safe.** Unique generated namespaces + per-requirement labels; tests
  run concurrently without cross-talk.
- **Machine-readable & requirement-tagged.** JUnit + coverage, every result carrying its
  `R-*` tags (convention in [`tests/README.md`](tests/README.md)).
- **Self-cleaning.** Fixtures removed on pass **or** fail; a leaked namespace/fixture is
  itself a failure (`skipDelete: false`, enforced delete/cleanup timeouts).
- **Versioned with the code.** Tests live next to what they verify and change only when the
  human-owned spec they encode changes — **never** weakened to go green (R-TEST-06).

## Layout
```
run.sh                      single-command runner (the entrypoint)
versions.env                pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
config/chainsaw-config.yaml  shared execution contract (timeouts, ns isolation, JUnit)
requirements/registry.yaml   mirror of EVERY spec R-* ID (coverage enumerates from here)
lib/
  common.sh                  tagging convention + test discovery/selection helpers
  coverage.py                requirement-coverage generator (stdlib only)
  status.py                  requirement-STATUS generator (stdlib only; --selftest)
  record_run.py              run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
  emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
runs/
  ledger.jsonl               TRACKED append-only proof-of-pass ledger (one row per
                             testcase per run) — the only source status.py trusts
  <UTC>-<short-sha>/         TRACKED per-run evidence: run.log + report.junit.xml
tests/
  l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
incluster/
  chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)
```

## Proof of pass — the run ledger (`runs/`)

`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
asked for — **a committed, machine-readable run ledger**.

```bash
./run.sh --record                     # runs normally, and captures the evidence
# -> conformance/runs/<UTC>-<short-sha>/run.log            (tee'd, carries the sentinels)
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml   (that run's JUnit)
# -> conformance/runs/ledger.jsonl                         (one appended row per testcase)
python3 lib/record_run.py --run-dir runs/<id> [--dry-run]  # or record an existing run dir
python3 lib/record_run.py --selftest                       # deterministic, no cluster
```

**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
that silently records itself is how a bad run becomes a permanent false proof. **Commit
the run dir and the ledger** — an uncommitted ledger proves nothing.

### Why a committed JUnit alone would be WORSE than nothing

The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
and `exit 0`, and their `check:` accepts either outcome —

```yaml
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
 || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
```

So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.

**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
verdict token), so the table below is never hand-maintained:

| Test | pass token (proof) | skip token (never proof) |
|---|---|---|
| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |

A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
token present in that test's log lines, and **no** skip token anywhere in them. A skip
token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
skips as JUnit `<skipped/>`, never as a green pass.

### Staleness — a proof is a claim about a tree, not a badge

`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
last commit touching that requirement's implementing manifests or its verifying test**.
Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
it never counts. Re-run with `--record` to refresh.

A requirement reads `proven` only when **every** verifying test does; one skipped, failed
or unrecorded test drags the whole requirement down. With an empty ledger `--status`
behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.

## How CI invokes it
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:

1. **No-cluster stage (every PR, fast):** `./run.sh --lint` and `./run.sh --coverage-only
   --strict-coverage --requirements <IDs the change touches>`. Validates every test parses
   + is tagged, regenerates coverage, **fails the PR** on tag/annotation drift (hygiene)
   and on any *touched* requirement lacking a test (merge gate). (L0 Kyverno CLI runs
   alongside — task T-0.5.)
2. **Cluster stage (L2/L3):** CI does **not** hold a kubeconfig. It **triggers the
   in-cluster Chainsaw Job** ([`incluster/`](incluster/README.md)), which runs `./run.sh`
   under the scoped `conformance-runner` ServiceAccount and writes JUnit + coverage to a
   results volume CI collects.
   - **Merge gate:** the Job runs `--requirements <IDs the change touches>`; that subset
     runs green **and** those requirements are covered to merge (R-TEST-03).
   - **Phase-exit gate:** the Job runs `--layer all --strict-coverage`; add `--require-all`
     only at the milestone where the full requirement set is expected covered. The phase
     suite must be green before the phase's exit criteria are met.

## Phase 0 status (no cluster yet)
No cluster exists, so L1/L2/L3 **cannot execute**. The behavior tests are **valid
scaffolds** (steps + assertions authored, tagged, `TODO(T-x.y)` where a later phase
delivers the fixture). In this phase `run.sh`:
- always generates the **coverage report** (cluster-independent), and
- emits a **skipped** JUnit listing every scaffold (so the gate is green and the artifact
  shape is real), and
- the **`meta/harness-selfcheck`** checks (reproducible output, JUnit+coverage emitted,
  everything tagged, zero safety-critical gaps) are what `run.sh --strict-coverage`
  actually enforces today.

As later phases deliver capabilities, each scaffold's `TODO`s are filled and the test flips
from *skipped* to *executing green* — landing coverage without re-architecting the harness.