# conformance/ — the home-k3s conformance harness (WS-TEST)

This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
on a demo, a screenshot, or "it reconciled": every requirement is proven by an
**automated, reproducible, behavior-based test** tagged with the requirement ID(s) it
verifies (spec [`docs/14`](../../home-network-k3s/docs/14-testing-and-conformance.md),
[ADR-0010](../../home-network-k3s/decisions/adr-0010-conformance-testing.md),
[ADR-0011](../../home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md)).

Tooling is fixed by ADR-0011: **Kyverno** for L0 (the CI policy gate + in-cluster
admission enforcer — owned by task T-0.5) and **Chainsaw** for **L1 rendering / L2
behavioral / L3 fault-injection** (this harness).

## The single command

```bash
./run.sh                              # full suite (skips L1/L2/L3 if no cluster)
./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
./run.sh --layer l2                   # one layer: l1 | l2 | l3 | meta | all
./run.sh --coverage-only              # regenerate the coverage report (no cluster)
./run.sh --status                     # derived requirement-STATUS report (no cluster)
./run.sh --lint                       # structural-validate test defs (no cluster)
./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
./run.sh --require-all                # phase-exit gate: every requirement must be covered
```

**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
- `--strict-coverage` — **hygiene** (always-on, WS-TEST owns it): fails only on
  tag/annotation drift or a tag absent from the registry. Uncovered future-phase
  requirements are *visible gaps in the report*, not a harness failure.
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
  those touched requirements are covered (a listed safety-critical one must have an L3 test).
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
  state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
  are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
  `DOCUMENTED_GAPS` so the exemption is explicit and reviewable. (`R-STATE-02` node-loss
  used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
  is covered, not exempted.)

### Coverage vs status — two different questions

`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
question — **"what is the real state of each requirement given what is actually in the
repo?"** — deriving three orthogonal axes per requirement:

1. **implementation** — is there a tracked artifact naming the requirement?
2. **reconciliation** — is that artifact reachable by walking `resources:` from
   `clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
   `spec.path`? This is what separates **authored** from **actually reconciled**: a
   commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
   reported as in-git-but-not-on-any-cluster.
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
   both as "covered"; status does not. TODO markers in a body always outrank an
   `executable: "true"` annotation, so ambiguity never upgrades a test.

**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
signal — so every requirement's proof state is reported as `not-derivable` rather than
defaulting to green. A status report that overstates completion is worse than none
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
run ledger**; until then the strongest claim the report makes is *wired and testable*.
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.

`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
the *same* script with the same config, so results are identical.

### Artifacts (always emitted, under `_artifacts/`, git-ignored)
| Path | What | Consumer |
|---|---|---|
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |

## Reproducibility contract (R-TEST-02 — what the harness guarantees)
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
  artifacts above — locally and in CI.
- **Hermetic.** Each Chainsaw test provisions its own namespace + fixtures/probes and tears
  them down (`config/chainsaw-config.yaml` auto-creates/auto-deletes the namespace; tests
  add `finally` for anything out-of-namespace). It never relies on pre-existing state.
- **Deterministic.** Tool versions pinned in [`versions.env`](versions.env); eventual
  consistency handled by **bounded polling with explicit timeouts** (config), never sleeps
  or wall-clock/ordering assumptions.
- **Isolated & parallel-safe.** Unique generated namespaces + per-requirement labels; tests
  run concurrently without cross-talk.
- **Machine-readable & requirement-tagged.** JUnit + coverage, every result carrying its
  `R-*` tags (convention in [`tests/README.md`](tests/README.md)).
- **Self-cleaning.** Fixtures removed on pass **or** fail; a leaked namespace/fixture is
  itself a failure (`skipDelete: false`, enforced delete/cleanup timeouts).
- **Versioned with the code.** Tests live next to what they verify and change only when the
  human-owned spec they encode changes — **never** weakened to go green (R-TEST-06).

## Layout
```
run.sh                      single-command runner (the entrypoint)
versions.env                pinned toolchain (Chainsaw/kubectl/Kyverno) — determinism
config/chainsaw-config.yaml  shared execution contract (timeouts, ns isolation, JUnit)
requirements/registry.yaml   mirror of EVERY spec R-* ID (coverage enumerates from here)
lib/
  common.sh                  tagging convention + test discovery/selection helpers
  coverage.py                requirement-coverage generator (stdlib only)
  status.py                  requirement-STATUS generator (stdlib only; --selftest)
  emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
tests/
  l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
incluster/
  chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)
```

## How CI invokes it
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:

1. **No-cluster stage (every PR, fast):** `./run.sh --lint` and `./run.sh --coverage-only
   --strict-coverage --requirements <IDs the change touches>`. Validates every test parses
   + is tagged, regenerates coverage, **fails the PR** on tag/annotation drift (hygiene)
   and on any *touched* requirement lacking a test (merge gate). (L0 Kyverno CLI runs
   alongside — task T-0.5.)
2. **Cluster stage (L2/L3):** CI does **not** hold a kubeconfig. It **triggers the
   in-cluster Chainsaw Job** ([`incluster/`](incluster/README.md)), which runs `./run.sh`
   under the scoped `conformance-runner` ServiceAccount and writes JUnit + coverage to a
   results volume CI collects.
   - **Merge gate:** the Job runs `--requirements <IDs the change touches>`; that subset
     runs green **and** those requirements are covered to merge (R-TEST-03).
   - **Phase-exit gate:** the Job runs `--layer all --strict-coverage`; add `--require-all`
     only at the milestone where the full requirement set is expected covered. The phase
     suite must be green before the phase's exit criteria are met.

## Phase 0 status (no cluster yet)
No cluster exists, so L1/L2/L3 **cannot execute**. The behavior tests are **valid
scaffolds** (steps + assertions authored, tagged, `TODO(T-x.y)` where a later phase
delivers the fixture). In this phase `run.sh`:
- always generates the **coverage report** (cluster-independent), and
- emits a **skipped** JUnit listing every scaffold (so the gate is green and the artifact
  shape is real), and
- the **`meta/harness-selfcheck`** checks (reproducible output, JUnit+coverage emitted,
  everything tagged, zero safety-critical gaps) are what `run.sh --strict-coverage`
  actually enforces today.

As later phases deliver capabilities, each scaffold's `TODO`s are filled and the test flips
from *skipped* to *executing green* — landing coverage without re-architecting the harness.