home-network-k3s-impl/conformance/tests/README.md

# Conformance tests — layout & requirement-tagging convention

Every test is a **Chainsaw `Test`** in its own directory:

```
tests/
  l0-static/           L0 — whole-repo invariants over the tracked tree (no cluster)
  l1-rendering/        L1 — ServiceDefinition -> rendered resources match golden
  l2-behavioral/       L2 — real runtime behavior (probes against a cluster)
  l3-fault-injection/  L3 — break something; assert fail-closed / no-data-loss
  meta/                harness self-checks (R-TEST-*), cluster-independent
    <test-name>/chainsaw-test.yaml
```

The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
coverage generator cross-checks the two.

### L0 static (`l0-static/`) — what belongs here, and what does not

docs/14's L0 row is *"Pre-apply, in CI · manifests/ServiceDefinitions obey invariants"*.
Kyverno (`platform/kyverno/`) covers the invariants expressible over a **single
resource**. This subtree covers the L0 invariants that are **properties of the repository
as a whole** and that no per-resource policy can state — e.g. *no plaintext secret exists
anywhere in the tracked tree*, *no tracked RoleBinding grants a workload identity a
privileged role*, *the spec-immutability gate actually blocks a modified spec*.

Each `l0-static/<name>/` dir carries a **`check.sh`**: a self-contained offline assertion
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:

- by `conformance/run.sh --layer l0` (and as the offline leg of a full run), via
  `lib/run_l1_offline.py`, producing a REAL pass/fail JUnit entry — never a skip; and
- by the merge gate's **`ci/l0.sh`** stage, so these assertions run on every PR; and
- by the wrapping Chainsaw `Test`, which runs the SAME `check.sh`, so an in-cluster run
  executes byte-identical assertions.

Two authoring rules specific to this layer:

1. **State the gap.** A statically-provable half of a requirement is still only half.
   Every `check.sh` header ends with an explicit *"what this does NOT prove"* list naming
   the cluster-layer remainder. Covering a requirement at L0 does not finish it.
2. **Exemptions must be self-policing.** Where a check must tolerate something (a
   deliberately-plaintext policy fixture, an `.example` template), the exemption is
   conditioned on a property that would break if the exemption were abused — e.g. the
   plaintext Secret fixture is exempt *only while* a `kyverno-test.yaml` asserts the
   policy rejects it. Never a bare path allowlist.

## Requirement-tagging convention (authoritative)

A test declares the requirement(s) it verifies **three** ways; the label is the source
of truth, the annotations must agree with it (the coverage generator fails on drift):

```yaml
metadata:
  name: default-deny
  labels:
    # ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
    conformance.home.k3s/R-NET-01: "true"
  annotations:
    # Human-readable canonical list — MUST equal the label set.
    conformance.home.k3s/requirements: "R-NET-01"
    # Declared layer (L0|L1|L2|L3|META) — must match the subtree.
    conformance.home.k3s/layer: "L2"
    # true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
    conformance.home.k3s/safety-critical: "false"
    # Given/When/Then intent, for the coverage report and reviewers.
    conformance.home.k3s/intent: "Given ... When ... Then ..."
```

Why a label **per requirement** (not a comma list): Kubernetes label *values* can't hold
a comma-separated list, and `chainsaw test --selector` selects on label **keys/values**.
One `conformance.home.k3s/<R-ID>=true` label per requirement makes both subset-selection
and coverage extraction exact and robust.

### How this drives the tooling
- **Subset runs (merge gate):** `run.sh --requirements R-NET-01,R-NET-06` resolves the
  R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once.
- **Coverage (`lib/coverage.py`):** reads these labels from every `chainsaw-test.yaml`,
  maps each registry requirement -> its verifying test(s), and flags gaps + any
  label/annotation mismatch + tags not in the registry.
- **JUnit:** results (real or skipped-scaffold) carry the R-IDs in `classname` +
  properties, so the JUnit itself stays traceable (R-TEST-04).

## Authoring rules (R-TEST-01/02/06)
- **Behavior, not existence.** Assert what the system *does* (a probe cannot connect),
  never that an object merely exists (a policy can exist and not enforce).
- **Given/When/Then** structure with `given-*/when-*/then-*` step names.
- **Hermetic & self-cleaning.** Rely on Chainsaw's per-test namespace + auto-teardown;
  put any out-of-namespace fixture cleanup in a `finally` block. A leaked fixture is a
  failure.
- **Bounded, deterministic waits.** Use assert/error polling within the config timeouts;
  never `sleep`-and-hope.
- **Safety-critical => fault-injection.** For a safety-critical requirement, actually
  break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
  the PM — do not relax the assertion or the requirement.
- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
  Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
  so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
  binding over the same ConfigMap Flux substitutes from:

  ```yaml
  spec:
    bindings:
      - name: baseDomain
        value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
  ```

  Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
  `($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
  full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
  which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
  over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
  the Service was assigned) — it is both stricter and environment-independent. Never take the
  environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
  ends up silently asserting the other cluster's expectations. L1 goldens are the one
  exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.

### How a scaffold declares itself (and how `status.py` tells one from a verifier)

Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test. `lib/status.py` classifies each
test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`,
`render-check.sh`, `check.sh` — and either signal below **outranks**
`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as
an executable test. Declare a scaffold in **both** positions:

1. **`# STATUS: scaffold — <what is missing>`** in the file's header comment block. The
   declared state must be the first token after `STATUS:` (executable tests say
   `# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`).
2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first
   non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`,
   or an inline comment on a missing fixture:
   `file: fixtures/smtp-sink.yaml   # TODO(T-4.1): hermetic SMTP capture sink`.

What is **not** a scaffold marker, and why the distinction is load-bearing: a marker
mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does
NOT prove" list), and a marker that is the **entire content of a quoted literal** — a
placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in
`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each
idiom a **standalone quoted literal in a list**; a marker with text attached (a regex
alternation, several idioms in one string) still classifies as a scaffold, because
ambiguity here only ever downgrades.