home-network-k3s-impl/conformance/tests

Conformance tests — layout & requirement-tagging convention

Every test is a Chainsaw Test in its own directory:

tests/
  l0-static/           L0 — whole-repo invariants over the tracked tree (no cluster)
  l1-rendering/        L1 — ServiceDefinition -> rendered resources match golden
  l2-behavioral/       L2 — real runtime behavior (probes against a cluster)
  l3-fault-injection/  L3 — break something; assert fail-closed / no-data-loss
  meta/                harness self-checks (R-TEST-*), cluster-independent
    <test-name>/chainsaw-test.yaml

The layer a test belongs to is its subtree and its .../layer annotation; the coverage generator cross-checks the two.

L0 static (l0-static/) — what belongs here, and what does not

docs/14's L0 row is "Pre-apply, in CI · manifests/ServiceDefinitions obey invariants". Kyverno (platform/kyverno/) covers the invariants expressible over a single resource. This subtree covers the L0 invariants that are properties of the repository as a whole and that no per-resource policy can state — e.g. no plaintext secret exists anywhere in the tracked tree, no tracked RoleBinding grants a workload identity a privileged role, the spec-immutability gate actually blocks a modified spec.

Each l0-static/<name>/ dir carries a check.sh: a self-contained offline assertion script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:

• by conformance/run.sh --layer l0 (and as the offline leg of a full run), via lib/run_l1_offline.py, producing a REAL pass/fail JUnit entry — never a skip; and
• by the merge gate's ci/l0.sh stage, so these assertions run on every PR; and
• by the wrapping Chainsaw Test, which runs the SAME check.sh, so an in-cluster run executes byte-identical assertions.

Two authoring rules specific to this layer:

1. State the gap. A statically-provable half of a requirement is still only half. Every check.sh header ends with an explicit "what this does NOT prove" list naming the cluster-layer remainder. Covering a requirement at L0 does not finish it.
2. Exemptions must be self-policing. Where a check must tolerate something (a deliberately-plaintext policy fixture, an .example template), the exemption is conditioned on a property that would break if the exemption were abused — e.g. the plaintext Secret fixture is exempt only while a kyverno-test.yaml asserts the policy rejects it. Never a bare path allowlist.

Requirement-tagging convention (authoritative)

A test declares the requirement(s) it verifies three ways; the label is the source of truth, the annotations must agree with it (the coverage generator fails on drift):

metadata:
  name: default-deny
  labels:
    # ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
    conformance.home.k3s/R-NET-01: "true"
  annotations:
    # Human-readable canonical list — MUST equal the label set.
    conformance.home.k3s/requirements: "R-NET-01"
    # Declared layer (L0|L1|L2|L3|META) — must match the subtree.
    conformance.home.k3s/layer: "L2"
    # true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
    conformance.home.k3s/safety-critical: "false"
    # Given/When/Then intent, for the coverage report and reviewers.
    conformance.home.k3s/intent: "Given ... When ... Then ..."

Why a label per requirement (not a comma list): Kubernetes label values can't hold a comma-separated list, and chainsaw test --selector selects on label keys/values. One conformance.home.k3s/<R-ID>=true label per requirement makes both subset-selection and coverage extraction exact and robust.

How this drives the tooling

• Subset runs (merge gate): run.sh --requirements R-NET-01,R-NET-06 resolves the R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once.
• Coverage (lib/coverage.py): reads these labels from every chainsaw-test.yaml, maps each registry requirement -> its verifying test(s), and flags gaps + any label/annotation mismatch + tags not in the registry.
• JUnit: results (real or skipped-scaffold) carry the R-IDs in classname + properties, so the JUnit itself stays traceable (R-TEST-04).

Authoring rules (R-TEST-01/02/06)

• Behavior, not existence. Assert what the system does (a probe cannot connect), never that an object merely exists (a policy can exist and not enforce).

• Given/When/Then structure with given-*/when-*/then-* step names.

• Hermetic & self-cleaning. Rely on Chainsaw's per-test namespace + auto-teardown; put any out-of-namespace fixture cleanup in a finally block. A leaked fixture is a failure.

• Bounded, deterministic waits. Use assert/error polling within the config timeouts; never sleep-and-hope.

• Safety-critical => fault-injection. For a safety-critical requirement, actually break the thing and assert fail-closed / no-data-loss, with finally restoring state.

• Never weaken a test to go green (R-TEST-06). If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement.

• Never write a per-environment value into a test (R-ENV-02). No hostname suffix, no Gateway VIP, no LB-pool range. One authored suite must run green against dev and prod, so a test asks the cluster it is pointed at what it is. The mechanism is a chainsaw binding over the same ConfigMap Flux substitutes from:

  spec:
    bindings:
      - name: baseDomain
        value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
  

  Declare it at test level (step outputs/bindings are step-scoped), then consume it: ($baseDomain) for a pod env: value, (join('.', ['myhost', $baseDomain])) where a full hostname is needed — including inside a fixtures/*.yaml applied with apply: file:, which chainsaw templates with the same bindings. Prefer reading the actual observed value over restating a range (e.g. assert the DNS answer equals GATEWAY_VIP, or equals the LB IP the Service was assigned) — it is both stricter and environment-independent. Never take the environment from a CLI flag or an exported variable: a flag someone forgets is how a suite ends up silently asserting the other cluster's expectations. L1 goldens are the one exception: they are rendering fixtures and carry the literal ${BASE_DOMAIN} token.

How a scaffold declares itself (and how status.py tells one from a verifier)

Scaffolds in this tree carry TODO(T-x.y) markers where a fixture/assertion lands as a later roadmap phase delivers the capability under test. lib/status.py classifies each test EXECUTABLE or SCAFFOLD from the test's own files — chainsaw-test.yaml, render-check.sh, check.sh — and either signal below outranks conformance.home.k3s/executable: "true", so a placeholder body can never be reported as an executable test. Declare a scaffold in both positions:

1. # STATUS: scaffold — <what is missing> in the file's header comment block. The declared state must be the first token after STATUS: (executable tests say # STATUS: EXECUTABLE / IMPLEMENTED / RUNNABLE).
2. A TODO/FIXME marker carrying a message, in the body — the file from its first non-comment line onward. echo "TODO(T-4.2): inject a fault to fire a rule", or an inline comment on a missing fixture: file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink.

What is not a scaffold marker, and why the distinction is load-bearing: a marker mentioned in the header prose (the Given/When/Then and the mandatory "what this does NOT prove" list), and a marker that is the entire content of a quoted literal — a placeholder detector's own vocabulary, e.g. SEED = ["TBD", "TODO", "FIXME", ...] in l0-static/placeholder-guard-coupling/check.sh. A whole-file "TODO" in text test used to classify that 32-assertion check as a scaffold and, through it, report the safety-critical R-STATE-03 as exec=1 scaffold=1. If your check must hold a marker as data, keep each idiom a standalone quoted literal in a list; a marker with text attached (a regex alternation, several idioms in one string) still classifies as a scaffold, because ambiguity here only ever downgrades.