jakman85andClaude Opus 4.8
9d8e03892f
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
69 lines
3.4 KiB
Markdown
69 lines
3.4 KiB
Markdown
# Conformance tests — layout & requirement-tagging convention
|
|
|
|
Every test is a **Chainsaw `Test`** in its own directory:
|
|
|
|
```
|
|
tests/
|
|
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
|
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
|
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
|
meta/ harness self-checks (R-TEST-*), cluster-independent
|
|
<test-name>/chainsaw-test.yaml
|
|
```
|
|
|
|
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
|
coverage generator cross-checks the two.
|
|
|
|
## Requirement-tagging convention (authoritative)
|
|
|
|
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
|
of truth, the annotations must agree with it (the coverage generator fails on drift):
|
|
|
|
```yaml
|
|
metadata:
|
|
name: default-deny
|
|
labels:
|
|
# ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
|
|
conformance.home.k3s/R-NET-01: "true"
|
|
annotations:
|
|
# Human-readable canonical list — MUST equal the label set.
|
|
conformance.home.k3s/requirements: "R-NET-01"
|
|
# Declared layer (L1|L2|L3|META) — must match the subtree.
|
|
conformance.home.k3s/layer: "L2"
|
|
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
|
conformance.home.k3s/safety-critical: "false"
|
|
# Given/When/Then intent, for the coverage report and reviewers.
|
|
conformance.home.k3s/intent: "Given ... When ... Then ..."
|
|
```
|
|
|
|
Why a label **per requirement** (not a comma list): Kubernetes label *values* can't hold
|
|
a comma-separated list, and `chainsaw test --selector` selects on label **keys/values**.
|
|
One `conformance.home.k3s/<R-ID>=true` label per requirement makes both subset-selection
|
|
and coverage extraction exact and robust.
|
|
|
|
### How this drives the tooling
|
|
- **Subset runs (merge gate):** `run.sh --requirements R-NET-01,R-NET-06` resolves the
|
|
R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once.
|
|
- **Coverage (`lib/coverage.py`):** reads these labels from every `chainsaw-test.yaml`,
|
|
maps each registry requirement -> its verifying test(s), and flags gaps + any
|
|
label/annotation mismatch + tags not in the registry.
|
|
- **JUnit:** results (real or skipped-scaffold) carry the R-IDs in `classname` +
|
|
properties, so the JUnit itself stays traceable (R-TEST-04).
|
|
|
|
## Authoring rules (R-TEST-01/02/06)
|
|
- **Behavior, not existence.** Assert what the system *does* (a probe cannot connect),
|
|
never that an object merely exists (a policy can exist and not enforce).
|
|
- **Given/When/Then** structure with `given-*/when-*/then-*` step names.
|
|
- **Hermetic & self-cleaning.** Rely on Chainsaw's per-test namespace + auto-teardown;
|
|
put any out-of-namespace fixture cleanup in a `finally` block. A leaked fixture is a
|
|
failure.
|
|
- **Bounded, deterministic waits.** Use assert/error polling within the config timeouts;
|
|
never `sleep`-and-hope.
|
|
- **Safety-critical => fault-injection.** For a safety-critical requirement, actually
|
|
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
|
|
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
|
|
the PM — do not relax the assertion or the requirement.
|
|
|
|
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
|
later roadmap phase delivers the capability under test.
|