jakman85andClaude Opus 4.8
4c21f9f6e3
chainsaw v0.2.12 has no test-level `spec.finally` (finally exists only on a
step), and a step-level `finally` accepts only catch-style ops (script/command/
delete/wait), NOT `apply`/`assert`. Both tests failed to parse ("unknown field").
- egress-class: drop the redundant test-level `spec.finally` delete-pods block —
chainsaw auto-deletes applied fixtures on cleanup (skipDelete: false). The
valid step-level `finally` that restores the egress-class label is kept.
- vpn-killswitch: move the SAFETY-CRITICAL self-healing tunnel restart out of the
invalid `spec.finally` and into the STEP-level `finally` of the fault step
`when-force-tunnel-down-then-no-leak`, implemented as a `script` (finally can't
use apply/assert) that `kubectl apply`s the privileged node-pinned restart Job
(name `tunnel-restart`, distinct from `tunnel-stop`) and `kubectl wait`s for it
to complete — the equivalent of `assert succeeded:1`, blocking teardown until
the host confirms the tunnel is up. This finally runs even if the no-leak /
internet-unaffected asserts in the same step's `try` fail, so no path leaves the
gateway tunnel down. The recovery step is now poll-only (restart already
happened in the prior step's finally).
Validation with the pinned binary (~/.local/bin/chainsaw v0.2.12):
`chainsaw lint test --file ...` => "The document is valid" for both; `chainsaw
test ... --no-cluster` loads both past the parse stage (no "unknown field"), then
fails only at runtime for lack of a cluster (expected). yamllint + run.sh
--strict-coverage clean (RC=0). No live apply / fault injection, no spec edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Conformance tests — layout & requirement-tagging convention
Every test is a Chainsaw Test in its own directory:
tests/
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
meta/ harness self-checks (R-TEST-*), cluster-independent
<test-name>/chainsaw-test.yaml
The layer a test belongs to is its subtree and its .../layer annotation; the
coverage generator cross-checks the two.
Requirement-tagging convention (authoritative)
A test declares the requirement(s) it verifies three ways; the label is the source of truth, the annotations must agree with it (the coverage generator fails on drift):
metadata:
name: default-deny
labels:
# ONE label per verified R-* ID. Machine-selectable + coverage source of truth.
conformance.home.k3s/R-NET-01: "true"
annotations:
# Human-readable canonical list — MUST equal the label set.
conformance.home.k3s/requirements: "R-NET-01"
# Declared layer (L1|L2|L3|META) — must match the subtree.
conformance.home.k3s/layer: "L2"
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
conformance.home.k3s/safety-critical: "false"
# Given/When/Then intent, for the coverage report and reviewers.
conformance.home.k3s/intent: "Given ... When ... Then ..."
Why a label per requirement (not a comma list): Kubernetes label values can't hold
a comma-separated list, and chainsaw test --selector selects on label keys/values.
One conformance.home.k3s/<R-ID>=true label per requirement makes both subset-selection
and coverage extraction exact and robust.
How this drives the tooling
- Subset runs (merge gate):
run.sh --requirements R-NET-01,R-NET-06resolves the R-IDs to the set of test dirs carrying those labels (union, deduped) and runs each once. - Coverage (
lib/coverage.py): reads these labels from everychainsaw-test.yaml, maps each registry requirement -> its verifying test(s), and flags gaps + any label/annotation mismatch + tags not in the registry. - JUnit: results (real or skipped-scaffold) carry the R-IDs in
classname+ properties, so the JUnit itself stays traceable (R-TEST-04).
Authoring rules (R-TEST-01/02/06)
- Behavior, not existence. Assert what the system does (a probe cannot connect), never that an object merely exists (a policy can exist and not enforce).
- Given/When/Then structure with
given-*/when-*/then-*step names. - Hermetic & self-cleaning. Rely on Chainsaw's per-test namespace + auto-teardown;
put any out-of-namespace fixture cleanup in a
finallyblock. A leaked fixture is a failure. - Bounded, deterministic waits. Use assert/error polling within the config timeouts;
never
sleep-and-hope. - Safety-critical => fault-injection. For a safety-critical requirement, actually
break the thing and assert fail-closed / no-data-loss, with
finallyrestoring state. - Never weaken a test to go green (R-TEST-06). If the spec can't be met, escalate to the PM — do not relax the assertion or the requirement.
Scaffolds in this tree carry TODO(T-x.y) markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test.