home-network-k3s/decisions/adr-0011-test-tooling-kyverno-chainsaw.md

ADR-0011 — Conformance test tooling: Kyverno + Chainsaw

• Status: Accepted
• Date: 2026-08-15
• Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05
• Refines: ADR-0010 (which fixed the approach and left the tool mix as the one open item)
• Supersedes: —

Context

ADR-0010 fixed the conformance approach (automated, reproducible, behavior-based, layered, requirement-tagged, gating) and left the concrete tool selection as the sole open item. A choice is now made by the operator/architect.

Decision

Use two tools from the Kyverno ecosystem, covering all test layers:

• Kyverno — the L0 static/policy gate (policy-as-code) and in-cluster admission enforcer. The Kyverno CLI (kyverno apply / kyverno test, incl. kyverno-json for arbitrary YAML/JSON like ServiceDefinitions) validates manifests and rendered resources in CI against platform invariants: every namespace carries a default-deny baseline, no plaintext secrets, required labels/egress-class set, ServiceDefinition schema conformance, exposure/egress sanity. The same policies also run in-cluster as an admission controller, rejecting non-conforming resources at reconcile time (defense in depth) — authored once, shared between CLI gate and admission.
• Chainsaw — the L1 rendering, L2 behavioral, and L3 fault-injection runner. Chainsaw's declarative test steps (apply/assert/error, script/command, try/catch/finally), bounded polling with explicit timeouts, per-test namespace isolation, and automatic cleanup directly satisfy the reproducibility contract (hermetic, deterministic, self-cleaning). Fault-injection (force the VPN tunnel down, cordon/kill a node, inject a known-bad image, trigger a restore) is expressed via Chainsaw script steps, with the fail-closed / no-data-loss outcome asserted by probe fixtures and finally steps restoring state.

Both emit structured, requirement-tagged results feeding the coverage report and the CI merge / phase exit gates (ADR-0010, ../docs/14-testing-and-conformance.md). Tests declare their R-* IDs (e.g. via labels/annotations/naming) so coverage is generated automatically.

Alternatives considered

• kuttl (for L1/L2) — Chainsaw is its spiritual successor with a richer assertion model, native scripting (needed for L3), and try/catch/finally cleanup; chosen over kuttl.
• Conftest/OPA (Rego) for L0 — viable, but Kyverno keeps policy-as-code in the same YAML/ecosystem as Chainsaw and can double as an in-cluster enforcer; chosen over introducing Rego.
• Ginkgo/Gomega (Go) for L3 — more power but more code and a second language/toolchain; Chainsaw script steps cover the platform's fault-injection needs declaratively. Revisit via a superseding ADR only if a scenario genuinely exceeds Chainsaw's orchestration.
• Testkube as an orchestrator — unnecessary layer for this scale; CI + Chainsaw + Kyverno CLI suffice.

Consequences

• A single, coherent ecosystem (Kyverno project) spans policy enforcement and behavior/fault tests — fewer tools, shared idioms.
• L3 fault-injection is expressed in Chainsaw script steps. L2/L3 run as an in-cluster Chainsaw Job on the dev/test cluster, triggered by CI — so CI holds no cluster credentials; the Job's own scoped ServiceAccount does. L0/L1 still run in CI without a cluster. (Resolves ADR-0010's open harness-location item.)
• The test Job's ServiceAccount is a privileged, scoped test identity (create/delete test namespaces + fixtures, patch NetworkPolicies, scale VPN sidecar/gateway, cordon/kill a node, inject a bad image, trigger a Velero restore). It must be least-privilege for that set, named, documented, never reused by workloads, and treated as its own audit item (R-SEC-02).
• Kyverno runs both as the CI policy gate and as an in-cluster admission enforcer (decided): non-conforming resources are rejected at reconcile time as well as in CI. This must not replace behavior tests — a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so L2/L3 Chainsaw tests remain mandatory. Admission enforcement adds an operational failure mode (a bad policy can block legitimate reconciliation); policies ship with their own tests and roll out via GitOps.
• Reproducibility properties (hermetic, deterministic, self-cleaning, requirement-tagged, JUnit + coverage) remain the acceptance bar; the tools serve them, never relax them.
• Pin tool versions (Kyverno, Chainsaw) for deterministic runs; upgrades flow through the normal update path with the suite re-green as the gate.