# ADR-0011 — Conformance test tooling: Kyverno + Chainsaw
- Status: Accepted
- Date: 2026-08-15
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05
- Refines: [ADR-0010](adr-0010-conformance-testing.md) (which fixed the *approach* and left the tool mix as the one open item)
- Supersedes: —

## Context
[ADR-0010](adr-0010-conformance-testing.md) fixed the conformance approach (automated, reproducible,
behavior-based, layered, requirement-tagged, gating) and left the concrete tool selection as the sole
open item. A choice is now made by the operator/architect.

## Decision
Use **two tools from the Kyverno ecosystem**, covering all test layers:

- **Kyverno** — the **L0 static/policy gate** (policy-as-code) **and in-cluster admission enforcer**.
  The Kyverno CLI (`kyverno apply` / `kyverno test`, incl. `kyverno-json` for arbitrary YAML/JSON
  like ServiceDefinitions) validates manifests and rendered resources in CI against platform
  invariants: every namespace carries a default-deny baseline, no plaintext secrets, required
  labels/egress-class set, ServiceDefinition schema conformance, exposure/egress sanity. **The same
  policies also run in-cluster as an admission controller**, rejecting non-conforming resources at
  reconcile time (defense in depth) — authored once, shared between CLI gate and admission.
- **Chainsaw** — the **L1 rendering, L2 behavioral, and L3 fault-injection** runner. Chainsaw's
  declarative test steps (apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded
  polling with explicit timeouts, per-test namespace isolation, and automatic cleanup directly
  satisfy the reproducibility contract (hermetic, deterministic, self-cleaning). Fault-injection
  (force the VPN tunnel down, cordon/kill a node, inject a known-bad image, trigger a restore) is
  expressed via Chainsaw `script` steps, with the fail-closed / no-data-loss outcome asserted by
  probe fixtures and `finally` steps restoring state.

Both emit structured, requirement-tagged results feeding the coverage report and the CI merge / phase
exit gates ([ADR-0010](adr-0010-conformance-testing.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)).
Tests declare their `R-*` IDs (e.g. via labels/annotations/naming) so coverage is generated
automatically.

## Alternatives considered
- **kuttl** (for L1/L2) — Chainsaw is its spiritual successor with a richer assertion model, native
  scripting (needed for L3), and try/catch/finally cleanup; chosen over kuttl.
- **Conftest/OPA (Rego)** for L0 — viable, but Kyverno keeps policy-as-code in the same
  YAML/ecosystem as Chainsaw and can double as an in-cluster enforcer; chosen over introducing Rego.
- **Ginkgo/Gomega (Go)** for L3 — more power but more code and a second language/toolchain; Chainsaw
  `script` steps cover the platform's fault-injection needs declaratively. Revisit via a superseding
  ADR only if a scenario genuinely exceeds Chainsaw's orchestration.
- **Testkube** as an orchestrator — unnecessary layer for this scale; CI + Chainsaw + Kyverno CLI
  suffice.

## Consequences
- A single, coherent ecosystem (Kyverno project) spans policy enforcement and behavior/fault tests —
  fewer tools, shared idioms.
- L3 fault-injection is expressed in Chainsaw `script` steps. **L2/L3 run as an in-cluster Chainsaw
  Job** on the dev/test cluster, **triggered by CI** — so CI holds no cluster credentials; the Job's
  own scoped ServiceAccount does. L0/L1 still run in CI without a cluster. (Resolves ADR-0010's open
  harness-location item.)
- The test Job's ServiceAccount is a **privileged, scoped test identity** (create/delete test
  namespaces + fixtures, patch NetworkPolicies, scale VPN sidecar/gateway, cordon/kill a node, inject
  a bad image, trigger a Velero restore). It must be least-privilege for that set, named, documented,
  never reused by workloads, and treated as its own audit item (R-SEC-02).
- Kyverno runs **both** as the CI policy gate **and** as an in-cluster admission enforcer (decided):
  non-conforming resources are rejected at reconcile time as well as in CI. This must not *replace*
  behavior tests — a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
  L2/L3 Chainsaw tests remain mandatory. Admission enforcement adds an operational failure mode
  (a bad policy can block legitimate reconciliation); policies ship with their own tests and roll out
  via GitOps.
- Reproducibility properties (hermetic, deterministic, self-cleaning, requirement-tagged, JUnit +
  coverage) remain the acceptance bar; the tools serve them, never relax them.
- Pin tool versions (Kyverno, Chainsaw) for deterministic runs; upgrades flow through the normal
  update path with the suite re-green as the gate.