jakman85andClaude Opus 4.8
e8c555cfeb
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
68 lines
4.7 KiB
Markdown
68 lines
4.7 KiB
Markdown
# ADR-0011 — Conformance test tooling: Kyverno + Chainsaw
|
|
- Status: Accepted
|
|
- Date: 2026-08-15
|
|
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05
|
|
- Refines: [ADR-0010](adr-0010-conformance-testing.md) (which fixed the *approach* and left the tool mix as the one open item)
|
|
- Supersedes: —
|
|
|
|
## Context
|
|
[ADR-0010](adr-0010-conformance-testing.md) fixed the conformance approach (automated, reproducible,
|
|
behavior-based, layered, requirement-tagged, gating) and left the concrete tool selection as the sole
|
|
open item. A choice is now made by the operator/architect.
|
|
|
|
## Decision
|
|
Use **two tools from the Kyverno ecosystem**, covering all test layers:
|
|
|
|
- **Kyverno** — the **L0 static/policy gate** (policy-as-code) **and in-cluster admission enforcer**.
|
|
The Kyverno CLI (`kyverno apply` / `kyverno test`, incl. `kyverno-json` for arbitrary YAML/JSON
|
|
like ServiceDefinitions) validates manifests and rendered resources in CI against platform
|
|
invariants: every namespace carries a default-deny baseline, no plaintext secrets, required
|
|
labels/egress-class set, ServiceDefinition schema conformance, exposure/egress sanity. **The same
|
|
policies also run in-cluster as an admission controller**, rejecting non-conforming resources at
|
|
reconcile time (defense in depth) — authored once, shared between CLI gate and admission.
|
|
- **Chainsaw** — the **L1 rendering, L2 behavioral, and L3 fault-injection** runner. Chainsaw's
|
|
declarative test steps (apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded
|
|
polling with explicit timeouts, per-test namespace isolation, and automatic cleanup directly
|
|
satisfy the reproducibility contract (hermetic, deterministic, self-cleaning). Fault-injection
|
|
(force the VPN tunnel down, cordon/kill a node, inject a known-bad image, trigger a restore) is
|
|
expressed via Chainsaw `script` steps, with the fail-closed / no-data-loss outcome asserted by
|
|
probe fixtures and `finally` steps restoring state.
|
|
|
|
Both emit structured, requirement-tagged results feeding the coverage report and the CI merge / phase
|
|
exit gates ([ADR-0010](adr-0010-conformance-testing.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)).
|
|
Tests declare their `R-*` IDs (e.g. via labels/annotations/naming) so coverage is generated
|
|
automatically.
|
|
|
|
## Alternatives considered
|
|
- **kuttl** (for L1/L2) — Chainsaw is its spiritual successor with a richer assertion model, native
|
|
scripting (needed for L3), and try/catch/finally cleanup; chosen over kuttl.
|
|
- **Conftest/OPA (Rego)** for L0 — viable, but Kyverno keeps policy-as-code in the same
|
|
YAML/ecosystem as Chainsaw and can double as an in-cluster enforcer; chosen over introducing Rego.
|
|
- **Ginkgo/Gomega (Go)** for L3 — more power but more code and a second language/toolchain; Chainsaw
|
|
`script` steps cover the platform's fault-injection needs declaratively. Revisit via a superseding
|
|
ADR only if a scenario genuinely exceeds Chainsaw's orchestration.
|
|
- **Testkube** as an orchestrator — unnecessary layer for this scale; CI + Chainsaw + Kyverno CLI
|
|
suffice.
|
|
|
|
## Consequences
|
|
- A single, coherent ecosystem (Kyverno project) spans policy enforcement and behavior/fault tests —
|
|
fewer tools, shared idioms.
|
|
- L3 fault-injection is expressed in Chainsaw `script` steps. **L2/L3 run as an in-cluster Chainsaw
|
|
Job** on the dev/test cluster, **triggered by CI** — so CI holds no cluster credentials; the Job's
|
|
own scoped ServiceAccount does. L0/L1 still run in CI without a cluster. (Resolves ADR-0010's open
|
|
harness-location item.)
|
|
- The test Job's ServiceAccount is a **privileged, scoped test identity** (create/delete test
|
|
namespaces + fixtures, patch NetworkPolicies, scale VPN sidecar/gateway, cordon/kill a node, inject
|
|
a bad image, trigger a Velero restore). It must be least-privilege for that set, named, documented,
|
|
never reused by workloads, and treated as its own audit item (R-SEC-02).
|
|
- Kyverno runs **both** as the CI policy gate **and** as an in-cluster admission enforcer (decided):
|
|
non-conforming resources are rejected at reconcile time as well as in CI. This must not *replace*
|
|
behavior tests — a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
|
|
L2/L3 Chainsaw tests remain mandatory. Admission enforcement adds an operational failure mode
|
|
(a bad policy can block legitimate reconciliation); policies ship with their own tests and roll out
|
|
via GitOps.
|
|
- Reproducibility properties (hermetic, deterministic, self-cleaning, requirement-tagged, JUnit +
|
|
coverage) remain the acceptance bar; the tools serve them, never relax them.
|
|
- Pin tool versions (Kyverno, Chainsaw) for deterministic runs; upgrades flow through the normal
|
|
update path with the suite re-green as the gate.
|