jakman85andClaude Opus 5.5
18f7ba8d0f
Retire Kasm from the catalog. Drop ACME revocation from R-TLS-02. Make L3 fault injection operator-run and exclude it from the merge gate (R-TEST-03, ADR-0026). Update the gap register to match; Homepage authentication is now ruled must-enforce and stays a gap until the implementation lands. Co-Authored-By: Claude Opus 5.5 <[email protected]>
3.1 KiB
ADR-0010 — Conformance testing: automated, reproducible, behavior-based
- Status: Accepted — stands, except that L3 fault injection does not gate merges (ADR-0026)
- Date: 2026-08-15
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05, R-TEST-06
- Supersedes: —
Context
Acceptance must be objective and repeatable, not a human eyeballing a demo. Every requirement
(R-*) must be verified by an automated behavior test that any machine can run and reproduce, that
gates merges and phase exits, and that traces back to the requirement it proves. Safety-critical
behaviors (default-deny, VPN kill-switch, data survival, DR, auto-rollback) demand fault-injection,
not happy-path checks. Additionally, agents must not weaken the target: specs are immutable to
agents; tests encode spec intent.
Decision
Adopt an automated, reproducible, behavior-based conformance suite as the sole definition of
acceptance, structured in layers (L0 static/policy, L1 rendering, L2 behavioral, L3 fault-injection,
L4 aggregate suite), specified in ../docs/14-testing-and-conformance.md.
The suite is requirement-tagged, single-command, hermetic, deterministic, and emits
JUnit + a requirement-coverage report. It gates:
- merges — the subset for the touched requirements must pass;
- phase exits — the phase's full suite must pass.
Tooling: the approach is fixed; the concrete mix is policy-as-code for L0 (e.g. Kyverno / Conftest-OPA), a declarative Kubernetes test runner for L1/L2 (e.g. Chainsaw or kuttl), and a small programmatic harness (e.g. Ginkgo/Gomega or a scripted runner) for L3 fault-injection that needs orchestration. The final tool selection is the one remaining open item and may be pinned by a short follow-up (still human-approved), but no choice may relax the properties above.
Alternatives considered
- Manual demonstrations / checklists — rejected: not reproducible, not gating, not traceable (violates R-TEST-01..04).
- Unit tests of manifests only (L0/L1) — necessary but insufficient: a policy object existing does not prove enforcement; behavior (L2) and fault-injection (L3) are required (R-TEST-05).
- A single heavyweight framework for everything — rejected: L3 fault-injection needs orchestration a declarative runner can't express, while L0/L1/L2 are cleaner declaratively. A layered mix fits better.
- Letting agents adjust tests/requirements to pass — explicitly prohibited (R-TEST-06); a failing spec is an escalation, not a test edit.
Consequences
- "Done" becomes objective and machine-checkable; progress = green, requirement-tagged suite.
- Every requirement needs a test; uncovered requirements are visible and blocking (R-TEST-04).
- Establishing L3 fault-injection (kill-switch, node-kill, bad-update, DR) is real work and is first-class in the roadmap (WS-TEST), not an afterthought.
- CI must have cluster access (or spin an ephemeral dev/test cluster) to run L2/L3.
- The suite is versioned with the implementation and evolves only when the human-owned spec it encodes changes.