jakman85andClaude Opus 4.8
e8c555cfeb
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
49 lines
3.0 KiB
Markdown
49 lines
3.0 KiB
Markdown
# ADR-0010 — Conformance testing: automated, reproducible, behavior-based
|
|
- Status: Accepted
|
|
- Date: 2026-08-15
|
|
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05, R-TEST-06
|
|
- Supersedes: —
|
|
|
|
## Context
|
|
Acceptance must be objective and repeatable, not a human eyeballing a demo. Every requirement
|
|
(`R-*`) must be verified by an automated behavior test that any machine can run and reproduce, that
|
|
gates merges and phase exits, and that traces back to the requirement it proves. Safety-critical
|
|
behaviors (default-deny, VPN kill-switch, data survival, DR, auto-rollback) demand fault-injection,
|
|
not happy-path checks. Additionally, agents must not weaken the target: specs are immutable to
|
|
agents; tests encode spec intent.
|
|
|
|
## Decision
|
|
Adopt an **automated, reproducible, behavior-based conformance suite** as the sole definition of
|
|
acceptance, structured in layers (L0 static/policy, L1 rendering, L2 behavioral, L3 fault-injection,
|
|
L4 aggregate suite), specified in [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md).
|
|
The suite is **requirement-tagged**, **single-command**, **hermetic**, **deterministic**, and emits
|
|
**JUnit + a requirement-coverage report**. It gates:
|
|
- **merges** — the subset for the touched requirements must pass;
|
|
- **phase exits** — the phase's full suite must pass.
|
|
|
|
Tooling: the *approach* is fixed; the concrete mix is **policy-as-code** for L0 (e.g. Kyverno /
|
|
Conftest-OPA), a **declarative Kubernetes test runner** for L1/L2 (e.g. Chainsaw or kuttl), and a
|
|
**small programmatic harness** (e.g. Ginkgo/Gomega or a scripted runner) for L3 fault-injection that
|
|
needs orchestration. The final tool selection is the one remaining open item and may be pinned by a
|
|
short follow-up (still human-approved), but no choice may relax the properties above.
|
|
|
|
## Alternatives considered
|
|
- **Manual demonstrations / checklists** — rejected: not reproducible, not gating, not traceable
|
|
(violates R-TEST-01..04).
|
|
- **Unit tests of manifests only (L0/L1)** — necessary but insufficient: a policy object existing
|
|
does not prove enforcement; behavior (L2) and fault-injection (L3) are required (R-TEST-05).
|
|
- **A single heavyweight framework for everything** — rejected: L3 fault-injection needs
|
|
orchestration a declarative runner can't express, while L0/L1/L2 are cleaner declaratively. A
|
|
layered mix fits better.
|
|
- **Letting agents adjust tests/requirements to pass** — explicitly prohibited (R-TEST-06); a
|
|
failing spec is an escalation, not a test edit.
|
|
|
|
## Consequences
|
|
- "Done" becomes objective and machine-checkable; progress = green, requirement-tagged suite.
|
|
- Every requirement needs a test; uncovered requirements are visible and blocking (R-TEST-04).
|
|
- Establishing L3 fault-injection (kill-switch, node-kill, bad-update, DR) is real work and is
|
|
first-class in the roadmap (WS-TEST), not an afterthought.
|
|
- CI must have cluster access (or spin an ephemeral dev/test cluster) to run L2/L3.
|
|
- The suite is versioned with the implementation and evolves only when the human-owned spec it
|
|
encodes changes.
|