Initial baseline: home platform (k3s + Longhorn) specification repository

Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision &
requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap
(phases + workstreams), and AI-agent coordination (roles, protocol,
definition-of-done, task board).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

docs/14-testing-and-conformance.md

@@ -0,0 +1,168 @@
+# 14 — Testing & Conformance
+
+Serves: **R-TEST-01..06**, and is the verification mechanism for **every other requirement**.
+Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
+
+> **This spec changes what "done" means.** Nothing is accepted on a manual demo, a screenshot, or
+> "it reconciled." Every requirement is proven by an **automated, reproducible, behavior-based test**
+> that a machine can run on demand and that is tagged with the requirement ID(s) it verifies.
+
+## Principles
+
+1. **Behavior-based (R-TEST-01).** Tests assert *observable system behavior* — Given a state, When
+   an action, Then an outcome — not implementation internals. "A pod with no allow rules cannot reach
+   Plex" is a behavior; "a CiliumNetworkPolicy object exists" is not (it may exist and still not
+   enforce). Test the behavior.
+2. **Reproducible (R-TEST-02).** Deterministic, hermetic, version-pinned, single-command, and
+   environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness
+   budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope.
+3. **Conformance gates delivery (R-TEST-03).** The suite blocks merges (for touched requirements)
+   and blocks phase exits (full phase suite). Green suite is the definition of progress.
+4. **Traceable (R-TEST-04).** Each test declares its `R-*` IDs; a generated coverage report maps
+   every requirement to its verifying test(s). Uncovered requirements are visible gaps.
+5. **Fault-injection for safety-critical behavior (R-TEST-05).** Default-deny, VPN kill-switch,
+   data-survival, DR restore, and auto-rollback are proven by *breaking things* and asserting
+   fail-closed / no-data-loss — not by happy-path checks.
+6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a
+   test or a requirement to get green; a genuine inability to meet the spec is an escalation to the
+   human, not a spec/test edit.
+
+## Reproducibility contract (what "reproducible" concretely requires)
+
+- **Single command** runs the suite (or a tagged subset) locally and in CI.
+- **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down;
+  it never depends on pre-existing mutable state or on another test's side effects.
+- **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit
+  timeouts for eventual consistency; no reliance on wall-clock timing or ordering.
+- **Isolated & parallel-safe**: unique namespaces/labels; tests can run concurrently without
+  cross-talk.
+- **Machine-readable output**: JUnit (or equivalent) plus a requirement-coverage report; every
+  result carries the `R-*` tags it verifies.
+- **Self-cleaning**: on pass or fail, fixtures are removed so a re-run starts clean (leaked fixtures
+  are themselves a test failure).
+- **Versioned with the code**: tests live in the implementation repo next to what they verify and
+  evolve only when the human-owned spec they encode changes.
+
+## Test layers
+
+| Layer | Runs | Verifies | Example |
+|---|---|---|---|
+| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
+| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
+| **L2 Behavioral/integration** | Against dev/test cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
+| **L3 Fault-injection** | Against dev/test cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
+| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
+
+## Behavior test catalog (normative — implementers author these as automated tests)
+
+Each maps to requirement IDs; phrased as Given/When/Then intent.
+
+- **Default-deny (R-NET-01):** *Given* a fresh workload with no allow rules, *When* it attempts any
+  in-cluster connection except DNS, *Then* the connection fails; DNS succeeds.
+- **Explicit allow / deny-undeclared (R-NET-02):** *Given* declared `dependencies`, *When* the
+  source reaches a declared dest port, *Then* it succeeds; *When* it reaches an **undeclared** dest,
+  *Then* it fails.
+- **Egress class (R-NET-05):** *Given* `egress: internet` vs `egress: vpn` workloads, *When* each
+  queries an external echo of its public IP, *Then* the internet one shows the home/WAN IP and the
+  vpn one shows the VPN IP.
+- **VPN kill-switch (R-NET-06, L3):** *Given* a `vpn` workload with a healthy tunnel, *When* the
+  tunnel is forced down, *Then* the workload has **zero** egress to the plain internet and
+  `internet`-class workloads are unaffected; *When* the tunnel recovers, connectivity returns.
+- **Egress flip (R-NET-07):** *Given* a running service, *When* its `egress` class is changed in
+  git and reconciled, *Then* its observed egress path changes with no workload rewrite.
+- **Data survival (R-STATE-01, L3):** *Given* a stateful service with written data, *When* the
+  workload is destroyed and re-applied from git, *Then* the data is intact.
+- **Node loss (R-STATE-02, L3):** *Given* a 3-replica volume, *When* a node is removed, *Then* the
+  volume stays available and the cluster reconverges.
+- **DR restore (R-STATE-03, L3):** *Given* an off-cluster backup, *When* a namespace + volumes are
+  restored into a clean state, *Then* the service returns with its data.
+- **Trusted TLS (R-TLS-01/02):** *Given* any exposed endpoint (incl. `internal`), *When* its cert is
+  inspected, *Then* it is valid, non-self-signed, unexpired; renewal occurs before expiry with no
+  manual step.
+- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
+  internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
+  reachable + TLS-valid.
+- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
+  *When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately
+  bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
+  service, *Then* it does not advance.
+- **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/
+  stop/destroy via `kubectl`/`virtctl`, *Then* it behaves; its disk survives rebuild; default-deny +
+  egress class apply.
+- **Alerting (R-OBS-02/04, L3):** *Given* a monitored condition, *When* a fault is injected, *Then*
+  an email alert is delivered and the dashboard reflects it.
+- **GitOps drift (R-GITOPS-03, L3):** *Given* a reconciled resource, *When* it is mutated live,
+  *Then* Flux reverts it to the git-declared state.
+- **Onboarding completeness (WS-APPS):** *Given* only a merged ServiceDefinition, *Then* the service
+  is fully wired (compute/storage/policy/egress/TLS/DNS/monitoring/update) with no manual step —
+  asserted by the union of the relevant tests above against that service.
+
+## Gating rules (R-TEST-03)
+
+- **Per-change (merge gate):** the conformance subset tagged with the requirement IDs the change
+  touches MUST pass before merge.
+- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
+  before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
+- **Regression:** the growing suite runs continuously against dev/test; a previously-green
+  requirement going red blocks further delivery until fixed.
+- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
+  state.
+
+## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md))
+
+The approach ([ADR-0010](../decisions/adr-0010-conformance-testing.md)) is realized with two tools
+from the **Kyverno ecosystem**:
+
+- **Kyverno** — **L0 static/policy gate _and_ in-cluster admission enforcer.** In CI, the Kyverno CLI
+  (`kyverno apply` / `kyverno test`, incl. `kyverno-json` for ServiceDefinitions) validates manifests
+  and rendered resources against platform invariants (default-deny baseline present, no plaintext
+  secrets, required labels/egress-class, schema conformance). **The same policies also run in-cluster
+  as an admission controller**, rejecting any non-conforming resource at reconcile time (defense in
+  depth — a resource that somehow bypassed CI still cannot be admitted). Enforcement **never replaces
+  behavior tests**: a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
+  L2/L3 Chainsaw tests remain mandatory. Policies are authored once and shared between the CLI gate
+  and the admission controller.
+- **Chainsaw** — **L1 rendering, L2 behavioral, and L3 fault-injection.** Declarative steps
+  (apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded polling with explicit
+  timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract.
+  Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore)
+  is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures,
+  with `finally` restoring state.
+
+Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs
+(labels/annotations/naming) so the coverage report is generated automatically; both tools' output
+feeds JUnit + the requirement-coverage report and the CI merge / phase-exit gates. Tool versions are
+pinned for determinism.
+
+### Where each layer runs
+- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
+  ServiceDefinition validation; fast merge-gate feedback.
+- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the dev/test cluster.
+  CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
+  needs a broad kubeconfig. This keeps credentials in-cluster and least-privileged (R-SEC-02).
+  - The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the
+    behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch
+    NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag,
+    trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and
+    documented, MUST NOT be reused by workloads, and its own reach is itself an audit item.
+  - The Job writes **JUnit + coverage artifacts** to a known location (log stream and/or a results
+    PVC/object) that CI collects to enforce the merge / phase-exit gates.
+  - Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run.
+
+## Acceptance gates (for the testing capability itself)
+- A single command runs the suite and emits requirement-tagged JUnit + coverage.
+- Every `R-*` requirement maps to at least one passing behavior test (coverage report proves it).
+- Safety-critical requirements each have a **fault-injection** test that fails closed / loses no
+  data.
+- Re-running the suite yields identical results (reproducibility) with no leaked fixtures.
+
+## Open questions
+- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
+- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
+  holds no cluster credentials.
+- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
+  in-cluster (defense in depth), in addition to the CI gate.
+- Delegated to WS-TEST at implementation time (not human decisions): the **exact RBAC scope** of the
+  test ServiceAccount (author it **least-privilege** for precisely the fault-injection actions it
+  performs, [07](07-security-and-secrets.md)) and the **results-artifact mechanism** (log stream vs
+  results PVC/object store). Both are implementer's discretion within the stated constraints.