home-network-k3s/docs/14-testing-and-conformance.md

14 — Testing & Conformance

Serves: R-TEST-01..06, and is the verification mechanism for every other requirement. Related: ADR-0010.

This spec changes what "done" means. Nothing is accepted on a manual demo, a screenshot, or "it reconciled." Every requirement is proven by an automated, reproducible, behavior-based test that a machine can run on demand and that is tagged with the requirement ID(s) it verifies.

Principles

1. Behavior-based (R-TEST-01). Tests assert observable system behavior — Given a state, When an action, Then an outcome — not implementation internals. "A pod with no allow rules cannot reach Plex" is a behavior; "a CiliumNetworkPolicy object exists" is not (it may exist and still not enforce). Test the behavior.
2. Reproducible (R-TEST-02). Deterministic, hermetic, version-pinned, single-command, and environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope.
3. Conformance gates delivery (R-TEST-03). The suite blocks merges (for touched requirements) and blocks phase exits (full phase suite). Green suite is the definition of progress. Fault injection is operator-run and does not gate individual merges (ADR-0026).
4. Traceable (R-TEST-04). Each test declares its R-* IDs; a generated coverage report maps every requirement to its verifying test(s). Uncovered requirements are visible gaps.
5. Fault-injection for safety-critical behavior (R-TEST-05). Default-deny, VPN kill-switch, data-survival, DR restore, and auto-rollback are proven by breaking things and asserting fail-closed / no-data-loss — not by happy-path checks.
6. Specs are the fixed target (R-TEST-06). Tests encode the spec's intent. Agents never weaken a test or a requirement to get green; a genuine inability to meet the spec is an escalation to the human, not a spec/test edit. The implementation pins the spec commit it is built against, and each spec advance is adopted explicitly (../CONTRIBUTING.md).

Reproducibility contract (what "reproducible" concretely requires)

• Single command runs the suite (or a tagged subset) locally and in CI, in the same pinned toolbox image in both places.
• Hermetic fixtures: each test provisions its own namespace/workloads/probes and tears them down; it never depends on pre-existing mutable state or on another test's side effects.
• Deterministic: fixed inputs, pinned tool/image versions, bounded retries with explicit timeouts for eventual consistency; no reliance on wall-clock timing or ordering.
• Isolated & parallel-safe: unique namespaces/labels; tests can run concurrently without cross-talk.
• Machine-readable output: JUnit (or equivalent) plus a requirement-coverage report; every result carries the R-* tags it verifies.
• Self-cleaning: on pass or fail, fixtures are removed so a re-run starts clean (leaked fixtures are themselves a test failure).
• Versioned with the code: tests live in the implementation repo next to what they verify and evolve only when the human-owned spec they encode changes.

Test layers

Layer	Runs	Verifies	Example
L0 Static/policy	Pre-apply, in CI	Manifests/ServiceDefinitions obey invariants	Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition
L1 Rendering	CI, no cluster	ServiceDefinition → rendered resources match golden expectations	egress: vpn renders the kill-switched egress path
L2 Behavioral/integration	Against the live cluster	Real runtime behavior	Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild
L3 Fault-injection	Against the live cluster, operator-run only	Fail-closed / no-data-loss under failure	VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore
L4 Conformance suite	CI + on-demand	Aggregate, requirement-tagged gate	The union of the above, reported per R-*

Behavior test catalog (normative — implementers author these as automated tests)

Each maps to requirement IDs; phrased as Given/When/Then intent.

• Default-deny (R-NET-01): Given a fresh workload with no allow rules, When it attempts any in-cluster connection except DNS, Then the connection fails; DNS succeeds.
• Explicit allow / deny-undeclared (R-NET-02): Given declared dependencies, When the source reaches a declared dest port, Then it succeeds; When it reaches an undeclared dest, Then it fails.
• Egress class (R-NET-05): Given egress: internet vs egress: vpn workloads, When each queries an external echo of its public IP, Then the internet one shows the home/WAN IP and the vpn one shows the VPN IP.
• VPN kill-switch (R-NET-06, L3): Given a vpn workload with a healthy tunnel, When the tunnel is forced down, Then the workload has zero egress to the plain internet and internet-class workloads are unaffected; When the tunnel recovers, connectivity returns.
• Egress flip (R-NET-07): Given a running service, When its egress class is changed in git and reconciled, Then its observed egress path changes with no workload rewrite.
• Data survival (R-STATE-01, L3): Given a stateful service with written data, When the workload is destroyed and re-applied from git, Then the data is intact.
• Node loss (R-STATE-02, L3): Given a 3-replica volume, When a node is removed, Then the volume stays available and the cluster reconverges.
• DR restore (R-STATE-03, L3): Given an off-cluster backup, When a namespace + volumes are restored into a clean state, Then the service returns with its data.
• Trusted TLS (R-TLS-01/02): Given any exposed endpoint (incl. internal), When its cert is inspected, Then it is valid, non-self-signed, unexpired; renewal occurs before expiry with no manual step.
• DNS/exposure (R-DNS-*): Given a service, When resolved, Then the record exists and internal clients hit the internal Gateway; internal is unreachable publicly, public is reachable + TLS-valid.
• Public ingress path (R-DNS-04/05, R-SEC-04): Given the tunnel ingress config (a deny-list of LAN/VPN-only hostnames ahead of a zone wildcard to the shared Gateway), When it is rendered, Then every rule targets only the shared Gateway or a 404, every LAN/VPN-only hostname 404s ahead of the wildcard, the last rule is http_status:404, and no rule targets the Kubernetes API (L0). When a LAN/VPN-only hostname is requested at the tunnel, Then it gets 404. When a public name is resolved via a public resolver from the LAN, Then it is reachable + TLS-valid (L2).
• Hands-off update (R-UPD-01/02, L3): Given an out-of-date service under a stable policy, When the update schedule elapses, Then a machine-authored change advances it through the merge gate with no human/AI edit; Given a deliberately bad update, Then health-gating auto-rolls-back and an email alert fires; Given a pinned service, Then it does not advance.
• VM lifecycle (R-PLAT-03, L2/L3): Given a kind: vm ServiceDefinition, When create/start/ stop/destroy via kubectl/virtctl, Then it behaves; its disk survives rebuild; default-deny + egress class apply.
• Alerting (R-OBS-02/04, L3): Given a monitored condition, When a fault is injected, Then an email alert is delivered and the dashboard reflects it.
• GitOps drift (R-GITOPS-03, L3): Given a reconciled resource, When it is mutated live, Then Flux reverts it to the git-declared state.
• Onboarding completeness (WS-APPS): Given only a merged ServiceDefinition, Then the service is fully wired (compute/storage/policy/egress/TLS/DNS/monitoring/update) with no manual step — asserted by the union of the relevant tests above against that service.

Gating rules (R-TEST-03)

• Per-change (merge gate): the conformance subset tagged with the requirement IDs the change touches MUST pass before merge, across the automatable layers (L0–L2).
• Per-phase (exit gate): the full conformance suite for the phase's requirements MUST pass before the phase's exit criteria are considered met (../roadmap/phases.md).
• Fault injection (L3) is operator-run (ADR-0026): no pipeline or schedule ever starts it. The operator runs it deliberately and records each result in the run ledger.
• Regression: the automatable layers run continuously; a previously-green requirement going red blocks further delivery until fixed.
• Coverage: a requirement with no passing test is not satisfied, regardless of implementation state.
• Non-answers are never green: a stage that did not run is never reported as passing (05; see known gaps).

Coverage, status and proof (R-TEST-04)

Three different questions, answered separately so that none is mistaken for another:

• Coverage — does a requirement-tagged test exist for each R-*?
• Status — derived from the repository alone: is the requirement implemented, is that implementation actually reconciled by Flux, and is its test executable or a scaffold?
• Proof — did a run actually pass? Proof is a committed, machine-readable run ledger. A test that skips itself is never proof, and a proof goes stale when the implementation or test moves on.

./conformance/run.sh --status in the implementation repo produces the live view. 16-known-gaps.md is a dated snapshot.

Tooling (pinned — ADR-0011)

The approach (ADR-0010) is realized with two tools from the Kyverno ecosystem:

• Kyverno — L0 static/policy gate and in-cluster admission enforcer. In CI, the Kyverno CLI (kyverno apply / kyverno test, incl. kyverno-json for ServiceDefinitions) validates manifests and rendered resources against platform invariants (default-deny baseline present, no plaintext secrets, required labels/egress-class, schema conformance). The same policies also run in-cluster as an admission controller, rejecting any non-conforming resource at reconcile time (defense in depth — a resource that somehow bypassed CI still cannot be admitted). Enforcement never replaces behavior tests: a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so L2/L3 Chainsaw tests remain mandatory. Policies are authored once and shared between the CLI gate and the admission controller.
  • ServiceDefinition validation is two complementary gates, not one (do not consolidate): the JSON Schema (schemas/servicedefinition.schema.json) checks shape — types, enums, required fields, exactly-one-of source, public ⇒ auth.mode≠none — as a fast, cluster-free PR check; Kyverno kyverno-json enforces the cross-cutting invariants the schema can't express (longhorn ⇒ size, irreplaceable⇒off-cluster-backup, label conventions). Both are L0 and run in CI; they overlap by design (defense in depth), not by redundancy.
• Chainsaw — L1 rendering, L2 behavioral, and L3 fault-injection. Declarative steps (apply/assert/error, script/command, try/catch/finally), bounded polling with explicit timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract. Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore) is expressed as Chainsaw script steps, asserting fail-closed / no-data-loss via probe fixtures, with finally restoring state. Cluster-free L0 and L1 checks are declared as Chainsaw tests for tagging and execute through the harness's offline runner, so they need no cluster.
• Repository invariants — L0 also carries whole-repo checks a per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC, the spec pin), which run on every gate regardless of the requirements a change touches.

Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their R-* IDs (labels/annotations/naming) so the coverage report is generated automatically; both tools' output feeds JUnit + the requirement-coverage report and the CI merge / phase-exit gates. Tool versions are pinned for determinism.

Where each layer runs

• L0 (Kyverno CLI) and L1 (rendering assertions) run in CI without a cluster — pure manifest/ ServiceDefinition validation; fast merge-gate feedback.
• L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job on the live cluster. CI triggers the L2 Job (and waits for it), and only the operator starts an L3 run (ADR-0026). The Job — not CI — holds cluster access, so CI never needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and least-privileged (R-SEC-02).
  • The Job runs under a dedicated, scoped test ServiceAccount whose RBAC grants exactly what the behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag, trigger a Velero restore). This is a privileged test identity — it MUST be scoped, named, and documented, MUST NOT be reused by workloads, and its own reach is itself an audit item.
  • The Job emits JUnit + coverage artifacts over its log stream, in an integrity-checked envelope that CI collects to enforce the merge / phase-exit gates.
  • Fixtures are hermetic and self-cleaning (finally steps); a leaked fixture fails the run.

Acceptance gates (for the testing capability itself)

• A single command runs the suite and emits requirement-tagged JUnit + coverage.
• Every R-* requirement maps to at least one passing behavior test (coverage report proves it).
• Safety-critical requirements each have a fault-injection test that fails closed / loses no data.
• Re-running the suite yields identical results (reproducibility) with no leaked fixtures.

Open questions

• Tool mix is resolved: Kyverno (L0) + Chainsaw (L1/L2/L3), ADR-0011.
• Harness location is resolved: L2/L3 run as an in-cluster Chainsaw Job, L2 triggered by CI and L3 by the operator; CI holds no broad cluster credential, only the admission-bounded trigger.
• Kyverno-as-admission-enforcer is resolved: yes — the L0 policies also enforce at admission in-cluster (defense in depth), in addition to the CI gate.
• The test ServiceAccount's RBAC is resolved: scoped least-privilege to precisely the fault-injection actions it performs (07), with its grants justified per test in the implementation repo.
• The results-artifact mechanism is resolved: the log stream, as above.