# 14 — Testing & Conformance

Serves: **R-TEST-01..06**, and is the verification mechanism for **every other requirement**.
Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).

> **This spec changes what "done" means.** Nothing is accepted on a manual demo, a screenshot, or
> "it reconciled." Every requirement is proven by an **automated, reproducible, behavior-based test**
> that a machine can run on demand and that is tagged with the requirement ID(s) it verifies.

## Principles

1. **Behavior-based (R-TEST-01).** Tests assert *observable system behavior* — Given a state, When
   an action, Then an outcome — not implementation internals. "A pod with no allow rules cannot reach
   Plex" is a behavior; "a CiliumNetworkPolicy object exists" is not (it may exist and still not
   enforce). Test the behavior.
2. **Reproducible (R-TEST-02).** Deterministic, hermetic, version-pinned, single-command, and
   environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness
   budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope.
3. **Conformance gates delivery (R-TEST-03).** The suite blocks merges (for touched requirements)
   and blocks phase exits (full phase suite). Green suite is the definition of progress. Fault
   injection is operator-run and does not gate individual merges (ADR-0026).
4. **Traceable (R-TEST-04).** Each test declares its `R-*` IDs; a generated coverage report maps
   every requirement to its verifying test(s). Uncovered requirements are visible gaps.
5. **Fault-injection for safety-critical behavior (R-TEST-05).** Default-deny, VPN kill-switch,
   data-survival, DR restore, and auto-rollback are proven by *breaking things* and asserting
   fail-closed / no-data-loss — not by happy-path checks.
6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a
   test or a requirement to get green; a genuine inability to meet the spec is an escalation to the
   human, not a spec/test edit. The implementation pins the spec commit it is built against, and each
   spec advance is adopted explicitly ([`../CONTRIBUTING.md`](../CONTRIBUTING.md)).

## Reproducibility contract (what "reproducible" concretely requires)

- **Single command** runs the suite (or a tagged subset) locally and in CI, in the same pinned
  toolbox image in both places.
- **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down;
  it never depends on pre-existing mutable state or on another test's side effects.
- **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit
  timeouts for eventual consistency; no reliance on wall-clock timing or ordering.
- **Isolated & parallel-safe**: unique namespaces/labels; tests can run concurrently without
  cross-talk.
- **Machine-readable output**: JUnit (or equivalent) plus a requirement-coverage report; every
  result carries the `R-*` tags it verifies.
- **Self-cleaning**: on pass or fail, fixtures are removed so a re-run starts clean (leaked fixtures
  are themselves a test failure).
- **Versioned with the code**: tests live in the implementation repo next to what they verify and
  evolve only when the human-owned spec they encode changes.

## Test layers

| Layer | Runs | Verifies | Example |
|---|---|---|---|
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
| **L2 Behavioral/integration** | Against the live cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
| **L3 Fault-injection** | Against the live cluster, operator-run only | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |

## Behavior test catalog (normative — implementers author these as automated tests)

Each maps to requirement IDs; phrased as Given/When/Then intent.

- **Default-deny (R-NET-01):** *Given* a fresh workload with no allow rules, *When* it attempts any
  in-cluster connection except DNS, *Then* the connection fails; DNS succeeds.
- **Explicit allow / deny-undeclared (R-NET-02):** *Given* declared `dependencies`, *When* the
  source reaches a declared dest port, *Then* it succeeds; *When* it reaches an **undeclared** dest,
  *Then* it fails.
- **Egress class (R-NET-05):** *Given* `egress: internet` vs `egress: vpn` workloads, *When* each
  queries an external echo of its public IP, *Then* the internet one shows the home/WAN IP and the
  vpn one shows the VPN IP.
- **VPN kill-switch (R-NET-06, L3):** *Given* a `vpn` workload with a healthy tunnel, *When* the
  tunnel is forced down, *Then* the workload has **zero** egress to the plain internet and
  `internet`-class workloads are unaffected; *When* the tunnel recovers, connectivity returns.
- **Egress flip (R-NET-07):** *Given* a running service, *When* its `egress` class is changed in
  git and reconciled, *Then* its observed egress path changes with no workload rewrite.
- **Data survival (R-STATE-01, L3):** *Given* a stateful service with written data, *When* the
  workload is destroyed and re-applied from git, *Then* the data is intact.
- **Node loss (R-STATE-02, L3):** *Given* a 3-replica volume, *When* a node is removed, *Then* the
  volume stays available and the cluster reconverges.
- **DR restore (R-STATE-03, L3):** *Given* an off-cluster backup, *When* a namespace + volumes are
  restored into a clean state, *Then* the service returns with its data.
- **Trusted TLS (R-TLS-01/02):** *Given* any exposed endpoint (incl. `internal`), *When* its cert is
  inspected, *Then* it is valid, non-self-signed, unexpired; renewal occurs before expiry with no
  manual step.
- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
  internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
  reachable + TLS-valid.
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config (a deny-list of
  LAN/VPN-only hostnames ahead of a zone wildcard to the shared Gateway), *When* it is rendered,
  *Then* every rule targets only the shared Gateway or a 404, every LAN/VPN-only hostname 404s ahead
  of the wildcard, the last rule is `http_status:404`, and no rule targets the Kubernetes API (L0).
  *When* a LAN/VPN-only hostname is requested at the tunnel, *Then* it gets 404. *When* a `public`
  name is resolved via a public resolver from the LAN, *Then* it is reachable + TLS-valid (L2).
- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
  *When* the update schedule elapses, *Then* a machine-authored change advances it through the merge
  gate with no human/AI edit; *Given* a deliberately
  bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
  service, *Then* it does not advance.
- **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/
  stop/destroy via `kubectl`/`virtctl`, *Then* it behaves; its disk survives rebuild; default-deny +
  egress class apply.
- **Alerting (R-OBS-02/04, L3):** *Given* a monitored condition, *When* a fault is injected, *Then*
  an email alert is delivered and the dashboard reflects it.
- **GitOps drift (R-GITOPS-03, L3):** *Given* a reconciled resource, *When* it is mutated live,
  *Then* Flux reverts it to the git-declared state.
- **Onboarding completeness (WS-APPS):** *Given* only a merged ServiceDefinition, *Then* the service
  is fully wired (compute/storage/policy/egress/TLS/DNS/monitoring/update) with no manual step —
  asserted by the union of the relevant tests above against that service.

## Gating rules (R-TEST-03)

- **Per-change (merge gate):** the conformance subset tagged with the requirement IDs the change
  touches MUST pass before merge, across the automatable layers (L0–L2).
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
  before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
- **Fault injection (L3) is operator-run** ([ADR-0026](../decisions/adr-0026-fault-injection-operator-run.md)):
  no pipeline or schedule ever starts it. The operator runs it deliberately and records each result
  in the run ledger.
- **Regression:** the automatable layers run continuously; a previously-green
  requirement going red blocks further delivery until fixed.
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
  state.
- **Non-answers are never green:** a stage that did not run is never reported as passing
  ([05](05-gitops-and-delivery.md); see [known gaps](16-known-gaps.md)).

## Coverage, status and proof (R-TEST-04)

Three different questions, answered separately so that none is mistaken for another:

- **Coverage** — does a requirement-tagged test exist for each `R-*`?
- **Status** — derived from the repository alone: is the requirement implemented, is that
  implementation actually reconciled by Flux, and is its test executable or a scaffold?
- **Proof** — did a run actually pass? Proof is a committed, machine-readable run ledger. A test that
  skips itself is never proof, and a proof goes stale when the implementation or test moves on.

`./conformance/run.sh --status` in the implementation repo produces the live view.
[`16-known-gaps.md`](16-known-gaps.md) is a dated snapshot.

## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md))

The approach ([ADR-0010](../decisions/adr-0010-conformance-testing.md)) is realized with two tools
from the **Kyverno ecosystem**:

- **Kyverno** — **L0 static/policy gate _and_ in-cluster admission enforcer.** In CI, the Kyverno CLI
  (`kyverno apply` / `kyverno test`, incl. `kyverno-json` for ServiceDefinitions) validates manifests
  and rendered resources against platform invariants (default-deny baseline present, no plaintext
  secrets, required labels/egress-class, schema conformance). **The same policies also run in-cluster
  as an admission controller**, rejecting any non-conforming resource at reconcile time (defense in
  depth — a resource that somehow bypassed CI still cannot be admitted). Enforcement **never replaces
  behavior tests**: a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
  L2/L3 Chainsaw tests remain mandatory. Policies are authored once and shared between the CLI gate
  and the admission controller.
  - **ServiceDefinition validation is two complementary gates, not one** (do not consolidate): the
    **JSON Schema** (`schemas/servicedefinition.schema.json`) checks *shape* — types, enums, required
    fields, exactly-one-of `source`, `public ⇒ auth.mode≠none` — as a fast, cluster-free PR check;
    **Kyverno `kyverno-json`** enforces the *cross-cutting invariants* the schema can't express
    (`longhorn ⇒ size`, irreplaceable⇒off-cluster-backup, label conventions). Both are L0 and run in
    CI; they overlap by design (defense in depth), not by redundancy.
- **Chainsaw** — **L1 rendering, L2 behavioral, and L3 fault-injection.** Declarative steps
  (apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded polling with explicit
  timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract.
  Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore)
  is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures,
  with `finally` restoring state. Cluster-free L0 and L1 checks are declared as Chainsaw tests for
  tagging and execute through the harness's offline runner, so they need no cluster.
- **Repository invariants** — L0 also carries whole-repo checks a per-resource policy cannot express
  (secret hygiene across every tracked file, least-privilege RBAC, the spec pin), which run on every
  gate regardless of the requirements a change touches.

Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs
(labels/annotations/naming) so the coverage report is generated automatically; both tools' output
feeds JUnit + the requirement-coverage report and the CI merge / phase-exit gates. Tool versions are
pinned for determinism.

### Where each layer runs
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
  ServiceDefinition validation; fast merge-gate feedback.
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the live cluster.
  CI **triggers** the L2 Job (and waits for it), and only the operator starts an L3 run
  (ADR-0026). The Job — not CI — holds cluster access, so CI never
  needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a
  fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and
  least-privileged (R-SEC-02).
  - The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the
    behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch
    NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag,
    trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and
    documented, MUST NOT be reused by workloads, and its own reach is itself an audit item.
  - The Job emits **JUnit + coverage artifacts** over its log stream, in an integrity-checked
    envelope that CI collects to enforce the merge / phase-exit gates.
  - Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run.

## Acceptance gates (for the testing capability itself)
- A single command runs the suite and emits requirement-tagged JUnit + coverage.
- Every `R-*` requirement maps to at least one passing behavior test (coverage report proves it).
- Safety-critical requirements each have a **fault-injection** test that fails closed / loses no
  data.
- Re-running the suite yields identical results (reproducibility) with no leaked fixtures.

## Open questions
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job**, L2 triggered by CI and L3 by the operator; CI
  holds no broad cluster credential, only the admission-bounded trigger.
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
  in-cluster (defense in depth), in addition to the CI gate.
- The test ServiceAccount's RBAC is **resolved**: scoped **least-privilege** to precisely the
  fault-injection actions it performs ([07](07-security-and-secrets.md)), with its grants justified
  per test in the implementation repo.
- The results-artifact mechanism is **resolved**: the log stream, as above.