docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 22:46:00 -06:00
1 parent 1ddc14b918
commit 250fb54154
43 files changed
+1672 -575

No files matched your search

+46 -21
View File
@@ -25,11 +25,13 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
fail-closed / no-data-loss — not by happy-path checks.
6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a
test or a requirement to get green; a genuine inability to meet the spec is an escalation to the
human, not a spec/test edit.
human, not a spec/test edit. The implementation pins the spec commit it is built against, and each
spec advance is adopted explicitly ([`../CONTRIBUTING.md`](../CONTRIBUTING.md)).
## Reproducibility contract (what "reproducible" concretely requires)
- **Single command** runs the suite (or a tagged subset) locally and in CI.
- **Single command** runs the suite (or a tagged subset) locally and in CI, in the same pinned
toolbox image in both places.
- **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down;
it never depends on pre-existing mutable state or on another test's side effects.
- **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit
@@ -49,8 +51,8 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|---|---|---|---|
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
| **L2 Behavioral/integration** | Against dev/test cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
| **L3 Fault-injection** | Against dev/test cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
| **L2 Behavioral/integration** | Against the live cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
| **L3 Fault-injection** | Against the live cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
## Behavior test catalog (normative — implementers author these as automated tests)
@@ -82,13 +84,15 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
reachable + TLS-valid.
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config, *When* it is
rendered, *Then* every route is a `public` service, the last rule is `http_status:404`, and no rule
targets the Kubernetes API (L0). *When* a non-`public` hostname is requested at the tunnel, *Then*
it gets 404. *When* a `public` name is resolved via a public resolver from the LAN, *Then* it is
reachable + TLS-valid (L2).
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config (a deny-list of
LAN/VPN-only hostnames ahead of a zone wildcard to the shared Gateway), *When* it is rendered,
*Then* every rule targets only the shared Gateway or a 404, every LAN/VPN-only hostname 404s ahead
of the wildcard, the last rule is `http_status:404`, and no rule targets the Kubernetes API (L0).
*When* a LAN/VPN-only hostname is requested at the tunnel, *Then* it gets 404. *When* a `public`
name is resolved via a public resolver from the LAN, *Then* it is reachable + TLS-valid (L2).
- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
*When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately
*When* the update schedule elapses, *Then* a machine-authored change advances it through the merge
gate with no human/AI edit; *Given* a deliberately
bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
service, *Then* it does not advance.
- **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/
@@ -108,10 +112,25 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
touches MUST pass before merge.
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
- **Regression:** the growing suite runs continuously against dev/test; a previously-green
- **Regression:** the growing suite runs continuously against the live cluster; a previously-green
requirement going red blocks further delivery until fixed.
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
state.
- **Non-answers are never green:** a stage that did not run is never reported as passing
([05](05-gitops-and-delivery.md); see [known gaps](16-known-gaps.md)).
## Coverage, status and proof (R-TEST-04)
Three different questions, answered separately so that none is mistaken for another:
- **Coverage** — does a requirement-tagged test exist for each `R-*`?
- **Status** — derived from the repository alone: is the requirement implemented, is that
implementation actually reconciled by Flux, and is its test executable or a scaffold?
- **Proof** — did a run actually pass? Proof is a committed, machine-readable run ledger. A test that
skips itself is never proof, and a proof goes stale when the implementation or test moves on.
`./conformance/run.sh --status` in the implementation repo produces the live view.
[`16-known-gaps.md`](16-known-gaps.md) is a dated snapshot.
## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md))
@@ -138,7 +157,11 @@ from the **Kyverno ecosystem**:
timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract.
Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore)
is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures,
with `finally` restoring state.
with `finally` restoring state. Cluster-free L0 and L1 checks are declared as Chainsaw tests for
tagging and execute through the harness's offline runner, so they need no cluster.
- **Repository invariants** — L0 also carries whole-repo checks a per-resource policy cannot express
(secret hygiene across every tracked file, least-privilege RBAC, the spec pin), which run on every
gate regardless of the requirements a change touches.
Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs
(labels/annotations/naming) so the coverage report is generated automatically; both tools' output
@@ -148,16 +171,18 @@ pinned for determinism.
### Where each layer runs
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
ServiceDefinition validation; fast merge-gate feedback.
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the dev/test cluster.
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the live cluster.
CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
needs a broad kubeconfig. This keeps credentials in-cluster and least-privileged (R-SEC-02).
needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a
fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and
least-privileged (R-SEC-02).
- The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the
behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch
NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag,
trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and
documented, MUST NOT be reused by workloads, and its own reach is itself an audit item.
- The Job writes **JUnit + coverage artifacts** to a known location (log stream and/or a results
PVC/object) that CI collects to enforce the merge / phase-exit gates.
- The Job emits **JUnit + coverage artifacts** over its log stream, in an integrity-checked
envelope that CI collects to enforce the merge / phase-exit gates.
- Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run.
## Acceptance gates (for the testing capability itself)
@@ -170,10 +195,10 @@ pinned for determinism.
## Open questions
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
holds no cluster credentials.
holds no broad cluster credential, only the admission-bounded trigger.
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
in-cluster (defense in depth), in addition to the CI gate.
- Delegated to WS-TEST at implementation time (not human decisions): the **exact RBAC scope** of the
test ServiceAccount (author it **least-privilege** for precisely the fault-injection actions it
performs, [07](07-security-and-secrets.md)) and the **results-artifact mechanism** (log stream vs
results PVC/object store). Both are implementer's discretion within the stated constraints.
- The test ServiceAccount's RBAC is **resolved**: scoped **least-privilege** to precisely the
fault-injection actions it performs ([07](07-security-and-secrets.md)), with its grants justified
per test in the implementation repo.
- The results-artifact mechanism is **resolved**: the log stream, as above.