docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
@@ -25,11 +25,13 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
fail-closed / no-data-loss — not by happy-path checks.
|
||||
6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a
|
||||
test or a requirement to get green; a genuine inability to meet the spec is an escalation to the
|
||||
human, not a spec/test edit.
|
||||
human, not a spec/test edit. The implementation pins the spec commit it is built against, and each
|
||||
spec advance is adopted explicitly ([`../CONTRIBUTING.md`](../CONTRIBUTING.md)).
|
||||
|
||||
## Reproducibility contract (what "reproducible" concretely requires)
|
||||
|
||||
- **Single command** runs the suite (or a tagged subset) locally and in CI.
|
||||
- **Single command** runs the suite (or a tagged subset) locally and in CI, in the same pinned
|
||||
toolbox image in both places.
|
||||
- **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down;
|
||||
it never depends on pre-existing mutable state or on another test's side effects.
|
||||
- **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit
|
||||
@@ -49,8 +51,8 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
|---|---|---|---|
|
||||
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
|
||||
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
|
||||
| **L2 Behavioral/integration** | Against dev/test cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
|
||||
| **L3 Fault-injection** | Against dev/test cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L2 Behavioral/integration** | Against the live cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
|
||||
| **L3 Fault-injection** | Against the live cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
|
||||
|
||||
## Behavior test catalog (normative — implementers author these as automated tests)
|
||||
@@ -82,13 +84,15 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
|
||||
internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
|
||||
reachable + TLS-valid.
|
||||
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config, *When* it is
|
||||
rendered, *Then* every route is a `public` service, the last rule is `http_status:404`, and no rule
|
||||
targets the Kubernetes API (L0). *When* a non-`public` hostname is requested at the tunnel, *Then*
|
||||
it gets 404. *When* a `public` name is resolved via a public resolver from the LAN, *Then* it is
|
||||
reachable + TLS-valid (L2).
|
||||
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config (a deny-list of
|
||||
LAN/VPN-only hostnames ahead of a zone wildcard to the shared Gateway), *When* it is rendered,
|
||||
*Then* every rule targets only the shared Gateway or a 404, every LAN/VPN-only hostname 404s ahead
|
||||
of the wildcard, the last rule is `http_status:404`, and no rule targets the Kubernetes API (L0).
|
||||
*When* a LAN/VPN-only hostname is requested at the tunnel, *Then* it gets 404. *When* a `public`
|
||||
name is resolved via a public resolver from the LAN, *Then* it is reachable + TLS-valid (L2).
|
||||
- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
|
||||
*When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately
|
||||
*When* the update schedule elapses, *Then* a machine-authored change advances it through the merge
|
||||
gate with no human/AI edit; *Given* a deliberately
|
||||
bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
|
||||
service, *Then* it does not advance.
|
||||
- **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/
|
||||
@@ -108,10 +112,25 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
touches MUST pass before merge.
|
||||
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
|
||||
before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
|
||||
- **Regression:** the growing suite runs continuously against dev/test; a previously-green
|
||||
- **Regression:** the growing suite runs continuously against the live cluster; a previously-green
|
||||
requirement going red blocks further delivery until fixed.
|
||||
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
|
||||
state.
|
||||
- **Non-answers are never green:** a stage that did not run is never reported as passing
|
||||
([05](05-gitops-and-delivery.md); see [known gaps](16-known-gaps.md)).
|
||||
|
||||
## Coverage, status and proof (R-TEST-04)
|
||||
|
||||
Three different questions, answered separately so that none is mistaken for another:
|
||||
|
||||
- **Coverage** — does a requirement-tagged test exist for each `R-*`?
|
||||
- **Status** — derived from the repository alone: is the requirement implemented, is that
|
||||
implementation actually reconciled by Flux, and is its test executable or a scaffold?
|
||||
- **Proof** — did a run actually pass? Proof is a committed, machine-readable run ledger. A test that
|
||||
skips itself is never proof, and a proof goes stale when the implementation or test moves on.
|
||||
|
||||
`./conformance/run.sh --status` in the implementation repo produces the live view.
|
||||
[`16-known-gaps.md`](16-known-gaps.md) is a dated snapshot.
|
||||
|
||||
## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md))
|
||||
|
||||
@@ -138,7 +157,11 @@ from the **Kyverno ecosystem**:
|
||||
timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract.
|
||||
Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore)
|
||||
is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures,
|
||||
with `finally` restoring state.
|
||||
with `finally` restoring state. Cluster-free L0 and L1 checks are declared as Chainsaw tests for
|
||||
tagging and execute through the harness's offline runner, so they need no cluster.
|
||||
- **Repository invariants** — L0 also carries whole-repo checks a per-resource policy cannot express
|
||||
(secret hygiene across every tracked file, least-privilege RBAC, the spec pin), which run on every
|
||||
gate regardless of the requirements a change touches.
|
||||
|
||||
Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs
|
||||
(labels/annotations/naming) so the coverage report is generated automatically; both tools' output
|
||||
@@ -148,16 +171,18 @@ pinned for determinism.
|
||||
### Where each layer runs
|
||||
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
|
||||
ServiceDefinition validation; fast merge-gate feedback.
|
||||
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the dev/test cluster.
|
||||
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the live cluster.
|
||||
CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
|
||||
needs a broad kubeconfig. This keeps credentials in-cluster and least-privileged (R-SEC-02).
|
||||
needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a
|
||||
fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and
|
||||
least-privileged (R-SEC-02).
|
||||
- The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the
|
||||
behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch
|
||||
NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag,
|
||||
trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and
|
||||
documented, MUST NOT be reused by workloads, and its own reach is itself an audit item.
|
||||
- The Job writes **JUnit + coverage artifacts** to a known location (log stream and/or a results
|
||||
PVC/object) that CI collects to enforce the merge / phase-exit gates.
|
||||
- The Job emits **JUnit + coverage artifacts** over its log stream, in an integrity-checked
|
||||
envelope that CI collects to enforce the merge / phase-exit gates.
|
||||
- Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run.
|
||||
|
||||
## Acceptance gates (for the testing capability itself)
|
||||
@@ -170,10 +195,10 @@ pinned for determinism.
|
||||
## Open questions
|
||||
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
|
||||
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
|
||||
holds no cluster credentials.
|
||||
holds no broad cluster credential, only the admission-bounded trigger.
|
||||
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
|
||||
in-cluster (defense in depth), in addition to the CI gate.
|
||||
- Delegated to WS-TEST at implementation time (not human decisions): the **exact RBAC scope** of the
|
||||
test ServiceAccount (author it **least-privilege** for precisely the fault-injection actions it
|
||||
performs, [07](07-security-and-secrets.md)) and the **results-artifact mechanism** (log stream vs
|
||||
results PVC/object store). Both are implementer's discretion within the stated constraints.
|
||||
- The test ServiceAccount's RBAC is **resolved**: scoped **least-privilege** to precisely the
|
||||
fault-injection actions it performs ([07](07-security-and-secrets.md)), with its grants justified
|
||||
per test in the implementation repo.
|
||||
- The results-artifact mechanism is **resolved**: the log stream, as above.
|
||||
Reference in new issue
Block a user