docs: apply the operator's rulings on Kasm, revocation and fault injection

Retire Kasm from the catalog. Drop ACME revocation from R-TLS-02. Make L3
fault injection operator-run and exclude it from the merge gate (R-TEST-03,
ADR-0026). Update the gap register to match; Homepage authentication is now
ruled must-enforce and stays a gap until the implementation lands.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 23:06:26 -06:00
1 parent c626349c7d
commit 18f7ba8d0f
13 files changed
+76 -41

No files matched your search

+11 -6
View File
@@ -17,7 +17,8 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness
budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope.
3. **Conformance gates delivery (R-TEST-03).** The suite blocks merges (for touched requirements)
and blocks phase exits (full phase suite). Green suite is the definition of progress.
and blocks phase exits (full phase suite). Green suite is the definition of progress. Fault
injection is operator-run and does not gate individual merges (ADR-0026).
4. **Traceable (R-TEST-04).** Each test declares its `R-*` IDs; a generated coverage report maps
every requirement to its verifying test(s). Uncovered requirements are visible gaps.
5. **Fault-injection for safety-critical behavior (R-TEST-05).** Default-deny, VPN kill-switch,
@@ -52,7 +53,7 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
| **L2 Behavioral/integration** | Against the live cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
| **L3 Fault-injection** | Against the live cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
| **L3 Fault-injection** | Against the live cluster, operator-run only | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
## Behavior test catalog (normative — implementers author these as automated tests)
@@ -109,10 +110,13 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
## Gating rules (R-TEST-03)
- **Per-change (merge gate):** the conformance subset tagged with the requirement IDs the change
touches MUST pass before merge.
touches MUST pass before merge, across the automatable layers (L0–L2).
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
- **Regression:** the growing suite runs continuously against the live cluster; a previously-green
- **Fault injection (L3) is operator-run** ([ADR-0026](../decisions/adr-0026-fault-injection-operator-run.md)):
no pipeline or schedule ever starts it. The operator runs it deliberately and records each result
in the run ledger.
- **Regression:** the automatable layers run continuously; a previously-green
requirement going red blocks further delivery until fixed.
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
state.
@@ -172,7 +176,8 @@ pinned for determinism.
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
ServiceDefinition validation; fast merge-gate feedback.
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the live cluster.
CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
CI **triggers** the L2 Job (and waits for it), and only the operator starts an L3 run
(ADR-0026). The Job — not CI — holds cluster access, so CI never
needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a
fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and
least-privileged (R-SEC-02).
@@ -194,7 +199,7 @@ pinned for determinism.
## Open questions
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job**, L2 triggered by CI and L3 by the operator; CI
holds no broad cluster credential, only the admission-bounded trigger.
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
in-cluster (defense in depth), in addition to the CI gate.