docs: apply the operator's rulings on Kasm, revocation and fault injection
Retire Kasm from the catalog. Drop ACME revocation from R-TLS-02. Make L3 fault injection operator-run and exclude it from the merge gate (R-TEST-03, ADR-0026). Update the gap register to match; Homepage authentication is now ruled must-enforce and stays a gap until the implementation lands. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
c626349c7d
commit
18f7ba8d0f
13 files changed
+76
-41
No files matched your search
@@ -17,7 +17,8 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness
|
||||
budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope.
|
||||
3. **Conformance gates delivery (R-TEST-03).** The suite blocks merges (for touched requirements)
|
||||
and blocks phase exits (full phase suite). Green suite is the definition of progress.
|
||||
and blocks phase exits (full phase suite). Green suite is the definition of progress. Fault
|
||||
injection is operator-run and does not gate individual merges (ADR-0026).
|
||||
4. **Traceable (R-TEST-04).** Each test declares its `R-*` IDs; a generated coverage report maps
|
||||
every requirement to its verifying test(s). Uncovered requirements are visible gaps.
|
||||
5. **Fault-injection for safety-critical behavior (R-TEST-05).** Default-deny, VPN kill-switch,
|
||||
@@ -52,7 +53,7 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
|
||||
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
|
||||
| **L2 Behavioral/integration** | Against the live cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
|
||||
| **L3 Fault-injection** | Against the live cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L3 Fault-injection** | Against the live cluster, operator-run only | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
|
||||
|
||||
## Behavior test catalog (normative — implementers author these as automated tests)
|
||||
@@ -109,10 +110,13 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
## Gating rules (R-TEST-03)
|
||||
|
||||
- **Per-change (merge gate):** the conformance subset tagged with the requirement IDs the change
|
||||
touches MUST pass before merge.
|
||||
touches MUST pass before merge, across the automatable layers (L0–L2).
|
||||
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
|
||||
before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
|
||||
- **Regression:** the growing suite runs continuously against the live cluster; a previously-green
|
||||
- **Fault injection (L3) is operator-run** ([ADR-0026](../decisions/adr-0026-fault-injection-operator-run.md)):
|
||||
no pipeline or schedule ever starts it. The operator runs it deliberately and records each result
|
||||
in the run ledger.
|
||||
- **Regression:** the automatable layers run continuously; a previously-green
|
||||
requirement going red blocks further delivery until fixed.
|
||||
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
|
||||
state.
|
||||
@@ -172,7 +176,8 @@ pinned for determinism.
|
||||
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
|
||||
ServiceDefinition validation; fast merge-gate feedback.
|
||||
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the live cluster.
|
||||
CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
|
||||
CI **triggers** the L2 Job (and waits for it), and only the operator starts an L3 run
|
||||
(ADR-0026). The Job — not CI — holds cluster access, so CI never
|
||||
needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a
|
||||
fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and
|
||||
least-privileged (R-SEC-02).
|
||||
@@ -194,7 +199,7 @@ pinned for determinism.
|
||||
|
||||
## Open questions
|
||||
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
|
||||
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
|
||||
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job**, L2 triggered by CI and L3 by the operator; CI
|
||||
holds no broad cluster credential, only the admission-bounded trigger.
|
||||
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
|
||||
in-cluster (defense in depth), in addition to the CI gate.
|
||||
|
||||
Reference in new issue
Block a user