test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience)
R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.
tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
GIVEN a 3-replica Longhorn volume with written data attached on node N,
WHEN N is removed from service (cordon + evict the attached workload, forcing
the volume to detach from N),
THEN a fresh workload on a SURVIVING node re-attaches the same volume and reads
the canary intact — volume stays available, cluster reconverges.
Safe-by-construction for the 3-node embedded-etcd dev cluster:
- Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
- Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
system pod is disturbed.
- Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
ALWAYS uncordons every node with THAT label and removes it — restored even on
failure/timeout, and never touches an operator-cordoned node.
- Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.
Bookkeeping:
- run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
(serial pass, matches the other Longhorn/KubeVirt L3 tests).
- incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
(cordon/uncordon/label); no eviction rights (no drain).
- README.md / coverage.py docstring: refresh the documented-gap example.
Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.
NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
d34c639269
commit
fdf5c69adc
5 files changed
+233
-15
No files matched your search
@@ -88,13 +88,15 @@ rules:
|
||||
# namespaces policy is enforced, the `chainsaw-conf-*` test namespaces MUST be in
|
||||
# its exclusion set (documented, audited exception — R-SEC-02).
|
||||
#
|
||||
# ---- R-STATE-02 (node loss): cordon/drain/uncordon a node (NOT in WS-TEST scope yet)
|
||||
# - apiGroups: [""]
|
||||
# resources: [nodes]
|
||||
# verbs: [get, list, patch] # cordon = patch spec.unschedulable
|
||||
# - apiGroups: [""]
|
||||
# resources: [pods/eviction]
|
||||
# verbs: [create] # drain
|
||||
# ---- R-STATE-02 (node loss, tests/l3-fault-injection/node-loss): the fault is
|
||||
# `kubectl cordon` + label of ONE node, reverted by `kubectl uncordon` + label removal
|
||||
# in the test's `finally`. All three (cordon/uncordon/label) are a `patch` on the node
|
||||
# object; `get`/`list` locate the victim + the finally cleanup selector. The test evicts
|
||||
# only its OWN writer pod via `delete pod` (already granted above) — it deliberately does
|
||||
# NOT `kubectl drain`, so no eviction rights and no reach over unrelated pods are needed.
|
||||
- apiGroups: [""]
|
||||
resources: [nodes]
|
||||
verbs: [get, list, patch] # cordon/uncordon/label = patch spec.unschedulable + metadata.labels
|
||||
#
|
||||
# ---- R-UPD-02 (bad-update auto-rollback): inject a bad image tag via automation
|
||||
# - apiGroups: [image.toolkit.fluxcd.io]
|
||||
|
||||
Reference in new issue
Block a user