R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.
tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
GIVEN a 3-replica Longhorn volume with written data attached on node N,
WHEN N is removed from service (cordon + evict the attached workload, forcing
the volume to detach from N),
THEN a fresh workload on a SURVIVING node re-attaches the same volume and reads
the canary intact — volume stays available, cluster reconverges.
Safe-by-construction for the 3-node embedded-etcd dev cluster:
- Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
- Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
system pod is disturbed.
- Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
ALWAYS uncordons every node with THAT label and removes it — restored even on
failure/timeout, and never touches an operator-cordoned node.
- Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.
Bookkeeping:
- run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
(serial pass, matches the other Longhorn/KubeVirt L3 tests).
- incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
(cordon/uncordon/label); no eviction rights (no drain).
- README.md / coverage.py docstring: refresh the documented-gap example.
Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.
NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 4.8
fdf5c69adc