test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience)
R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.
tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
GIVEN a 3-replica Longhorn volume with written data attached on node N,
WHEN N is removed from service (cordon + evict the attached workload, forcing
the volume to detach from N),
THEN a fresh workload on a SURVIVING node re-attaches the same volume and reads
the canary intact — volume stays available, cluster reconverges.
Safe-by-construction for the 3-node embedded-etcd dev cluster:
- Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
- Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
system pod is disturbed.
- Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
ALWAYS uncordons every node with THAT label and removes it — restored even on
failure/timeout, and never touches an operator-cordoned node.
- Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.
Bookkeeping:
- run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
(serial pass, matches the other Longhorn/KubeVirt L3 tests).
- incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
(cordon/uncordon/label); no eviction rights (no drain).
- README.md / coverage.py docstring: refresh the documented-gap example.
Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.
NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
d34c639269
commit
fdf5c69adc
5 files changed
+233
-15
No files matched your search
+7
-4
@@ -49,7 +49,6 @@ FORCE_NO_CLUSTER=0
|
||||
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
|
||||
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
|
||||
# exemption is explicit and reviewable, not silent.
|
||||
# R-STATE-02 (node-loss) — delivered with Longhorn/infra (WS-STORAGE).
|
||||
# R-ENV-02 (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in
|
||||
# coverage.md). The coarse path->requirement map keys
|
||||
# clusters/* AND ci/* to it and the R-ENV-02 doc token
|
||||
@@ -71,8 +70,12 @@ FORCE_NO_CLUSTER=0
|
||||
# authoritative signal is the `conformance.home.k3s/R-*` label on a test, which still is.
|
||||
# The PATH-MAP half remains: ci/touched-requirements.sh keys clusters/* and ci/* to
|
||||
# R-ENV-02 + R-GITOPS-01 wholesale, which is why those two must stay exempted here. Tighten
|
||||
# the path map (or scaffold the two requirements) and this list shrinks to R-STATE-02.
|
||||
DOCUMENTED_GAPS="R-STATE-02 R-ENV-02 R-GITOPS-01"
|
||||
# the path map (or scaffold the two requirements) and this list empties.
|
||||
#
|
||||
# R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
|
||||
# but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
|
||||
# longer a gap and has been removed from this list; the coverage report counts it as covered.
|
||||
DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01"
|
||||
|
||||
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
|
||||
|
||||
@@ -154,7 +157,7 @@ select_dirs() {
|
||||
# DIRECTORY basename (stable); override via CONF_HEAVY_TESTS to add/remove without code
|
||||
# changes. Listed here even for tests still scaffolded (vm-lifecycle, dr-restore, ...) so
|
||||
# the serialization is already correct the moment they grow real KubeVirt/Longhorn bodies.
|
||||
CONF_HEAVY_TESTS="${CONF_HEAVY_TESTS:-vm-lifecycle vm-data-survival vm-default-deny data-survival dr-restore}"
|
||||
CONF_HEAVY_TESTS="${CONF_HEAVY_TESTS:-vm-lifecycle vm-data-survival vm-default-deny data-survival dr-restore node-loss}"
|
||||
|
||||
conf_is_heavy_dir() {
|
||||
# $1 = test dir; true if its basename is in the heavy set.
|
||||
|
||||
Reference in new issue
Block a user