test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience)

R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.

tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
  GIVEN a 3-replica Longhorn volume with written data attached on node N,
  WHEN  N is removed from service (cordon + evict the attached workload, forcing
        the volume to detach from N),
  THEN  a fresh workload on a SURVIVING node re-attaches the same volume and reads
        the canary intact — volume stays available, cluster reconverges.

Safe-by-construction for the 3-node embedded-etcd dev cluster:
  - Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
    evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
  - Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
    system pod is disturbed.
  - Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
    ALWAYS uncordons every node with THAT label and removes it — restored even on
    failure/timeout, and never touches an operator-cordoned node.
  - Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.

Bookkeeping:
  - run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
    (serial pass, matches the other Longhorn/KubeVirt L3 tests).
  - incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
    (cordon/uncordon/label); no eviction rights (no drain).
  - README.md / coverage.py docstring: refresh the documented-gap example.

Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.

NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-27 18:12:33 -05:00
1 parent d34c639269
commit fdf5c69adc
5 files changed
+233 -15

No files matched your search

+7 -4
View File
@@ -49,7 +49,6 @@ FORCE_NO_CLUSTER=0
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
# exemption is explicit and reviewable, not silent.
# R-STATE-02 (node-loss) — delivered with Longhorn/infra (WS-STORAGE).
# R-ENV-02 (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in
# coverage.md). The coarse path->requirement map keys
# clusters/* AND ci/* to it and the R-ENV-02 doc token
@@ -71,8 +70,12 @@ FORCE_NO_CLUSTER=0
# authoritative signal is the `conformance.home.k3s/R-*` label on a test, which still is.
# The PATH-MAP half remains: ci/touched-requirements.sh keys clusters/* and ci/* to
# R-ENV-02 + R-GITOPS-01 wholesale, which is why those two must stay exempted here. Tighten
# the path map (or scaffold the two requirements) and this list shrinks to R-STATE-02.
DOCUMENTED_GAPS="R-STATE-02 R-ENV-02 R-GITOPS-01"
# the path map (or scaffold the two requirements) and this list empties.
#
# R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
# but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
# longer a gap and has been removed from this list; the coverage report counts it as covered.
DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01"
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
@@ -154,7 +157,7 @@ select_dirs() {
# DIRECTORY basename (stable); override via CONF_HEAVY_TESTS to add/remove without code
# changes. Listed here even for tests still scaffolded (vm-lifecycle, dr-restore, ...) so
# the serialization is already correct the moment they grow real KubeVirt/Longhorn bodies.
CONF_HEAVY_TESTS="${CONF_HEAVY_TESTS:-vm-lifecycle vm-data-survival vm-default-deny data-survival dr-restore}"
CONF_HEAVY_TESTS="${CONF_HEAVY_TESTS:-vm-lifecycle vm-data-survival vm-default-deny data-survival dr-restore node-loss}"
conf_is_heavy_dir() {
# $1 = test dir; true if its basename is in the heavy set.