test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience)

R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.

tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
  GIVEN a 3-replica Longhorn volume with written data attached on node N,
  WHEN  N is removed from service (cordon + evict the attached workload, forcing
        the volume to detach from N),
  THEN  a fresh workload on a SURVIVING node re-attaches the same volume and reads
        the canary intact — volume stays available, cluster reconverges.

Safe-by-construction for the 3-node embedded-etcd dev cluster:
  - Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
    evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
  - Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
    system pod is disturbed.
  - Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
    ALWAYS uncordons every node with THAT label and removes it — restored even on
    failure/timeout, and never touches an operator-cordoned node.
  - Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.

Bookkeeping:
  - run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
    (serial pass, matches the other Longhorn/KubeVirt L3 tests).
  - incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
    (cordon/uncordon/label); no eviction rights (no drain).
  - README.md / coverage.py docstring: refresh the documented-gap example.

Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.

NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-27 18:12:33 -05:00
1 parent d34c639269
commit fdf5c69adc
5 files changed
+233 -15

No files matched your search

+9 -7
View File
@@ -88,13 +88,15 @@ rules:
# namespaces policy is enforced, the `chainsaw-conf-*` test namespaces MUST be in
# its exclusion set (documented, audited exception — R-SEC-02).
#
# ---- R-STATE-02 (node loss): cordon/drain/uncordon a node (NOT in WS-TEST scope yet)
# - apiGroups: [""]
# resources: [nodes]
# verbs: [get, list, patch] # cordon = patch spec.unschedulable
# - apiGroups: [""]
# resources: [pods/eviction]
# verbs: [create] # drain
# ---- R-STATE-02 (node loss, tests/l3-fault-injection/node-loss): the fault is
# `kubectl cordon` + label of ONE node, reverted by `kubectl uncordon` + label removal
# in the test's `finally`. All three (cordon/uncordon/label) are a `patch` on the node
# object; `get`/`list` locate the victim + the finally cleanup selector. The test evicts
# only its OWN writer pod via `delete pod` (already granted above) — it deliberately does
# NOT `kubectl drain`, so no eviction rights and no reach over unrelated pods are needed.
- apiGroups: [""]
resources: [nodes]
verbs: [get, list, patch] # cordon/uncordon/label = patch spec.unschedulable + metadata.labels
#
# ---- R-UPD-02 (bad-update auto-rollback): inject a bad image tag via automation
# - apiGroups: [image.toolkit.fluxcd.io]