R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.
tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
GIVEN a 3-replica Longhorn volume with written data attached on node N,
WHEN N is removed from service (cordon + evict the attached workload, forcing
the volume to detach from N),
THEN a fresh workload on a SURVIVING node re-attaches the same volume and reads
the canary intact — volume stays available, cluster reconverges.
Safe-by-construction for the 3-node embedded-etcd dev cluster:
- Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
- Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
system pod is disturbed.
- Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
ALWAYS uncordons every node with THAT label and removes it — restored even on
failure/timeout, and never touches an operator-cordoned node.
- Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.
Bookkeeping:
- run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
(serial pass, matches the other Longhorn/KubeVirt L3 tests).
- incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
(cordon/uncordon/label); no eviction rights (no drain).
- README.md / coverage.py docstring: refresh the documented-gap example.
Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.
NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
TASK 1 — reliably single-pass green L2/L3:
Split run_chainsaw into a LIGHT pass (concurrent, config parallelism) and a
HEAVY pass (serial, parallel=1). The KubeVirt-CDI/Longhorn/DR tests
(vm-lifecycle, vm-data-survival, vm-default-deny, data-survival, dr-restore)
spike embedded-etcd on the small dev VMs; running them serially in their own
pass keeps their etcd spikes from ever coinciding with another test's setup —
the coincidence that let one heavy test's apiserver blip fail-close Kyverno
admission and abort every in-flight test at 0.01s. Heavy set is keyed by test
dir basename and overridable via CONF_HEAVY_TESTS. Each pass writes its own
JUnit; lib/merge_junit.py stitches them into the one stable report.junit.xml
CI collects, so the split is invisible downstream. Overall rc is the union of
both passes (no masking — real failures still surface).
Proof: full --layer l2,l3 run now completes single-pass green: 24/24 passed,
0 failures/errors/skipped. vpn-killswitch, which previously failed with an
apiserver EOF (etcd blip), passed; no mass environmental setup-aborts.
TASK 2 (T-4.1) — author conformance/tests/l3-fault-injection/alerting/
fixtures/smtp-sink.yaml: a hermetic Mailpit SMTP capture sink (Service
smtp-sink, SMTP 1025, HTTP API 8025) the alerting test applies as its mail
sink. restricted-PodSecurity compliant, in-memory (no PVC). Unblocks the
previously-aborting alerting scaffold; it now passes.
Touches conformance/ + ci-read content only — no apps/ or platform/ workload.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
VMs did not boot on the dev cluster. Three distinct live-verified causes,
all fixed GitOps-side:
1. containerDisk root failed at OCI init:
exec "/container-disk-binary/usr/bin/container-disk": no such file.
KubeVirt v1.9 supplies that binary to the volumerootdisk init container
via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA
`ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver
silently prunes the image volume + its mount -> binary missing -> crash.
(Proved: a server-side dry-run of a pod with an `image` volume drops it.)
Fix: convert the VM roots in all three tests to CDI DataVolumes imported
onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume
path). This is also the spec-required disk model (docs/06, R-STATE-01).
vm-lifecycle now PASSES green live (VMI Running on start + restart).
2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is
UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under
KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out ->
local-fs.target fails -> cloud-init runcmd/poweroff never runs.
Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests
and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI.
3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes
until one went NotReady and VMs were evicted). Cause: useEmulation:true
in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/
kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though
all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a
virt-launcher compute container = "No such file"; qemu accel = tcg.)
Fix: useEmulation:false so the kvm device is requested and VMs run on real
(nested) KVM. Requires Flux reconcile to take effect live (PM action).
The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller
(294 labels, hypervisor=true), device plugins, and VMI scheduling all work.
Also raised test exec timeouts to 30m (CDI import + boot) and, in
vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy
so the CDI importer's egress to quay.io is not severed by the policy.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Runnable, self-skipping (green until image.toolkit CRDs exist). Proves: an ImagePolicy
auto-advances to the newest STABLE tag on schedule with no edit and never a pre-release
(R-UPD-01/02, via podinfo ImageRepository/ImagePolicy status); a deliberately bad update
is health-gated and AUTO-ROLLS-BACK to the last-good revision while the old ReplicaSet
keeps serving, and the UpdateHealthGateFailed alert is wired (R-UPD-02/03); and an
exact-range (pinned) policy does not advance past its pin (R-UPD-03). Hermetic to the
chainsaw per-test namespace; block-style contains() checks per chainsaw v0.2.12.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
chainsaw v0.2.12 has no test-level `spec.finally` (finally exists only on a
step), and a step-level `finally` accepts only catch-style ops (script/command/
delete/wait), NOT `apply`/`assert`. Both tests failed to parse ("unknown field").
- egress-class: drop the redundant test-level `spec.finally` delete-pods block —
chainsaw auto-deletes applied fixtures on cleanup (skipDelete: false). The
valid step-level `finally` that restores the egress-class label is kept.
- vpn-killswitch: move the SAFETY-CRITICAL self-healing tunnel restart out of the
invalid `spec.finally` and into the STEP-level `finally` of the fault step
`when-force-tunnel-down-then-no-leak`, implemented as a `script` (finally can't
use apply/assert) that `kubectl apply`s the privileged node-pinned restart Job
(name `tunnel-restart`, distinct from `tunnel-stop`) and `kubectl wait`s for it
to complete — the equivalent of `assert succeeded:1`, blocking teardown until
the host confirms the tunnel is up. This finally runs even if the no-leak /
internet-unaffected asserts in the same step's `try` fail, so no path leaves the
gateway tunnel down. The recovery step is now poll-only (restart already
happened in the prior step's finally).
Validation with the pinned binary (~/.local/bin/chainsaw v0.2.12):
`chainsaw lint test --file ...` => "The document is valid" for both; `chainsaw
test ... --no-cluster` loads both past the parse stage (no "unknown field"), then
fails only at runtime for lack of a cluster (expected). yamllint + run.sh
--strict-coverage clean (RC=0). No live apply / fault injection, no spec edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for
R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress
Gateway with a host nftables kill-switch), replacing the obsolete in-cluster
gluetun design.
- fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented
RBAC/PodSecurity/namespace requirements for node-level fault injection.
- vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN);
WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged,
hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl
fails, never returns the WAN IP = no leak) while internet-class is unaffected;
WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an
idempotent start Job (self-healing) so a mid-test failure never leaves the
gateway tunnel down.
- incluster/rbac.yaml: grant the runner least-privilege for the delivered tests
(namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/
pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the
ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC).
Offline validation: python yaml parse (all docs), `run.sh --lint` (structural),
`run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/
07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Was a scaffold (fixture TODO, fault-injection a no-op echo). Implemented against the
live datapath: fixtures/allowed-pair.yaml = default-deny baseline + source + dest(:8080)
+ the sole explicit allow. The test verifies source->dest works (ALLOWED_OK), then
REVOKES both halves of the allow (real fault: kubectl delete networkpolicy), and asserts
connectivity fails CLOSED (FAIL_CLOSED_OK) while DNS still resolves (DNS_OK). Proves
default-deny is datapath-enforced and fails closed on revocation. DNS check asserts the
resolved answer, since busybox nslookup exits non-zero on a successful forward lookup.
PASSES (74.8s).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
My previous commit converted =~ to contains() but kept flow style
'check: { (contains($stdout, 'X')): true }' — the comma inside contains() is a
YAML flow-map separator, so 9 tests failed yamllint ('expected , or }'). Convert
those checks to block style (matching data-survival), where a comma in the key
scalar is fine. Gate PASS; data-survival still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Every chainsaw `check` used the regex operator `=~` (`($stdout =~ 'X'): true`),
which chainsaw's expression engine rejects ('SyntaxError: Unknown char: ~') — so
EVERY such test failed on the assertion even when the behavior passed. Proven with
data-survival (R-STATE-01): the Longhorn canary genuinely SURVIVED pod destruction
(DATA_SURVIVED_OK) but the test FAILED on the `=~` syntax. Converted all 10 tests to
JMESPath `contains($stdout, 'X')` (data-survival's alternation → contains||contains).
data-survival now PASSES; the other 9 (later-phase components) are syntactically fixed
for when their phases deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 4.8