home-network-k3s-impl/conformance/runs/20260919T052225Z-9339d5a/run.log

[run.sh] Coverage report first (always)
[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
[run.sh] LIGHT pass: no tests selected.
[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
Version: 0.2.12
Loading config (/work/conformance/config/chainsaw-config.yaml)...
- Using test file: chainsaw-test
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
- SkipDelete false
- FailFast false
- ReportFormat 'JUNIT-TEST'
- ReportName 'chainsaw-report-heavy'
- ReportPath '/work/conformance/_artifacts'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 1m30s
- AssertTimeout 3m0s
- CleanupTimeout 2m0s
- DeleteTimeout 2m0s
- ErrorTimeout 1m0s
- ExecTimeout 3m0s
- DeletionPropagationPolicy Background
- Parallel 1
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
Loading values...
Running tests...
=== RUN   chainsaw
=== PAUSE chainsaw
=== CONT  chainsaw
=== RUN   chainsaw/node-loss
=== PAUSE chainsaw/node-loss
=== CONT  chainsaw/node-loss
    l.go:52: | 05:22:26 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
        === COMMAND
        /usr/bin/sh -c set -euo pipefail
        NS="$NAMESPACE"
        
        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
          echo NODE_LOSS_SKIPPED_OK
          exit 0
        fi
        
        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
        READY="$(kubectl get nodes \
          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
          | grep -c '^True$' || true)"
        echo "ready nodes: ${READY}"
        if [ "${READY:-0}" -lt 3 ]; then
          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
          echo NODE_LOSS_SKIPPED_OK
          exit 0
        fi
        
        CANARY="node-loss-canary-$(date +%s%N)"
        
        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
        #    writer pod that writes the canary and flushes it to the replicated volume. --
        kubectl -n "$NS" apply -f - <<EOF
        apiVersion: v1
        kind: PersistentVolumeClaim
        metadata:
          name: survivor
        spec:
          accessModes: [ReadWriteOnce]
          storageClassName: longhorn
          resources:
            requests:
              storage: 1Gi
        EOF
        
        apply_pod() {
          kubectl -n "$NS" apply -f - <<EOF
        apiVersion: v1
        kind: Pod
        metadata:
          name: $1
        spec:
          restartPolicy: Never
          containers:
            - name: app
              image: busybox:1.36.1
              command: ["sh", "-c", "sleep 3600"]
              volumeMounts:
                - name: data
                  mountPath: /data
          volumes:
            - name: data
              persistentVolumeClaim:
                claimName: survivor
        EOF
        }
        
        apply_pod writer
        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
        
        # The victim node is the one the attached volume currently lives on.
        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
        echo "victim node (currently hosts the attached volume): $VICTIM"
        
        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
        # runner is least-privilege and may not have longhorn.io read rights) — the
        # data-availability assertion below is the authoritative proof, not this.
        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
        if [ -n "$PVNAME" ]; then
          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
        fi
        
        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
        #    Label it with THIS test's unique namespace so `finally` can always find and
        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
        #    pods on the node are never touched.)
        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
        kubectl cordon "$VICTIM"
        echo "cordoned + labelled victim node: $VICTIM"
        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
        echo "evicted writer off the victim node; volume detached from $VICTIM"
        
        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
        #    the volume there from its replicas on the other nodes. --
        apply_pod reader
        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
        if [ "$READER_NODE" = "$VICTIM" ]; then
          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
          exit 1
        fi
        
        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
        echo "recovered='$GOT' expected='$CANARY'"
        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
        echo "volume available + data intact from a surviving node"
        
        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
        # (bounded) for robustness to settle back to available (healthy|degraded — both
        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
        # Non-fatal for the same least-privilege reason as the pre-fault log.
        if [ -n "$PVNAME" ]; then
          for i in $(seq 1 30); do
            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
            echo "post-fault robustness[$i]=$ROB"
            case "$ROB" in healthy|degraded) break ;; esac
            sleep 10
          done
        fi
        
        echo NODE_LOSS_VOLUME_AVAILABLE_OK
    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
        === STDOUT
        ready nodes: 3
        persistentvolumeclaim/survivor created
        pod/writer created
        pod/writer condition met
        wrote canary: node-loss-canary-1789795346371286043
        victim node (currently hosts the attached volume): k3s-prod-2
        pre-fault robustness=healthy replicas=3
        node/k3s-prod-2 labeled
        node/k3s-prod-2 cordoned
        cordoned + labelled victim node: k3s-prod-2
        pod "writer" deleted from chainsaw-advanced-boxer namespace
        evicted writer off the victim node; volume detached from k3s-prod-2
        pod/reader created
        pod/reader condition met
        reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
        recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
        volume available + data intact from a surviving node
        post-fault robustness[1]=healthy
        NODE_LOSS_VOLUME_AVAILABLE_OK
    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
        === COMMAND
        /usr/bin/sh -c set -uo pipefail
        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
          kubectl uncordon "${n#node/}" || true
          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
          echo "restored node to service: ${n#node/}"
        done
    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
        === STDOUT
        node/k3s-prod-2 uncordoned
        node/k3s-prod-2 unlabeled
        restored node to service: k3s-prod-2
    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
        === COMMAND
        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
        === STDOUT
        pod "reader" deleted from chainsaw-advanced-boxer namespace
        persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
        cleanup best-effort; chainsaw namespace teardown finalizes
    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
    l.go:52: | 05:24:21 | node-loss | @chainsaw                               | CLEANUP   | END   |
--- PASS: chainsaw (0.00s)
    --- PASS: chainsaw/node-loss (115.86s)
PASS
Tests Summary...
- Passed  tests 1
- Failed  tests 0
- Skipped tests 0
Done.
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md