docs(prod): node 1's DIMMs are not a matched pair

Surfaced by the new node_health DIMM reporting, unrelated to the memattr bug and NOT a fault: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity is right, but different rank organisation and rated speed mean the pair runs at the slower 2667. Recorded in §0 where the symmetry decision gets made, because §2 requires the three nodes to end up with identical layouts and the previous '2x16 GB Micron' note hid the mismatch. Serves: R-ENV-03. Task: T-7.1.
feat(node_health): detect the EFI memattr bug on every run, two independent ways
2026-09-01 22:31:26 -05:00 · 2026-09-01 22:30:45 -05:00 · 2026-09-01 22:16:22 -05:00 · 2026-09-01 22:14:14 -05:00 · 2026-09-01 22:10:07 -05:00 · 2026-09-01 21:54:12 -05:00
16 changed files with 2107 additions and 48 deletions
@@ -27,6 +27,17 @@ Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<pro

  *** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***

+AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
+------------------------------------------------------
+Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
+it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
+the log of every run of that test. A section-blind grep therefore reads a skip token out
+of the source of every skip-gated test and can never record `proven` for any of them: the
+ledger fails closed so completely that it records nothing and proves nothing. Tokens are
+consequently read from SECTIONS (see classify_sections): a pass token only from
+`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
+pass — within genuine output, which is the only place either was ever evidence.
+
 That is the fake-green this project refuses, and it would be worse than no ledger at all
 because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
 PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
@@ -149,10 +160,117 @@ _SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"
 # mis-classifying a SKIP token as a pass would manufacture one.
 _SKIP_TOKEN_RE = re.compile(r"_SKIP")

-# A Chainsaw log line header:  `| 20:41:32 | node-loss | step-name | TRY | RUN | ...`
-# Field 2 is the test name; it is how stdout is attributed back to a testcase.
+# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
+# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
+#     l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
+# Field 2 is the test name; it is how output is attributed back to a testcase. The
+# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
 _LOG_HEADER_RE = re.compile(
-    r"^\s*\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
+    r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
+    r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
+
+# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
+# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
+
+
+def _strip_noise(line: str) -> str:
+    return _ANSI_RE.sub("", line).replace("\b", "")
+
+
+# ===================================================================== output sections
+# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
+# -----------------------------------------------------------------------------
+# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
+# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
+# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
+# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
+# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
+# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
+# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
+# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
+# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
+#
+# So tokens are attributed to SECTIONS, never to raw lines:
+#     l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |     <- header, no section
+#         === COMMAND                                            <- marker (indent 8)
+#         /usr/bin/sh -c set -euo pipefail                       <- source, indent >= 8
+#           echo NODE_LOSS_SKIPPED_OK                            <- SOURCE, not output
+#     l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |    <- indent 4 => section ends
+#         === STDOUT                                             <- marker
+#         NODE_LOSS_VOLUME_AVAILABLE_OK                          <- the REAL emission
+#
+# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
+# (`=== RUN   chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
+# after the verb, so they match neither the indent nor the end-of-line anchor; they are
+# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
+# section that would swallow the whole log.
+_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
+_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
+
+# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
+# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
+# never echoes script source, so there is nothing to be confused by and the legacy
+# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
+# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
+# "" and are evidence for nothing.
+UNSECTIONED = "*"
+
+# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
+# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
+# never manufacture a proof.
+PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
+
+# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
+# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
+# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
+# one. Unknown section names therefore fail closed in both directions.
+SOURCE_SECTIONS = frozenset({"COMMAND"})
+
+
+def classify_sections(log_text: str) -> list[tuple[str, str]]:
+    """[(section, line)] for every line of a run log.
+
+    `section` is the enclosing `=== NAME` block, "" for a line inside no block, or
+    UNSECTIONED for every line when the log declares no sections at all. A block runs from
+    its marker until the first non-blank line indented LESS than the marker (blank lines do
+    not end it) or until the next marker; the marker line itself is content of nothing.
+    """
+    rows: list[tuple[str, str]] = []
+    cur, cur_indent, sectioned = "", 0, False
+    for raw in log_text.splitlines():
+        line = _strip_noise(raw)
+        m = _SECTION_MARKER_RE.match(line)
+        if m and m.group(2) not in _GO_TEST_MARKERS:
+            sectioned = True
+            cur, cur_indent = m.group(2), len(m.group(1))
+            rows.append(("", line))
+            continue
+        if cur:
+            if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
+                rows.append((cur, line))
+                continue
+            cur = ""
+        rows.append(("", line))
+    if not sectioned:
+        return [(UNSECTIONED, line) for _, line in rows]
+    return rows
+
+
+def scan_texts(lines) -> tuple[str, str]:
+    """(text a PASS token may be read from, text a SKIP token may be read from).
+
+    Accepts either [(section, line)] pairs from classify_sections() or a plain list of
+    strings, which is treated as an unsectioned log (legacy whole-log scanning).
+    """
+    pass_lines, skip_lines = [], []
+    for item in lines:
+        section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
+        if section in PASS_SECTIONS:
+            pass_lines.append(line)
+        if section and section not in SOURCE_SECTIONS:
+            skip_lines.append(line)
+    return "\n".join(pass_lines), "\n".join(skip_lines)

 _RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")

@@ -203,28 +321,32 @@ def sentinel_map(tests_dir: Path) -> dict[str, dict]:


 # =============================================================================== log
-def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[str]], bool]:
-    """Split a run log into per-testcase line buckets.
+def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
+                                                            bool]:
+    """Split a run log into per-testcase buckets of (section, line) pairs.

    Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
-    script's stdout underneath that header, so the test name is carried by a state machine:
-    a header naming a KNOWN test switches the current bucket, every following line lands in
-    it. Requiring the name to be known is what stops run.sh's own `[run.sh]` chatter (and
-    any other pipe-shaped line) from opening a bogus bucket.
+    script's sections underneath that header, so the test name is carried by a state
+    machine: a header naming a KNOWN test switches the current bucket, every following line
+    lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
+    chatter (and any other pipe-shaped line) from opening a bogus bucket.
+
+    Section labels come from classify_sections() and travel WITH the lines, so a bucket
+    still knows which of its lines were script source and which were real output.

    Returns (buckets, attributed) where `attributed` is False when the log contained no
    recognisable header at all — the signal for the degraded global scan in decide().
    """
-    buckets: dict[str, list[str]] = {}
+    buckets: dict[str, list[tuple[str, str]]] = {}
    cur: str | None = None
    attributed = False
-    for line in log_text.splitlines():
+    for section, line in classify_sections(log_text):
        m = _LOG_HEADER_RE.match(line)
        if m and m.group(1) in known:
            cur = m.group(1)
            attributed = True
        if cur is not None:
-            buckets.setdefault(cur, []).append(line)
+            buckets.setdefault(cur, []).append((section, line))
    return buckets, attributed


@@ -253,7 +375,7 @@ def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
      1. JUnit failure/error            -> failed        (a red run is recorded as red)
      2. JUnit <skipped/>               -> junit-skipped  (never green; ci/lib.sh rule)
      3. ANY declared skip token seen   -> skipped        (BEATS a green JUnit and beats
-                                                           the pass token: if any gated
+           IN REAL OUTPUT                                  the pass token: if any gated
                                                           step self-skipped, the run did
                                                           not prove the requirement)
      4. no sentinel declared           -> proven only for L0/L1, whose skips are already
@@ -264,6 +386,13 @@ def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
    Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
    asserts `pass || skip`, so a missing pass token means a step did not reach its proven
    branch, whatever the JUnit says.
+
+    Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
+    only in `=== STDOUT`, a skip token counts in any section that is not script source.
+    That narrows WHERE a token is read from and changes nothing about what it means — skip
+    still outranks pass, and it outranks it within genuine output, which is the only place
+    either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
+    strings (an unsectioned log; see scan_texts / UNSECTIONED).
    """
    if junit_result in ("failed", "error"):
        return (None, "failed", f"JUnit recorded {junit_result}")
@@ -271,12 +400,12 @@ def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
        return (None, "junit-skipped",
                "JUnit <skipped/> — the test did not execute; a skip is never green")

-    text = "\n".join(lines)
-    seen_skip = [t for t in sent["skip"] if t in text]
+    pass_text, skip_text = scan_texts(lines)
+    seen_skip = [t for t in sent["skip"] if t in skip_text]
    if seen_skip:
        return (seen_skip[0], "skipped",
-                f"run log carries the SELF-SKIP token {seen_skip[0]!r} — the test exited 0 "
-                f"without exercising the behaviour, so this is NOT proof")
+                f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
+                f"exited 0 without exercising the behaviour, so this is NOT proof")

    if not sent["pass"] and not sent["skip"]:
        if layer not in SENTINEL_EXEMPT_LAYERS:
@@ -288,14 +417,14 @@ def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
                f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
                f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")

-    missing = [t for t in sent["pass"] if t not in text]
+    missing = [t for t in sent["pass"] if t not in pass_text]
    if not missing:
        return (sent["terminal_pass"], "proven",
-                f"run log carries every declared pass token "
+                f"run log OUTPUT carries every declared pass token "
                f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
    return (None, "unknown-sentinel-absent",
-            f"run log lacks pass token(s) {', '.join(missing)} for this test [{scope}] — "
-            f"refusing to infer proof from the JUnit alone")
+            f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
+            f"[{scope}] — refusing to infer proof from the JUnit alone")


 # ============================================================================= junit
@@ -387,9 +516,13 @@ def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
    junit = run_dir / "report.junit.xml"
    log = run_dir / "run.log"
    smap = sentinel_map(tests_dir)
-    buckets, attributed = attribute_log(log.read_text(errors="replace"), set(smap))
+    log_text = log.read_text(errors="replace")
+    buckets, attributed = attribute_log(log_text, set(smap))
    uniq = unique_tokens(smap)
-    all_lines = log.read_text(errors="replace").splitlines()
+    all_pairs = classify_sections(log_text)
+    sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
+    section_note = ("tokens read from real-output sections only"
+                    if sectioned else "log declares no === sections; whole-log scan")
    ts = run_timestamp(run_dir, junit)

    rows: list[dict] = []
@@ -403,17 +536,18 @@ def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
        reqs = sent.get("requirements") or case["requirements"]

        if name in buckets:
-            lines, scope = buckets[name], "log scoped to this testcase"
+            lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
        elif not attributed:
            # Degraded mode: the log had no recognisable Chainsaw header (a redirected or
            # reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
            # exactly one test, so a token another test emitted can never prove this one.
-            lines = all_lines
+            lines = all_pairs
            sent = dict(sent,
                        **{"pass": [t for t in sent["pass"] if t in uniq],
                           "skip": [t for t in sent["skip"] if t in uniq]})
            sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
-            scope = "DEGRADED: unattributable log, globally-unique tokens only"
+            scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
+                     + section_note)
        else:
            lines, scope = [], "no log lines attributed to this testcase"

@@ -552,6 +686,98 @@ def _selftest() -> int:
    check("an unknown/blank layer fails CLOSED — sentinel required",
          proof == "unknown-no-sentinel-declared")

+    # --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
+    # Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
+    # `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
+    # inside `=== COMMAND` on every run of a skip-gated test.
+    nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
+          "skip": ["NODE_LOSS_SKIPPED_OK"],
+          "terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+    _CMD = ("    l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |\n"
+            "        === COMMAND\n"
+            "        /usr/bin/sh -c set -euo pipefail\n"
+            "        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
+            "          echo NODE_LOSS_SKIPPED_OK\n"
+            "\n"
+            "          exit 0\n"
+            "        fi\n"
+            "        echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
+
+    ran = classify_sections(_CMD + "        === STDOUT\n"
+                            "        ready nodes: 3\n"
+                            "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
+    sent, proof, _ = decide("L3", "passed", nl, ran, "t")
+    check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
+          proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+    selfskipped = classify_sections(_CMD + "        === STDOUT\n"
+                                    "        SKIP: 'longhorn' StorageClass absent\n"
+                                    "        NODE_LOSS_SKIPPED_OK\n")
+    sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
+    check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
+          proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
+
+    neither = classify_sections(_CMD + "        === STDOUT\n        ready nodes: 3\n")
+    _, proof, _ = decide("L3", "passed", nl, neither, "t")
+    check("both tokens in COMMAND and neither in STDOUT is NOT proven",
+          proof == "unknown-sentinel-absent")
+
+    both_out = classify_sections("        === STDOUT\n"
+                                 "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                                 "        NODE_LOSS_SKIPPED_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, both_out, "t")
+    check("skip still BEATS pass when both are in genuine output", proof == "skipped")
+
+    stderr_skip = classify_sections("        === STDERR\n        NODE_LOSS_SKIPPED_OK\n"
+                                    "        === STDOUT\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
+    check("a skip token in a non-source section (STDERR) still costs the proof",
+          proof == "skipped")
+
+    stderr_pass = classify_sections("        === STDERR\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
+    check("a pass token OUTSIDE stdout cannot prove anything",
+          proof == "unknown-sentinel-absent")
+
+    # --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
+    gorows = classify_sections("=== RUN   chainsaw\n=== PAUSE chainsaw\n"
+                               "=== CONT  chainsaw/node-loss\n"
+                               "        === COMMAND\n        echo NODE_LOSS_SKIPPED_OK\n"
+                               "    l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
+                               "--- PASS: chainsaw/node-loss (131.09s)\n")
+    secs = [sec for sec, _ in gorows]
+    check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
+          secs[:3] == ["", "", ""])
+    check("a section covers its indented body and ends at a less-indented line",
+          secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
+    check("a log with no === markers stays wholly scannable (UNSECTIONED)",
+          {sec for sec, _ in classify_sections("plain line\n  another\n")} == {UNSECTIONED})
+
+    # --- the preserved run itself, as a fixture ----------------------------------------
+    # runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
+    # (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
+    fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
+    if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
+        rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
+        row = next((r for r in rows if r["test"] == "node-loss"), None)
+        check("preserved real L3 run records node-loss/R-STATE-02 as proven",
+              row is not None and row["proof"] == "proven"
+              and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
+              and row["requirements"] == ["R-STATE-02"])
+        check("preserved real L3 run attributes (no DEGRADED fallback needed)",
+              row is not None and "DEGRADED" not in row["note"])
+        pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
+        secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
+        check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
+              secs == {"COMMAND"})
+        secs = {sec for sec, line in pairs
+                if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+        check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
+
    # --- log attribution --------------------------------------------------------------
    log = ("[run.sh] Coverage report first\n"
           "    | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
@@ -561,10 +787,22 @@ def _selftest() -> int:
    b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
    check("log lines are attributed to the testcase whose header preceded them",
          attributed
-          and any("NODE_LOSS_SKIPPED_OK" in l for l in b["node-loss"])
-          and not any("NODE_LOSS_SKIPPED_OK" in l for l in b["dr-restore"]))
+          and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
+          and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
    check("a pass token does NOT leak across testcase boundaries",
-          not any("DR_RESTORED" in l for l in b["node-loss"]))
+          not any("DR_RESTORED" in l for _, l in b["node-loss"]))
+    check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
+          all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
+
+    # Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
+    # regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
+    real_hdr = ("    l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
+                "\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
+                "        === STDOUT\n        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
+    check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
+          attributed_real and "node-loss" in b2
+          and ("STDOUT", "        NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
    _, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
    check("an unattributable log is reported as such (degraded mode)", not attributed2)

@@ -108,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io"
 #   conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there.
 #   ci/tests          — gate unit-test fixtures embed example R-IDs as DATA.
 #   registry.yaml     — the requirement mirror itself, not an implementation of anything.
-IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/")
+# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the
+# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as
+# implementing evidence makes `last_change` move every time proof is recorded, so a row's
+# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is
+# committed — recording the proof destroys it. The ledger is the proof record; it must not
+# participate in the freshness computation about itself. (Found live 2026-09-01 on the first
+# PROVEN row.)
+IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/")
 IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)


@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="131.090121" />
+  </testsuite>
+</testsuites>
@@ -0,0 +1,232 @@
+[1m[run.sh][0m Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[1m[run.sh][0m LIGHT pass: no tests selected.
+[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 23:51:18 | [32;1mnode-loss[0;22m | [32;1m@chainsaw                              [0;22m | [32;1mCREATE[0;22m    | [32;1mOK[0;22m    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:51:18 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m       | [36;1mBEGIN[0;22m |
+    l.go:52: | 23:51:18 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1788306678898809444
+        victim node (currently hosts the attached volume): k3s-dev-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-dev-2 labeled
+        node/k3s-dev-2 cordoned
+        cordoned + labelled victim node: k3s-dev-2
+        pod "writer" deleted from chainsaw-fond-marlin namespace
+        evicted writer off the victim node; volume detached from k3s-dev-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
+        recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m       | [36;1mEND[0;22m   |
+    l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m   | [36;1mBEGIN[0;22m |
+    l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        node/k3s-dev-2 uncordoned
+        node/k3s-dev-2 unlabeled
+        restored node to service: k3s-dev-2
+    l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-fond-marlin namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m   | [36;1mEND[0;22m   |
+    l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1m@chainsaw                              [0;22m | [36;1mCLEANUP[0;22m   | [36;1mBEGIN[0;22m |
+    l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1m@chainsaw                              [0;22m | [32;1mDELETE[0;22m    | [32;1mOK[0;22m    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:53:29 | [36;1mnode-loss[0;22m | [36;1m@chainsaw                              [0;22m | [36;1mCLEANUP[0;22m   | [36;1mEND[0;22m   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (131.09s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="149.053701" />
+  </testsuite>
+</testsuites>
@@ -0,0 +1,671 @@
+[1m[run.sh][0m Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[1m[run.sh][0m LIGHT pass: no tests selected.
+[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 00:19:07 | [32;1mdr-restore[0;22m | [32;1m@chainsaw                                [0;22m | [32;1mCREATE[0;22m    | [32;1mOK[0;22m    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:19:07 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster      [0;22m | [36;1mTRY[0;22m       | [36;1mBEGIN[0;22m |
+    l.go:52: | 00:19:07 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster      [0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster      [0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true http://192.168.1.9:9000 velero
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=velero  endpoint=http://192.168.1.9:9000
+        backup endpoint host: 192.168.1.9
+        backup endpoint resolves to: 192.168.1.9
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 81)
+        endpoint is not any Pod IP (checked 186)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=192.168.1.9 bucket=velero
+    l.go:52: | 00:19:11 | [32;1mdr-restore[0;22m | [32;1massert-backup-target-is-off-cluster      [0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster      [0;22m | [36;1mTRY[0;22m       | [36;1mEND[0;22m   |
+    l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled             [0;22m | [36;1mTRY[0;22m       | [36;1mBEGIN[0;22m |
+    l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled             [0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled             [0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 00:19:11 | [32;1mdr-restore[0;22m | [32;1massert-backups-are-scheduled             [0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled             [0;22m | [36;1mTRY[0;22m       | [36;1mEND[0;22m   |
+    l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m       | [36;1mBEGIN[0;22m |
+    l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
+        namespace/chainsaw-cunning-hen-dr-src created
+        namespace/chainsaw-cunning-hen-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1788308352223263220
+        original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
+        backup.velero.io/chainsaw-cunning-hen-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=Completed
+        backup chainsaw-cunning-hen-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-cunning-hen-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-cunning-hen-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
+        recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 00:21:16 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m       | [36;1mEND[0;22m   |
+    l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m   | [36;1mBEGIN[0;22m |
+    l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
+        velero objects for chainsaw-cunning-hen cleaned up (ns velero)
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
+        released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 00:21:28 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m   | [36;1mEND[0;22m   |
+    l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1m@chainsaw                                [0;22m | [36;1mCLEANUP[0;22m   | [36;1mBEGIN[0;22m |
+    l.go:52: | 00:21:28 | [32;1mdr-restore[0;22m | [32;1m@chainsaw                                [0;22m | [32;1mDELETE[0;22m    | [32;1mOK[0;22m    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:21:36 | [36;1mdr-restore[0;22m | [36;1m@chainsaw                                [0;22m | [36;1mCLEANUP[0;22m   | [36;1mEND[0;22m   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (149.05s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
@@ -0,0 +1,2 @@
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}
@@ -25,20 +25,187 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
 |---|---|---|---|
 | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
 | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
-| Model | HP EliteDesk 800 G5 Desktop Mini | | |
-| CPU | i7-9700T, 8c/8t, VT-x | | |
-| RAM | 31 GiB | | |
-| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | |
-| OS | Debian 13 (trixie), kernel 6.12 | | |
-| Firmware | HP R21 Ver. 02.26.00 | | |
-| `/dev/kvm` | present | | |
+| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | |
+| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | |
+| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | |
+| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | |
+| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | |
+| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | |
+| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | |
+| `/dev/kvm` | present | present | |
+| Serial | MXL1052NGN | MXL0073949 | |
+
+**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check;
+unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**),
+`DIMM3` a Micron **16**ATF2G64HZ-**2G6**E1 (DDR4-**2667**). Both are 16 GB so capacity is correct,
+but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded
+only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch.
+
+**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a
+healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica
+placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's
+NVMe is the drive node 1 should have had.

 Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).

 ---

+## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01)
+
+> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory
+> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine.
+> Keeping the correction visible because the wrong framing nearly cost money and a board RMA.
+
+`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware
+publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before
+reserving memory based on its contents.** Fixed upstream in **6.13**.
+
+### The evidence that breaks it open
+
+The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full
+15.9 GiB two boots earlier**:
+
+| Boot | `Memory: … available` | reserved | memattr line |
+|---|---|---|---|
+| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` |
+| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
+| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
+
+The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in
+that shape, and none is fixed by a reboot.
+
+### Mechanism (verified against kernel source)
+
+`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`:
+
+```c
+if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; }
+tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
+memblock_reserve(efi_mem_attr_table, tbl_size);
+```
+
+**`num_entries` and `desc_size` are never validated.** With a corrupt table:
+- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)*
+- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s
+  the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)*
+
+That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` =
+DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero.
+
+### BOTH prod nodes are affected — node 1 has only been lucky
+
+| Node | memattr version field | Outcome |
+|---|---|---|
+| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** |
+| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB |
+| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB |
+
+Node 1 is one unlucky boot away from the identical fault. **Fix both.**
+
+### Mechanism, traced to the byte (second, independent confirmation)
+
+A second research pass reached the same root cause by a different route and pinned the arithmetic:
+
+- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The
+  kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that
+  `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it.
+- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and
+  when bit 31 lands the result is **negative**.
+- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to
+  ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical
+  address above `0xbeeb0018` is reserved.**
+- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages`
+  (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`.
+- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by
+  `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is
+  therefore positive proof the reservation already happened** — it is a consequence, not the cause.
+
+The cleanest A/B tell between the two nodes is one line of dmesg:
+
+| | `NODE_DATA(0) allocated` | meaning |
+|---|---|---|
+| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation |
+| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement |
+
+Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
+node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.
+
+### THERE IS NO KERNEL CMDLINE WORKAROUND
+
+`efi_memattr_init()` is called from `efi_config_parse_tables()` under only
+`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()`
+**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap`
+**do not prevent the reservation.** Booting a fixed kernel is the only real fix.
+
+### The fix
+
+**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is
+clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message
+describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable
+branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints
+`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory.
+
+The hardening landed exactly one release too late for Debian 13: the bug has existed since the
+feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up
+`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim
+*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably
+lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the
+release lifetime, and the fix is small and already upstream.
+
+Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382)
+(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the
+table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook
+Studio G5**, 16 GiB → 1.5 GiB).
+
+### What NOT to do
+
+- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is
+  a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be
+  installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6.
+- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full
+  R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries.
+- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket:
+  they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They
+  may appear to "fix" it. They are not fixes.
+
+### Hypotheses killed, and why — so they are not re-run
+
+- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by
+  an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP
+  documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control.
+- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by
+  `hp-bioscfg` on this platform.
+- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB.
+- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by
+  measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`.
+  Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never
+  `memblock_add()`ed — which would have left `present` at ~0, not 3340288.
+- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`.
+- **MTRR trim** — MTRRs near-identical to node 1, no trim warning.
+- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal.
+- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly
+  explained: everything from `0xbeeb0018` upward.
+
 ## 1. Firmware / BIOS (per node, before OS install)

+> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes
+> **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a
+> `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes",
+> which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
+> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye.
+>
+> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4,
+> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is
+> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means
+> F10 on the box.
+>
+> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is
+> achievable today with reads alone, and would have caught node 2's drift on its first health run.
+> Converging it automatically additionally requires provisioning a BIOS Setup Password on every
+> node, which is a real security decision (a stored credential that can rewrite firmware) and is
+> the operator's call, not a default.
+
+
 - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
      boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
 - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
@@ -50,6 +217,80 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di
      nodes match.
 - [ ] **Firmware version recorded** in §0 and consistent across all three nodes.

+### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01
+
+**There is no default BIOS password, and no factory default exists to look up.** HP business
+desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we
+observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5
+describes both purely as things the owner configures in Computer Setup.
+
+**Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents
+clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once).
+**Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely,
+turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The
+permissive default is the right setting for a homelab.
+
+#### CORRECTION — error `0x4` is NOT an authorisation failure
+
+I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That
+was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI
+`SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a
+different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have
+unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting`
+returns:
+
+| Code | Meaning |
+|---|---|
+| 0 | Success |
+| 1 | Not Supported |
+| **4** | **Failed** (generic) |
+| 6 | Access Denied |
+
+So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and
+the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither
+of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up
+in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`.
+
+**Most likely actual cause (inferred, testable):** with an empty password the driver emits an
+*untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when
+blank (`Password = "<utf-16/>"`). The driver only prepends that prefix for a *non-empty* string. So
+setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would
+work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this
+firmware already makes the driver log `Prerequisites size value exceeded the maximum number of
+elements supported or data may be malformed` at boot.
+
+**`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and
+`new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM;
+nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read.
+
+#### Recommendation
+
+1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff
+   `current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV
+   win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10.
+2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision
+   one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to
+   `current_password`, retry one benign attribute. Return `0` confirms the `<utf-16/>` theory; `6`
+   would mean a password mismatch; another `4` means the driver is incompatible with R21 and the
+   fallback is `hp-repsetup`.
+3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup
+   Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it
+   needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**.
+   Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a
+   security downgrade.
+4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in
+   principle — the private key never leaves the workstation, each change is individually signed, with
+   anti-replay — but provisioning and every signature require Windows-only tooling. For three
+   machines whose firmware settings change roughly never, the operational cost far exceeds the risk
+   reduction. BCU is Windows/WinPE-only.
+5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full
+   pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) —
+   functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via
+   `hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is
+   persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the
+   password in cleartext on the command line — it would land in process listings and shell history
+   unless wrapped. Store it in the Ansible vault **and** offline.
+
 ## 2. Disk layout — **OPEN DECISION, blocks the build**

 `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
@@ -87,11 +87,13 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10

 [k3s_prod_secondary]
-# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md
-# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so
-# no play can accidentally target a host that isn't there; uncomment each line only once
-# that node is actually on the bench and answering SSH.
-# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
+# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
+# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
+# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
+# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
+# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
+# node 3 remains commented until it is racked.
+k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
 # k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12

 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
@@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI,
  EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified
  against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM).
- **Never `become`s unless it must.** Only SMART needs root.
+- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring
+  buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM
+  inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and
+  the privileged second pass re-runs the *same* collector with `--privileged` so the
+  parsing has one home. Without escalation those checks SKIP with the exact reason.

 ## What it checks

@@ -44,15 +48,51 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
 | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
+| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
 | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
+| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |

 Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
 in task logic.

+
+## The EFI memattr check
+
+`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
+
+HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
+does not validate it before reserving memory from its contents —
+`drivers/firmware/efi/memattr.c` multiplies two never-validated fields
+(`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the
+table address to the top of RAM. Which of two kernel-log lines appears decides whether the
+node keeps its memory, and it is **decided afresh on every boot**:
+
+| kernel log | meaning | verdict |
+|---|---|---|
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table |
+| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working |
+| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS |
+| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected |
+| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected |
+
+The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and
+differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before
+it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing.
+
+**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
+`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
+workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
+rebooting/CMOS/BIOS/USB changes only move where the table lands.
+
+`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node
+whose kernel log cannot be read is still not unmeasured — and a future memory loss with a
+different cause is still caught.
+
 ## Output

 1. A human summary printed in the play output.
@@ -45,6 +45,33 @@ node_health_thresholds:
  # Memory (percentage of MemTotal still available)
  mem_available_pct_warn: 15
  mem_available_pct_fail: 5
+  # --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ----------------------------
+  # Two independent measurements of the same question, both fed by memory.physical_total:
+  #
+  #   unmanaged  /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own
+  #              memory map says exists, that never reached the allocator. Needs no root
+  #              and no tools, works identically on metal and in a VM.
+  #   shortfall  MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is
+  #              installed. Needs root, and only bare metal reports it. Independent of the
+  #              kernel's memory map entirely, so it also catches RAM the firmware never
+  #              handed the kernel in the first place.
+  #
+  # MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and
+  # not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2%
+  # k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state.
+  # The healthy figure is the kernel image, memmap and firmware reservations; it is a
+  # near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10%
+  # keeps clearance for that. There is no plausible tuning of a threshold in this band
+  # that changes any of those five verdicts.
+  #
+  # The DIMM shortfall runs a little higher than the unmanaged figure on the same node
+  # (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and
+  # the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the
+  # kernel's to manage. Same bands: the signal being caught is 83%, not 3%.
+  mem_unmanaged_pct_warn: 10
+  mem_unmanaged_pct_fail: 25
+  mem_dimm_shortfall_pct_warn: 10
+  mem_dimm_shortfall_pct_fail: 25
  # Temperatures (deg C, any sensor)
  temp_warn_c: 70
  temp_fail_c: 85
@@ -97,10 +97,22 @@ def read_kv(path, sep=":"):
    return out


+def which(tool):
+    """Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits.
+
+    `dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the
+    unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed
+    but unreachable would be a false reason on a SKIP, which is the one thing this
+    collector may never do."""
+    return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+
+
 def run(argv, timeout=20):
    """Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent."""
-    if shutil.which(argv[0]) is None:
+    exe = which(argv[0])
+    if exe is None:
        return None, "", "%s not installed" % argv[0]
+    argv = [exe] + list(argv[1:])
    try:
        proc = subprocess.run(
            argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout
@@ -284,9 +296,285 @@ def collect_memory():
        edac["ce_total"] = ce_total
        edac["ue_total"] = ue_total
    mem["edac"] = edac
+    # Physically present vs actually usable — the cause-independent memory-loss signal.
+    mem["zones"] = collect_zoneinfo(mem["mem_total_kb"])
+    # Firmware's own inventory of what is physically installed. Root-only; attempted
+    # here anyway so the unprivileged pass records the exact reason it could not read.
+    mem["devices"] = collect_memory_devices()
    return mem


+# --- Memory the kernel has but never hands out --------------------------------------
+# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
+# no DMI. Per zone it reports:
+#   spanned  pages in the zone's address range, holes included
+#   present  spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says
+#            physically exists here
+#   managed  present MINUS everything memblock reserved before the buddy allocator was
+#            handed the rest (kernel image, memmap, initrd, crashkernel, firmware
+#            reservations). sum(managed) IS MemTotal, exactly, on every node measured.
+# So `present - managed` is, by construction, "RAM this machine has that the OS will
+# never allocate", whatever the reason. On a healthy node it is the couple of hundred
+# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and
+# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% —
+# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude
+# separation, which is why this needs no clever tuning to be trustworthy.
+#
+# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if
+# a future firmware, kernel or hypervisor lost the memory for an entirely different
+# reason, and it needs neither root nor a parseable kernel log.
+def collect_zoneinfo(mem_total_kb):
+    raw = read_text("/proc/zoneinfo", optional=False)
+    page_kb = os.sysconf("SC_PAGE_SIZE") // 1024
+    out = {
+        "available": raw is not None,
+        "reason": None if raw is not None else "/proc/zoneinfo could not be read",
+        "page_size_kb": page_kb,
+        "zones": [],
+        "present_kb": None,
+        "managed_kb": None,
+        "unmanaged_kb": None,
+        "unmanaged_pct": None,
+    }
+    if raw is None:
+        return out
+    node = zone = None
+    cur = {}
+    present_pages = managed_pages = 0
+    for line in raw.splitlines():
+        fields = line.split()
+        if line.startswith("Node ") and "zone" in line:
+            # "Node 0, zone   Normal"
+            if cur:
+                out["zones"].append(cur)
+            node, zone = fields[1].rstrip(","), fields[-1]
+            cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None}
+        elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"):
+            try:
+                cur[fields[0]] = int(fields[1])
+            except ValueError:
+                cur[fields[0]] = None
+    if cur:
+        out["zones"].append(cur)
+    # An empty parse is a PARSE failure, not "this node has no memory" — say so and stay
+    # null rather than emitting a confident 0.
+    if not out["zones"]:
+        out["available"] = False
+        out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it"
+        return out
+    for z in out["zones"]:
+        present_pages += z["present"] or 0
+        managed_pages += z["managed"] or 0
+    out["present_kb"] = present_pages * page_kb
+    out["managed_kb"] = managed_pages * page_kb
+    out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"]
+    if out["present_kb"] > 0:
+        out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1)
+    # Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the
+    # two readings disagree and the discrepancy itself is worth showing.
+    out["matches_mem_total"] = (
+        None if (mem_total_kb is None or out["managed_kb"] is None)
+        else abs(out["managed_kb"] - mem_total_kb) <= page_kb
+    )
+    return out
+
+
+# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) -----------------------------------
+# The independent answer to "how much RAM is physically in this box", from the firmware's
+# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads
+# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that
+# failure is recorded as the SKIP reason — never as zero DIMMs.
+#
+# On a VM there is usually either no type-17 structure at all or a single synthetic entry
+# that merely restates the configured guest RAM; both are handled by reporting what was
+# found and letting the check SKIP when nothing populated was reported.
+DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def collect_memory_devices():
+    out = {
+        "available": False,
+        "reason": None,
+        "devices": [],
+        "slots": 0,
+        "populated_slots": 0,
+        "installed_kb": None,
+    }
+    if which("dmidecode") is None:
+        out["reason"] = ("dmidecode is not installed on this node; it ships with "
+                         "roles/node_baseline (node_baseline_packages). "
+                         "This role never installs packages.")
+        return out
+    rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30)
+    if rc != 0:
+        out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % (
+            rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output")
+        return out
+    # Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which
+    # may or may not hold a module ("Size: No Module Installed").
+    total_kb = 0
+    saw_size = False
+    for block in stdout.split("\n\n"):
+        if "Memory Device" not in block:
+            continue
+        fields = {}
+        for line in block.splitlines():
+            key, sep, value = line.partition(":")
+            if sep:
+                fields[key.strip()] = value.strip()
+        if "Size" not in fields:
+            continue
+        out["slots"] += 1
+        size_kb = None
+        parts = fields["Size"].split()
+        if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS:
+            try:
+                size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()]
+            except ValueError:
+                size_kb = None
+        if size_kb:
+            saw_size = True
+            total_kb += size_kb
+            out["populated_slots"] += 1
+        out["devices"].append(
+            {
+                "locator": fields.get("Locator"),
+                "bank_locator": fields.get("Bank Locator"),
+                # Verbatim, so an unparsed spelling is visible rather than silently null.
+                "size_raw": fields["Size"],
+                "size_kb": size_kb,
+                "type": fields.get("Type"),
+                "speed": fields.get("Speed"),
+                "configured_speed": fields.get("Configured Memory Speed"),
+                "manufacturer": fields.get("Manufacturer"),
+                "part_number": fields.get("Part Number"),
+                "rank": fields.get("Rank"),
+                # Serial deliberately NOT collected: the artifact already carries drive
+                # serials and a DIMM serial adds identifying data no check reads.
+            }
+        )
+    out["available"] = saw_size
+    if not saw_size:
+        out["reason"] = ("dmidecode ran but reported no populated memory device "
+                         "(%d slot structure(s) found) — typical of a VM with no SMBIOS "
+                         "type 17 inventory" % out["slots"])
+    else:
+        out["installed_kb"] = total_kb
+    return out
+
+
+# --- Kernel ring buffer: the EFI Memory Attributes Table ----------------------------
+# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative
+# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
+# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
+# the product). The kernel prints exactly one of two lines, and WHICH one decides whether
+# the node keeps its RAM:
+#
+#   "Failed to map EFI Memory Attributes table @ 0x..."   the table's garbage version read
+#       <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW.
+#   "Unexpected EFI Memory Attributes table version <N>"  the garbage version read > 2, so
+#       the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
+#       firmware is still publishing a corrupt table and the outcome is decided afresh by
+#       where the table lands on every boot. That is luck, not immunity.
+#
+# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots
+# before it lost 13.3 GiB), which is precisely why it is detected on every health run
+# instead of diagnosed once.
+#
+# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
+# every node here). The unprivileged pass therefore records WHY it could not read, and
+# roles/node_health re-runs this collector with `--privileged` when escalation works.
+BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:")
+EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
+MEMATTR_FAILED_RE = re.compile(
+    r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
+MEMATTR_VERSION_RE = re.compile(
+    r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I)
+MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
+# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
+# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
+# captured separately, above, and quoting it as evidence would bury the one line that
+# matters in 200 characters of unrelated table addresses.
+MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I)
+# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line
+# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict.
+MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available")
+MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved")
+
+
+def collect_klog():
+    out = {
+        "available": False,
+        "reason": None,
+        "boot_covered": None,
+        "memattr_state": None,
+        "memattr_published": None,
+        "memattr_table_address": None,
+        "memattr_version": None,
+        "memattr_lines": [],
+        "memory_line": None,
+    }
+    rc, stdout, stderr = run(["dmesg"], timeout=30)
+    if rc is None:
+        out["reason"] = "dmesg is not installed on this node"
+        return out
+    if rc != 0:
+        detail = (stderr or stdout).strip().splitlines()
+        out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % (
+            rc, os.geteuid(), detail[-1] if detail else "no output",
+            " — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG"
+            if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "")
+        return out
+
+    out["available"] = True
+    # Does the buffer still reach back to early boot? The ring buffer WRAPS, and every
+    # memattr line is printed in the first fraction of a second. If the boot header is
+    # gone then so may be a memattr line that was printed — absence would prove nothing,
+    # and the check must SKIP rather than PASS. This is the difference between
+    # "unaffected" and "could not tell", and the role is not allowed to blur it.
+    out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout))
+
+    published = EFI_CONFIG_RE.search(stdout)
+    if published:
+        out["memattr_published"] = True
+        out["memattr_table_address"] = published.group(1)
+    elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M):
+        # UEFI booted and the config-table line is present, but it lists no MEMATTR=.
+        out["memattr_published"] = False
+    # else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None.
+
+    for line in stdout.splitlines():
+        if MEMATTR_LINE_RE.search(line):
+            out["memattr_lines"].append(line.strip())
+    joined = "\n".join(out["memattr_lines"])
+    if MEMATTR_CORRUPT_RE.search(joined):
+        # Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected.
+        out["memattr_state"] = "corrupt_rejected"
+    elif MEMATTR_FAILED_RE.search(joined):
+        out["memattr_state"] = "failed_to_map"
+        addr = MEMATTR_FAILED_RE.search(joined).group(1)
+        if addr:
+            out["memattr_table_address"] = addr
+    elif MEMATTR_VERSION_RE.search(joined):
+        out["memattr_state"] = "bad_version"
+        out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1))
+    else:
+        out["memattr_state"] = "clean"
+
+    mem_line = MEMORY_LINE_RE.search(stdout)
+    if mem_line:
+        full = [l for l in stdout.splitlines() if mem_line.group(0) in l]
+        reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None
+        out["memory_line"] = {
+            "available_kb": int(mem_line.group(1)),
+            "total_kb": int(mem_line.group(2)),
+            "reserved_kb": int(reserved.group(1)) if reserved else None,
+            "raw": full[0].strip() if full else None,
+        }
+    return out
+
+
 # --- Thermals -----------------------------------------------------------------------
 def collect_thermal():
    zones = []
@@ -708,15 +996,34 @@ def collect_os():
 def collect_tools():
    tools = {}
    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
-        tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+        tools[tool] = which(tool)
    return tools


 def main():
+    # --privileged: re-collect ONLY the fields that genuinely need root, for the second
+    # pass roles/node_health makes when (and only when) escalation is proven to work
+    # (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place —
+    # re-implementing the kernel-log grep in Jinja would be a second source of truth for
+    # the same fact, and the two would drift.
+    if "--privileged" in sys.argv[1:]:
+        doc = {
+            "schema": SCHEMA,
+            "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "collector_uid": os.geteuid(),
+            "klog": collect_klog(),
+            "memory": {"devices": collect_memory_devices()},
+        }
+        doc["unreadable"] = UNREADABLE
+        json.dump(doc, sys.stdout, indent=2, sort_keys=True)
+        sys.stdout.write("\n")
+        return
+
    doc = {
        "schema": SCHEMA,
        "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
        "collector_uid": os.geteuid(),
+        "klog": collect_klog(),
        "os": collect_os(),
        "dmi": collect_dmi(),
        "cpu": collect_cpu(),
@@ -387,6 +387,83 @@
               ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
               ~ avail_pct | string ~ '%)') }}

+# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the
+# one cause we have actually hit here; this one asks the blunter question — "is this
+# machine using the RAM it has?" — and answers it from two sources that share NOTHING
+# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some
+# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml.
+- name: Assess — usable memory vs the memory this machine physically has
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'memory.physical_total',
+      'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
+      'status': phys_status,
+      'detail': phys_detail }] }}"
+  vars:
+    ansible_become: false
+    th: "{{ node_health_thresholds }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+
+    # 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
+    #    actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
+    #    tools, identical on metal and in a VM — so this half is available almost always.
+    z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
+    z_status: >-
+      {{ 'SKIP' if z_pct is not number
+         else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
+               else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
+
+    # 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
+    #    VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
+    #    never as a zero DIMM count. Independent of the kernel's memory map entirely.
+    inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
+    d_pct: >-
+      {{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
+         if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
+         else none }}
+    d_status: >-
+      {{ 'SKIP' if d_pct is not number
+         else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
+               else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
+
+    # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
+    # SKIPs must never add up to a PASS.
+    phys_status: >-
+      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
+         | default('SKIP', true) }}
+    # What the kernel log says, IF it was read — quoted here only as attribution for a
+    # shortfall this check measured on its own. The verdict above never depends on it.
+    phys_cause: >-
+      {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
+           firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
+           if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
+         else ('. The kernel log was NOT read here, so this shortfall is unattributed —
+                see check firmware.efi_memattr')
+           if not (node_health_facts.klog.available | bool)
+         else '' }}
+    phys_detail: >-
+      {{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
+           ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
+           ~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
+             was never handed to the allocator (/proc/zoneinfo)')
+          if z_pct is number
+          else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
+         ~ '. '
+         ~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
+             ~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
+             ~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
+             ~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
+             ~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
+            if d_pct is number
+            else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
+         ~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
+              reservations and (on this platform) 64 MiB of graphics stolen memory.
+              A shortfall of this size is memory the machine HAS and cannot use')
+            if phys_status in ['WARN', 'FAIL'] else '')
+         ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
+
 - name: Assess — swap (kubelet requires it off)
  ansible.builtin.set_fact:
    node_health_checks: "{{ node_health_checks + [{
@@ -674,3 +751,127 @@
               ~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
               ~ (' — serial numbers not read (root-only DMI fields)'
                  if d.product_serial is none else '')) }}
+
+# --- EFI Memory Attributes Table ------------------------------------------------------
+# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
+# what follows is only what this check needs to be read correctly.
+#
+# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
+# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c
+# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s
+# the product — everything from the table address to the top of RAM. The kernel prints
+# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field
+# happens to read > 2:
+#
+#   "Failed to map EFI Memory Attributes table @ 0x..."  version read <= 2, the kernel
+#       TRUSTED the table. The node has lost the memory. FAIL.
+#   "Unexpected EFI Memory Attributes table version <N>"  version read > 2, the kernel
+#       bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a
+#       corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS:
+#       k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself
+#       booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one
+#       unlucky boot from losing 83% of its RAM is not passing a health check.
+#
+# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
+# so it cannot be diagnosed once. It has to be measured on every health run.
+#
+# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
+# ONLY if the kernel log was actually read AND still reaches back to early boot — every
+# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
+# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
+# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
+# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
+# log at all, so a node whose log is unreadable is still not unmeasured.
+- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'firmware.efi_memattr',
+      'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
+      'status': memattr_status,
+      'detail': memattr_detail }] }}"
+  vars:
+    ansible_become: false
+    k: "{{ node_health_facts.klog }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    ma_state: "{{ k.memattr_state | default(none) }}"
+    memattr_status: >-
+      {{ 'SKIP' if not (k.available | bool)
+         else ('FAIL' if ma_state == 'failed_to_map'
+               else ('WARN' if ma_state == 'bad_version'
+                     else ('PASS' if ma_state == 'corrupt_rejected'
+                           else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
+    # One remediation, quoted by both findings. NOTE what it must NOT say: there is no
+    # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
+    # could take effect. Suggesting one would be a fix that quietly does nothing.
+    memattr_fix: >-
+      FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
+      upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
+      bogus", first released in v6.13 and absent from v6.12 and from the current
+      linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
+      There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
+      bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
+      unplugging USB only change where the table lands and may APPEAR to fix it — they are
+      a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+    # The symptom, measured independently of this log (see memory.physical_total).
+    memattr_corroboration: >-
+      {{ (' Measured independently of this log: MemTotal '
+          ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+          ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
+          ~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away
+            from the allocator (/proc/zoneinfo) — check memory.physical_total.')
+         if ((z.available | bool) and z.unmanaged_pct is number) else '' }}
+    # The kernel log lines themselves, verbatim. A verdict the operator cannot check
+    # against the evidence is not much better than a wrong one.
+    memattr_evidence: >-
+      {{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
+          if (k.memattr_lines | default([]) | length) > 0 else '')
+         ~ ((' At boot the kernel reported '
+             ~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+             ~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
+             ~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
+             ~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
+            if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
+            else '') }}
+    memattr_detail: >-
+      {%- if not (k.available | bool) -%}
+      the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
+      Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
+      lives only in dmesg.{{ memattr_corroboration }}
+      {%- elif ma_state == 'failed_to_map' -%}
+      THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory
+      Attributes table at {{ k.memattr_table_address | default('an unreported address') }}
+      and reserved memory from the corrupt table anyway — everything from that address
+      upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'bad_version' -%}
+      EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table
+      here too; this boot survived only because its garbage version field read
+      {{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out
+      before reserving anything. The outcome is re-decided on EVERY boot: if that field
+      reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right
+      now and this node is one unlucky boot from the failure state.{{ memattr_evidence }}
+      {{ memattr_fix }}
+      {%- elif ma_state == 'corrupt_rejected' -%}
+      the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
+      validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is
+      intact and stays intact across reboots; no action.{{ memattr_evidence }}
+      {%- elif not (k.boot_covered | bool) -%}
+      NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
+      back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
+      the ring buffer, which wraps). Every memattr diagnostic is printed within the first
+      fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
+      pass. Read /var/log/kern.log for this boot, or re-run after a
+      reboot.{{ memattr_corroboration }}
+      {%- elif k.memattr_published is sameas true -%}
+      the firmware publishes an EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }} and the kernel
+      accepted it without complaint, in a log that still reaches back to early boot. Not
+      affected on this boot.
+      {%- elif k.memattr_published is sameas false -%}
+      this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
+      lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
+      guest.
+      {%- else -%}
+      no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
+      so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
+      boot, so that absence is measured, not assumed.
+      {%- endif -%}
@@ -0,0 +1,62 @@
+---
+# The SECOND pass of the collector, run with escalation, for the two facts that are
+# genuinely root-gated:
+#
+#   * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an
+#     unprivileged read gets EPERM. This is where the EFI Memory Attributes Table
+#     diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference
+#     between a node that has lost 13.3 GiB and one that has not.
+#   * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed
+#     DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem.
+#
+# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a
+# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line-
+# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND
+# source of truth for the same fact, and the two would drift. `--privileged` makes the
+# collector emit exactly those subtrees and nothing else, so the parsing stays in one
+# place and is identical whichever pass produced it.
+#
+# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass
+# actually failed to read one of them (main.yml checks both). On a node where
+# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule
+# is that everything readable WITHOUT root is read without root.
+#
+# If escalation is unavailable the checks that consume these SKIP with the exact reason
+# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line"
+# and "could not read the kernel log" are different answers and the role says which.
+
+- name: Re-collect the root-only facts with escalation
+  ansible.builtin.script:
+    cmd: node_health_collect.py --privileged
+    executable: /usr/bin/python3
+  become: true
+  vars:
+    ansible_become: true
+  register: node_health_privileged_raw
+  changed_when: false   # strictly read-only; it can never change the node
+
+- name: Assert the privileged payload is the schema this role understands
+  ansible.builtin.assert:
+    that:
+      - (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1'
+    fail_msg: >-
+      Privileged collector pass returned an unexpected schema; role and collector are
+      out of step.
+    quiet: true
+  vars:
+    ansible_become: false
+
+# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive
+# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine
+# would REPLACE it and throw away the reasons the first pass recorded.
+- name: Merge the privileged facts over the unprivileged ones
+  ansible.builtin.set_fact:
+    node_health_facts: >-
+      {{ node_health_facts | combine({
+           'klog': priv.klog,
+           'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}),
+           'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])),
+         }) }}
+  vars:
+    ansible_become: false
+    priv: "{{ node_health_privileged_raw.stdout | from_json }}"
@@ -4,9 +4,11 @@
 # ORDER OF OPERATIONS
 #   1. Collect everything readable WITHOUT root (one script, one round trip).
 #   2. Probe whether privilege escalation actually works on this host.
-#   3. Only if it does AND smartmontools is installed: read SMART.
-#   4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
-#   5. Render the human summary + write the JSON artifact.
+#   3. Only if it does, and only for what the first pass could NOT read: re-run the same
+#      collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
+#   4. Only if it does AND smartmontools is installed: read SMART.
+#   5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#   6. Render the human summary + write the JSON artifact.
 #
 # BECOME HANDLING — the subtle bit, read before editing.
 #   `ansible_become` set as an inventory/group var (which is exactly what
@@ -43,6 +45,16 @@
  ansible.builtin.include_tasks: probe_privilege.yml
  when: node_health_try_privileged | bool

+# Only when the unprivileged pass actually failed to read one of them. On a node with
+# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
+# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
+- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
+  ansible.builtin.include_tasks: collect_privileged.yml
+  when:
+    - node_health_root_available | bool
+    - (not (node_health_facts.klog.available | bool))
+      or (not (node_health_facts.memory.devices.available | bool))
+
 - name: Read SMART data (privileged)
  ansible.builtin.include_tasks: collect_smart.yml
  when:
@@ -45,6 +45,15 @@
 {% endif %}
 Memory       {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
 ECC/EDAC     {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
+{% set mz = node_health_facts.memory.zones %}
+{% set md = node_health_facts.memory.devices %}
+ Present RAM  {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }}
+ DIMMs        {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }}
+{% for dimm in md.devices | default([]) if dimm.size_kb %}
+              {{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }}
+{% endfor %}
+{% set klog = node_health_facts.klog %}
+ EFI memattr  {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
 Load         {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}

 --------------------------------------------------------------------------------