feat(prod): kernel fix + disk layout — prod nodes ready for k3s

Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.

=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.

roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).

Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.

THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.

§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.

TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
  - MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
    2's real value was -1569980416, which did not match, so state fell through to `clean` and
    the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
    exact bug the check exists to find. Now (-?\d+).
  - bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
    Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.

=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).

Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).

=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.

Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.

The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.

k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.

LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.

DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.

Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.

OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.

Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.

Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

clusters/base/cilium.yaml

@@ -0,0 +1,60 @@
+---
+# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
+# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
+#
+# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
+#
+# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
+# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
+# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
+# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
+#
+#   * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
+#   * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
+#     tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
+#     `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
+#     the valve the 2026-09-01 substitution incident wanted and did not have.
+#   * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
+#     `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
+#
+# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
+# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
+# expressed as an explicit edge instead of an accident of co-tenancy.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: cilium
+  namespace: flux-system
+spec:
+  # A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
+  # kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
+  interval: 1h
+  retryInterval: 2m
+  # 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
+  # rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
+  timeout: 10m
+  prune: true
+  # TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
+  wait: true
+  # Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
+  # by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
+  # is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
+  # revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
+  # `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/cilium
+  # MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
+  # kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
+  # this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
+  # renders empty). That is the intended fail-closed behaviour — but it means the stanza is
+  # load-bearing for the CNI coming up at all. Do not remove it.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
+  # oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).

clusters/base/cluster-vars.yaml

@@ -74,6 +74,17 @@ data:
 
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
+  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
+  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
+  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
+  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
+  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
+  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
+  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
+  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
+  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
+  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
+  LONGHORN_DATA_PATH: "/var/lib/longhorn"
 
   # --- observability sizing (docs/10) ---
   MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long

clusters/base/kustomization.yaml

@@ -5,6 +5,13 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - flux-system.yaml     # GitRepository (reconciliation source)
+  # Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
+  # of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
+  # Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
+  # `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
+  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
+  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
+  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).

clusters/base/platform.yaml

@@ -11,7 +11,18 @@ spec:
   interval: 10m
   retryInterval: 2m
   timeout: 5m
-  prune: true          # drift correction: delete resources removed from git (R-GITOPS-03)
+  # Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
+  # on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
+  # live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
+  # `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
+  # any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
+  #
+  # LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
+  # Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
+  # made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
+  # Cilium. Before moving ANY platform/* component in future, check whether it is the sole
+  # declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
+  prune: true
   wait: true           # block `apps` until platform is healthy
   # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
   #
@@ -40,6 +51,13 @@ spec:
   # health-gates its own children (see clusters/dev/sync.yaml).
   dependsOn:
     - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+    # THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
+    # (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
+    # suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
+    # nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
+    # is healthy. It is also revision-gated, which is what makes the prune-safety argument above
+    # deterministic rather than a race.
+    - name: cilium
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/dev/kustomization.yaml

@@ -190,6 +190,16 @@ patches:
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
+      # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
+      # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
+      # else would orphan every one of them (they are not recoverable from the old path by
+      # rescheduling). Restated explicitly even though it equals the base default, exactly like the
+      # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
+      # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/var/lib/longhorn"
       # smaller retention / volumes to keep the dev workstation footprint light.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

clusters/prod/kustomization.yaml

@@ -157,6 +157,20 @@ patches:
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
+      # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
+      # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
+      # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
+      # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
+      # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
+      #
+      # [parity] MUST equal node_baseline_longhorn_data_path in
+      # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
+      # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
+      # path on the OS filesystem instead of the data partition and /var fills up.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/data/longhorn"
       # longer retention / larger volumes on prod NVMe.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

conformance/.gitignore

@@ -8,3 +8,22 @@ _artifacts/
 
 # Python bytecode from lib/ (status.py imports coverage.py); never committed.
 __pycache__/
+
+# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
+# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
+# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
+# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
+# fact, so it must survive in git.
+#
+# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
+# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
+# the `conformance/runs/` directory itself, so re-including the files here works (git
+# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
+# live under _artifacts/).
+#
+# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
+# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
+# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
+!runs/
+!runs/**
+!runs/**/*.junit.xml

conformance/README.md

@@ -22,6 +22,7 @@ behavioral / L3 fault-injection** (this harness).
 ./run.sh --lint                       # structural-validate test defs (no cluster)
 ./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
 ./run.sh --require-all                # phase-exit gate: every requirement must be covered
+./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
 ```
 
 **Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -105,13 +106,89 @@ lib/
   common.sh                  tagging convention + test discovery/selection helpers
   coverage.py                requirement-coverage generator (stdlib only)
   status.py                  requirement-STATUS generator (stdlib only; --selftest)
+  record_run.py              run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
   emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
+runs/
+  ledger.jsonl               TRACKED append-only proof-of-pass ledger (one row per
+                             testcase per run) — the only source status.py trusts
+  <UTC>-<short-sha>/         TRACKED per-run evidence: run.log + report.junit.xml
 tests/
   l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
 incluster/
   chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)
 ```
 
+## Proof of pass — the run ledger (`runs/`)
+
+`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
+proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
+root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
+only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
+asked for — **a committed, machine-readable run ledger**.
+
+```bash
+./run.sh --record                     # runs normally, and captures the evidence
+# -> conformance/runs/<UTC>-<short-sha>/run.log            (tee'd, carries the sentinels)
+# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml   (that run's JUnit)
+# -> conformance/runs/ledger.jsonl                         (one appended row per testcase)
+python3 lib/record_run.py --run-dir runs/<id> [--dry-run]  # or record an existing run dir
+python3 lib/record_run.py --selftest                       # deterministic, no cluster
+```
+
+**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
+that silently records itself is how a bad run becomes a permanent false proof. **Commit
+the run dir and the ledger** — an uncommitted ledger proves nothing.
+
+### Why a committed JUnit alone would be WORSE than nothing
+
+The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
+`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
+and `exit 0`, and their `check:` accepts either outcome —
+
+```yaml
+(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+ || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+```
+
+So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
+A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
+R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
+
+**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
+`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
+`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
+verdict token), so the table below is never hand-maintained:
+
+| Test | pass token (proof) | skip token (never proof) |
+|---|---|---|
+| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
+| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
+| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
+| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
+| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
+
+A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
+token present in that test's log lines, and **no** skip token anywhere in them. A skip
+token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
+`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
+non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
+skips as JUnit `<skipped/>`, never as a green pass.
+
+### Staleness — a proof is a claim about a tree, not a badge
+
+`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
+last commit touching that requirement's implementing manifests or its verifying test**.
+Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
+dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
+it never counts. Re-run with `--record` to refresh.
+
+A requirement reads `proven` only when **every** verifying test does; one skipped, failed
+or unrecorded test drags the whole requirement down. With an empty ledger `--status`
+behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
+
 ## How CI invokes it
 CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
 

conformance/lib/record_run.py

@@ -0,0 +1,949 @@
+#!/usr/bin/env python3
+"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
+
+WHY THIS EXISTS
+---------------
+`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
+fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
+`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
+`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
+that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
+and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
+tracked (see conformance/.gitignore for the negations that let it survive).
+
+THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
+------------------------------------------------------------------
+The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
+StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
+`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
+injection/dr-restore:
+
+    (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+     || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+
+So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
+(verified against the committed reports). Therefore:
+
+  *** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
+
+AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
+------------------------------------------------------
+Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
+it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
+the log of every run of that test. A section-blind grep therefore reads a skip token out
+of the source of every skip-gated test and can never record `proven` for any of them: the
+ledger fails closed so completely that it records nothing and proves nothing. Tokens are
+consequently read from SECTIONS (see classify_sections): a pass token only from
+`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
+pass — within genuine output, which is the only place either was ever evidence.
+
+That is the fake-green this project refuses, and it would be worse than no ledger at all
+because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
+PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
+files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
+unless the log carries that test's own pass token and carries NO skip token.
+
+HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
+--------------------------------------------------------
+Hard-coding a token table would rot the moment a test is edited. Instead every token is
+parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
+`check:` block is a declared sentinel. Classification is by the project's naming convention
+(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
+— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
+and this tool follows automatically; delete them and the test simply becomes unprovable
+rather than silently proven.
+
+  node-loss   pass NODE_LOSS_VOLUME_AVAILABLE_OK              skip NODE_LOSS_SKIPPED_OK
+  dr-restore  pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK     skip DR_RESTORE_SKIPPED_OK
+              (+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
+
+WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
+-----------------------------------------------------------
+The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
+dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
+any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
+pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
+can express "skipped" as "passed", and only they require a sentinel — see
+SENTINEL_REQUIRED_LAYERS.
+
+LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
+  ts_utc            run-dir timestamp, else the JUnit's mtime      (when)
+  git_sha           `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
+                    modified — a proof captured from a dirty tree is NOT honoured by
+                    status.py, because what ran is not what is committed  (what)
+  env               --env / CONF_ENV (cluster overlay the run targeted)
+  chainsaw_version  CHAINSAW_VERSION from versions.env (the pin IS the truth)
+  kube_version      `kubectl version` server gitVersion, else "unknown"
+  layer             L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
+  test              testcase name (== the test dir basename, asserted by convention)
+  requirements[]    the `conformance.home.k3s/R-*` labels on the test — the SAME
+                    authoritative source coverage.py uses, not the JUnit classname
+  junit_result      passed | failed | error | skipped
+  sentinel          the DECIDING token grepped from run.log, or null
+  proof             proven | skipped | failed | junit-skipped
+                    | unknown-no-sentinel-declared | unknown-sentinel-absent
+  run_id            the runs/<dir> name (provenance: which log backs this row)
+  note              one line saying WHY this row landed on that proof state
+
+`proven` is emitted on exactly one path (see decide()); every other path is non-green.
+
+RECORDING IS EXPLICIT, NEVER IMPLICIT
+-------------------------------------
+Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
+how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
+ledger row at all.
+
+EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
+  0  rows appended.
+  1  a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
+  3  the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
+     NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
+
+Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
+status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+_LIB = Path(__file__).resolve().parent
+if str(_LIB) not in sys.path:
+    sys.path.insert(0, str(_LIB))
+# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
+# tree, and the lint stage scans untracked files. Same guard as status.py.
+sys.dont_write_bytecode = True
+import coverage as cov  # noqa: E402  — reuse the registry/test parsers, don't fork them
+
+CONF_ROOT = _LIB.parent
+REPO_ROOT = CONF_ROOT.parent
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
+# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
+# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
+# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
+# by JUnit alone.
+SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
+
+# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
+# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
+# harmless for its own report, but here a mis-read layer decides whether a sentinel is
+# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
+_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
+                     ("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
+                     ("/meta/", "META"))
+
+
+def layer_of(path: Path) -> str:
+    rel = str(path)
+    for frag, layer in _LAYER_BY_SUBTREE:
+        if frag in rel:
+            return layer
+    return ""
+
+# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
+# Both quote styles are accepted; the repo uses single quotes throughout.
+_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
+
+# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
+# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
+# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
+# mis-classifying a SKIP token as a pass would manufacture one.
+_SKIP_TOKEN_RE = re.compile(r"_SKIP")
+
+# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
+# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
+#     l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
+# Field 2 is the test name; it is how output is attributed back to a testcase. The
+# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
+_LOG_HEADER_RE = re.compile(
+    r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
+    r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
+
+# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
+# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
+
+
+def _strip_noise(line: str) -> str:
+    return _ANSI_RE.sub("", line).replace("\b", "")
+
+
+# ===================================================================== output sections
+# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
+# -----------------------------------------------------------------------------
+# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
+# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
+# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
+# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
+# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
+# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
+# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
+# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
+# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
+#
+# So tokens are attributed to SECTIONS, never to raw lines:
+#     l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |     <- header, no section
+#         === COMMAND                                            <- marker (indent 8)
+#         /usr/bin/sh -c set -euo pipefail                       <- source, indent >= 8
+#           echo NODE_LOSS_SKIPPED_OK                            <- SOURCE, not output
+#     l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |    <- indent 4 => section ends
+#         === STDOUT                                             <- marker
+#         NODE_LOSS_VOLUME_AVAILABLE_OK                          <- the REAL emission
+#
+# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
+# (`=== RUN   chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
+# after the verb, so they match neither the indent nor the end-of-line anchor; they are
+# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
+# section that would swallow the whole log.
+_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
+_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
+
+# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
+# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
+# never echoes script source, so there is nothing to be confused by and the legacy
+# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
+# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
+# "" and are evidence for nothing.
+UNSECTIONED = "*"
+
+# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
+# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
+# never manufacture a proof.
+PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
+
+# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
+# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
+# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
+# one. Unknown section names therefore fail closed in both directions.
+SOURCE_SECTIONS = frozenset({"COMMAND"})
+
+
+def classify_sections(log_text: str) -> list[tuple[str, str]]:
+    """[(section, line)] for every line of a run log.
+
+    `section` is the enclosing `=== NAME` block, "" for a line inside no block, or
+    UNSECTIONED for every line when the log declares no sections at all. A block runs from
+    its marker until the first non-blank line indented LESS than the marker (blank lines do
+    not end it) or until the next marker; the marker line itself is content of nothing.
+    """
+    rows: list[tuple[str, str]] = []
+    cur, cur_indent, sectioned = "", 0, False
+    for raw in log_text.splitlines():
+        line = _strip_noise(raw)
+        m = _SECTION_MARKER_RE.match(line)
+        if m and m.group(2) not in _GO_TEST_MARKERS:
+            sectioned = True
+            cur, cur_indent = m.group(2), len(m.group(1))
+            rows.append(("", line))
+            continue
+        if cur:
+            if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
+                rows.append((cur, line))
+                continue
+            cur = ""
+        rows.append(("", line))
+    if not sectioned:
+        return [(UNSECTIONED, line) for _, line in rows]
+    return rows
+
+
+def scan_texts(lines) -> tuple[str, str]:
+    """(text a PASS token may be read from, text a SKIP token may be read from).
+
+    Accepts either [(section, line)] pairs from classify_sections() or a plain list of
+    strings, which is treated as an unsectioned log (legacy whole-log scanning).
+    """
+    pass_lines, skip_lines = [], []
+    for item in lines:
+        section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
+        if section in PASS_SECTIONS:
+            pass_lines.append(line)
+        if section and section not in SOURCE_SECTIONS:
+            skip_lines.append(line)
+    return "\n".join(pass_lines), "\n".join(skip_lines)
+
+_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
+
+
+# ========================================================================= sentinels
+def parse_sentinels(test_yaml: Path) -> dict:
+    """Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
+
+    Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
+    the LAST non-skip token in file order — the test's verdict token, i.e. the one its
+    final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
+    A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
+    unprovable rather than proven; see decide().
+    """
+    try:
+        text = test_yaml.read_text()
+    except (OSError, UnicodeDecodeError):
+        return {"pass": [], "skip": [], "terminal_pass": None}
+    toks = _SENTINEL_RE.findall(text)
+    passes, skips = [], []
+    for t in toks:
+        (skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
+    # De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
+    def uniq(xs: list[str]) -> list[str]:
+        return list(dict.fromkeys(xs))
+    passes, skips = uniq(passes), uniq(skips)
+    return {"pass": passes, "skip": skips,
+            "terminal_pass": passes[-1] if passes else None}
+
+
+def sentinel_map(tests_dir: Path) -> dict[str, dict]:
+    """{test-name: sentinels} for every test in the tree.
+
+    Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
+    name=...>`. This repo keeps that equal to the test's directory basename; the mapping
+    is by name, so it stays correct if that ever drifts.
+    """
+    out: dict[str, dict] = {}
+    for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
+        meta = cov.parse_test(f)
+        name = meta["name"] or f.parent.name
+        s = parse_sentinels(f)
+        s["dir"] = str(f.parent)
+        s["layer"] = layer_of(f) or meta["layer"]
+        s["requirements"] = meta["labels"]
+        out[name] = s
+    return out
+
+
+# =============================================================================== log
+def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
+                                                            bool]:
+    """Split a run log into per-testcase buckets of (section, line) pairs.
+
+    Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
+    script's sections underneath that header, so the test name is carried by a state
+    machine: a header naming a KNOWN test switches the current bucket, every following line
+    lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
+    chatter (and any other pipe-shaped line) from opening a bogus bucket.
+
+    Section labels come from classify_sections() and travel WITH the lines, so a bucket
+    still knows which of its lines were script source and which were real output.
+
+    Returns (buckets, attributed) where `attributed` is False when the log contained no
+    recognisable header at all — the signal for the degraded global scan in decide().
+    """
+    buckets: dict[str, list[tuple[str, str]]] = {}
+    cur: str | None = None
+    attributed = False
+    for section, line in classify_sections(log_text):
+        m = _LOG_HEADER_RE.match(line)
+        if m and m.group(1) in known:
+            cur = m.group(1)
+            attributed = True
+        if cur is not None:
+            buckets.setdefault(cur, []).append((section, line))
+    return buckets, attributed
+
+
+def unique_tokens(smap: dict[str, dict]) -> set[str]:
+    """Tokens claimed by EXACTLY ONE test in the tree.
+
+    Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
+    tests, so in an unattributable log it can never be evidence for either; but
+    `NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
+    """
+    owners: dict[str, set[str]] = {}
+    for name, s in smap.items():
+        for t in s["pass"] + s["skip"]:
+            owners.setdefault(t, set()).add(name)
+    return {t for t, o in owners.items() if len(o) == 1}
+
+
+# ============================================================================ decide
+def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
+           scope: str) -> tuple[str | None, str, str]:
+    """The ONE place a testcase becomes `proven` — or, far more often, does not.
+
+    Returns (sentinel, proof, note).
+
+    Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
+      1. JUnit failure/error            -> failed        (a red run is recorded as red)
+      2. JUnit <skipped/>               -> junit-skipped  (never green; ci/lib.sh rule)
+      3. ANY declared skip token seen   -> skipped        (BEATS a green JUnit and beats
+           IN REAL OUTPUT                                  the pass token: if any gated
+                                                           step self-skipped, the run did
+                                                           not prove the requirement)
+      4. no sentinel declared           -> proven only for L0/L1, whose skips are already
+                                           JUnit skips; unknown-no-sentinel-declared for
+                                           the Chainsaw layers
+      5. EVERY declared pass token seen -> proven
+      6. anything else                  -> unknown-sentinel-absent
+    Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
+    asserts `pass || skip`, so a missing pass token means a step did not reach its proven
+    branch, whatever the JUnit says.
+
+    Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
+    only in `=== STDOUT`, a skip token counts in any section that is not script source.
+    That narrows WHERE a token is read from and changes nothing about what it means — skip
+    still outranks pass, and it outranks it within genuine output, which is the only place
+    either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
+    strings (an unsectioned log; see scan_texts / UNSECTIONED).
+    """
+    if junit_result in ("failed", "error"):
+        return (None, "failed", f"JUnit recorded {junit_result}")
+    if junit_result == "skipped":
+        return (None, "junit-skipped",
+                "JUnit <skipped/> — the test did not execute; a skip is never green")
+
+    pass_text, skip_text = scan_texts(lines)
+    seen_skip = [t for t in sent["skip"] if t in skip_text]
+    if seen_skip:
+        return (seen_skip[0], "skipped",
+                f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
+                f"exited 0 without exercising the behaviour, so this is NOT proof")
+
+    if not sent["pass"] and not sent["skip"]:
+        if layer not in SENTINEL_EXEMPT_LAYERS:
+            return (None, "unknown-no-sentinel-declared",
+                    f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
+                    f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
+                    f"refusing to call it proven")
+        return (None, "proven",
+                f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
+                f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
+
+    missing = [t for t in sent["pass"] if t not in pass_text]
+    if not missing:
+        return (sent["terminal_pass"], "proven",
+                f"run log OUTPUT carries every declared pass token "
+                f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
+    return (None, "unknown-sentinel-absent",
+            f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
+            f"[{scope}] — refusing to infer proof from the JUnit alone")
+
+
+# ============================================================================= junit
+def junit_cases(path: Path) -> list[dict]:
+    """[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
+    root = ET.parse(path).getroot()
+    cases: list[dict] = []
+    for c in root.iter("testcase"):
+        if c.find("failure") is not None:
+            result = "failed"
+        elif c.find("error") is not None:
+            result = "error"
+        elif c.find("skipped") is not None:
+            result = "skipped"
+        else:
+            result = "passed"
+        props = {p.get("name"): (p.get("value") or "")
+                 for p in c.iter("property") if p.get("name")}
+        reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
+        layer = (props.get("layer") or "").upper()
+        if not layer:
+            layer = (c.get("classname") or "").split(":", 1)[0].upper()
+        cases.append({"name": c.get("name") or "?", "layer": layer,
+                      "requirements": reqs, "result": result})
+    return cases
+
+
+# =============================================================================== git
+def git_sha(repo: Path) -> str:
+    """HEAD, suffixed `-dirty` when TRACKED files differ from it.
+
+    Untracked files are deliberately ignored, matching status.py's `tracked_files()`
+    posture (untracked scratch is not repo state) — and because a `--record` run creates
+    untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
+    A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
+    the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
+    """
+    try:
+        sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
+                             capture_output=True, text=True, check=True).stdout.strip()
+    except (subprocess.CalledProcessError, OSError):
+        return "unknown"
+    dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--"],
+                           capture_output=True).returncode != 0
+    return f"{sha}-dirty" if dirty else sha
+
+
+def pinned_chainsaw_version(conf_root: Path) -> str:
+    """CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
+    f = conf_root / "versions.env"
+    if f.is_file():
+        m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
+        if m:
+            return m.group(1)
+    return "unknown"
+
+
+def server_kube_version() -> str:
+    """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
+    for args in (["kubectl", "version", "-o", "json"],
+                 ["kubectl", "version", "--output=json"]):
+        try:
+            p = subprocess.run(args, capture_output=True, text=True, timeout=15)
+        except (OSError, subprocess.TimeoutExpired):
+            continue
+        if p.returncode != 0 or not p.stdout.strip():
+            continue
+        try:
+            v = json.loads(p.stdout).get("serverVersion") or {}
+        except json.JSONDecodeError:
+            continue
+        if v.get("gitVersion"):
+            return v["gitVersion"]
+    return "unknown"
+
+
+def run_timestamp(run_dir: Path, junit: Path) -> str:
+    """UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
+    m = _RUN_ID_TS_RE.match(run_dir.name)
+    if m:
+        t = m.group(1)
+        return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
+    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
+
+
+# ============================================================================ record
+def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
+               kube_version: str, sha: str) -> list[dict]:
+    junit = run_dir / "report.junit.xml"
+    log = run_dir / "run.log"
+    smap = sentinel_map(tests_dir)
+    log_text = log.read_text(errors="replace")
+    buckets, attributed = attribute_log(log_text, set(smap))
+    uniq = unique_tokens(smap)
+    all_pairs = classify_sections(log_text)
+    sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
+    section_note = ("tokens read from real-output sections only"
+                    if sectioned else "log declares no === sections; whole-log scan")
+    ts = run_timestamp(run_dir, junit)
+
+    rows: list[dict] = []
+    for case in junit_cases(junit):
+        name = case["name"]
+        sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
+                               "layer": case["layer"], "requirements": []})
+        layer = sent.get("layer") or case["layer"] or "?"
+        # Requirements come from the test's LABELS (coverage.py's authoritative source);
+        # the JUnit property is the fallback for a testcase with no matching test file.
+        reqs = sent.get("requirements") or case["requirements"]
+
+        if name in buckets:
+            lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
+        elif not attributed:
+            # Degraded mode: the log had no recognisable Chainsaw header (a redirected or
+            # reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
+            # exactly one test, so a token another test emitted can never prove this one.
+            lines = all_pairs
+            sent = dict(sent,
+                        **{"pass": [t for t in sent["pass"] if t in uniq],
+                           "skip": [t for t in sent["skip"] if t in uniq]})
+            sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
+            scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
+                     + section_note)
+        else:
+            lines, scope = [], "no log lines attributed to this testcase"
+
+        sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
+        rows.append({
+            "ts_utc": ts,
+            "git_sha": sha,
+            "env": env,
+            "chainsaw_version": chainsaw_version,
+            "kube_version": kube_version,
+            "layer": layer,
+            "test": name,
+            "requirements": reqs,
+            "junit_result": case["result"],
+            "sentinel": sentinel,
+            "proof": proof,
+            "run_id": run_dir.name,
+            "note": note,
+        })
+    return rows
+
+
+def append_rows(ledger: Path, rows: list[dict]) -> None:
+    """Append-only, one compact JSON object per line. Never rewrites existing rows."""
+    ledger.parent.mkdir(parents=True, exist_ok=True)
+    with ledger.open("a", encoding="utf-8") as fh:
+        for r in rows:
+            fh.write(json.dumps(r, sort_keys=True) + "\n")
+
+
+# ========================================================================== selftest
+def _selftest() -> int:
+    """Deterministic, no-cluster self-test of every decision rule (house pattern:
+    `status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
+
+    The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
+    assertion standing between this ledger and a permanent false proof of R-STATE-03.
+    """
+    import tempfile
+    ok = True
+
+    def check(label: str, cond: bool) -> None:
+        nonlocal ok
+        if not cond:
+            sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
+            ok = False
+
+    # --- token derivation, against the dr-restore/node-loss shape ---------------------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
+            "  name: dr-restore\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-03: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-03\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo OFF_CLUSTER_TARGET_OK'\n"
+            "        check:\n"
+            "          (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
+            "    - script:\n        content: 'echo x'\n        check:\n"
+            "          (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
+        s = parse_sentinels(tdir / "chainsaw-test.yaml")
+        check("pass tokens derived from the test file",
+              s["pass"] == ["OFF_CLUSTER_TARGET_OK",
+                            "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
+        check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
+              s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
+        check("terminal pass token is the LAST non-skip token",
+              s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
+
+        smap = sentinel_map(root / "tests")
+        check("sentinel_map keys on metadata.name and carries layer + labels",
+              smap["dr-restore"]["layer"] == "L3"
+              and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
+
+    # --- decide(): the SKIP-is-never-proven contract ----------------------------------
+    l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
+          "skip": ["DR_RESTORE_SKIPPED_OK"],
+          "terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
+
+    # *** THE CASE THIS WHOLE FILE EXISTS FOR ***
+    sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
+    check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
+          proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
+                          "DR_RESTORE_SKIPPED_OK"], "t")
+    check("a skip token BEATS a pass token in the same log (worst wins)",
+          proof == "skipped")
+
+    _, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
+    check("green JUnit with no sentinel in the log is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "passed", l3, [], "t")
+    check("green JUnit with NO log lines attributed is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "failed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("a failing JUnit outranks a pass token in the log", proof == "failed")
+
+    _, proof, _ = decide("L3", "skipped", l3, [], "t")
+    check("JUnit <skipped/> records junit-skipped, never green",
+          proof == "junit-skipped")
+
+    multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
+    _, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
+    check("every declared pass token must be present, not just the terminal one",
+          proof == "unknown-sentinel-absent")
+    _, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
+    check("all pass tokens present == proven", proof == "proven")
+
+    none = {"pass": [], "skip": [], "terminal_pass": None}
+    _, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
+    check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("L1", "passed", none, [], "t")
+    check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
+          proof == "proven")
+    _, proof, _ = decide("L0", "skipped", none, [], "t")
+    check("an L0 JUnit skip is still never green", proof == "junit-skipped")
+    _, proof, _ = decide("META", "passed", none, [], "t")
+    check("META (a Chainsaw layer) needs a sentinel like L2/L3",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("", "passed", none, [], "t")
+    check("an unknown/blank layer fails CLOSED — sentinel required",
+          proof == "unknown-no-sentinel-declared")
+
+    # --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
+    # Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
+    # `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
+    # inside `=== COMMAND` on every run of a skip-gated test.
+    nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
+          "skip": ["NODE_LOSS_SKIPPED_OK"],
+          "terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+    _CMD = ("    l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |\n"
+            "        === COMMAND\n"
+            "        /usr/bin/sh -c set -euo pipefail\n"
+            "        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
+            "          echo NODE_LOSS_SKIPPED_OK\n"
+            "\n"
+            "          exit 0\n"
+            "        fi\n"
+            "        echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
+
+    ran = classify_sections(_CMD + "        === STDOUT\n"
+                            "        ready nodes: 3\n"
+                            "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
+    sent, proof, _ = decide("L3", "passed", nl, ran, "t")
+    check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
+          proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+    selfskipped = classify_sections(_CMD + "        === STDOUT\n"
+                                    "        SKIP: 'longhorn' StorageClass absent\n"
+                                    "        NODE_LOSS_SKIPPED_OK\n")
+    sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
+    check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
+          proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
+
+    neither = classify_sections(_CMD + "        === STDOUT\n        ready nodes: 3\n")
+    _, proof, _ = decide("L3", "passed", nl, neither, "t")
+    check("both tokens in COMMAND and neither in STDOUT is NOT proven",
+          proof == "unknown-sentinel-absent")
+
+    both_out = classify_sections("        === STDOUT\n"
+                                 "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                                 "        NODE_LOSS_SKIPPED_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, both_out, "t")
+    check("skip still BEATS pass when both are in genuine output", proof == "skipped")
+
+    stderr_skip = classify_sections("        === STDERR\n        NODE_LOSS_SKIPPED_OK\n"
+                                    "        === STDOUT\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
+    check("a skip token in a non-source section (STDERR) still costs the proof",
+          proof == "skipped")
+
+    stderr_pass = classify_sections("        === STDERR\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
+    check("a pass token OUTSIDE stdout cannot prove anything",
+          proof == "unknown-sentinel-absent")
+
+    # --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
+    gorows = classify_sections("=== RUN   chainsaw\n=== PAUSE chainsaw\n"
+                               "=== CONT  chainsaw/node-loss\n"
+                               "        === COMMAND\n        echo NODE_LOSS_SKIPPED_OK\n"
+                               "    l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
+                               "--- PASS: chainsaw/node-loss (131.09s)\n")
+    secs = [sec for sec, _ in gorows]
+    check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
+          secs[:3] == ["", "", ""])
+    check("a section covers its indented body and ends at a less-indented line",
+          secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
+    check("a log with no === markers stays wholly scannable (UNSECTIONED)",
+          {sec for sec, _ in classify_sections("plain line\n  another\n")} == {UNSECTIONED})
+
+    # --- the preserved run itself, as a fixture ----------------------------------------
+    # runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
+    # (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
+    fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
+    if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
+        rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
+        row = next((r for r in rows if r["test"] == "node-loss"), None)
+        check("preserved real L3 run records node-loss/R-STATE-02 as proven",
+              row is not None and row["proof"] == "proven"
+              and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
+              and row["requirements"] == ["R-STATE-02"])
+        check("preserved real L3 run attributes (no DEGRADED fallback needed)",
+              row is not None and "DEGRADED" not in row["note"])
+        pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
+        secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
+        check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
+              secs == {"COMMAND"})
+        secs = {sec for sec, line in pairs
+                if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+        check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
+
+    # --- log attribution --------------------------------------------------------------
+    log = ("[run.sh] Coverage report first\n"
+           "    | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
+           "        NODE_LOSS_SKIPPED_OK\n"
+           "    | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
+           "        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
+    b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
+    check("log lines are attributed to the testcase whose header preceded them",
+          attributed
+          and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
+          and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
+    check("a pass token does NOT leak across testcase boundaries",
+          not any("DR_RESTORED" in l for _, l in b["node-loss"]))
+    check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
+          all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
+
+    # Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
+    # regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
+    real_hdr = ("    l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
+                "\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
+                "        === STDOUT\n        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
+    check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
+          attributed_real and "node-loss" in b2
+          and ("STDOUT", "        NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
+    _, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
+    check("an unattributable log is reported as such (degraded mode)", not attributed2)
+
+    smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
+             "b": {"pass": ["DNS_OK"], "skip": []}}
+    check("only tokens owned by exactly one test may be used in degraded mode",
+          unique_tokens(smap2) == {"A_ONLY_OK"})
+
+    # --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "node-loss"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "kind: Test\nmetadata:\n  name: node-loss\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-02: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-02\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo hi'\n        check:\n"
+            "          (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
+            " || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
+        rd = root / "runs" / "20260901T120000Z-abc1234"
+        rd.mkdir(parents=True)
+        (rd / "report.junit.xml").write_text(
+            '<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
+            'name="node-loss"><properties>'
+            '<property name="requirements" value="R-STATE-02"/>'
+            '<property name="layer" value="L3"/></properties></testcase>'
+            "</testsuite></testsuites>")
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      SKIP: 'longhorn' StorageClass absent\n"
+            "      NODE_LOSS_SKIPPED_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
+              len(rows) == 1 and rows[0]["junit_result"] == "passed"
+              and rows[0]["proof"] == "skipped"
+              and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
+        check("end-to-end: row carries ts from the run-dir name and the R-ID label",
+              rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
+              and rows[0]["requirements"] == ["R-STATE-02"])
+
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
+              rows[0]["proof"] == "proven"
+              and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+        ledger = root / "ledger.jsonl"
+        append_rows(ledger, rows)
+        append_rows(ledger, rows)
+        check("ledger is append-only JSONL (one parsable object per line)",
+              len(ledger.read_text().strip().splitlines()) == 2
+              and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
+
+    if not ok:
+        return 1
+    sys.stderr.write("record_run.py selftest: PASS\n")
+    return 0
+
+
+# ============================================================================== main
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--run-dir", type=Path,
+                    help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"append-only ledger (default {DEFAULT_LEDGER})")
+    ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
+    ap.add_argument("--repo", type=Path, default=REPO_ROOT)
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
+    ap.add_argument("--chainsaw-version", default=None,
+                    help="override the versions.env pin (recorded verbatim)")
+    ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
+                    help="override the `kubectl version` probe (recorded verbatim)")
+    ap.add_argument("--dry-run", action="store_true",
+                    help="print the rows that WOULD be appended; write nothing")
+    ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
+    args = ap.parse_args()
+
+    if args.selftest:
+        return _selftest()
+    if args.run_dir is None:
+        sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
+        return 1
+
+    run_dir = args.run_dir.resolve()
+    junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
+    # BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
+    # recording from it alone is exactly the false proof this tool exists to prevent.
+    missing = [p.name for p in (junit, log) if not p.is_file()]
+    if missing:
+        sys.stderr.write(
+            f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
+            f"The JUnit alone cannot distinguish a self-skip from a real pass "
+            f"(see module docstring). SKIPPED (not green).\n")
+        return 3
+
+    try:
+        rows = build_rows(
+            run_dir, args.tests, args.env,
+            args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
+            args.kube_version or server_kube_version(),
+            git_sha(args.repo))
+    except ET.ParseError as e:
+        sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
+        return 1
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
+        return 1
+
+    if not rows:
+        sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
+                         f"SKIPPED (not green).\n")
+        return 3
+
+    for r in rows:
+        print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
+              f"({','.join(r['requirements']) or 'untagged'}) "
+              f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
+    counts: dict[str, int] = {}
+    for r in rows:
+        counts[r["proof"]] = counts.get(r["proof"], 0) + 1
+    print(f"[record] {len(rows)} testcase(s): "
+          + ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
+
+    if args.dry_run:
+        print("[record] --dry-run: ledger NOT written.")
+        return 0
+    try:
+        append_rows(args.ledger, rows)
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
+        return 1
+    print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
+    print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/lib/status.py

@@ -15,26 +15,42 @@ This generator derives, from the REPOSITORY ALONE, three orthogonal axes per req
                        from a Flux Kustomization entrypoint (i.e. actually reconciled)?
   2. VERIFICATION    — is there a requirement-tagged test, at which layer, and is it an
                        EXECUTABLE test or a TODO scaffold?
-  3. PROOF           — has that test actually passed?  *** DELIBERATELY NOT DERIVED. ***
+  3. PROOF           — has that test actually passed?  Derived ONLY from the committed
+                       run ledger, and ONLY as far as that ledger's evidence reaches.
 
-WHAT THIS TOOL REFUSES TO DERIVE (and why)
-------------------------------------------
-PROOF-OF-PASS IS NOT DERIVABLE FROM REPO STATE. Nothing tracked in git records a test
-result: `conformance/_artifacts/` and `results/` are git-ignored, and the only prose
-record of a green run lives in DECISIONS-LOG.md — hand-maintained, unstructured, and
-exactly the kind of source the operator has ruled out. The project's own culture is that
-"merged" is never "done" (agents/definition-of-done.md, R-TEST-03), so this report will
-NOT infer, guess, or default a requirement to green. Every requirement's proof state is
-reported as `not-derivable`, and the headline says so. If an untracked JUnit artifact
-happens to be lying in the workspace it is surfaced ONLY as a quarantined advisory with
-its mtime — never counted, never mixed into a status.
+PROOF — WHAT CHANGED, AND WHAT DID NOT
+--------------------------------------
+This report used to say `not-derivable` for every requirement, because nothing tracked in
+git recorded a test result and the only prose record of a green run lived in
+DECISIONS-LOG.md — hand-maintained, unstructured, exactly the source the operator ruled
+out. The fix this docstring asked for now exists: `conformance/runs/ledger.jsonl`, an
+append-only ledger written by `lib/record_run.py` from a run's JUnit **and its run log**
+(see conformance/.gitignore for the negations that keep both in git).
 
-The honest fix is a committed, machine-readable run ledger; until that exists, the most
-this report can say is "wired and testable", never "proven".
+THE REFUSAL POSTURE IS UNCHANGED. Evidence replaced "not derivable" only where evidence
+exists; nowhere is optimism substituted for it:
+
+  * A ledger row reaches `proven` only if `record_run.py` saw that test's own PASS
+    sentinel token in the run log. The L2/L3 tests SELF-SKIP GREEN — Chainsaw records
+    `passed` on `DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries no
+    stdout — so a skip token maps to `skipped` and can never become proof.
+  * A proof is honoured only while it still describes the current code: if any of the
+    requirement's implementing manifests or verifying tests changed in a commit that is
+    NOT an ancestor of the run's `git_sha`, the row degrades to `stale-proof`, which is
+    not green. A run recorded from a dirty tree (`<sha>-dirty`) is never honoured.
+  * A requirement with no ledger row, or whose verifying tests did not ALL report proven,
+    stays `not-derivable` / `unknown` — never rounded up.
+  * The untracked `_artifacts/report.junit.xml` is STILL only a quarantined advisory with
+    its mtime. It is not the ledger, it is not evidence, and it feeds no status.
+
+With an empty or absent ledger this report behaves exactly as it did before: every
+requirement `not-derivable`, headline PROVEN 0, "the most this report can say is wired and
+testable, never proven".
 
 READS (all read-only)
   requirements/registry.yaml     the authoritative R-* mirror  (via lib/coverage.py)
   tests/**/chainsaw-test.yaml    requirement tags + layer + executable annotation
+  runs/ledger.jsonl              COMMITTED proof-of-pass rows (via lib/record_run.py)
   clusters/<env>/               Flux entrypoint -> the reconciliation graph
   `git ls-files`                 TRACKED files only; untracked scratch is not repo state
   ../home-network-k3s/docs/00-vision-and-scope.md   OPTIONAL registry<->spec parity check
@@ -92,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io"
 #   conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there.
 #   ci/tests          — gate unit-test fixtures embed example R-IDs as DATA.
 #   registry.yaml     — the requirement mirror itself, not an implementation of anything.
-IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/")
+# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the
+# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as
+# implementing evidence makes `last_change` move every time proof is recorded, so a row's
+# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is
+# committed — recording the proof destroys it. The ledger is the proof record; it must not
+# participate in the freshness computation about itself. (Found live 2026-09-01 on the first
+# PROVEN row.)
+IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/")
 IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)
 
 
@@ -343,6 +366,169 @@ def spec_requirement_ids(spec_root: Path) -> set[str] | None:
     return set(RID_RE.findall(doc.read_text()))
 
 
+# ========================================================================== ledger
+# PROOF axis. Everything here reads `conformance/runs/ledger.jsonl` (written ONLY by an
+# explicit `lib/record_run.py` / `run.sh --record` invocation) and downgrades relentlessly.
+# Nothing in this section may ever turn absent, ambiguous or aged evidence into green.
+
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# Severity ranking, WORST FIRST. A requirement's proof is the MINIMUM over its verifying
+# tests, so one failed/skipped/missing test drags the whole requirement down and a single
+# proven test can never carry the others. `stale-proof` ranks below `proven` on purpose:
+# it is a real past pass that no longer describes the current code, and it is NOT green.
+PROOF_ORDER = {
+    "failed": 0,              # a ledger row recorded a JUnit failure/error
+    "unknown": 1,             # ran, but the log could not evidence pass vs self-skip
+    "not-derivable": 2,       # no ledger row for this test at all (the pre-ledger default)
+    "junit-skipped": 3,       # JUnit <skipped/> — did not execute
+    "skipped": 4,             # SELF-SKIPPED green: exited 0 without exercising the behaviour
+    "stale-proof": 5,         # proven at a sha the implementation/test has since moved past
+    "proven": 6,              # the ONLY green state
+}
+# record_run.py's two "we cannot tell" states collapse to one reported state here; the
+# per-test detail in status.json keeps the distinction for whoever has to fix it.
+_PROOF_ALIASES = {
+    "unknown-no-sentinel-declared": "unknown",
+    "unknown-sentinel-absent": "unknown",
+}
+
+
+def read_ledger(path: Path) -> list[dict]:
+    """Parse the append-only JSONL ledger. A malformed line is SKIPPED, never guessed at.
+
+    Absent or empty file -> [] , which reproduces this report's pre-ledger behaviour
+    exactly (every requirement `not-derivable`, headline PROVEN 0).
+    """
+    if not path.is_file():
+        return []
+    rows: list[dict] = []
+    for line in path.read_text(errors="replace").splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        try:
+            obj = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(obj, dict) and obj.get("test"):
+            rows.append(obj)
+    return rows
+
+
+def latest_by_test(rows: list[dict]) -> dict[str, dict]:
+    """Most recent ledger row per test name (ts_utc, then file order as the tiebreak).
+
+    Latest wins rather than best-ever: a test that has since gone back to self-skipping
+    must stop reading as proven.
+    """
+    latest: dict[str, dict] = {}
+    for i, r in enumerate(rows):
+        key = r["test"]
+        prev = latest.get(key)
+        if prev is None or (r.get("ts_utc", ""), i) >= (prev.get("ts_utc", ""),
+                                                       prev["_i"]):
+            latest[key] = {**r, "_i": i}
+    return latest
+
+
+class _GitOracle:
+    """Small cached wrapper for the two git questions the staleness rule asks."""
+
+    def __init__(self, repo: Path):
+        self.repo = repo
+        self._last: dict[tuple[str, ...], str | None] = {}
+        self._anc: dict[tuple[str, str], bool] = {}
+
+    def _git(self, *args: str) -> subprocess.CompletedProcess:
+        return subprocess.run(["git", "-C", str(self.repo), *args],
+                              capture_output=True, text=True)
+
+    def last_change(self, paths: tuple[str, ...]) -> str | None:
+        """Commit that last touched any of `paths`; None if git knows none of them."""
+        if not paths:
+            return None
+        if paths in self._last:
+            return self._last[paths]
+        p = self._git("log", "-1", "--format=%H", "--", *paths)
+        sha = p.stdout.strip() if p.returncode == 0 else ""
+        self._last[paths] = sha or None
+        return self._last[paths]
+
+    def is_ancestor(self, older: str, newer: str) -> bool:
+        """True iff `older` is an ancestor of (or equal to) `newer`.
+
+        False whenever either rev is unresolvable — an unknown or `-dirty` sha (a run
+        recorded from a modified working tree) can never satisfy the freshness test, which
+        is the intended fail-closed direction.
+        """
+        key = (older, newer)
+        if key in self._anc:
+            return self._anc[key]
+        ok = self._git("merge-base", "--is-ancestor", older, newer).returncode == 0
+        self._anc[key] = ok
+        return ok
+
+
+def derive_proof(git: _GitOracle, rid: str, verifiers: list[dict],
+                 impl_files: list[dict], latest: dict[str, dict]) -> dict:
+    """PROOF for one requirement, from the ledger, with the staleness rule applied.
+
+    STALENESS RULE — a proof is a claim about a specific tree, not a permanent badge:
+      relevant paths = every implementing artifact this report found for the requirement
+                       + every verifying test's directory
+      last_change    = `git log -1 -- <relevant paths>`
+      A row's `proven` is honoured only if `last_change` is an ANCESTOR of the row's
+      `git_sha`, i.e. the run happened at or after the newest change to the code it
+      claims to prove. Otherwise it degrades to `stale-proof` (not green). A `-dirty`
+      sha, a sha this clone does not have (rebased/dropped), or a path git has never
+      seen (an uncommitted test) all fail the ancestry test and degrade the same way.
+
+    The requirement's state is the MINIMUM over its verifying tests: every verifying test
+    must be proven and fresh before the requirement reads `proven`. A test with no ledger
+    row contributes `not-derivable`, so a partially-recorded run cannot manufacture green.
+    """
+    paths = sorted({f["path"] for f in impl_files}
+                   | {os.path.dirname(v["path"]) for v in verifiers if v.get("path")})
+    last_change = git.last_change(tuple(paths))
+
+    per_test: list[dict] = []
+    for v in verifiers:
+        row = latest.get(v["name"])
+        if row is None:
+            per_test.append({"test": v["name"], "state": "not-derivable",
+                             "reason": "no ledger row — this test has never been "
+                                       "recorded by run.sh --record / record_run.py"})
+            continue
+        base = _PROOF_ALIASES.get(row.get("proof", ""), row.get("proof", "unknown"))
+        if base not in PROOF_ORDER:
+            base = "unknown"
+        state, reason = base, row.get("note", "")
+        if base == "proven":
+            sha = row.get("git_sha", "")
+            if last_change and not git.is_ancestor(last_change, sha):
+                state = "stale-proof"
+                reason = (f"proven at {sha[:12]}, but the implementing manifests / test "
+                          f"changed in {last_change[:12]}, which that run predates")
+        per_test.append({"test": v["name"], "state": state, "reason": reason,
+                         "run_id": row.get("run_id"), "ts_utc": row.get("ts_utc"),
+                         "git_sha": row.get("git_sha"), "env": row.get("env"),
+                         "sentinel": row.get("sentinel"),
+                         "junit_result": row.get("junit_result")})
+
+    if not per_test:
+        state = "not-derivable"
+        reason = "no requirement-tagged test exists to be proven"
+    else:
+        worst = min(per_test, key=lambda d: PROOF_ORDER[d["state"]])
+        state, reason = worst["state"], worst["reason"]
+        if state == "not-derivable" and not reason:
+            reason = "no committed ledger row for this requirement's verifying test(s)"
+
+    return {"state": state, "reason": reason, "last_change_sha": last_change,
+            "evidence_paths": paths, "per_test": per_test}
+
+
 # ========================================================================== build
 IMPL_ORDER = {"none": 0, "off-cluster": 1, "unwired": 2, "reconciled-partial": 3,
               "reconciled": 4}
@@ -369,7 +555,13 @@ LABELS = {
 
 def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, list[dict]],
           applied: set[str], gitops_roots: set[str], spec_ids: set[str] | None,
-          env: str) -> dict:
+          env: str, ledger_rows: list[dict] | None = None) -> dict:
+    # PROOF axis inputs. `ledger_rows=None` / `[]` reproduces the pre-ledger behaviour:
+    # `latest` is empty, so every requirement derives `not-derivable` exactly as before.
+    ledger_rows = ledger_rows or []
+    latest = latest_by_test(ledger_rows)
+    git = _GitOracle(repo)
+
     classified = [classify_test(t) for t in tests]
     by_req_tests: dict[str, list[dict]] = {r["id"]: [] for r in registry}
     for t, c in zip(tests, classified):
@@ -439,11 +631,9 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
                 "blocked_by": sorted({t["blocked_by"] for t in ts if t["blocked_by"]}),
                 "tests": ts,
             },
-            # NEVER derived. See the module docstring.
-            "proof": {"state": "not-derivable",
-                      "reason": "no committed, machine-readable record of a test run "
-                                "exists in the repo (_artifacts/ and results/ are "
-                                "git-ignored); DECISIONS-LOG.md prose is not a signal"},
+            # Derived ONLY from the committed ledger, and only as far as it reaches.
+            # With no ledger row this stays `not-derivable`, as it always was.
+            "proof": derive_proof(git, rid, ts, ev, latest),
             "status": LABELS[(impl_state, verify_state)],
         }
 
@@ -492,8 +682,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
             and v["verification"]["state"] == "executable"),
         "safety_critical_total": count(lambda v: v["safety_critical"]),
         "safety_critical_without_executable_l3": len(safety_scaffold),
-        "proven": 0,
-        "proven_derivable": False,
+        # PROVEN is now a real count — but only of requirements whose EVERY verifying
+        # test has a committed ledger row that saw its pass sentinel, at a sha the
+        # implementation has not moved past. `stale-proof` and `skipped` are counted
+        # separately and are NOT green.
+        "proven": count(lambda v: v["proof"]["state"] == "proven"),
+        "proof_stale": count(lambda v: v["proof"]["state"] == "stale-proof"),
+        "proof_skipped": count(lambda v: v["proof"]["state"] == "skipped"),
+        "proof_failed": count(lambda v: v["proof"]["state"] == "failed"),
+        "proof_unknown": count(lambda v: v["proof"]["state"] == "unknown"),
+        "proof_not_derivable": count(
+            lambda v: v["proof"]["state"] in ("not-derivable", "junit-skipped")),
+        "ledger_rows": len(ledger_rows),
+        "proven_derivable": bool(ledger_rows),
         "coverage_style_covered": count(lambda v: v["verification"]["state"] != "none"),
     }
     summary["uncited_by_any_reconciled_manifest"] = count(
@@ -552,16 +753,39 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append(f"| Safety-critical requirements whose L3 test is a scaffold | "
              f"**{s['safety_critical_without_executable_l3']}** of "
              f"{s['safety_critical_total']} |")
-    o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
-    o.append("")
-    o.append("> **Proof is deliberately not derived.** Nothing tracked in git records a "
-             "test result: `conformance/_artifacts/` and `results/` are git-ignored, and "
-             "the only record of a green run is DECISIONS-LOG.md prose — hand-maintained, "
-             "which is the very thing this report replaces. Per "
-             "`agents/definition-of-done.md` a requirement is done only when its tagged "
-             "test **passes**, so the strongest claim this report can make is "
-             "*wired and testable*. Closing the gap needs a committed, machine-readable "
-             "run ledger; until then, treat every row below as unproven.\n")
+    if s["proven_derivable"]:
+        o.append(f"| **PROVEN (every verifying test passed, evidenced)** | "
+                 f"**{s['proven']}** |")
+        o.append(f"| Proof exists but is STALE (code changed after the run) | "
+                 f"{s['proof_stale']} |")
+        o.append(f"| Verifying test SELF-SKIPPED green (not proof) | {s['proof_skipped']} |")
+        o.append(f"| Verifying test FAILED in the recorded run | {s['proof_failed']} |")
+        o.append(f"| Ran, but the log could not evidence pass vs self-skip | "
+                 f"{s['proof_unknown']} |")
+        o.append(f"| No committed ledger row yet | {s['proof_not_derivable']} |")
+        o.append("")
+        o.append(f"> **Proof comes from `conformance/runs/ledger.jsonl`** "
+                 f"({s['ledger_rows']} row(s)), written only by an explicit "
+                 f"`run.sh --record` / `lib/record_run.py` invocation. A row reaches "
+                 f"`proven` ONLY when the run log carried that test's own pass sentinel "
+                 f"token — the L2/L3 tests self-skip GREEN (Chainsaw records `passed` on "
+                 f"`DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries "
+                 f"no stdout), so a skip token maps to `skipped` and can never become "
+                 f"proof. A proof is honoured only while the implementing manifests and "
+                 f"the test itself have not changed since the recorded `git_sha`; "
+                 f"otherwise it degrades to `stale-proof`, which is not green.\n")
+    else:
+        o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
+        o.append("")
+        o.append("> **No proof is derivable: `conformance/runs/ledger.jsonl` is absent or "
+                 "empty.** The ledger is the committed, machine-readable run record; until "
+                 "a run is captured with `run.sh --record`, nothing tracked in git records "
+                 "a test result (`conformance/_artifacts/` and `results/` are git-ignored, "
+                 "and DECISIONS-LOG.md prose is hand-maintained — the very thing this "
+                 "report replaces). Per `agents/definition-of-done.md` a requirement is "
+                 "done only when its tagged test **passes**, so the strongest claim this "
+                 "report can make is *wired and testable*; treat every row below as "
+                 "unproven.\n")
     o.append(f"> For contrast, `coverage.md` counts **{s['coverage_style_covered']}** "
              f"requirements as covered — that count includes the "
              f"{s['verify_scaffold_only']} whose only verifying test is a TODO scaffold.\n")
@@ -612,7 +836,15 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
             if ver["blocked_by"]:
                 vcell += f" · blocked-by: {'; '.join(ver['blocked_by'])}"
         sc = " ⚠safety" if v["safety_critical"] else ""
-        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | not-derivable |")
+        pr = v["proof"]
+        if pr["state"] == "not-derivable":
+            pcell = "not-derivable"
+        else:
+            runs = sorted({d["run_id"] for d in pr["per_test"] if d.get("run_id")})
+            pcell = f"**{pr['state']}**" if pr["state"] == "proven" else pr["state"]
+            if runs:
+                pcell += f" ({', '.join(runs)})"
+        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | {pcell} |")
 
     if rep["safety_critical_without_executable_l3"]:
         o.append("\n## Safety-critical requirements whose L3 fault test is a SCAFFOLD "
@@ -701,22 +933,76 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append("| executable vs scaffold | TODO markers in the test body outrank the "
              "`executable` annotation; a real body without the annotation still counts "
              "and is flagged | conservative — ambiguity never upgrades |")
-    o.append("| **test actually passed** | **NOT DERIVED** — nothing in git records a "
-             "run result | refused |")
+    o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
+             "every verifying test: JUnit result **plus** that test's own pass sentinel "
+             "token grepped from the run log | exact where a run was recorded, refused "
+             "everywhere else — a self-skip token or a missing/unrecognised sentinel "
+             "NEVER yields `proven` |")
+    o.append("| proof still current | the row's `git_sha` must be a descendant of the "
+             "last commit touching the requirement's implementing manifests or its test "
+             "| conservative — an unresolvable or `-dirty` sha degrades to `stale-proof` |")
     o.append("| cluster liveness (pods/CRs healthy) | **NOT DERIVED** — no cluster is "
              "contacted; `reconciled` means Flux *would* apply it, not that it is "
              "healthy | refused |")
     o.append("| phase/board status | **NOT DERIVED** — `agents/task-board.md` is stale "
              "and is not a signal | refused |")
 
+    o.append("\n## Run ledger — the committed proof-of-pass evidence\n")
+    if not s["proven_derivable"]:
+        o.append("- `conformance/runs/ledger.jsonl` holds no rows. Capture a run with "
+                 "`./run.sh --record` (recording is never implicit: a run that silently "
+                 "records itself is how a bad run becomes a permanent false proof), then "
+                 "commit `conformance/runs/<run-id>/` **with its `run.log`** and the "
+                 "ledger. The log is the load-bearing half — Chainsaw's JUnit carries no "
+                 "stdout, so without it a self-skip is indistinguishable from a pass.")
+    else:
+        o.append(f"- {s['ledger_rows']} row(s) in `conformance/runs/ledger.jsonl`. "
+                 f"Per-test evidence for every non-`not-derivable` requirement below; "
+                 f"full rows (env, chainsaw/kube versions, sentinel, note) are in "
+                 f"`status.json` under `by_requirement.<R-ID>.proof.per_test`.\n")
+        o.append("| Requirement | Proof | Test | Ledger state | Sentinel | Run |")
+        o.append("|---|---|---|---|---|---|")
+        for rid, v in rep["by_requirement"].items():
+            pr = v["proof"]
+            if pr["state"] == "not-derivable":
+                continue
+            for d in pr["per_test"]:
+                o.append(f"| {rid} | {pr['state']} | `{d['test']}` | {d['state']} | "
+                         f"`{d.get('sentinel') or '—'}` | {d.get('run_id') or '—'} |")
+        stale = [(rid, v) for rid, v in rep["by_requirement"].items()
+                 if v["proof"]["state"] == "stale-proof"]
+        if stale:
+            o.append("\n### Stale proofs — a real pass that no longer describes this tree\n")
+            o.append("These requirements DID pass, but their implementing manifests or "
+                     "their verifying test changed in a commit the recorded run predates. "
+                     "A stale proof is not green; re-run with `--record`.\n")
+            for rid, v in stale:
+                pr = v["proof"]
+                o.append(f"- **{rid}** — {pr['reason']}")
+        skipped = [(rid, v) for rid, v in rep["by_requirement"].items()
+                   if v["proof"]["state"] == "skipped"]
+        if skipped:
+            o.append("\n### SELF-SKIPPED green — recorded, and deliberately NOT proof\n")
+            o.append("Chainsaw reported `passed` for these, because the test exited 0 "
+                     "after emitting its skip token: a precondition (Longhorn "
+                     "StorageClass, Velero CRDs, >=3 Ready nodes) was absent, so the "
+                     "behaviour was never exercised. This is exactly the row a "
+                     "JUnit-only ledger would have recorded as green.\n")
+            for rid, v in skipped:
+                toks = ", ".join(sorted({d.get("sentinel") or "?"
+                                         for d in v["proof"]["per_test"]
+                                         if d["state"] == "skipped"}))
+                o.append(f"- **{rid}** — skip token(s): `{toks}`")
+
     o.append("\n## Advisory — untracked local run artifact (NOT counted)\n")
     if advisory:
         o.append(f"- `{advisory['path']}` (untracked, git-ignored), last written "
                  f"{advisory['mtime']}: {advisory['tests']} testcase(s), "
                  f"{advisory['failures']} failure(s), {advisory['skipped']} skipped.")
         o.append(f"- requirement IDs it names: {', '.join(advisory['requirements']) or '—'}")
-        o.append("- This is one developer's local workspace, not repo state. It is shown "
-                 "for orientation only and contributes to NO status above.")
+        o.append("- This is one developer's local workspace, not repo state, and it is NOT "
+                 "the ledger. It is shown for orientation only and contributes to NO "
+                 "status above — a JUnit alone cannot tell a self-skip from a pass.")
     else:
         o.append("- none present.")
     return "\n".join(o) + "\n"
@@ -841,6 +1127,122 @@ def _selftest() -> int:
               classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
               == "scaffold")
 
+    # ---------------------------------------------------------------- PROOF / ledger
+    # These are the assertions that stand between this report and a fake green.
+    check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
+          read_ledger(Path("/nonexistent/ledger.jsonl")) == [])
+
+    def row(**kw):
+        base = {"ts_utc": "2026-09-01T12:00:00Z", "git_sha": "deadbeef", "env": "dev",
+                "chainsaw_version": "v0.2.12", "kube_version": "v1.36.4", "layer": "L3",
+                "test": "dr-restore", "requirements": ["R-STATE-03"],
+                "junit_result": "passed", "sentinel": None, "proof": "skipped",
+                "run_id": "20260901T120000Z-deadbee", "note": ""}
+        base.update(kw)
+        return base
+
+    verifiers = [{"name": "dr-restore",
+                  "path": "conformance/tests/l3-fault-injection/dr-restore/"
+                          "chainsaw-test.yaml"}]
+
+    with tempfile.TemporaryDirectory() as td:
+        gr = Path(td)
+        env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
+               "GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
+
+        def git(*a):
+            return subprocess.run(["git", "-C", str(gr), *a], capture_output=True,
+                                  text=True, env=env)
+
+        git("init", "-q", "-b", "main")
+        (gr / "platform").mkdir()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03\n")
+        git("add", "-A"); git("commit", "-qm", "one")
+        sha_a = git("rev-parse", "HEAD").stdout.strip()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03 v2\n")
+        git("add", "-A"); git("commit", "-qm", "two")
+        sha_b = git("rev-parse", "HEAD").stdout.strip()
+
+        oracle = _GitOracle(gr)
+        impl = [{"path": "platform/velero.yaml"}]
+
+        # *** THE CASE THIS WHOLE FEATURE EXISTS FOR ***
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="skipped", junit_result="passed",
+                                              sentinel="DR_RESTORE_SKIPPED_OK",
+                                              git_sha=sha_b)]))
+        check("a SELF-SKIP row on a GREEN JUnit is 'skipped', NEVER 'proven'",
+              pr["state"] == "skipped")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a proven row at the CURRENT sha is honoured", pr["state"] == "proven")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_a,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("STALENESS: proven at a sha the implementation has moved past degrades "
+              "to stale-proof", pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=f"{sha_b}-dirty",
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a run recorded from a DIRTY tree is never honoured as proof",
+              pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha="0" * 40)]))
+        check("a git_sha this clone cannot resolve degrades, never upgrades",
+              pr["state"] == "stale-proof")
+
+        for bad, expect in (("failed", "failed"),
+                            ("junit-skipped", "junit-skipped"),
+                            ("unknown-sentinel-absent", "unknown"),
+                            ("unknown-no-sentinel-declared", "unknown"),
+                            ("something-new-from-the-future", "unknown")):
+            pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                              latest_by_test([row(proof=bad, git_sha=sha_b)]))
+            check(f"ledger state {bad!r} maps to {expect!r}, never to proven",
+                  pr["state"] == expect)
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl, {})
+        check("no ledger row at all stays not-derivable",
+              pr["state"] == "not-derivable")
+
+        # Two verifying tests, only one proven: the requirement must NOT read proven.
+        two = verifiers + [{"name": "data-survival",
+                            "path": "conformance/tests/l3-fault-injection/"
+                                    "data-survival/chainsaw-test.yaml"}]
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b)]))
+        check("a requirement is proven only when EVERY verifying test is",
+              pr["state"] == "not-derivable")
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b),
+                                          row(test="data-survival", proof="skipped",
+                                              git_sha=sha_b)]))
+        check("worst-of wins across verifying tests (skip drags proven down)",
+              pr["state"] == "skipped")
+
+        # Latest row wins, not best-ever: a test that regressed to self-skipping must
+        # stop reading as proven.
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(ts_utc="2026-09-01T10:00:00Z",
+                                              proof="proven", git_sha=sha_b),
+                                          row(ts_utc="2026-09-02T10:00:00Z",
+                                              proof="skipped", git_sha=sha_b)]))
+        check("the LATEST ledger row wins, not the best one ever recorded",
+              pr["state"] == "skipped")
+
+        # A malformed ledger line is skipped, never guessed at.
+        lf = gr / "ledger.jsonl"
+        lf.write_text('{"test":"a","proof":"proven"}\nnot json at all\n\n'
+                      '{"proof":"proven"}\n')
+        rows = read_ledger(lf)
+        check("malformed / test-less ledger lines are dropped, not guessed at",
+              len(rows) == 1 and rows[0]["test"] == "a")
+
     if not ok:
         return 1
     sys.stderr.write("status.py selftest: PASS\n")
@@ -865,6 +1267,10 @@ def main() -> int:
                     help="exit 1 on defects this tool owns: registry/spec drift, "
                          "executable-annotation conflicts, or a safety-critical "
                          "requirement whose only L3 test is a scaffold")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"committed proof-of-pass ledger written by lib/record_run.py "
+                         f"(default {DEFAULT_LEDGER}). Absent or empty => every "
+                         f"requirement stays `not-derivable`, as before the ledger existed")
     ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
     args = ap.parse_args()
 
@@ -895,7 +1301,12 @@ def main() -> int:
     applied, nodes = reconciled_files(repo, entry)
     gitops_roots = {n["resolved"].split("/", 1)[0] for n in nodes if n["exists"]}
     impl = scan_implementation(repo, tracked_files(repo))
-    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env)
+    ledger_rows = read_ledger(args.ledger)
+    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env,
+                ledger_rows)
+    rep["ledger"] = {"path": os.path.relpath(args.ledger, REPO_ROOT),
+                     "rows": len(ledger_rows),
+                     "present": args.ledger.is_file()}
     rep["flux_graph"] = nodes
     rep["gitops_roots"] = sorted(gitops_roots)
     rep["reconciled_file_count"] = len(applied)
@@ -915,7 +1326,11 @@ def main() -> int:
           f"(+{s['wired_or_partial_and_testable'] - s['wired_and_testable']} partial) | "
           f"scaffold-only {s['verify_scaffold_only']} | "
           f"safety-critical w/o executable L3 "
-          f"{s['safety_critical_without_executable_l3']} | PROVEN: not derivable")
+          f"{s['safety_critical_without_executable_l3']} | "
+          + (f"PROVEN: {s['proven']} (stale {s['proof_stale']}, self-skipped "
+             f"{s['proof_skipped']}, failed {s['proof_failed']}, unknown "
+             f"{s['proof_unknown']}) from {s['ledger_rows']} ledger row(s)"
+             if s["proven_derivable"] else "PROVEN: not derivable (empty run ledger)"))
     print(f"[status] wrote {args.out/'status.md'} and {args.out/'status.json'}")
 
     rc = 0

conformance/run.sh

@@ -14,12 +14,33 @@
 #   ./run.sh --status                 # DERIVED STATUS report only (no cluster): per
 #                                     #   requirement, is it implemented, is it actually
 #                                     #   reconciled by Flux, is its test executable or a
-#                                     #   scaffold. Proof-of-pass is deliberately NOT
-#                                     #   derived — see lib/status.py.
+#                                     #   scaffold, and whether the committed run ledger
+#                                     #   (runs/ledger.jsonl) actually proves it — see
+#                                     #   lib/status.py. No ledger row => not-derivable.
 #   ./run.sh --lint                   # structural-validate test defs (no cluster)
 #   ./run.sh --strict-coverage        # hygiene gate: fail on tag/annotation drift
 #   ./run.sh --requirements R-NET-01  # ALSO asserts those IDs are covered (merge gate)
 #   ./run.sh --require-all            # phase-exit: every requirement must be covered
+#   ./run.sh --record                 # RUN AND CAPTURE: tee the whole run to
+#                                     #   runs/<UTC>-<short-sha>/run.log, copy the JUnit
+#                                     #   beside it, and append one ledger row per
+#                                     #   testcase to runs/ledger.jsonl (lib/record_run.py)
+#
+# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
+# --------------------------------------------------------------------------
+# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
+# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
+# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
+# that — `conformance/runs/` (see conformance/.gitignore for the negations).
+#
+# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
+# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
+#
+# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
+# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
+# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
+# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
+# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
 #
 # Artifacts (always written under conformance/_artifacts/, gitignored):
 #   _artifacts/report.junit.xml        JUnit (Chainsaw output, or skipped-scaffold shim)
@@ -28,6 +49,11 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Run ledger (TRACKED — commit these; --record only):
+#   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
+#   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
+#   runs/ledger.jsonl                         append-only, one row per testcase per run
+#
 # Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
 # execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
 # green — the harness is proven reproducible while later phases fill in behavior.
@@ -47,12 +73,17 @@ STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
 PYTHON="${PYTHON:-python3}"
 
+# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
+# under `tee`. Captured BEFORE parsing so nothing is normalized away.
+ORIG_ARGS=("$@")
+
 LAYER="all"
 REQS=""
 MODE="run"          # run | coverage-only | status | lint
 STRICT_COVERAGE=0
 REQUIRE_ALL=0
 FORCE_NO_CLUSTER=0
+RECORD=0
 
 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
 # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
@@ -83,7 +114,9 @@ FORCE_NO_CLUSTER=0
 # longer a gap and has been removed from this list; the coverage report counts it as covered.
 DOCUMENTED_GAPS=""
 
-usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
+# Print the header block (line 2 through the closing `####` rule, which is dropped) as
+# help. Bounded by the rule rather than a fixed line number so the header can grow.
+usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
 
 while [ $# -gt 0 ]; do
   case "$1" in
@@ -95,6 +128,7 @@ while [ $# -gt 0 ]; do
     --strict-coverage) STRICT_COVERAGE=1; shift ;;
     --require-all)    REQUIRE_ALL=1; shift ;;
     --no-cluster)     FORCE_NO_CLUSTER=1; shift ;;
+    --record)         RECORD=1; shift ;;
     -h|--help)        usage; exit 0 ;;
     *) echo "unknown arg: $1" >&2; usage; exit 2 ;;
   esac
@@ -104,6 +138,62 @@ mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
 
 log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
 
+###############################################################################
+# --record: capture this run as committed evidence, then hand it to the ledger.
+#
+# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
+# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
+# operator-visible output — no second, differently-behaving "recording mode" to drift.
+# CONF_RECORD_CHILD guards against recursion.
+#
+# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
+# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
+# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
+###############################################################################
+if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
+  case "${MODE}" in
+    run) ;;
+    # --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
+    # test, so a ledger row from one of them would be a row about nothing.
+    *) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
+  esac
+  RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
+  RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
+  RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
+  mkdir -p "${RUN_DIR}"
+  # Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
+  # ever capture a report THIS run produced. Copying a stale artifact would pair someone
+  # else's result with this run's log — a fabricated pairing, which is precisely the class
+  # of false proof the ledger exists to prevent.
+  rm -f "${JUNIT_OUT}"
+  # Strip --record from the argv handed to the child; everything else passes through.
+  CHILD_ARGS=()
+  for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
+    [ "$a" = "--record" ] || CHILD_ARGS+=("$a")
+  done
+  log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
+  set +e
+  CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
+    | tee "${RUN_DIR}/run.log"
+  RUN_RC=${PIPESTATUS[0]}
+  set -e
+  if [ -f "${JUNIT_OUT}" ]; then
+    cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
+  else
+    log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
+  fi
+  # Explicit, separate step: read the JUnit + the log and append one row per testcase.
+  # rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
+  # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
+  # to the real ledger; operators should never set it.
+  "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
+    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
+    --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
+  log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
+  log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
+  exit "${RUN_RC}"
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no

conformance/runs/20260901T235117Z-e9a4b66/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="131.090121" />
+  </testsuite>
+</testsuites>

conformance/runs/20260901T235117Z-e9a4b66/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 23:51:18 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1788306678898809444
+        victim node (currently hosts the attached volume): k3s-dev-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-dev-2 labeled
+        node/k3s-dev-2 cordoned
+        cordoned + labelled victim node: k3s-dev-2
+        pod "writer" deleted from chainsaw-fond-marlin namespace
+        evicted writer off the victim node; volume detached from k3s-dev-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
+        recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-dev-2 uncordoned
+        node/k3s-dev-2 unlabeled
+        restored node to service: k3s-dev-2
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-fond-marlin namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:53:29 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (131.09s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260902T001907Z-fdc3265/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="149.053701" />
+  </testsuite>
+</testsuites>

conformance/runs/20260902T001907Z-fdc3265/run.log

@@ -0,0 +1,671 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 00:19:07 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true http://192.168.1.9:9000 velero
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=velero  endpoint=http://192.168.1.9:9000
+        backup endpoint host: 192.168.1.9
+        backup endpoint resolves to: 192.168.1.9
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 81)
+        endpoint is not any Pod IP (checked 186)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=192.168.1.9 bucket=velero
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
+        namespace/chainsaw-cunning-hen-dr-src created
+        namespace/chainsaw-cunning-hen-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1788308352223263220
+        original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
+        backup.velero.io/chainsaw-cunning-hen-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=Completed
+        backup chainsaw-cunning-hen-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-cunning-hen-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-cunning-hen-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
+        recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
+        velero objects for chainsaw-cunning-hen cleaned up (ns velero)
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
+        released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:21:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (149.05s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -0,0 +1,2 @@
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}

infra/PROD-PROVISIONING.md

@@ -25,20 +25,455 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
 |---|---|---|---|
 | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
 | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
-| Model | HP EliteDesk 800 G5 Desktop Mini | | |
-| CPU | i7-9700T, 8c/8t, VT-x | | |
-| RAM | 31 GiB | | |
-| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | |
-| OS | Debian 13 (trixie), kernel 6.12 | | |
-| Firmware | HP R21 Ver. 02.26.00 | | |
-| `/dev/kvm` | present | | |
+| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini |
+| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x |
+| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) |
+| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
+| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
+| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 |
+| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 |
+| `/dev/kvm` | present | present | present |
+| Serial | MXL1052NGN | MXL0073949 | 8CC015307N |
+
+**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check;
+unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**),
+`DIMM3` a Micron **16**ATF2G64HZ-**2G6**E1 (DDR4-**2667**). Both are 16 GB so capacity is correct,
+but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded
+only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch.
+
+**The nodes are NOT hardware-symmetric — and with node 3 in (2026-09-03) the asymmetry is
+THREE-WAY, not a node-1 outlier.** Node 1 has a ~2011 SATA SSD with real wear; node 2 a healthy
+modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent health (100% life,
+0 grown bad blocks). **Three nodes, three different drive models, and only one is NVMe.** §2 requires
+an identical disk layout across all three because Longhorn replica placement assumes symmetry — so
+this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should
+have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs**
+(2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually
+usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table
+too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to
+the §0b bug on exactly the same terms as node 1 and needs the same >= 6.13 kernel. Node 3 therefore
+WIDENS §0b's blast radius from "both prod nodes" to **all three**; it is not a control machine.
+Every node also carries **exactly one** physical disk with **no unpartitioned spare**, so
+`node_baseline_longhorn_device` / `node_baseline_etcd_device` have nothing to claim on any of the
+three without repartitioning or adding drives — the §2 decision cannot be satisfied by node 3 either.
 
 Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).
 
 ---
 
+## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01)
+
+> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory
+> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine.
+> Keeping the correction visible because the wrong framing nearly cost money and a board RMA.
+
+`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware
+publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before
+reserving memory based on its contents.** Fixed upstream in **6.13**.
+
+### The evidence that breaks it open
+
+The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full
+15.9 GiB two boots earlier**:
+
+| Boot | `Memory: … available` | reserved | memattr line |
+|---|---|---|---|
+| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` |
+| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
+| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
+
+The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in
+that shape, and none is fixed by a reboot.
+
+### Mechanism (verified against kernel source)
+
+`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`:
+
+```c
+if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; }
+tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
+memblock_reserve(efi_mem_attr_table, tbl_size);
+```
+
+**`num_entries` and `desc_size` are never validated.** With a corrupt table:
+- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)*
+- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s
+  the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)*
+
+That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` =
+DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero.
+
+### ALL THREE prod nodes are affected — none is immune
+
+Seven boots observed across the three machines. **The MAT address moves boot-to-boot on every
+node with settings held constant**, and each node has been seen on more than one path:
+
+| Node | Boot | `MEMATTR=` | Kernel's read | Memory |
+|---|---|---|---|---|
+| 1 | −1 | `0xbeee6018` | *(no message — table read as VALID)* | 32.4 G ✅ |
+| 1 | 0 | `0xbeec9298` | version **1030** (> 2 → bail) | 32.4 G ✅ |
+| 2 | −2 | `0xbeadc018` | version **1463964749** (> 2 → bail) | 15.9 G ✅ |
+| 2 | −1 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ |
+| 2 | 0 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ |
+| 3 | −1 | `0xbee8a018` | version **1463964749** (> 2 → bail) | 32.4 G ✅ |
+| 3 | 0 | `0xbeece018` | *(no message — table read as VALID)* | 32.4 G ✅ |
+
+So there are **three** outcomes, not two: bail-out on a garbage version > 2, a *valid* parse with a
+sane size, and the catastrophic path. `k3s-prod-3` publishes a MAT at `0xbeece018` exactly like the
+others — the *absence* of a memattr diagnostic line on its current boot means the kernel read a
+**valid** table, which is the opposite of immunity. Dev nodes publish no MAT at all (Proxmox
+SeaBIOS/OVMF) and are genuinely unaffected.
+
+**All three prod nodes are on the same lottery and all three need the ≥ 6.13 kernel.**
+
+### The corrupting bytes are the OS boot chain, not the firmware or the DRAM
+
+The garbage version `1463964749` = `0x5742544D` appeared on **node 2 AND node 3** — two different
+machines, identical corrupting bytes. That rules out both firmware-specific behaviour and a DRAM
+fault: the overwriter is the **shared OS boot chain** (shim / GRUB / EFI stub / initrd placement),
+writing the same data into the page the firmware pointed at. This is the documented failure mode —
+the firmware places the MAT in memory the OS classifies as plain `System RAM`, and whoever allocates
+there next overwrites it.
+
+It also means the boot chain's own allocation footprint — initrd size, module set, boot device — is
+part of the causal chain, which matters for the experiment design below.
+
+
+
+### Mechanism, traced to the byte (second, independent confirmation)
+
+A second research pass reached the same root cause by a different route and pinned the arithmetic:
+
+- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The
+  kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that
+  `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it.
+- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and
+  when bit 31 lands the result is **negative**.
+- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to
+  ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical
+  address above `0xbeeb0018` is reserved.**
+- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages`
+  (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`.
+- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by
+  `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is
+  therefore positive proof the reservation already happened** — it is a consequence, not the cause.
+
+The cleanest A/B tell between the two nodes is one line of dmesg:
+
+| | `NODE_DATA(0) allocated` | meaning |
+|---|---|---|
+| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation |
+| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement |
+
+Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
+node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.
+
+### CORRECTION (2026-09-03) — "Failed to map" does NOT mean memory was lost
+
+An earlier version of this section, and the `firmware.efi_memattr` health check built from it, treated
+the log line `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` as proof of memory loss.
+**A live counter-example falsifies that.** `k3s-prod-2`, after being power-cycled on 2026-09-03:
+
+```
+MEMATTR=0xbeeb0018                                             <-- the SAME address as both bad boots
+[ 0.106493] efi: memattr: Failed to map ... @ 0xbeeb0018       <-- the SAME line
+MemTotal: 15857732 kB                                          <-- 15.1 GiB of 16 GB. FULL MEMORY.
+```
+
+**Two different kernel functions were being conflated:**
+
+- **`efi_memattr_init()`** does the `memblock_reserve()`. Catastrophic loss requires `tbl_size` to be
+  **negative** — signed `int`, sign-extended through `phys_addr_t`, clamped to `PHYS_ADDR_MAX`, so
+  everything above the table is reserved. A garbage-but-**positive** `tbl_size` reserves only a modest
+  wrong amount and costs nothing noticeable.
+- **`efi_memattr_apply_permissions()`** does the later `memremap()`, and **this is what prints
+  "Failed to map"**. It fails for *any* sufficiently wrong size — negative or merely large-positive.
+
+So the line proves **the table was corrupt and the kernel trusted it**, not that memory was lost. The
+cost is set by the *value* of the garbage on that boot, which is why the same address yielded 2.6 GiB
+twice and 15.1 GiB once.
+
+**This also refutes the address→outcome hypothesis outright** — it is not merely untested, it is
+disproven. The MAT address does not determine the outcome; the garbage value at it does.
+
+The health check has been re-based accordingly: the measured shortfall is computed **once** into a
+shared fact and both `memory.physical_total` and `firmware.efi_memattr` read it, so they can no longer
+contradict each other. `Failed to map` + normal memory is now **WARN** ("the bug fired and this boot
+survived it"), and FAIL requires a measured shortfall.
+
+### SECOND FAULT — node 2 hangs on a warm reboot, staying powered
+
+On 2026-09-03 `k3s-prod-2` failed to return from the **first** scripted reboot of the survey below. Its
+journal ends cleanly mid-shutdown (services stopping, `eno1: NIC Link is Down`) and then stops. The box
+stayed **powered on but off the LAN for ~38 minutes** — no ping, no SSH, ARP INCOMPLETE — until the
+operator held the power button and cold-booted it by hand.
+
+**This is arguably a worse prod defect than the memory bug.** A node that cannot survive an unattended
+warm reboot cannot be rolled in an HA cluster: every kernel update and maintenance window becomes a
+physical visit, and `roles/node_health` cannot see a machine that is off the network.
+
+Leading candidate: node 2 is the **only** node with `Recover OS after Boot Failure = Enable`. A box
+parked at a firmware recovery prompt awaiting a keypress presents exactly this way. Unproven — it needs
+eyes on the screen during a hang — but it promotes that setting from "coincidence" in the BIOS diff to a
+live availability defect. **Worth disabling on all three nodes regardless.**
+
+Consequence for testing: **the reboot survey is not viable unattended on node 2** until this is
+understood or out-of-band recovery exists.
+
+### THERE IS NO KERNEL CMDLINE WORKAROUND
+
+`efi_memattr_init()` is called from `efi_config_parse_tables()` under only
+`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()`
+**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap`
+**do not prevent the reservation.** Booting a fixed kernel is the only real fix.
+
+### The fix
+
+**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is
+clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message
+describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable
+branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints
+`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory.
+
+The hardening landed exactly one release too late for Debian 13: the bug has existed since the
+feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up
+`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim
+*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably
+lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the
+release lifetime, and the fix is small and already upstream.
+
+Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382)
+(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the
+table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook
+Studio G5**, 16 GiB → 1.5 GiB).
+
+### Will 6.12 ever get the fix? — researched 2026-09-02. **Not on its own.**
+
+| Question | Answer |
+|---|---|
+| In `linux-6.12.y` today? | **No.** Newest commit touching `memattr.c` on that branch is 2024-07-12. |
+| How long has it been available? | In mainline since **v6.13, 2025-01-19** — ~19 months and ~97 6.12.y point releases. |
+| Tagged `Cc: stable`? | **No.** Neither commit. |
+| Tagged `Fixes:`? | **No** — and this is structural, not an oversight. The unvalidated arithmetic has been there since the file was created in **2016**; it is not a regression, so there is no bad commit to point a `Fixes:` at. **Both automatic feeds — the `Fixes:`-scanning scripts and AUTOSEL — had nothing to grab.** |
+| Picked up by AUTOSEL? | **Never surfaced** for any branch. |
+| Has anyone asked? | **No.** Nothing on `[email protected]`, and nothing in all 8,027 open+archived Debian bugs against `src:linux`. We would be first. |
+| 6.12 LTS horizon | **EOL December 2028** — outlives Debian 13's own support window, so a backport is *not* time-barred. |
+
+**Every automatic path has already had its chance and produced nothing.** The probability that 6.12.y
+acquires this commit spontaneously is close to zero.
+
+**But the request is unusually strong, and the cost is one email.** The decisive card: the cherry-pick
+is **byte-clean**. Debian trixie 6.12.94-1's `drivers/firmware/efi/memattr.c` hashes to
+`164203429fa7a…`, **identical to the upstream patch's pre-image blob** — the file has not diverged at
+all between v6.12 and the commit's parent, so `git cherry-pick 8fbe4c49c0cc` applies with zero
+conflicts and zero adjustment. Neither the stable team nor Debian would have any backporting work to
+do, which removes the usual reason such requests are declined. The commit also meets every criterion
+in `stable-kernel-rules.rst`: in mainline, 17 insertions (limit 100 lines), fixes "a hardware quirk"
+and "a real bug that bothers people" — the enumerated categories.
+
+**Debian rebases**, so an upstream pickup reaches us automatically: trixie carries `linux 6.12.94-1`
+in *trixie-security* whose changelog opens *"New upstream stable update: ChangeLog-6.12.91"*. No
+action beyond `apt upgrade`.
+
+#### Plan
+
+1. **Run `trixie-backports` on both prod nodes now**, as an explicitly *provisional* measure. Any
+   6.13+ kernel carries the fix. Record the standing cost: backports packages are provided as-is,
+   without the main archive's security guarantees.
+2. **Email `[email protected]`** — this is `stable-kernel-rules.rst` Option 2 for an
+   already-merged patch. CC `[email protected]` and **Ard Biesheuvel `<[email protected]>`**,
+   who both authored and committed the fix as EFI maintainer. Include: the commit ID and subject;
+   **6.12.y only**; the byte-clean-apply proof above (it pre-empts the main objection); and the
+   reproduction — three consecutive boots on one machine, two outcomes, **byte-identical
+   `BIOS-e820` across all three** (the detail that proves the divergence is in the MAT, not the
+   memory map), 16 GB → 2.6 GiB, plus a second identical machine taking the safe path only by
+   chance. Attach both dmesgs. Corroboration exists:
+   [Ubuntu LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (HP ZBook G5,
+   16 GiB → 1.5 GiB) and SUSE bugzilla 1231465, cited in the commit message itself.
+3. **File a Debian BTS bug as secondary**, against `src:linux` via `reportbug` (the bug-script
+   metadata is mandatory), severity `important`, referencing the lore thread. Debian's kernel
+   handbook §9.1.7 allows cherry-picking a patch already accepted upstream — which this is,
+   maintainer-authored with two Acked-bys — so if stable@ goes quiet Debian can act on its own.
+4. **Trigger to make the deviation permanent:** no stable@ pickup within ~2 release cycles (4–6
+   weeks) and no Debian movement ⇒ encode the backports kernel in the provisioning IaC for every
+   prod node and document it as a standing deviation from the distro baseline.
+
+**If patching by hand, only `8fbe4c49c0cc` is needed.** The follow-up `5d0faa8e8369` loosens a
+heuristic that was rejecting *legitimate* tables (false kexec warnings in Meta's fleet); it is not
+required for this symptom and depends on the first commit anyway.
+
+### Is it a BIOS setting? — three-way diff, researched 2026-09-03. **No.**
+
+The operator's hypothesis, tested properly now that there are two working nodes and one broken one.
+All **240** attributes read on all three; identical key sets; identical firmware everywhere (same
+BIOS version string, ME 12.1.1.1022, GOP 9.0.1085, Super I/O 8.9.22). So every difference is settings
+or hardware, not firmware revision.
+
+**The discriminating set — both working nodes agree, the broken node differs.** 26 attributes differ
+in total; after excluding per-machine noise (serials, UUIDs, MACs, asset tags, Born On Date, Feature
+Byte, SKU, memory size, storage device names) and settings falling on a different partition, seven
+remain:
+
+| Attribute | nodes 1 & 3 (work) | node 2 (broken) | Verdict |
+|---|---|---|---|
+| **Fast Boot** | Disable | **Enable** | **plausible probability-shifter — the only live candidate** |
+| Legacy Boot Options | Disable | Enable | **coincidence — the setting is INERT, see below** |
+| Microsoft / Windows / MS Option ROM UEFI CA 2023 | Enable | Disable | coincidence (sub-page `db` size at most) |
+| TPM Activation Policy | F1 to Boot | Allow user to reject | coincidence |
+| Recover OS after Boot Failure | Disable | Enable | coincidence (but a policy problem in its own right) |
+
+**`After Power Loss` is eliminated by the third node** — node 3 *works* and shares node 2's
+`Power Off`; node 1 is the odd one out. It was never mechanistically credible (PCH/ME power policy,
+applied before firmware executes). It remains a §1 checklist violation on **two** nodes now.
+
+**`Legacy Boot Options` is affirmatively inert**, which a naive diff gets wrong. Three attributes are
+**identical on all three nodes** and settle it: `Configure Legacy Support and Secure Boot` =
+*"Legacy Support Disable and Secure Boot Enable"*, `Configure Option ROM Launch Policy` = *"All
+UEFI"*, `UEFI Boot Options` = Enable. `Legacy Boot Options` is only the boot-order checkbox; the
+master switch that actually instantiates a CSM has legacy support disabled on node 2 too. No CSM is
+built, no legacy option ROM is shadowed. The second-strongest candidate has **no boot-time
+allocation footprint at all**. Secure Boot is likewise not a differential — all three log
+`Secure boot enabled` and load the same HP DB cert.
+
+**`Fast Boot` is the one credible mechanism**, and for a *different reason* than the hypothesis this
+document previously recorded and killed. HP documents it verbatim: *"reduces boot up time by
+bypassing boot to USB, CD-ROM, and PXE. **This skips some preboot initialization steps.**"* Skipped
+preboot init ⇒ fewer boot-services allocations before the MAT is installed ⇒ a different heap top ⇒
+**the MAT lands at a different address**. That is exactly the perturbation that flipped the
+near-identical Dell case in [LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382),
+where the reporter's "fix" was **unplugging a USB mouse**. (The old Fast-Boot-caches-MRC-training
+theory stays dead — byte-identical `BIOS-e820` refuted it.)
+
+#### Verdict: the hypothesis is not supported, and this diff cannot support it even in principle
+
+1. **No setting can be the cause.** Node 2 booted good once and bad twice with settings constant, and
+   the MAT address moves every boot on all three machines.
+2. **`Fast Boot` may shift the odds.** That is the honest ceiling. There is currently zero evidence
+   that it does.
+3. **A fatal confound:** the `1=3 ≠ 2` partition is *perfectly collinear* with two hardware
+   differences that have at least as strong a mechanism — node 2 is the only **single-DIMM** node and
+   the only **NVMe-booting** node. Nodes 1 and 3 are both dual-DIMM and both SATA. **Adding node 3
+   did not break the confound; it doubled it.** A different boot device means a different initrd
+   module set and size, hence a different EFI-stub allocation footprint — the very thing that decides
+   whether the MAT page gets overwritten.
+
+**No model-specific documentation exists.** HP publishes nothing mentioning the EFI Memory Attributes
+Table, MAT placement, or an NX/DEP option on this generation. The only memory-adjacent options
+(`Option ROM Launch Policy`, `DMA Protection`, `Pre-boot DMA protection`) are identical fleet-wide. No
+HP advisory matches. The R21 changelog never mentions memory, and these nodes run **02.26.00 dated
+05/05/2026 — newer than anything HP has indexed publicly**, so there is no BIOS update to chase.
+
+#### The experiment, if it is ever worth running
+
+Two statistics kill the obvious approach. Node 2's failure rate is known only to **n = 3** (2 bad of
+3); the 95% Clopper–Pearson interval on p(bad) is ≈ **0.09–0.99**. Against a null that loose, even
+8 clean boots after a change prove nothing. **A baseline arm is mandatory**, and: detecting
+*elimination* needs ~**20 boots per arm**; detecting a mere odds shift (0.67 → 0.33) needs ~**50 per
+arm** for 80% power. **Do not design for an odds shift — design for elimination and report
+"inconclusive" otherwise.**
+
+**Zero-touch first step, and the one actually worth doing:** 20 scripted reboots of node 2 with
+*nothing changed*, logging `MEMATTR=` and the `Memory: …K/…K available` line per boot. That
+establishes the real p(bad) we lack, and tests the causal chain directly — **is the outcome predicted
+by where the MAT landed?** Present data hint yes: both failures at exactly `0xbeeb0018`, all five
+successes elsewhere. If the bad outcome maps onto a narrow address range, the mechanism is settled
+without touching anything.
+
+**If more is wanted, swap hardware rather than BIOS settings** — it crosses settings against memory
+topology in one move, needs no firmware write (writes are refused; a BIOS change means a physical F10
+visit per arm), and is work §2 wants anyway: move node 2's single Samsung DIMM into node 3 and node
+3's matched pair into node 2, then 20 boots each. Fault follows the DIMM ⇒ topology, settings
+exonerated. Fault stays on node 2 ⇒ node 2 implicated, and only then is Fast Boot worth testing.
+Fault disappears ⇒ single-DIMM topology. ~1 hour unattended.
+
+**None of this changes the action.** The fix is already known, is not in dispute, and applies to all
+three nodes: kernel ≥ 6.13.
+
+### Reboot survey — 40 boots, run 2026-09-03
+
+`playbooks/memattr-survey.yml` (+ `roles/memattr_survey`), reboots opt-in via
+`memattr_survey_reboots_authorised`, resumable from an on-node ledger, artifacts in
+`_artifacts/memattr-survey/`.
+
+| node | bad | n | p(bad) | 95% Clopper–Pearson |
+|---|---:|---:|---:|---|
+| prod-1 | 0 | 20 | 0.00 | 0.000 – 0.168 |
+| **prod-2** | — | **0** | — | **no data — hung on its first reboot** |
+| prod-3 | 0 | 20 | 0.00 | 0.000 – 0.168 |
+| 1+3 pooled | 0 | 40 | 0.00 | 0.000 – 0.088 |
+
+**The arm the experiment was designed around produced nothing.** Node 2's prior 2-of-3 is unchanged.
+And 0/20 is **not** elimination — it still permits p(bad) up to 17% per node.
+
+What the 40 boots *did* establish:
+
+- **Addresses are node-specific and highly concentrated.** Node 1 used only **3** distinct addresses
+  across 20 boots (15 of them at `0xbeec9298`); node 3 used 8. **No address was ever shared between
+  nodes.** This is a lottery with very few tickets per machine — which is exactly why 20 clean boots on
+  nodes that only ever drew safe tickets says little about the node that drew a bad one.
+- **`MEMATTR` → kernel parse was a perfect function across all 7 addresses seen more than once**
+  (37 boots), with both readings present in the sample. Same address, same parse, every time. Combined
+  with the counter-example above, the picture is: the *address* determines what the kernel finds, and the
+  *garbage value there* determines whether it is survivable.
+- **The corrupting bytes are stable per machine, not fleet-wide.** Node 1's garbage version was `1030` on
+  **all 22** of its bail boots across three different addresses; node 3's was `1463964749` on all 12. So
+  the earlier "shared OS boot chain" inference needs refining: the boot chain remains the plausible
+  writer, but its footprint is machine-specific.
+
+**Ansible defect worth knowing repo-wide:** a looped `include_tasks` under the default `linear` strategy
+is flattened per host, so hosts run **serially**. The first survey attempt rebooted node 1 twenty times
+while nodes 2 and 3 sat idle. Fixed with `strategy: free` plus a preflight assert that `forks` ≥ host
+count. It is silent and costs 3× wall clock.
+
+### What NOT to do
+
+- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is
+  a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be
+  installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6.
+- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full
+  R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries.
+- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket:
+  they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They
+  may appear to "fix" it. They are not fixes.
+
+### Hypotheses killed, and why — so they are not re-run
+
+- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by
+  an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP
+  documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control.
+- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by
+  `hp-bioscfg` on this platform.
+- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB.
+- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by
+  measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`.
+  Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never
+  `memblock_add()`ed — which would have left `present` at ~0, not 3340288.
+- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`.
+- **MTRR trim** — MTRRs near-identical to node 1, no trim warning.
+- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal.
+- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly
+  explained: everything from `0xbeeb0018` upward.
+
 ## 1. Firmware / BIOS (per node, before OS install)
 
+> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes
+> **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a
+> `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes",
+> which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
+> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye.
+>
+> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4,
+> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is
+> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means
+> F10 on the box.
+>
+> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is
+> achievable today with reads alone, and would have caught node 2's drift on its first health run.
+> Converging it automatically additionally requires provisioning a BIOS Setup Password on every
+> node, which is a real security decision (a stored credential that can rewrite firmware) and is
+> the operator's call, not a default.
+
+
 - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
       boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
 - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
@@ -50,7 +485,81 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di
       nodes match.
 - [ ] **Firmware version recorded** in §0 and consistent across all three nodes.
 
-## 2. Disk layout — **OPEN DECISION, blocks the build**
+### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01
+
+**There is no default BIOS password, and no factory default exists to look up.** HP business
+desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we
+observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5
+describes both purely as things the owner configures in Computer Setup.
+
+**Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents
+clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once).
+**Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely,
+turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The
+permissive default is the right setting for a homelab.
+
+#### CORRECTION — error `0x4` is NOT an authorisation failure
+
+I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That
+was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI
+`SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a
+different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have
+unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting`
+returns:
+
+| Code | Meaning |
+|---|---|
+| 0 | Success |
+| 1 | Not Supported |
+| **4** | **Failed** (generic) |
+| 6 | Access Denied |
+
+So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and
+the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither
+of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up
+in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`.
+
+**Most likely actual cause (inferred, testable):** with an empty password the driver emits an
+*untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when
+blank (`Password = "<utf-16/>"`). The driver only prepends that prefix for a *non-empty* string. So
+setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would
+work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this
+firmware already makes the driver log `Prerequisites size value exceeded the maximum number of
+elements supported or data may be malformed` at boot.
+
+**`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and
+`new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM;
+nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read.
+
+#### Recommendation
+
+1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff
+   `current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV
+   win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10.
+2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision
+   one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to
+   `current_password`, retry one benign attribute. Return `0` confirms the `<utf-16/>` theory; `6`
+   would mean a password mismatch; another `4` means the driver is incompatible with R21 and the
+   fallback is `hp-repsetup`.
+3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup
+   Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it
+   needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**.
+   Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a
+   security downgrade.
+4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in
+   principle — the private key never leaves the workstation, each change is individually signed, with
+   anti-replay — but provisioning and every signature require Windows-only tooling. For three
+   machines whose firmware settings change roughly never, the operational cost far exceeds the risk
+   reduction. BCU is Windows/WinPE-only.
+5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full
+   pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) —
+   functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via
+   `hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is
+   persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the
+   password in cleartext on the command line — it would land in process listings and shell history
+   unless wrapped. Store it in the Ansible vault **and** offline.
+
+## 2. Disk layout — **DECIDED (operator, 2026-09-03) · REVISED (operator, 2026-09-04)**
 
 `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
 further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
@@ -67,6 +576,218 @@ sda  477G  C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
 └─sda5 433G  /srv          <- the only large free space
 ```
 
+### THE DECISION (operator, 2026-09-03; layout REVISED 2026-09-04) — made, applied, and measured-not-assumed
+
+| | |
+|---|---|
+| **Data partition** | the **existing `/srv` partition** on each node's single physical disk (~432 GB nodes 1–2, ~422 GB node 3), re-purposed, re-labelled **`k3s-data`** and mounted at the neutral **`/data`** |
+| **Longhorn store** | **`/data/longhorn`** — a *subdirectory* of that partition |
+| **k3s `--data-dir`** | **`/data/k3s`** — the *sibling* subdirectory. Holds **embedded etcd AND the containerd image store** |
+| **etcd** | **no dedicated device** — it lives at `/data/k3s/server/db`, i.e. on the data partition, **not** on the OS filesystem |
+| **Shared-device I/O contention** | **ACCEPTED, knowingly** — unchanged by the 2026-09-04 revision |
+
+#### THE 2026-09-04 REVISION, AND THE CAPACITY EVIDENCE THAT FORCED IT
+
+The 2026-09-03 ruling gave this partition to Longhorn **only** and left k3s at its default
+`/var/lib/rancher/k3s`. That was wrong, for a reason that is arithmetic rather than judgement.
+
+**`/var` on these nodes is a separate partition of 11.8 GiB** (11.6 on node 3) — see the
+partition map above; it is `/dev/sda3` / `/dev/nvme0n1p3`, not the 30 GiB `/`. k3s's data dir
+puts **two** things on it: etcd's database and WAL (`server/db`) and, far larger, the
+**containerd image store** (`agent/containerd`).
+
+**Measured on the fully-deployed dev cluster** — the same platform and app set prod will run:
+
+| | |
+|---|---|
+| `agent/containerd` per node | **21–25 GB** |
+| total `/var` per node | **58–70 GB** |
+| prod `/var` capacity | **11.8 GiB** |
+
+So prod would have **filled `/var` during its first image pulls** — not eventually, not under
+load, but on the way up. And because etcd was on that same partition, a full `/var` means
+**etcd cannot write its WAL**. That is cluster death, not degradation: a wedged etcd on all
+three servers is an unavailable control plane, and it would have happened during bring-up.
+
+**The fix is to move k3s's whole data dir onto the 424 GB partition**, which is why that
+partition's mountpoint and filesystem label are no longer Longhorn-specific — it now serves
+**two tenants**. `/var` retains only the OS's own usage (503 MB, 5% used, measured 2026-09-04).
+
+**Why now.** Nothing was installed on prod when this was decided — verified 2026-09-04: no
+`/usr/local/bin/k3s`, no `/etc/rancher/k3s`, no `/var/lib/rancher` on any node. Moving a data
+dir before first install is a one-line inventory change; after bring-up it means draining and
+re-imaging three servers.
+
+**Why `/data` and `LABEL=k3s-data`.**
+
+- **`/data`** — short, neutral, tenant-agnostic, and *not* under `/var`. Deliberately **not
+  `/srv`**: `/srv` is the *claim mountpoint* the role discovers the device at, and the
+  idempotency contract is that `/srv` is gone from `/proc/mounts` and `/etc/fstab` once the
+  claim has happened. Re-using it as the destination would make "claimed" and "not yet
+  claimed" indistinguishable.
+- **`LABEL=k3s-data`** — deliberately **not** the bare `data`. Guard #4 of the role's assert
+  wall accepts a partition whose label is empty *or already ours*, and names `data` as an
+  example of somebody else's named volume; labelling ours `data` would make a stranger's
+  `data` volume look already-claimed and weaken that guard. `k3s-data` is cluster-scoped
+  rather than tenant-scoped, and matches the `k3s-etcd` label convention already in the repo.
+
+**The shared-device I/O risk is UNCHANGED and still accepted.** Longhorn and etcd were already
+going to share this spindle under the 2026-09-03 ruling — the revision changes *which*
+filesystem etcd sits on, not *which device*. The acceptance rationale and its exit condition
+below stand verbatim.
+
+**How prod's data dir is scoped away from dev.** `k3s_data_dir` is a **single** variable read
+by **two** roles: `roles/k3s_server` turns it into `--data-dir` in the install flags, and
+`roles/node_baseline` (via `node_baseline_k3s_data_dir`, which defers to it) creates the
+directory `0700` on the data partition and derives etcd's location from it
+(`node_baseline_etcd_mount` = `<data-dir>/server/db`). The role default is **k3s's own
+default**, `/var/lib/rancher/k3s`, so **dev is unchanged** — and dev also has a *dedicated
+etcd device* mounted at `server/db` under that path, which moving dev's data dir would orphan.
+Prod overrides it in one line in `inventory/group_vars/k3s_prod/vars.yml`, the same seam
+`kube_vip_address` / `k3s_api_host` / `node_baseline_backports_kernel` use, so a fourth prod
+node inherits it from group membership rather than from anyone remembering.
+
+**Longhorn's `defaultDataPath` follows the R-ENV-02 knob pattern, not a base edit.**
+`platform/longhorn/helmrelease.yaml` referenced a hardcoded `/var/lib/longhorn`; it now
+references `${LONGHORN_DATA_PATH}`, a `cluster-vars` knob with a base default and an explicit
+value in **both** overlays:
+
+| | |
+|---|---|
+| `clusters/base/cluster-vars.yaml` | `/var/lib/longhorn` (base default = the known-good dev value, same convention as `GATEWAY_VIP`) |
+| `clusters/dev` | `/var/lib/longhorn` — **must never change**; the live dev cluster has 18 volumes whose replicas are on disk under that path |
+| `clusters/prod` | `/data/longhorn` |
+
+`kustomize-controller` runs with `--feature-gates=StrictPostBuildSubstitutions=true`, so an
+undefined variable fails the Kustomization outright — hence base-first, then both overlays.
+`[parity]`: the prod value MUST equal `node_baseline_longhorn_data_path` in
+`group_vars/k3s_prod/vars.yml`. Ansible creates and mounts the directory; Helm tells Longhorn
+to use it. If the two disagree, Longhorn writes replicas onto the OS filesystem and `/var`
+fills up — the exact failure this revision exists to prevent.
+
+**Verified 2026-09-04 that dev's rendered output is byte-identical.** All 12 Flux
+Kustomizations that declare `postBuild.substituteFrom cluster-vars` were rendered
+(`kubectl kustomize <spec.path> | flux envsubst --strict`, substituting the dev overlay's
+*effective* cluster-vars) both before and after the change: 12/12 identical per-file SHA-256,
+and one aggregate SHA-256 over every rendered byte, `b34c5d3d…89e7d`, unchanged. The only
+dev-side difference anywhere is the presence of the new knob in the ConfigMap itself, carrying
+dev's existing literal. The same harness shows prod's render changing on exactly one line
+(`defaultDataPath: /var/lib/longhorn` → `/data/longhorn`), which is what proves the comparison
+could have detected a regression.
+
+**Per-node device paths — they are NOT the same, and that is why nothing hardcodes them.**
+Verified 2026-09-03 with `findmnt --mountpoint /srv` on each node:
+
+| Node | Boot media | `/srv` partition | Size | Contents before the claim |
+|---|---|---|---|---|
+| `k3s-prod-1` | SATA SSD `C400-MTFDDAC512M` | `/dev/sda5` | 432.5 GiB ext4 | `lost+found` only (`du -sh` 20K; `df` 2.1 M of 425 G) |
+| `k3s-prod-2` | **NVMe** `PM9A1 Samsung` | **`/dev/nvme0n1p5`** | 432.5 GiB ext4 | `lost+found` only (same) |
+| `k3s-prod-3` | SATA SSD `WDC WDS500G2B0A` | `/dev/sda5` | 422.2 GiB ext4 | `lost+found` only (same) |
+
+The `df` "5.1% used" that made `/srv` look occupied is **ext4's default 5% root-reserved
+blocks**, not data — confirmed per node by `ls -A /srv` (only `lost+found`) and
+`du -sh --exclude=lost+found /srv` (20 K). Nothing was destroyed.
+
+**How the role resolves the device — discovery, not per-host inventory.** A single
+`k3s_prod` group value cannot be right when node 2 is NVMe, and per-host `host_vars` would
+freeze a *fact about today's hardware* into inventory, where it goes stale the first time a
+node is re-imaged or a disk is swapped — and the stale value's failure mode is `mkfs` on the
+wrong partition. So `group_vars/k3s_prod/vars.yml` declares **intent**
+(`node_baseline_longhorn_source: claim-mountpoint`,
+`node_baseline_longhorn_claim_mountpoint: /srv`) and
+`roles/node_baseline/tasks/disks.yml` asks the **node** which block device is mounted there,
+then proves the answer is safe before writing anything. Six independent asserts, all
+fail-closed: it is a partition or whole disk (not tmpfs/NFS/bind), it is **not** backing
+`/`, `/boot`, `/boot/efi`, `/usr`, `/var`, `/home`, `/tmp` or active swap (that set is built
+from the node's *own* gathered facts plus `swapon`, never a hardcoded partition list), it is
+already ext4, it carries no other filesystem label, it is ≥ 100 GiB, and it is **empty**.
+
+**Unmount → relabel, not reformat.** `/srv` is already formatted and already mounted, and
+`community.general.filesystem` will not touch a device that has a filesystem — it would
+silently no-op and then the `LABEL=longhorn` mount would fail. The role therefore unmounts
+`/srv`, removes its `/etc/fstab` entry (installer-written, pinned by UUID — leaving it would
+re-mount the partition at `/srv` underneath Longhorn on the next boot), and stamps the label
+with `e2label`. Relabelling rather than reformatting is deliberate: the partition is already
+ext4 and already verified empty, so `mkfs` buys nothing, and if some assert were ever too
+permissive the worst case is a renamed filesystem instead of a destroyed one.
+
+**Idempotency guard.** A second run is a no-op, not a reformat. After the claim the
+partition carries the ext4 label `k3s-data` and `/srv` is gone from `/proc/mounts` *and*
+`/etc/fstab` — so "what is mounted at `/srv`" would find nothing on run 2. The role pivots
+on the durable on-disk evidence instead: `blkid -L k3s-data` returning a device means the
+node is already claimed, and **every** destructive task is gated on that returning nothing.
+
+**The 2026-09-04 transition, handled idempotently.** All three nodes had *already* been
+claimed under the previous naming (`LABEL=longhorn` at `/var/lib/longhorn`). Because the
+idempotency pivot asks "does the **current** label exist?", changing the label makes an
+already-claimed node answer *no* — and it would then walk into the greenfield claim path and
+fail closed at "the claim mountpoint must actually be a mountpoint", `/srv` having been
+consumed by that first claim. Fail-closed is correct, but it is not convergence, so the
+transition is **declared** rather than inferred:
+`node_baseline_data_retired_fs_label: longhorn` + `node_baseline_data_retired_mount:
+/var/lib/longhorn` in `group_vars/k3s_prod/vars.yml`. The role then unmounts the retired
+mountpoint, drops its `fstab` entry and re-stamps the label with `e2label` — a **metadata
+write, never a reformat**, exactly as in the claim path — having first proved that the retired
+mountpoint is still a mount, that it is backed by the very device the retired label names, and
+that it is **empty** so nothing is orphaned. The common `LABEL=`-based mount task then brings
+it up at `/data`, and the two per-tenant subdirectories are created (`0755` Longhorn, `0700`
+k3s, since the latter holds the etcd DB, the node token and the image store).
+
+The transition block is **one-time and self-retiring**: on every subsequent run
+`blkid -L k3s-data` succeeds and none of it is reachable. The two retired-* lines are safe to
+delete from `group_vars` once the fleet is converged (it is, as of 2026-09-04).
+
+**Three independent reasons the LIVE dev cluster cannot enter that transition** — it has 18
+Longhorn volumes on `/var/lib/longhorn` and unmounting it would orphan every replica:
+(1) both retired-* vars default to `""` and are set only in the prod-scoped `group_vars`;
+(2) the path is gated on `longhorn_source == 'claim-mountpoint'`, and dev is `device` — the
+same structural gate that keeps dev out of the claim path; (3) the emptiness assert, which
+fails closed on a partition with replica directories even if (1) and (2) were both defeated.
+Dev's mountpoint, label and k3s data dir are also simply **not changing**.
+
+**Run evidence (2026-09-04).** Full `node-baseline.yml` against `k3s_prod`: run 1 converged
+(5–6 changed per node), run 2 **`changed=0` on all three nodes** with every transition and
+claim task skipping. Per node afterwards: `blkid -L k3s-data` → `/dev/sda5` (nodes 1, 3) /
+`/dev/nvme0n1p5` (node 2); `blkid -L longhorn` → gone; `/data` mounted with 403 G / 394 G
+available; `/var/lib/longhorn` and `/srv` no longer mountpoints; `fstab` carrying exactly
+`LABEL=k3s-data /data ext4 defaults,nofail,x-systemd.device-timeout=5s 0 0`; and `/var` down
+to **5% used, ~11 GiB free**.
+
+**Dev can never reach this path.** Both role defaults stay `source: device`, so the LIVE
+dev cluster keeps its dedicated `/dev/vdb` (Longhorn) and `/dev/vdc` (etcd) and the claim
+tasks are structurally unreachable there — they are gated on `claim-mountpoint`, which only
+the prod-scoped `group_vars` sets.
+
+### The accepted risk, and its exit condition
+
+Dev was burned by exactly this shape: a Longhorn rebuild starved etcd WAL fsync
+(`docs/plans/ssd-etcd-isolation.md`), which is why dev has a third, dedicated etcd device.
+**Prod accepts the same shared-device contention** because the failure mode is materially
+cheaper here: dev's contention was on a **spinning HDD** with seek-bound queueing; every
+prod node is **SSD or NVMe** end-to-end, where concurrent rebuild I/O costs latency rather
+than collapsing it. The three nodes also have no second device to give etcd — one physical
+disk each, already fully partitioned, no free slot verified populated.
+
+**Accepted to be MEASURED, not assumed.** Exit condition, in order of preference:
+
+- **Measure it before it matters.** Under a deliberate Longhorn replica rebuild, watch
+  `etcd_disk_wal_fsync_duration_seconds` p99 and `etcd_disk_backend_commit_duration_seconds`
+  p99, plus `etcd_server_leader_changes_seen_total`. If fsync p99 stays inside etcd's
+  practical budget (~10 ms wal fsync / ~25 ms backend commit) with no leader churn, the
+  acceptance holds and this becomes a recorded measurement rather than an assumption.
+- **If it does not hold:** add an M.2 NVMe per node (chassis slot to be verified
+  physically) and flip `node_baseline_etcd_source` back to `device` — the role still
+  supports that path unchanged, and it is the dev shape.
+
+**~~Known, separate consequence — `/var` is only 12 GiB.~~ RESOLVED 2026-09-04 by moving
+k3s's data dir.** This was recorded here on 2026-09-03 as a capacity item "to watch or
+repartition". Re-measuring it against dev turned it from a watch item into a blocker: the
+container image store alone is **21–25 GB per node** on the fully-deployed dev cluster and
+`/var` totals **58–70 GB**, against prod's **11.8 GiB**. So prod would have filled `/var`
+during its first image pulls, and with etcd on the same partition a full `/var` means etcd
+cannot write its WAL. The fix was neither "watch" nor "repartition" but `--data-dir` — see
+"THE 2026-09-04 REVISION" above. `/var` now carries only the OS's own usage (503 MB, 5%).
+
 Consequences to settle before imaging nodes 2 and 3:
 
 - [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
@@ -100,24 +821,43 @@ Consequences to settle before imaging nodes 2 and 3:
       a member corrupts — this drive has already logged 32 of them.
       (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
       `DECISIONS-LOG.md`. It is not part of this verdict.)
-- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
-      (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
-      Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
-      prod must override them in `group_vars`. On a single-disk node there is no second device at
-      all, so one of:
-      - **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
-        SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
-        spindle.
-      - **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
-        Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
-      - **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
-        physical failure domain.
-- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
-- [ ] Record the chosen device paths here and set them in
-      `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
-      `/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
-      `node-baseline` run against a node still carrying those placeholders now stops on an explicit
-      assert naming this section, instead of failing obscurely at `mkfs` time.
+- [x] **Layout DECIDED and APPLIED 2026-09-03** — see "THE DECISION" above. `roles/node_baseline`
+      no longer expects a block-device path for either store; it takes a declared **source
+      enum** per environment (`roles/node_baseline/tasks/disks.yml`):
+      - **Longhorn** — `node_baseline_longhorn_source: claim-mountpoint` +
+        `node_baseline_longhorn_claim_mountpoint: /srv`. The `/dev/vdb` default (a virtio
+        name that does not exist on bare metal) is unused on prod; the real partition is
+        discovered on the node and guarded, because it is `/dev/sda5` on nodes 1 and 3 and
+        `/dev/nvme0n1p5` on node 2.
+      - **etcd** — `node_baseline_etcd_source: shared-data-partition` (was `root-filesystem`
+        until 2026-09-04). "No dedicated device" is a first-class, explicitly-configured state
+        that the `disks` assert **accepts**, not a blank variable that the tasks happen to
+        skip — and the *name* of that state now describes reality, because etcd is no longer
+        on the OS filesystem. The distinction is load-bearing, not cosmetic:
+        `shared-data-partition` is **asserted**, per node on every run, by checking that the
+        filesystem backing the etcd data dir really is the data partition's mount. Declaring
+        `k3s_data_dir: /data/k3s` does not by itself achieve that — had the mount silently
+        failed, `/data` would be a plain directory on `/` and etcd plus 21–25 GB of images
+        would land back on the OS filesystem, invisibly, until the disk filled. Leaving the
+        value at `root-filesystem` would have been a lie inventory told about the cluster, and
+        `root-filesystem` makes no such claim and gets no such assert. The role also still
+        asserts nothing is already mounted at the etcd data dir (so a half-migrated node
+        cannot be inherited silently) and reports which filesystem etcd actually lands on
+        (measured 2026-09-04: `/data`, 403 G / 394 G available, vs `/var`'s 10.7 GiB).
+      - The two rejected options stand rejected: **add a second drive** (no verified free slot;
+        the exit condition above reopens it if measurement fails) and **repartition** (same
+        single failure domain, plus a destructive re-image of three working nodes for no
+        isolation the SSDs do not already give).
+- [x] **All three nodes get the identical layout.** Same declaration, same mountpoint, same
+      resulting `/data` + `/data/longhorn` + `/data/k3s`. Not the same *device path* and not
+      the same *size* (432.5 / 432.5 / 422.2 GiB) — Longhorn tolerates unequal node capacity;
+      it is replica *placement* that assumes a symmetric layout, and the layout is symmetric.
+      Confirmed per node 2026-09-04 by `findmnt`/`lsblk`/`blkid` after the transition.
+- [x] **Chosen paths recorded here and set in `group_vars`.** The BLOCKED placeholders in
+      `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` are replaced — by the source
+      enums above, plus deliberately-unusable `/dev/NOT-A-DEVICE-…` values for the two device
+      variables so that flipping a source back to `device` without supplying a real path
+      still fails on the role's assert instead of silently inheriting dev's `/dev/vdb`.
 - [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
       is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
       tasks this decision blocks — see §5's drive-health box for the exact command.
@@ -130,6 +870,138 @@ Consequences to settle before imaging nodes 2 and 3:
 - [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
       `roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
       so the same role covers bare metal. Nothing to port.
+- [x] **KERNEL: a standing, deliberate deviation from the distro baseline — `linux-image-amd64`
+      from `trixie-backports` (DONE on all three nodes, 2026-09-04).**
+
+      **What.** `roles/node_baseline` gained a `kernel`-tagged block that enables
+      `trixie-backports` as a deb822 apt source and installs the `linux-image-amd64`
+      **metapackage** from it. Applied 2026-09-04; all three nodes landed on
+      **`linux-image-amd64` 7.1.8-1~bpo13+1 → kernel `7.1.8+deb13-amd64`**, rebooted one at a
+      time, and are running it.
+
+      ```bash
+      ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
+      ```
+
+      **Why.** §0b. HP's R21 firmware publishes a corrupt EFI Memory Attributes Table; Linux
+      < 6.13 reserves memory from it without validating the size, and `k3s-prod-2` has booted
+      with 2.6 GiB of its 16 GB, twice. The fix is upstream commit **`8fbe4c49c0cc`**
+      *"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, first
+      released in **v6.13**, hardened by `5d0faa8e8369`). It is absent from v6.12 and from
+      `linux-6.12.y`, carries neither `Cc: stable` nor `Fixes:`, and there is **no kernel
+      cmdline workaround**. A k3s **server** that intermittently comes up with 2.6 GiB will
+      fail under etcd plus its workload share on any reboot, so this is a **prerequisite to
+      installing k3s**, not a follow-up.
+
+      **How it is scoped.** `node_baseline_backports_kernel` defaults to **false** in
+      `roles/node_baseline/defaults/main.yml` and is set **true** exactly once, in
+      `inventory/group_vars/k3s_prod/vars.yml` — the same seam `kube_vip_address` /
+      `k3s_api_host` / `cluster_kubeconfig_name` use. Dev's Proxmox VMs publish no MAT and are
+      genuinely unaffected; they must not be dragged onto a backports kernel as a side effect
+      of sharing the role, and a fourth prod node must inherit the deviation from **group
+      membership**, not from a `k3s-prod-*` hostname test. Verified: a `--tags kernel` run
+      against dev skips all 8 tasks on all 3 hosts.
+
+      **Security cost, accepted.** `*-backports` packages are provided **as-is** and are
+      **not** covered by the Debian Security Team's guarantees for the main archive — no
+      security-tracker commitment, no DSAs. Recorded here as the price of the nodes keeping
+      their RAM.
+
+      **No apt pin, deliberately.** The backports `Release` file carries `NotAutomatic: yes`
+      + `ButAutomaticUpgrades: yes`, which apt already reads as *"priority 100 so nothing is
+      taken from backports by accident, 500 for a package explicitly installed from there so
+      it keeps getting updates."* Writing an explicit `Pin-Priority: 100` in
+      `preferences.d` would look like belt-and-braces and would in fact **switch off the
+      automatic security upgrades for the very kernel this deviation installs.** Enabling the
+      source is the whole configuration.
+
+      **Metapackage, not a pinned ABI.** A frozen `linux-image-6.19.14+deb13-amd64` would stop
+      receiving even backports' own updates the moment the ABI rolls. The metapackage tracks
+      whatever backports ships (>= 6.13 by construction). The **exact** version each node
+      landed on is recorded on the node itself at `/etc/node-baseline-backports-kernel`,
+      written by the same role, along with the rollback path below.
+
+      **Rollback.**
+      1. *Non-destructive, try first.* The stock kernels are still installed
+         (`6.12.107+deb13-amd64` and `6.12.94+deb13-amd64` on all three) and selectable from
+         GRUB → *Advanced options for Debian GNU/Linux*. That answers "is the new kernel the
+         problem?" without uninstalling anything — it also reinstates the memattr bug for that
+         boot.
+      2. *Permanent.* Set `node_baseline_backports_kernel: false` in
+         `group_vars/k3s_prod/vars.yml` **first** (otherwise the next Ansible run puts it
+         back), then on each node:
+         ```bash
+         apt-get install --reinstall linux-image-amd64/trixie
+         apt-get purge linux-image-7.1.8+deb13-amd64
+         rm /etc/apt/sources.list.d/trixie-backports.sources
+         apt-get update && update-grub && reboot
+         ```
+
+      **The role does NOT reboot**, and that is deliberate — see §0b "SECOND FAULT". Reboots
+      are a serialized, supervised operator step: one node at a time, verify each returns
+      before touching the next, never node 2 first. On 2026-09-04 all three returned in
+      **39–41 s** (node 1 39 s, node 3 41 s, node 2 39 s); node 2 did **not** reproduce its
+      2026-09-03 hang, but one clean warm reboot is not a fix and that defect stays open.
+
+      **Exit condition — how this deviation ends.** If `8fbe4c49c0cc` is ever picked up into
+      `linux-6.12.y` (and therefore into Debian's 6.12 rebase, which reaches us by plain
+      `apt upgrade`), this can be removed entirely: set `node_baseline_backports_kernel:
+      false`, roll back per the note on each node, and delete the `kernel` block from
+      `roles/node_baseline`. §0b records why that pickup will not happen on its own and what
+      request would cause it.
+- [x] **Verification of the fix, and a correction to how it is verified (2026-09-04).**
+
+      §0b says *"A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table
+      detected!`"*. That is true but **conditional, and it did not happen on any of the three
+      nodes.** `efi_memattr_init()` on >= 6.13 has **three** gates and only the third prints
+      that line:
+
+      | gate | message | reachable when |
+      |---|---|---|
+      | `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
+      | implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, `desc_size` does not |
+      | `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* `desc_size` read plausibly, entry count does not |
+
+      **All three are rejections**, and which one a boot prints is decided by *which field*
+      that boot's garbage corrupts. On 7.1.8 all three nodes hit the **first** gate:
+
+      | node | `MEMATTR=` | kernel line on 7.1.8 | MemTotal | reserved at boot |
+      |---|---|---|---|---|
+      | 1 | `0xbee84018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1030` | 31.1 GiB | 0.9 GiB |
+      | 2 | `0xbeeb0018` | `efi: memattr: Unexpected EFI Memory Attributes table version -1569980416` | 15.4 GiB | 0.6 GiB |
+      | 3 | `0xbeecc018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` | 31.1 GiB | 0.9 GiB |
+
+      Node 2 is the result that matters: **`0xbeeb0018` is the exact address that produced
+      2.6 GiB twice**, and on 7.1.8 it reserved 634,904 K instead of 13,913,856 K.
+
+      **Boot-independent proof the fix is compiled in**, since the log line is a lottery: the
+      running kernel image itself contains the string. Decompressing `/boot/vmlinuz-$(uname
+      -r)` on all three nodes yields, verbatim —
+
+      ```
+      efi: memattr: [Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)
+      efi: memattr: Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)
+      ```
+
+      — neither of which exists in 6.12. That is the positive proof; the `dmesg` line is not
+      available on demand.
+
+      **Two bugs in `roles/node_health` were fixed as a consequence**, because without them
+      the health check could not show the fix landing:
+      - `MEMATTR_VERSION_RE` in `files/node_health_collect.py` matched `(\d+)` only. The
+        kernel prints that field with `%d` — **signed** — so node 2's `-1569980416` did not
+        match, the state fell through to `clean`, and the node was reported **PASS "the kernel
+        accepted it without complaint"**: a false pass on the exact bug the check exists to
+        find. Now `(-?\d+)`.
+      - `firmware.efi_memattr` treated `bad_version` as WARN unconditionally, so a node on a
+        fixed kernel would have stayed WARN forever. The verdict now keys on the running
+        kernel version (`memattr_kernel_fixed`, `>= 6.13`): the same log line is WARN on
+        < 6.13 (a lucky boot) and PASS on >= 6.13 (the validation working). The pre-existing
+        `corrupt_rejected` PASS branch is unchanged and remains reachable.
+
+      **`playbooks/node-health.yml -e node_health_hosts=k3s_prod`, after the reboots:
+      `firmware.efi_memattr` = PASS on all three, `memory.physical_total` = PASS on all
+      three.** Dev re-run unchanged.
 
 ## 4. Network (per node)
 

infra/ansible/README.md

@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOP
 > ```
 > (Both take the same target-group override as the rest — see **Targeting prod** below.)
 
+> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
+> ```
+> ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
+> ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey
+> ```
+> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
+> until `-e memattr_survey_reboots_authorised=true` is passed. See
+> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
+
 ## Re-provisioning a node (tofu -replace / fresh OS)
 A node rebuild wipes **everything on the host filesystem** — including host-level config
 that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
@@ -109,6 +118,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
 `playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
 keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
 bring-up chain.
+`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
+for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
+role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
+can be mistyped; inventory membership cannot.
 
 ```bash
 # dev — completely unchanged, no extra vars

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -87,24 +87,155 @@ kube_vip_address: "192.168.2.9"
 # — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
 cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
 
-# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
-# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
-# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
-# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
-# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
-# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
-# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
+# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
+# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
+# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
+# backports kernel as a side effect of sharing the role), and this line switches it on for
+# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
+# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
+# rather than from anyone remembering.
 #
-# The overrides below are deliberately present-but-broken: leaving them UNSET would
-# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
-# hardware — a confusing failure that looks like a wiring bug. This value fails just
-# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
-# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
-# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
-# updates these two lines. All three nodes must end up with an identical layout
-# (Longhorn replica placement assumes symmetry) before any of this is used for real.
-node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
-node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
+# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
+# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
+# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
+# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
+# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
+# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
+# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
+# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
+# in §3.
+#
+# COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
+# guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
+# the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
+# back to false, roll back per the note the role writes to
+# /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
+# roles/node_baseline.
+#
+# The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
+# FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
+node_baseline_backports_kernel: true
+
+# --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
+# Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
+#
+#   DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
+#                     disk, claimed and mounted at the NEUTRAL mountpoint `/data`
+#                     (LABEL=k3s-data), then split between TWO tenants:
+#                       /data/longhorn   Longhorn's dataPath
+#                       /data/k3s        k3s --data-dir (etcd AND the containerd image store)
+#   ETCD           -> NO dedicated device. It lives inside /data/k3s on that partition.
+#
+# ================================ THE 2026-09-04 REVISION ================================
+# The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
+# /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
+# data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
+# deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
+# would therefore have filled /var during its FIRST image pulls — and with etcd on the same
+# partition, a full /var means etcd cannot write its WAL. That is cluster death, not
+# degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
+# mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
+# tenants. Nothing was installed on prod when this was decided, which is exactly why it was
+# decided then — it is cheap now and very expensive after bring-up.
+#
+# THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
+# were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
+# contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
+# MEASURED, not assumed — exit condition in §2.
+# =========================================================================================
+#
+# Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
+# not as a device path and a blank. That distinction is the point of this block:
+#
+# 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
+#    k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
+#    (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
+#    Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
+#    freezes a fact about today's hardware into inventory, where it goes stale silently the
+#    first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
+#    nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
+#    So inventory declares the INTENT ("the partition currently mounted at /srv") and the
+#    role discovers the device on the node, then proves it is safe to claim (a partition or
+#    disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
+#    unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
+#
+# 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
+#    describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
+#    layout etcd is inside /data/k3s on the claimed data partition, which is neither a
+#    dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
+#    Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
+#    and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
+#    on every run, that the filesystem backing the etcd data dir really is /data. If the
+#    mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
+#    WAL would land back on the OS filesystem, invisibly, until the disk filled. The
+#    `root-filesystem` value makes no such claim and gets no such assert — which is exactly
+#    why the two are distinct enum values rather than one with a comment.
+#
+# 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
+#    the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
+#    the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
+#    this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
+#    k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
+#    Longhorn volumes are untouched.
+node_baseline_longhorn_source: claim-mountpoint
+node_baseline_longhorn_claim_mountpoint: /srv
+node_baseline_etcd_source: shared-data-partition
+
+# --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
+# `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
+# CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
+# /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
+# as the destination would make "claimed" and "not claimed" indistinguishable.
+# `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
+# partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
+# else's named volume — so labelling ours `data` would make a stranger's `data` volume look
+# already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
+# `k3s-etcd` label convention already in this repo.
+node_baseline_data_mount: /data
+node_baseline_data_fs_label: k3s-data
+# Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
+# equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
+# side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
+# default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
+node_baseline_longhorn_data_path: /data/longhorn
+
+# --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
+# The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
+# `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
+# label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
+# would walk into the greenfield claim path — which fails closed at "the claim mountpoint
+# must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
+# is right but it is not convergence, so the transition is DECLARED here: the role unmounts
+# the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
+# the partition is empty (nothing orphaned) and is the device the retired label names.
+# Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
+# SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
+# then skip on the retired-label lookup and nothing else depends on them.
+node_baseline_data_retired_fs_label: longhorn
+node_baseline_data_retired_mount: /var/lib/longhorn
+
+# --- k3s --data-dir (§2) -----------------------------------------------------------
+# THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
+# claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
+# `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
+# which defers to this variable) creates the directory 0700 on the data partition and derives
+# etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
+# Ansible side cannot disagree with itself.
+#
+# Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
+# it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
+# dir would orphan that mount under a running cluster.
+k3s_data_dir: /data/k3s
+
+# Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
+# source uses a device path any more, but leaving these UNSET would let prod silently
+# inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
+# metal) the moment anyone flipped a source back to `device`. These values can never parse
+# as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
+# before mkfs is ever reached.
+node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
+node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
 
 # --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
 # roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1

infra/ansible/inventory/hosts.ini

@@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 
 # =============================================================================
 # PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
-# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
-# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
-# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
-# nothing has been run against node 1, and the operator has decided the CLUSTER stays
-# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
-# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
-# later member-add dance.
+# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
+# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
+# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
+# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
+# check and the node-baseline PACKAGE floor has been run, and the operator has decided
+# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
+# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
+# single-server-etcd window, no later member-add dance.
 #
 # KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
 # hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
@@ -87,12 +88,23 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
 
 [k3s_prod_secondary]
-# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md
-# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so
-# no play can accidentally target a host that isn't there; uncomment each line only once
-# that node is actually on the bench and answering SSH.
-# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
-# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
+# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
+# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
+# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
+# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
+# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
+# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
+# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
+# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
+# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
+# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
+# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
+# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
+# three-way, not a node-1 outlier.
+# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
+# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
+k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
+k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
 
 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
 # pattern; named to match the group_vars directory so Ansible auto-loads it).

infra/ansible/playbooks/memattr-survey.yml

@@ -0,0 +1,137 @@
+---
+# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
+# R-ENV-03, R-TEST-02).
+#
+# WHAT QUESTION THIS ANSWERS
+#   infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
+#   EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
+#   Linux 6.12 reserves everything from the table address to the top of RAM without
+#   validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
+#   playbook measures them:
+#
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
+#        failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
+#        fleet-wide were at other addresses. If bad outcomes map onto a narrow address
+#        range, the causal chain closes with nothing touched.
+#     2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+#   §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
+#   is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
+#   a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
+#   boots per arm; detecting a mere odds shift needs ~50.
+#
+# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
+# package installs, no hardware moves, no node_baseline run. Its job is to establish real
+# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
+# a Fast Boot flip, a 6.13 kernel) compares against it.
+#
+# HOW TO RUN IT
+#   Dry run first — every stage except the reboots, so the pipeline is proven before 45
+#   minutes of wall clock are committed:
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml'
+#
+#   The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml \
+#           -e memattr_survey_reboots_authorised=true'
+#
+#   Fewer rounds, or one node:
+#     -e memattr_survey_rounds=5        --limit k3s-prod-2
+#
+#   RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
+#   rounds are never lost or double-counted. Analysis alone, over existing artifacts:
+#     ansible-playbook playbooks/memattr-survey.yml --tags analyse
+#
+# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
+# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
+# not a member of memattr_survey_required_group — the dev cluster is live, and a survey
+# that reboots it 20 times because of a mistyped --limit is exactly the accident worth
+# engineering against.
+#
+# HOW TO READ THE OUTPUT
+#   infra/ansible/_artifacts/memattr-survey/  (gitignored, regenerated every run)
+#     <host>.rounds.jsonl  one JSON line per round, mirrored off the node after EVERY round
+#     <host>.boots.json    every boot the persistent journal holds, re-derived in one pass
+#                          at the end — the AUTHORITATIVE record
+#     summary.md           the analysis, also printed at the end of the run
+#   See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
+#   with NO memattr log line is a healthy third state and not a missing measurement.
+
+# =====================================================================================
+# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
+# before a single reboot. That is the right asymmetry: the cost of a false stop is a
+# re-run, the cost of a false start is rebooting a machine that was doing something.
+# =====================================================================================
+- name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  any_errors_fatal: true
+  tags: [survey]
+  tasks:
+    - name: Preflight and baseline probe
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: preflight
+
+# =====================================================================================
+# PLAY 2 — THE SURVEY.
+#
+# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
+# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
+# keep collecting.
+#
+# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
+# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
+# that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
+# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
+# blocks belongs to exactly one host, so the other two sit idle through it and the survey
+# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
+# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
+# appeared in the output. `free` lets each host walk its own task list at its own pace,
+# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
+#
+# Consequences worth knowing before editing: output INTERLEAVES between hosts (every
+# progress line is host-tagged for exactly this reason), and `run_once` is unreliable
+# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
+# directory create on every host instead. Concurrency is also capped by `forks`
+# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
+# than letting a bigger fleet silently serialise again.
+# =====================================================================================
+- name: memattr survey — reboot rounds and journal harvest
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  strategy: free
+  tags: [survey]
+  tasks:
+    - name: Run the reboot rounds
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: rounds
+
+    - name: Harvest every retained boot from the persistent journal
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: harvest
+
+# =====================================================================================
+# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
+# even if every surveyed node died, because the artifacts are already on disk and the
+# analysis counts rounds from THOSE rather than from Ansible's host state. A node that
+# vanished shows up as a short arm with a wider interval, not as a missing report.
+# =====================================================================================
+- name: memattr survey — analyse
+  hosts: localhost
+  gather_facts: false
+  become: false
+  tags: [analyse]
+  tasks:
+    - name: Produce the analysis
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: analyse

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
 # the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
 k3s_disable: "traefik,servicelb,local-storage"
 
+# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
+# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
+# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
+# image store (agent/containerd) — which is by far the largest tenant.
+#
+# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
+# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
+# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
+# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
+# running cluster's entire state directory out from under it and orphan that etcd mount.
+#
+# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
+# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
+# node inherits it from group membership rather than from anyone remembering. It is set
+# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
+# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
+# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
+# this same variable). One line moves both.
+#
+# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
+# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
+# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
+# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
+# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
+#
+# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
+# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
+# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
+# Restart handler for a value that is not changing. The install flags are the correct and
+# sufficient place, and prod has no k3s installed yet.
+k3s_data_dir: "/var/lib/rancher/k3s"
+
 # Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
 k3s_server_mode: "secondary"
 

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -22,6 +22,7 @@
       --write-kubeconfig-mode=644
       {% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
       --node-ip={{ k3s_node_ip }}
+      --data-dir={{ k3s_data_dir }}
 
 # On an ALREADY-installed cluster the install task above is idempotent (creates:
 # k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in

infra/ansible/roles/memattr_survey/README.md

@@ -0,0 +1,201 @@
+# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery
+
+**Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02
+**Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism
+is settled there and this role does **not** re-derive it.
+
+## The question
+
+§0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes
+Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it,
+and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from
+the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.
+
+Two things §0b leaves unmeasured, and this role measures:
+
+1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed
+   failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes
+   fleet-wide were at other addresses. If bad outcomes map onto a narrow address range,
+   the causal chain closes without touching hardware or firmware.
+2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each.
+
+## Why 20 rounds
+
+Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson
+interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is
+worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a
+mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds`
+is a variable; lowering it below 20 produces an arm that cannot answer the question it was
+built for, and the analysis will say so.
+
+## This is the BASELINE arm
+
+**Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware
+moves, no `node_baseline` run. The role installs nothing and configures nothing; its only
+write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's
+own allocation footprint — initrd size, module set, boot device — is *part of the causal
+chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that
+the kernel release and cmdline were constant on every node, for exactly this reason.
+
+## Running it
+
+```bash
+# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml'
+
+# The real thing. Reboots are OPT-IN; the interlock is deliberate.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml \
+      -e memattr_survey_reboots_authorised=true'
+
+# Re-analyse existing artifacts, no hardware touched.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml --tags analyse'
+```
+
+Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`,
+`-e memattr_survey_hosts=<group>`.
+
+20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are
+independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That
+parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count;
+preflight asserts the latter. See the playbook comment for why the default linear
+strategy silently serialises a looped `include_tasks` (it was measured, not assumed).
+
+## Safety — verified, not assumed
+
+The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That
+premise is re-checked on **every** run, because a document and a machine drift apart:
+
+| Check | Failure mode it prevents |
+|---|---|
+| Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. |
+| `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. |
+| `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. |
+| `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. |
+| `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. |
+
+Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the
+whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a
+false start is a rebooted production machine.
+
+## Fail-safe and resume
+
+* **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node
+  that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the
+  observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that
+  host only** and the others keep collecting. The abort is announced immediately and again
+  in the report — which counts rounds from the **artifacts**, not from Ansible's host
+  state, so even a node that goes fully unreachable (which `rescue` cannot catch) still
+  appears as a short arm with a wider interval, never as a silently missing report.
+* **Resumable.** Round numbers come from the on-node ledger, so re-running after an
+  interruption continues where it stopped and can neither lose nor duplicate a round. The
+  boot found on a resume is *not* re-appended: it is already round N of the interrupted
+  run, and double-counting it would bias the sample toward whichever outcome survived the
+  interruption.
+* **The abort is durable, not scrollback.** `<host>.aborted.json` records the round, the
+  failing task and the reason, so the report can say *why* an arm is short instead of
+  leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every
+  run and re-written only if that run aborts. Its *absence* is not proof of success — a
+  node can be lost abruptly enough that Ansible calls it unreachable before the rescue
+  runs — so completeness is still judged on round count, and the marker only supplies the
+  reason when there is one.
+* **Analysis survives everything.** It is a separate play on `localhost`, so it runs even
+  if every surveyed node died. With a node down, `--tags analyse` is the way in: a full
+  run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host.
+
+## Belt and braces on the data
+
+Two independent records, joined on `boot_id`:
+
+| | written by | authoritative for |
+|---|---|---|
+| `<host>.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** |
+| `<host>.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** |
+| `<host>.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** |
+
+The journal does not care whether the driver was alive: the boot happened, the kernel
+logged it, journald persisted it. So the harvest can only ever be a *superset* of the
+ledger, and a round the driver failed to append still shows up.
+
+## Reading the output
+
+Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as
+`roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never
+committed. `summary.md` is also printed at the end of the run.
+
+### The verdict is measured, never inferred from a log line
+
+This is the subtlest part of the design. There are **three** kernel readings, not two:
+
+| `kernel_read` | What the kernel logged | Memory |
+|---|---|---|
+| `valid` | **nothing at all** — it parsed a table it considered valid | intact |
+| `bail-version` | `Unexpected EFI Memory Attributes table version <garbage>` | intact — the *lucky* path, and it looks alarming |
+| `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** |
+
+Silence is therefore *evidence of health*, not a missing measurement — which is why the
+record carries an explicit `memattr_diagnostic_present: false` rather than an empty field.
+And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`,
+*later* than the `memblock_reserve()` that did the damage: it is a consequence, not the
+cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and
+`kernel_read` is recorded alongside it as an independent observation. §5a of the report
+cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.
+
+Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed
+clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything
+landing between them is reported as `indeterminate` and excluded from `n` — a new third
+thing deserves human eyes, not a coin flip into a bucket.
+
+### What is and is not pooled
+
+Boots the survey did **not cause** are shown in the per-round table for context and
+excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot
+each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so
+pooling it would double-count prior evidence into a run whose point is fresh trials).
+
+### The statistics
+
+`files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is.
+
+* **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the
+  Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly
+  instead of needing a special case. A bare fraction is what this exercise exists to stop
+  reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
+* **The address↔outcome question is tested three ways**, because the tempting version is
+  circular:
+  * **4b determinism** — was any single address seen with *both* outcomes? One such
+    address falsifies "the address determines the outcome" outright.
+  * **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed
+    up as a p-value: the bad set is defined after seeing the data.
+  * **4d the pre-registered prediction** — §0b, written before this run, names
+    `0xbeeb0018`. Fisher exact against this run's boots only. This is the one
+    non-circular test available.
+  * **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a
+    run produces no bad boot, `verdict` has no variance to explain, but `kernel_read`
+    does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in
+    §0b's mechanism. An address that reproduces the same *parse* boot after boot is the
+    address doing causal work, even when none of those parses was catastrophic.
+
+## Files
+
+| Path | What it is |
+|---|---|
+| `tasks/preflight.yml` | Safety gate, resume point, baseline probe |
+| `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock |
+| `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue |
+| `tasks/harvest.yml` | The authoritative journal pass |
+| `tasks/analyse.yml` | Controller-side analysis |
+| `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable |
+| `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report |
+
+## What this cannot tell you
+
+It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute
+the difference between nodes, because node 2 is simultaneously the only single-DIMM node
+and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled
+rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS
+settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this
+role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all
+three nodes, and that is not in dispute.

infra/ansible/roles/memattr_survey/defaults/main.yml

@@ -0,0 +1,92 @@
+---
+# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
+# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
+# settled and is NOT re-derived here.
+#
+# THE QUESTION THIS ROLE ANSWERS
+#   §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
+#   Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
+#        observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
+#        successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
+#        address range, the causal chain is closed with no hardware or firmware touched.
+#     2. Is the fault node-2-specific, or does any node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+# WHY 20 ROUNDS AND NOT FIVE
+#   Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
+#   interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
+#   boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
+#   needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
+#   not a round number.
+#
+# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
+#   No BIOS writes, no kernel change, no package installs, no hardware moves. The role
+#   installs nothing and configures nothing; its only write to a node is its own append-
+#   only ledger under /var/lib. Anything that alters the boot chain's allocation
+#   footprint — and per §0b that includes the initrd, the module set and the boot device —
+#   changes the variable under test and voids the arm.
+
+# --- Sample size --------------------------------------------------------------------
+# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
+# machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
+memattr_survey_rounds: 20
+
+# --- Safety interlock ----------------------------------------------------------------
+# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
+# preflight, a baseline probe of the current boot, the journal harvest and the analysis.
+# That is a genuinely useful dry run and it is also how you prove the pipeline works
+# before committing 45 minutes of wall clock to it.
+memattr_survey_reboots_authorised: false
+
+# The role refuses to run against a host that is not in this group. The prod boxes carry
+# no k3s and nothing is running on them, which is the entire reason rebooting them is
+# sanctioned; the dev cluster is live and must never be caught by a stray --limit.
+memattr_survey_required_group: k3s_prod
+
+# Unit names whose presence or activity means something IS running on this node. Any hit
+# aborts the whole playbook before the first reboot (see tasks/preflight.yml).
+memattr_survey_forbidden_units:
+  - k3s
+  - k3s-agent
+  - kubelet
+  - containerd
+  - docker
+# Paths whose existence means k3s has been installed here, even if not currently running.
+memattr_survey_forbidden_paths:
+  - /etc/rancher/k3s
+  - /var/lib/rancher/k3s
+  - /usr/local/bin/k3s
+
+# --- Reboot behaviour -----------------------------------------------------------------
+# A node that does not answer within this is declared lost: the role stops rebooting THAT
+# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
+# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
+# slow boot.
+memattr_survey_reboot_timeout: 600
+memattr_survey_connect_timeout: 20
+# Let journald settle before reading the new boot's kernel log. The kernel log itself is
+# complete long before SSH is up; this is purely margin.
+memattr_survey_post_reboot_delay: 15
+
+# --- On-node ledger (belt) -------------------------------------------------------------
+# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
+# the controller dies. This is what makes the survey resumable: re-running skips rounds
+# already recorded here rather than starting over.
+memattr_survey_state_dir: /var/lib/memattr-survey
+memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
+
+# --- Controller artifacts (braces) ------------------------------------------------------
+# Same convention as roles/node_health and conformance/_artifacts/: generated output under
+# a `_artifacts/` directory that .gitignore excludes, never committed.
+#   <host>.rounds.jsonl  the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
+#                        mid-survey still leaves every completed round on the controller
+#   <host>.boots.json    the journal harvest — every retained boot re-derived in one
+#                        pass. THIS is the authoritative record; the ledger is the backup.
+#   summary.md           the analysis (also printed)
+memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
+memattr_survey_write_artifacts: true
+
+# Print each round's one-line result as it happens. A 45-minute unattended run with no
+# output is indistinguishable from a hung one.
+memattr_survey_show_progress: true

infra/ansible/roles/memattr_survey/files/memattr_analyse.py

@@ -0,0 +1,521 @@
+#!/usr/bin/env python3
+"""memattr_analyse.py — turn a memattr survey's artifacts into the answer.
+
+Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads
+  <host>.boots.json    the journal harvest — authoritative, one entry per retained boot
+  <host>.rounds.jsonl  the per-round ledger — authoritative for ROUND NUMBERS
+and joins them on boot_id.
+
+WHAT IT ANSWERS, in order:
+
+  1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated
+     loudly and up front, because every number below it is weaker than it looks.
+
+  2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing
+     this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and
+     completely different evidence. Clopper-Pearson is the exact (conservative) binomial
+     interval — no normal approximation, correct at 0/n and n/n where the approximation is
+     degenerate and where a clean arm is most likely to land.
+
+  3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three
+     ways, deliberately, because the tempting version of this test is circular:
+       (a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such
+           address falsifies "the address determines the outcome" outright, and no amount
+           of correlation elsewhere repairs it.
+       (b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported
+           as sorted min/max per outcome plus the count of good addresses falling inside
+           the bad range. This is descriptive; it is NOT dressed up as a p-value, because
+           the bad set was chosen after seeing the data.
+       (c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts
+           that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots
+           this run observed, which is the one non-circular test available.
+
+  4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory
+     ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell.
+
+Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import math
+import os
+import sys
+from collections import Counter, defaultdict
+
+# §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad
+# both times, while all five successes fleet-wide were at other addresses.
+PREREGISTERED_BAD_ADDRESS = "0xbeeb0018"
+
+
+# --- Exact binomial interval ----------------------------------------------------------
+def binom_tail_ge(n: int, k: int, p: float) -> float:
+    """P(X >= k) for X ~ Bin(n, p)."""
+    if k <= 0:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1))
+
+
+def binom_tail_le(n: int, k: int, p: float) -> float:
+    """P(X <= k) for X ~ Bin(n, p)."""
+    if k >= n:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1))
+
+
+def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
+    """Exact 95% (by default) binomial confidence interval for k successes in n trials.
+
+    Solved by bisection on the exact tail probabilities rather than via the Beta/F
+    quantile identity, so it needs nothing outside the standard library and the
+    boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing:
+      lower bound solves P(X >= k | p) = alpha/2,  and is 0 when k = 0
+      upper bound solves P(X <= k | p) = alpha/2,  and is 1 when k = n
+    """
+    if n == 0:
+        return (0.0, 1.0)
+
+    def bisect(fn, target, lo, hi):
+        for _ in range(200):
+            mid = (lo + hi) / 2
+            if fn(mid) < target:
+                lo = mid
+            else:
+                hi = mid
+        return (lo + hi) / 2
+
+    lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0)
+    upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0)
+    return (lower, upper)
+
+
+def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float:
+    """Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only.
+
+    Used ONLY for the pre-registered prediction (address == the §0b address vs outcome),
+    never for a bad set defined after the fact — that would be a garden of forking paths
+    with a p-value stapled on.
+    """
+    n = a + b + c + d
+    if n == 0:
+        return 1.0
+    row1, col1 = a + b, a + c
+
+    def prob(x):
+        return (
+            math.comb(row1, x)
+            * math.comb(n - row1, col1 - x)
+            / math.comb(n, col1)
+        )
+
+    observed = prob(a)
+    lo = max(0, col1 - (n - row1))
+    hi = min(row1, col1)
+    # 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating
+    # float noise in the equality case.
+    return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9)))
+
+
+# --- Loading ---------------------------------------------------------------------------
+def load_host(art_dir: str, host: str) -> dict:
+    """Harvest (authoritative outcomes) joined to ledger (authoritative round numbers)."""
+    with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle:
+        harvest = json.load(handle)
+
+    rounds_by_boot: dict[str, int] = {}
+    ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl")
+    ledger: list[dict] = []
+    if os.path.exists(ledger_path):
+        with open(ledger_path, encoding="utf-8") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                try:
+                    record = json.loads(line)
+                except json.JSONDecodeError:
+                    continue
+                ledger.append(record)
+                if record.get("boot_id") and isinstance(record.get("round"), int):
+                    rounds_by_boot[record["boot_id"]] = record["round"]
+
+    boots = []
+    for boot in harvest.get("boots", []):
+        entry = dict(boot)
+        entry["round"] = rounds_by_boot.get(boot.get("boot_id"))
+        boots.append(entry)
+    boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0)))
+
+    # Written by the per-round rescue when a node stops coming back. Its absence is not
+    # proof of success — a node can be lost so abruptly that Ansible calls it unreachable
+    # before the rescue runs — so completeness is still judged on round COUNT, and this
+    # only supplies the reason when one exists.
+    abort = None
+    abort_path = os.path.join(art_dir, f"{host}.aborted.json")
+    if os.path.exists(abort_path):
+        try:
+            with open(abort_path, encoding="utf-8") as handle:
+                abort = json.load(handle)
+        except (OSError, json.JSONDecodeError):
+            abort = {"reason": "abort marker present but unreadable"}
+
+    return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort}
+
+
+def discover_hosts(art_dir: str) -> list[str]:
+    return sorted(
+        os.path.basename(p)[: -len(".boots.json")]
+        for p in glob.glob(os.path.join(art_dir, "*.boots.json"))
+    )
+
+
+# --- Rendering ---------------------------------------------------------------------------
+def gib(kb) -> str:
+    return "?" if kb is None else f"{kb / 1048576:.2f}"
+
+
+def pct(x) -> str:
+    return "?" if x is None else f"{100 * x:.0f}%"
+
+
+def render(hosts: list[dict], expected_rounds: int) -> str:
+    out: list[str] = []
+    w = out.append
+
+    all_boots = [b for h in hosts for b in h["boots"]]
+    # Only boots this survey CAUSED count toward any statistic. Excluded, deliberately:
+    #   * pre-existing journal boots (round None) — observed under an unknown selection
+    #     process, and already counted once in §0b's table;
+    #   * round 0, the boot each node was found sitting on — it is a boot nobody chose,
+    #     but for node 2 it is literally §0b's "boot 0" row, so pooling it would double-
+    #     count prior evidence into a run whose entire point is fresh, independent trials.
+    # Both are still SHOWN in the per-round table, because context is not contamination.
+    surveyed = [b for b in all_boots if b.get("round") not in (None, 0)]
+
+    w("# EFI Memory Attributes Table — baseline reboot survey")
+    w("")
+    w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no")
+    w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this")
+    w("measures p(bad) per node and tests whether the outcome is predicted by the address")
+    w("the firmware published the table at.")
+    w("")
+
+    # --- 1. Completeness -----------------------------------------------------------------
+    w("## 1. Completeness — read this before any number below it")
+    w("")
+    w("| node | rounds recorded | target | journal boots harvested | short? |")
+    w("|---|---:|---:|---:|---|")
+    short_arms = []
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        is_short = n < expected_rounds
+        if is_short:
+            short_arms.append((h["host"], n))
+        w(
+            f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | "
+            f"{'**YES — arm incomplete**' if is_short else 'no'} |"
+        )
+    w("")
+    if short_arms:
+        w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms))
+        w("> Every interval below is correspondingly wider than planned.")
+        w("")
+        for h in hosts:
+            if h.get("abort"):
+                a = h["abort"]
+                w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of "
+                  f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}")
+                w(f"> (failing task: {a.get('failed_task', 'unknown')})")
+                w("")
+
+    # --- 2. Per-round table ---------------------------------------------------------------
+    w("## 2. Per-round results")
+    w("")
+    for h in hosts:
+        installed = h["harvest"].get("smbios_installed_kb")
+        w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)")
+        w("")
+        w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |")
+        w("|---:|---|---|---|---:|---:|---:|---|")
+        for b in h["boots"]:
+            rnd = b.get("round")
+            label = "pre" if rnd is None else str(rnd)
+            usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb")
+            w(
+                f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | "
+                f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | "
+                f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | "
+                f"{gib(usable)} | {pct(b.get('usable_fraction'))} | "
+                f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |"
+            )
+        w("")
+        w("`round` = survey round. `pre` = a boot already in the journal before the survey;")
+        w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED")
+        w("from every statistic below. `kernel read`: `valid` = table")
+        w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage")
+        w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =")
+        w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from")
+        w("the presence of a log line.")
+        w("")
+
+    # --- 3. p(bad) -------------------------------------------------------------------------
+    w("## 3. p(bad) per node — 95% Clopper-Pearson")
+    w("")
+    w("| node | bad | n | p(bad) | 95% CI | reading |")
+    w("|---|---:|---:|---:|---|---|")
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        bad = sum(1 for b in rounds if b.get("verdict") == "bad")
+        lo, hi = clopper_pearson(bad, n)
+        if n == 0:
+            reading = "no data"
+        elif bad == 0:
+            reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}"
+        elif bad == n:
+            reading = f"bad every time in {n}; p(bad) at least {lo:.2f}"
+        else:
+            reading = f"p(bad) between {lo:.2f} and {hi:.2f}"
+        w(
+            f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | "
+            f"{lo:.2f} – {hi:.2f} | {reading} |"
+        )
+    w("")
+    indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")]
+    if indet:
+        w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are")
+        w("> reported as `indeterminate` rather than forced into a bucket. They are")
+        w("> excluded from n above. Inspect them by hand.")
+        w("")
+
+    # --- 4. Address vs outcome -----------------------------------------------------------
+    w("## 4. Is the outcome predicted by where the table landed?")
+    w("")
+    by_addr: dict[str, Counter] = defaultdict(Counter)
+    addr_hosts: dict[str, set] = defaultdict(set)
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        by_addr[addr][b.get("verdict")] += 1
+        addr_hosts[addr].add(b.get("host") or "?")
+
+    w("### 4a. Every address observed in the survey")
+    w("")
+    w("| MEMATTR | good | bad | other | nodes | verdict for this address |")
+    w("|---|---:|---:|---:|---|---|")
+    ambiguous = []
+    for addr in sorted(by_addr, key=lambda a: (a == "none", a)):
+        counts = by_addr[addr]
+        good, bad = counts.get("good", 0), counts.get("bad", 0)
+        other = sum(v for k, v in counts.items() if k not in ("good", "bad"))
+        if good and bad:
+            note = "**BOTH — address does not determine outcome**"
+            ambiguous.append(addr)
+        elif bad:
+            note = "always bad"
+        elif good:
+            note = "always good"
+        else:
+            note = "—"
+        w(
+            f"| `{addr}` | {good} | {bad} | {other} | "
+            f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |"
+        )
+    w("")
+
+    w("### 4b. Determinism")
+    w("")
+    if not by_addr:
+        w("No surveyed boots — nothing to test.")
+    elif ambiguous:
+        w(f"**FALSIFIED for {len(ambiguous)} address(es):** "
+          + ", ".join(f"`{a}`" for a in ambiguous)
+          + " was seen with BOTH outcomes. The address alone cannot determine the outcome.")
+    else:
+        w("**Not falsified.** No address in this survey was ever seen with both outcomes —")
+        w("consistent with the address determining the outcome, though with this many")
+        w("distinct addresses most were seen only once, which makes the test weak.")
+        singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1)
+        w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)")
+    w("")
+
+    w("### 4c. Separation of the address ranges")
+    w("")
+    bad_addrs = sorted({b["memattr_address"] for b in surveyed
+                        if b.get("verdict") == "bad" and b.get("memattr_address")})
+    good_addrs = sorted({b["memattr_address"] for b in surveyed
+                         if b.get("verdict") == "good" and b.get("memattr_address")})
+    if bad_addrs and good_addrs:
+        blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16)
+        inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi]
+        w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`")
+        w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`")
+        w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**"
+          + (" — the ranges interleave, so 'a narrow bad range' does not hold"
+             if inside else " — the ranges are disjoint"))
+        w("")
+        w("Descriptive only. The bad set was defined after seeing the data, so this is a")
+        w("picture, not a test.")
+    elif not bad_addrs:
+        w("No bad boot was observed in this survey, so there is no bad range to separate.")
+    else:
+        w("No good boot was observed in this survey.")
+    w("")
+
+    w("### 4d. The pre-registered prediction")
+    w("")
+    w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the")
+    w("bad address (both prior failures were there; all five prior successes elsewhere).")
+    w("Tested against this survey's boots only:")
+    w("")
+    a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    w(f"| | bad | good |")
+    w(f"|---|---:|---:|")
+    w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |")
+    w(f"| elsewhere | {c} | {d} |")
+    w("")
+    if a + b_ == 0:
+        w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the")
+        w("prediction had no opportunity to be right or wrong. That is itself informative:")
+        w("the address moves every boot, so a specific address is not a reproducible handle.")
+    else:
+        p = fisher_exact_2x2(a, b_, c, d)
+        w(f"Fisher exact (two-sided) p = **{p:.4g}**"
+          + ("  — the prediction holds." if p < 0.05 and a and not b_
+             else "  — not significant at 0.05." if p >= 0.05
+             else "  — significant, but read the cell counts before believing it."))
+    w("")
+
+    w("### 4e. Does the address predict how the KERNEL READ the table?")
+    w("")
+    w("The strongest test this data can support when no bad boot occurred. `verdict` has")
+    w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both")
+    w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:")
+    w("the kernel reads whatever bytes happen to be at the address the firmware published.")
+    w("If the address determines the reading, the address is doing causal work, one step")
+    w("short of the catastrophic path.")
+    w("")
+    reads_by_addr: dict[str, set] = defaultdict(set)
+    n_by_addr: Counter = Counter()
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        reads_by_addr[addr].add(b.get("kernel_read"))
+        n_by_addr[addr] += 1
+    repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1}
+    inconsistent = {a: r for a, r in repeated.items() if len(r) > 1}
+    distinct_reads = {r for rs in reads_by_addr.values() for r in rs}
+    w(f"- distinct addresses: **{len(reads_by_addr)}**, of which "
+      f"**{len(repeated)}** were observed more than once")
+    w(f"- distinct kernel readings across the survey: "
+      + ", ".join(f"`{r}`" for r in sorted(distinct_reads)))
+    w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**")
+    w("")
+    if len(distinct_reads) < 2:
+        w("Only one reading occurred in the whole survey, so there is nothing to predict.")
+    elif inconsistent:
+        w("**The address does NOT determine the reading:** "
+          + "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items()))
+    elif not repeated:
+        w("No address recurred, so the test could not be run.")
+    else:
+        w(f"**The address determines the reading, on every address observed more than once**")
+        w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both")
+        w("readings present in the sample). This is direct evidence for the causal chain")
+        w("§0b describes: the same address reproduces the same parse, boot after boot.")
+        w("It does NOT establish which addresses are catastrophic — no bad boot occurred —")
+        w("only that WHERE the table lands is what decides what the kernel finds there.")
+    w("")
+    w("| MEMATTR | boots | kernel reading(s) |")
+    w("|---|---:|---|")
+    for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)):
+        marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else ""
+        w(f"| `{addr}` | {n_by_addr[addr]} | "
+          f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |")
+    w("")
+
+    # --- 5. Corroboration -------------------------------------------------------------------
+    w("## 5. Corroborating evidence")
+    w("")
+    w("### 5a. Kernel's own reading vs the measured outcome")
+    w("")
+    cross: dict[tuple, int] = Counter()
+    for b in surveyed:
+        cross[(b.get("kernel_read"), b.get("verdict"))] += 1
+    w("| kernel read | verdict | boots |")
+    w("|---|---|---:|")
+    for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))):
+        w(f"| {read} | {verdict} | {count} |")
+    w("")
+    disagreements = [
+        (read, verdict, count)
+        for (read, verdict), count in cross.items()
+        if (read == "failed-map") != (verdict == "bad")
+    ]
+    if disagreements:
+        w("> **The log line and the measurement DISAGREE** in "
+          f"{sum(c for _, _, c in disagreements)} boot(s): "
+          + "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements)
+          + ". That is a finding in its own right — §0b's model says `failed-map` and only")
+        w("> `failed-map` costs memory.")
+    else:
+        w("`failed-map` and only `failed-map` cost memory, in every boot observed —")
+        w("exactly as §0b's mechanism predicts, and independent of the address question.")
+    w("")
+
+    w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)")
+    w("")
+    w("| verdict | NODE_DATA end address | boots |")
+    w("|---|---|---:|")
+    nd: Counter = Counter()
+    for b in surveyed:
+        line = b.get("node_data_line") or ""
+        end = line.split("-")[-1].rstrip("]") if "-" in line else "?"
+        nd[(b.get("verdict"), end)] += 1
+    for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
+        w(f"| {verdict} | `{end}` | {count} |")
+    w("")
+
+    # --- 6. Invariants ------------------------------------------------------------------
+    w("## 6. Nothing under test changed")
+    w("")
+    w("| node | kernels seen | distinct cmdlines | SMBIOS installed |")
+    w("|---|---|---:|---|")
+    for h in hosts:
+        kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")})
+        cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")}
+        w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | "
+          f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |")
+    w("")
+    w("More than one kernel or more than one cmdline on a node would void that node's arm.")
+    w("")
+    return "\n".join(out) + "\n"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--artifact-dir", required=True)
+    parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted")
+    parser.add_argument("--out", default=None, help="also write the report here")
+    args = parser.parse_args()
+
+    names = discover_hosts(args.artifact_dir)
+    if not names:
+        print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr)
+        return 1
+    hosts = [load_host(args.artifact_dir, name) for name in names]
+    report = render(hosts, args.rounds)
+    if args.out:
+        with open(args.out, "w", encoding="utf-8") as handle:
+            handle.write(report)
+    sys.stdout.write(report)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

infra/ansible/roles/memattr_survey/files/memattr_probe.py

@@ -0,0 +1,489 @@
+#!/usr/bin/env python3
+"""memattr_probe.py — read one boot's EFI Memory Attributes Table outcome off a node.
+
+Part of roles/memattr_survey (infra/ansible). See that role's README.md for the
+question this answers; see infra/PROD-PROVISIONING.md §0b for the settled mechanism.
+
+WHY A SCRIPT AND NOT A HANDFUL OF `command:` TASKS
+  The record for one boot needs six sources cross-referenced (kernel log config-table
+  line, kernel log memattr diagnostic, kernel log `Memory:` line, kernel log
+  `NODE_DATA` line, /proc/meminfo, SMBIOS) and one derived verdict. Doing that in
+  Jinja across six registered results is unreadable and would put the classification
+  logic in a template instead of in one testable place. It also has to be exactly the
+  same logic for a live boot and for a boot harvested out of the journal, which a
+  single --all-boots mode gives for free.
+
+MODES
+  (default)      one JSON record for the CURRENT boot, on stdout
+  --all-boots    one JSON object with a `boots` list: EVERY boot the persistent
+                 journal still holds, re-derived from scratch. This is the
+                 AUTHORITATIVE record — it survives the driver script dying mid-run.
+  --status       {"completed_rounds": N, "records": M} from the on-node ledger, so the
+                 playbook can resume without re-deriving anything.
+
+  --round N      stamp the record with the survey round number
+  --append PATH  ALSO append the record to PATH as one JSON line (the on-node ledger),
+                 created 0644 with parents. Appended with O_APPEND in one write() so a
+                 concurrent reader never sees a half line.
+
+VERDICT IS MEASURED, NOT INFERRED FROM A LOG LINE
+  A node whose kernel reads a VALID table logs no memattr diagnostic at all and is
+  perfectly healthy; a node whose kernel bails out on a garbage version > 2 logs a
+  scary warning and is ALSO perfectly healthy. Only the third state — "Failed to map"
+  — costs memory, and even that message is emitted by a LATER function than the one
+  that did the damage. So the verdict comes from usable RAM versus SMBIOS installed
+  RAM, and `kernel_read` is recorded ALONGSIDE it as an independent observation. If
+  the two ever disagree, that disagreement is itself the finding.
+
+EXIT STATUS is 0 whenever a record could be produced, even a mostly-null one: a field
+this script could not read comes back as null with a reason in `notes`, never as an
+empty string and never as a hard failure. A survey that aborts on one unreadable field
+loses the other 19 rounds.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+
+SCHEMA = "memattr-survey/1"
+
+# --- Verdict thresholds -------------------------------------------------------------
+# Observed on this fleet: a good boot yields usable/installed of 0.95-0.98 (kernel code,
+# reserved firmware regions and struct page overhead account for the rest); the bad path
+# yields 0.15-0.17 (everything above the table address is reserved). Two orders of
+# separation, so the thresholds are nowhere near either cluster. Anything landing in the
+# gap is deliberately NOT called: it is a new third thing and wants human eyes.
+VERDICT_GOOD_MIN_FRACTION = 0.85
+VERDICT_BAD_MAX_FRACTION = 0.60
+
+# `Memory: 2568544K/16546088K available (... 13958980K reserved ...)`
+RE_MEMORY_LINE = re.compile(
+    r"^Memory:\s+(?P<available>\d+)K/(?P<present>\d+)K available\b.*?"
+    r"(?:(?P<reserved>\d+)K reserved)?"
+)
+# `efi: ACPI=0x... SMBIOS=0x... MEMATTR=0xbeeb0018 MOKvar=0x... INITRD=0x...`
+RE_MEMATTR_ADDR = re.compile(r"\bMEMATTR=(?P<addr>0x[0-9a-fA-F]+)")
+# `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749`
+RE_TABLE_VERSION = re.compile(r"table version\s+(?P<version>-?\d+)")
+# `NODE_DATA(0) allocated [mem 0xbee85680-0xbeeaffff]` — per §0b the cleanest A/B tell:
+# on the bad path pgdat is forced immediately BELOW the table address instead of landing
+# at the top of RAM. Captured because it is free and it is independent evidence.
+RE_NODE_DATA = re.compile(r"^NODE_DATA\(\d+\) allocated \[mem (?P<range>[0-9a-fx]+-[0-9a-fx]+)\]")
+RE_CMDLINE = re.compile(r"^Command line:\s*(?P<cmdline>.*)$")
+# dmidecode type 17 `Size: 16 GB` / `Size: 8192 MB` / `Size: No Module Installed`
+RE_DMI_SIZE = re.compile(r"^\s*Size:\s+(?P<n>\d+)\s+(?P<unit>[kKmMgGtT]B)\s*$")
+
+DMI_UNIT_KB = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def run(argv: list[str], timeout: int = 60) -> tuple[int, str, str]:
+    """Run a command, never raise. Returns (rc, stdout, stderr); rc 127 if absent."""
+    try:
+        proc = subprocess.run(
+            argv, capture_output=True, text=True, timeout=timeout, check=False
+        )
+    except FileNotFoundError:
+        return 127, "", f"{argv[0]}: not found"
+    except subprocess.TimeoutExpired:
+        return 124, "", f"{' '.join(argv)}: timed out after {timeout}s"
+    except OSError as exc:  # pragma: no cover - defensive
+        return 126, "", f"{' '.join(argv)}: {exc}"
+    return proc.returncode, proc.stdout, proc.stderr
+
+
+def read_text(path: str) -> str | None:
+    try:
+        with open(path, "r", encoding="utf-8", errors="replace") as handle:
+            return handle.read()
+    except OSError:
+        return None
+
+
+def kernel_log(boot: str) -> tuple[list[str], str | None]:
+    """Kernel ring buffer for `boot` ('0' = current, '-1' = previous, or a boot ID).
+
+    `-o cat` is deliberate: it yields the message VERBATIM with no host/facility
+    decoration, which is what the survey wants to record, and it is stable across
+    systemd versions. `-k` restricts to _TRANSPORT=kernel.
+    """
+    rc, out, err = run(["journalctl", "-k", "-b", boot, "-o", "cat", "--no-pager"], timeout=180)
+    if rc != 0:
+        return [], f"journalctl -k -b {boot} failed (rc={rc}): {err.strip()[:200]}"
+    return out.splitlines(), None
+
+
+def list_boots() -> tuple[list[dict], str | None]:
+    """Every boot the journal still holds, oldest first.
+
+    Prefers `--list-boots -o json` (systemd >= 250, so always on Debian 13) and falls
+    back to parsing the table, because a survey that cannot enumerate boots has lost
+    its authoritative record and must say so rather than silently reporting one boot.
+    """
+    rc, out, err = run(["journalctl", "--list-boots", "-o", "json", "--no-pager"], timeout=120)
+    if rc == 0:
+        try:
+            parsed = json.loads(out)
+        except json.JSONDecodeError as exc:
+            return [], f"--list-boots -o json unparseable: {exc}"
+        boots = [
+            {
+                "index": int(entry.get("index", 0)),
+                "boot_id": entry.get("boot_id"),
+                "first_entry_usec": entry.get("first_entry"),
+                "last_entry_usec": entry.get("last_entry"),
+            }
+            for entry in parsed
+        ]
+        return sorted(boots, key=lambda b: b["index"]), None
+
+    rc, out, err = run(["journalctl", "--list-boots", "--no-pager"], timeout=120)
+    if rc != 0:
+        return [], f"journalctl --list-boots failed (rc={rc}): {err.strip()[:200]}"
+    boots = []
+    for line in out.splitlines():
+        fields = line.split()
+        if len(fields) < 2 or not re.fullmatch(r"-?\d+", fields[0]):
+            continue  # header row
+        boots.append(
+            {
+                "index": int(fields[0]),
+                "boot_id": fields[1],
+                "first_entry_usec": None,
+                "last_entry_usec": None,
+            }
+        )
+    return sorted(boots, key=lambda b: b["index"]), None
+
+
+def journal_storage() -> dict:
+    """Is the journal PERSISTENT? The harvest is only authoritative if it is.
+
+    /var/log/journal existing is what makes journald write to disk; /run/log/journal is
+    the volatile fallback and would mean every reboot erases the record.
+    """
+    return {
+        "var_log_journal": os.path.isdir("/var/log/journal"),
+        "run_log_journal": os.path.isdir("/run/log/journal"),
+        "persistent": os.path.isdir("/var/log/journal"),
+    }
+
+
+def smbios_installed_kb() -> tuple[int | None, str | None]:
+    """Installed RAM per SMBIOS type-17 records — the GROUND TRUTH for the verdict.
+
+    Deliberately not /proc/meminfo (that is the number under test) and not dmesg's
+    `Memory: x/yK` denominator (which is already post-e820 and post-reservation).
+    Hardware does not change between boots, so one read applies to harvested boots too.
+    """
+    rc, out, err = run(["dmidecode", "-t", "17"], timeout=60)
+    if rc != 0:
+        return None, f"dmidecode -t 17 failed (rc={rc}): {err.strip()[:200]}"
+    total = 0
+    seen = 0
+    for line in out.splitlines():
+        match = RE_DMI_SIZE.match(line)
+        if not match:
+            continue  # "No Module Installed" and every non-Size line
+        total += int(match.group("n")) * DMI_UNIT_KB[match.group("unit").lower()]
+        seen += 1
+    if seen == 0:
+        return None, "dmidecode -t 17 reported no populated memory devices"
+    return total, None
+
+
+def meminfo_total_kb() -> tuple[int | None, str | None]:
+    text = read_text("/proc/meminfo")
+    if text is None:
+        return None, "/proc/meminfo unreadable"
+    for line in text.splitlines():
+        if line.startswith("MemTotal:"):
+            return int(line.split()[1]), None
+    return None, "/proc/meminfo has no MemTotal"
+
+
+def classify_kernel_read(memattr_addr, diagnostics: list[str]) -> tuple[str, int | None]:
+    """What did the kernel make of the table? THREE outcomes, not two (§0b).
+
+    Returns (classification, table_version_if_reported).
+
+      no-table      firmware published no MEMATTR pointer at all (the dev nodes)
+      valid         a MEMATTR pointer AND no diagnostic => the kernel parsed a table it
+                    considered valid. This is SILENCE AS EVIDENCE, and it is a healthy
+                    state, not a missing measurement.
+      bail-version  garbage version > 2 => kernel warned and bailed. Memory intact.
+                    The LUCKY path, and it looks alarming in the log.
+      failed-map    the catastrophic path. The message comes from
+                    efi_memattr_apply_permissions() and is therefore PROOF the bogus
+                    memblock_reserve() already happened upstream of it.
+      bogus-size    a kernel >= 6.13 rejecting the table. Should never appear on 6.12;
+                    if it does, the kernel under test changed and the arm is void.
+    """
+    version = None
+    for line in diagnostics:
+        match = RE_TABLE_VERSION.search(line)
+        if match:
+            version = int(match.group("version"))
+    joined = " | ".join(diagnostics)
+    if "Failed to map" in joined:
+        return "failed-map", version
+    if "Corrupted EFI Memory Attributes Table" in joined:
+        return "bogus-size", version
+    if "Unexpected EFI Memory Attributes table version" in joined:
+        return "bail-version", version
+    if diagnostics:
+        return "diagnostic-unrecognised", version
+    if memattr_addr is None:
+        return "no-table", version
+    return "valid", version
+
+
+def derive_verdict(usable_kb, installed_kb) -> tuple[str, float | None]:
+    if usable_kb is None or not installed_kb:
+        return "unknown", None
+    fraction = usable_kb / installed_kb
+    if fraction >= VERDICT_GOOD_MIN_FRACTION:
+        return "good", fraction
+    if fraction <= VERDICT_BAD_MAX_FRACTION:
+        return "bad", fraction
+    return "indeterminate", fraction
+
+
+def build_record(
+    *,
+    boot_selector: str,
+    boot_meta: dict | None,
+    installed_kb,
+    installed_note,
+    round_number,
+    live: bool,
+) -> dict:
+    notes: list[str] = []
+    if installed_note:
+        notes.append(installed_note)
+
+    lines, log_note = kernel_log(boot_selector)
+    if log_note:
+        notes.append(log_note)
+
+    config_table_line = None
+    memattr_addr = None
+    diagnostics: list[str] = []
+    memory_line = None
+    node_data_line = None
+    cmdline = None
+
+    for line in lines:
+        stripped = line.strip()
+        if memattr_addr is None:
+            match = RE_MEMATTR_ADDR.search(stripped)
+            if match and stripped.startswith("efi:"):
+                memattr_addr = match.group("addr").lower()
+                config_table_line = stripped
+        # Every memattr-related diagnostic, verbatim. Matched on the pr_fmt prefix the
+        # kernel actually uses (`efi: memattr: `) AND on the message text, so a prefix
+        # change upstream cannot silently drop the line.
+        if stripped.startswith("efi: memattr:") or "EFI Memory Attributes" in stripped:
+            if stripped != config_table_line:
+                diagnostics.append(stripped)
+        if memory_line is None and stripped.startswith("Memory:"):
+            if RE_MEMORY_LINE.match(stripped):
+                memory_line = stripped
+        if node_data_line is None:
+            match = RE_NODE_DATA.match(stripped)
+            if match:
+                node_data_line = stripped
+        if cmdline is None:
+            match = RE_CMDLINE.match(stripped)
+            if match:
+                cmdline = match.group("cmdline").strip()
+
+    available_kb = present_kb = reserved_kb = None
+    if memory_line:
+        match = RE_MEMORY_LINE.match(memory_line)
+        available_kb = int(match.group("available"))
+        present_kb = int(match.group("present"))
+        reserved = match.group("reserved")
+        reserved_kb = int(reserved) if reserved else None
+    else:
+        notes.append("no `Memory: .../... available` line found in this boot's kernel log")
+
+    kernel_read, table_version = classify_kernel_read(memattr_addr, diagnostics)
+
+    # /proc/meminfo describes the CURRENT boot only. For a harvested past boot the
+    # equivalent measurement is the kernel's own `Memory: available` figure, which
+    # tracks MemTotal to within ~7% on both paths. Which one was used is recorded, so
+    # nobody has to guess later.
+    mem_total_kb, meminfo_note = (None, None)
+    if live:
+        mem_total_kb, meminfo_note = meminfo_total_kb()
+        if meminfo_note:
+            notes.append(meminfo_note)
+    usable_kb = mem_total_kb if mem_total_kb is not None else available_kb
+    basis = "MemTotal" if mem_total_kb is not None else "dmesg Memory: available"
+    verdict, fraction = derive_verdict(usable_kb, installed_kb)
+
+    record = {
+        "schema": SCHEMA,
+        "round": round_number,
+        "host": os.uname().nodename,
+        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+        "recorded_at_epoch": int(time.time()),
+        "live_boot": live,
+        "boot_index": (boot_meta or {}).get("index"),
+        "boot_id": (boot_meta or {}).get("boot_id"),
+        "boot_first_entry_usec": (boot_meta or {}).get("first_entry_usec"),
+        "kernel_release": os.uname().release,
+        "kernel_cmdline": cmdline,
+        # --- the independent variable -------------------------------------------------
+        "efi_config_table_line": config_table_line,
+        "memattr_address": memattr_addr,
+        "memattr_diagnostics": diagnostics,
+        "memattr_diagnostic_count": len(diagnostics),
+        # An explicit marker, never an empty field: "the kernel said nothing" is a
+        # measurement (it read a VALID table), not a gap in the data.
+        "memattr_diagnostic_present": bool(diagnostics),
+        "memattr_table_version": table_version,
+        "kernel_read": kernel_read,
+        # --- the dependent variable ---------------------------------------------------
+        "memory_line": memory_line,
+        "memory_available_kb": available_kb,
+        "memory_present_kb": present_kb,
+        "memory_reserved_kb": reserved_kb,
+        "mem_total_kb": mem_total_kb,
+        "smbios_installed_kb": installed_kb,
+        "usable_kb": usable_kb,
+        "usable_fraction": round(fraction, 5) if fraction is not None else None,
+        "verdict": verdict,
+        "verdict_basis": f"{basis} vs SMBIOS installed",
+        # --- corroborating evidence ---------------------------------------------------
+        "node_data_line": node_data_line,
+        "notes": notes,
+    }
+    return record
+
+
+def append_record(path: str, record: dict) -> str | None:
+    """One O_APPEND write of one line, so a reader never sees a torn record."""
+    try:
+        parent = os.path.dirname(path) or "."
+        os.makedirs(parent, mode=0o755, exist_ok=True)
+        payload = (json.dumps(record, sort_keys=True) + "\n").encode()
+        fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
+        try:
+            os.write(fd, payload)
+            os.fsync(fd)  # the point of the ledger is surviving the next reboot
+        finally:
+            os.close(fd)
+    except OSError as exc:
+        return f"could not append to {path}: {exc}"
+    return None
+
+
+def ledger_status(path: str) -> dict:
+    """Resume support: how far did a previous run get?
+
+    Round numbers come from the ledger itself rather than a line count, so the
+    baseline round 0 record and any hand-added line cannot shift the numbering.
+    """
+    rounds: list[int] = []
+    records = 0
+    malformed = 0
+    if os.path.exists(path):
+        with open(path, "r", encoding="utf-8", errors="replace") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                records += 1
+                try:
+                    parsed = json.loads(line)
+                except json.JSONDecodeError:
+                    malformed += 1
+                    continue
+                value = parsed.get("round")
+                if isinstance(value, int):
+                    rounds.append(value)
+    return {
+        "schema": SCHEMA,
+        "ledger": path,
+        "records": records,
+        "malformed_records": malformed,
+        "completed_rounds": max(rounds) if rounds else 0,
+        "rounds_recorded": sorted(rounds),
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--round", type=int, default=None, help="survey round number to stamp")
+    parser.add_argument("--append", default=None, help="on-node ledger to append to")
+    parser.add_argument("--all-boots", action="store_true", help="harvest every retained boot")
+    parser.add_argument("--status", default=None, metavar="LEDGER", help="report resume state")
+    args = parser.parse_args()
+
+    if args.status:
+        print(json.dumps(ledger_status(args.status), indent=2, sort_keys=True))
+        return 0
+
+    installed_kb, installed_note = smbios_installed_kb()
+    boots, boots_note = list_boots()
+    storage = journal_storage()
+
+    if args.all_boots:
+        records = []
+        for boot in boots:
+            # Address boots by BOOT ID, not by relative index: indices shift under the
+            # harvester's feet if journald rotates mid-harvest, and the ID is what the
+            # per-round ledger recorded, so the two records join cleanly.
+            selector = boot["boot_id"] or str(boot["index"])
+            records.append(
+                build_record(
+                    boot_selector=selector,
+                    boot_meta=boot,
+                    installed_kb=installed_kb,
+                    installed_note=installed_note,
+                    round_number=None,
+                    live=(boot["index"] == 0),
+                )
+            )
+        payload = {
+            "schema": SCHEMA,
+            "host": os.uname().nodename,
+            "harvested_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "journal_storage": storage,
+            "smbios_installed_kb": installed_kb,
+            "boots_listed": len(boots),
+            "notes": [n for n in [boots_note, installed_note] if n],
+            "boots": records,
+        }
+        print(json.dumps(payload, indent=2, sort_keys=True))
+        return 0
+
+    current = next((b for b in boots if b["index"] == 0), None)
+    record = build_record(
+        boot_selector="0",
+        boot_meta=current,
+        installed_kb=installed_kb,
+        installed_note=installed_note,
+        round_number=args.round,
+        live=True,
+    )
+    record["journal_storage"] = storage
+    if boots_note:
+        record["notes"].append(boots_note)
+    if args.append:
+        note = append_record(args.append, record)
+        if note:
+            record["notes"].append(note)
+    print(json.dumps(record, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

infra/ansible/roles/memattr_survey/tasks/analyse.yml

@@ -0,0 +1,42 @@
+---
+# Controller-side analysis. Runs on localhost over the artifact directory, so it is
+# independent of whether any surveyed node is still alive — and so it can be re-run on
+# its own (`--tags analyse`) without touching hardware.
+#
+# The statistics live in files/memattr_analyse.py, not in Jinja: Clopper-Pearson and
+# Fisher belong somewhere testable, and a confidence interval assembled out of template
+# filters is a confidence interval nobody will ever check.
+
+- name: Locate the survey artifacts
+  ansible.builtin.find:
+    paths: "{{ memattr_survey_artifact_dir }}"
+    patterns: "*.boots.json"
+  register: memattr_survey_artifacts
+
+- name: Report that there is nothing to analyse
+  ansible.builtin.debug:
+    msg: >-
+      No *.boots.json under {{ memattr_survey_artifact_dir }} — no survey has produced a
+      harvest yet. Run the playbook with -e memattr_survey_reboots_authorised=true.
+  when: memattr_survey_artifacts.matched | int == 0
+
+- name: Analyse the survey
+  ansible.builtin.command:
+    argv:
+      - /usr/bin/env
+      - python3
+      - "{{ role_path }}/files/memattr_analyse.py"
+      - "--artifact-dir"
+      - "{{ memattr_survey_artifact_dir }}"
+      - "--rounds"
+      - "{{ memattr_survey_rounds }}"
+      - "--out"
+      - "{{ memattr_survey_artifact_dir }}/summary.md"
+  register: memattr_survey_analysis
+  changed_when: false
+  when: memattr_survey_artifacts.matched | int > 0
+
+- name: Show the analysis
+  ansible.builtin.debug:
+    msg: "{{ memattr_survey_analysis.stdout.split('\n') }}"
+  when: memattr_survey_analysis is not skipped

infra/ansible/roles/memattr_survey/tasks/harvest.yml

@@ -0,0 +1,66 @@
+---
+# The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from
+# scratch in one pass by the same code that produced the per-round ledger.
+#
+# WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and
+# the append, that round exists on the machine but not in the ledger. The journal does not
+# care whether the driver was alive — the boot happened, the kernel logged it, journald
+# persisted it. So the harvest can only ever be a superset of the ledger, and the analysis
+# joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and
+# any boot the ledger missed still shows up.
+#
+# This runs even for a node that aborted mid-survey, which is exactly when it matters most.
+
+- name: Harvest every retained boot from the persistent journal
+  ansible.builtin.script:
+    cmd: memattr_probe.py --all-boots
+    executable: /usr/bin/python3
+  register: memattr_survey_harvest_raw
+  changed_when: false   # strictly a read of the journal
+
+- name: Parse the harvest
+  ansible.builtin.set_fact:
+    memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}"
+
+- name: Ensure the artifact directory exists (on the controller)
+  ansible.builtin.file:
+    path: "{{ memattr_survey_artifact_dir }}"
+    state: directory
+    mode: "0755"
+  delegate_to: localhost
+  become: false
+  # NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and
+  # run_once is unreliable there. `state: directory` is idempotent, so three hosts each
+  # asserting the directory exists is cheaper than a conditional that might not fire.
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Write the harvested boot record
+  ansible.builtin.copy:
+    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json"
+    mode: "0644"
+    content: "{{ memattr_survey_harvest | to_nice_json }}"
+  delegate_to: localhost
+  become: false
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Mirror the final ledger to the controller
+  ansible.builtin.fetch:
+    src: "{{ memattr_survey_state_file }}"
+    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
+    flat: true
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Harvest summary
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots
+      from the journal
+      ({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad,
+      {{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good,
+      {{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other)
+      {%- if memattr_survey_aborted | default(false) | bool %}
+      — NOTE: this node ABORTED ({{ memattr_survey_abort_reason }})
+      {%- endif %}

infra/ansible/roles/memattr_survey/tasks/main.yml

@@ -0,0 +1,23 @@
+---
+# memattr_survey — see defaults/main.yml for the question, README.md for how to run and
+# read it, and infra/PROD-PROVISIONING.md §0b for the settled mechanism this measures.
+#
+# ORDER OF OPERATIONS
+#   1. preflight.yml — refuse to touch a node that is not prod, not idle, or has no
+#      persistent journal; read the resume point; record the boot we found it on.
+#   2. round.yml x N — reboot, measure, append, mirror. Per-node fail-safe.
+#   3. harvest.yml   — re-derive every retained boot from the journal (authoritative).
+#
+# The playbook splits these across plays so that preflight is any_errors_fatal (one unsafe
+# node stops everything) while the rounds are not (one dead node stops only itself), and
+# so that the controller-side analysis runs even if every node dies. This file is the
+# whole-role entry point kept for completeness; the playbook calls the stages directly.
+
+- name: Preflight and baseline
+  ansible.builtin.include_tasks: preflight.yml
+
+- name: Run the survey rounds
+  ansible.builtin.include_tasks: rounds.yml
+
+- name: Harvest the authoritative record
+  ansible.builtin.include_tasks: harvest.yml

infra/ansible/roles/memattr_survey/tasks/preflight.yml

@@ -0,0 +1,149 @@
+---
+# Safety gate. Runs with any_errors_fatal at the play level (playbooks/memattr-survey.yml),
+# so a violation on ANY node stops the entire playbook before a single reboot — the
+# correct posture when the failure mode is "rebooted a machine that was doing something".
+#
+# VERIFY, DO NOT ASSUME. The premise that makes these reboots safe is that prod carries no
+# k3s and runs nothing. That premise is checked here on every run rather than trusted from
+# a document, because the document and the machine drift independently.
+
+- name: Assert the target host is in the prod group
+  # The dev cluster is LIVE. This is the one check that a mistyped --limit or a copied
+  # command line cannot slip past: membership is read from the inventory, not the pattern.
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_required_group in (group_names | default([]))
+    fail_msg: >-
+      REFUSING TO SURVEY {{ inventory_hostname }}: it is not a member of
+      '{{ memattr_survey_required_group }}' (groups: {{ group_names | default([]) | join(', ') }}).
+      This role reboots its targets. Point it at prod, or override
+      memattr_survey_required_group deliberately.
+    quiet: true
+
+- name: Check whether any cluster or container runtime unit is active
+  # `is-active` returns non-zero for inactive/unknown units, which is the expected answer
+  # here, so the rc is data and not a failure.
+  ansible.builtin.command:
+    argv: "{{ ['systemctl', 'is-active'] + memattr_survey_forbidden_units }}"
+  register: memattr_survey_units
+  changed_when: false
+  failed_when: false
+
+- name: Check whether k3s has been installed on this node
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  loop: "{{ memattr_survey_forbidden_paths }}"
+  register: memattr_survey_paths
+  changed_when: false
+
+- name: Assert nothing is running on this node
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_units.stdout_lines | reject('equalto', 'inactive') | reject('equalto', 'unknown') | list | length == 0
+      - memattr_survey_paths.results | selectattr('stat.exists') | list | length == 0
+    fail_msg: >-
+      REFUSING TO REBOOT {{ inventory_hostname }} — it is not idle.
+      units {{ memattr_survey_forbidden_units | zip(memattr_survey_units.stdout_lines) | map('join', '=') | join(' ') }};
+      present paths: {{ memattr_survey_paths.results | selectattr('stat.exists') | map(attribute='item') | list | join(', ') | default('none', true) }}.
+      The survey is only sanctioned because these boxes carry no k3s and run nothing.
+    quiet: true
+
+- name: Clear any abort marker from a previous run
+  # A marker left by a run that has since been resumed successfully would make this run's
+  # report describe the LAST failure rather than this one. The marker means "this run
+  # aborted", so it is cleared at the start of every run and re-written only if it does.
+  ansible.builtin.file:
+    path: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
+    state: absent
+  delegate_to: localhost
+  become: false
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Assert forks allows every node to be surveyed concurrently
+  # The survey play runs `strategy: free` so the nodes reboot in parallel, but forks is a
+  # hard cap on that: with fewer forks than hosts the survey quietly serialises and takes
+  # a multiple of the wall clock it should. Cheap to satisfy (`-f <n>`), so assert rather
+  # than discover it three hours later.
+  ansible.builtin.assert:
+    that:
+      - (lookup('ansible.builtin.config', 'DEFAULT_FORKS') | int) >= (ansible_play_hosts_all | length)
+    fail_msg: >-
+      forks={{ lookup('ansible.builtin.config', 'DEFAULT_FORKS') }} but this survey targets
+      {{ ansible_play_hosts_all | length }} hosts. The rounds would serialise. Re-run with
+      -f {{ ansible_play_hosts_all | length }} (or higher).
+    quiet: true
+  run_once: true
+
+- name: Read the node's resume state and baseline boot
+  # One call does double duty: it reports how far a previous run got (so this one resumes)
+  # and, with --round 0, records the boot we found the node on. Round 0 is a free data
+  # point — it is a boot nobody chose, so it is not selected for by the survey.
+  ansible.builtin.script:
+    cmd: >-
+      memattr_probe.py --status {{ memattr_survey_state_file | quote }}
+    executable: /usr/bin/python3
+  register: memattr_survey_status_raw
+  changed_when: false
+
+- name: Record the resume point
+  ansible.builtin.set_fact:
+    memattr_survey_status: "{{ memattr_survey_status_raw.stdout | from_json }}"
+    memattr_survey_aborted: false
+    memattr_survey_abort_reason: ""
+
+- name: Assert the probe schema is the one this role understands
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_status.schema == 'memattr-survey/1'
+    fail_msg: >-
+      Probe returned schema {{ memattr_survey_status.schema | default('<none>') }};
+      this role understands memattr-survey/1. Role and probe are out of step.
+    quiet: true
+
+- name: Probe the boot the node is currently on
+  # On a FIRST run this is stamped round 0 and appended: the boot the node was already
+  # sitting on is a free, unselected data point. On a RESUME it is read but NOT appended,
+  # because that boot is already round N of the interrupted run and double-counting it
+  # would bias the sample toward whatever outcome happened to survive the interruption.
+  ansible.builtin.script:
+    cmd: >-
+      memattr_probe.py
+      {% if memattr_survey_status.completed_rounds | int == 0 %}
+      --round 0 --append {{ memattr_survey_state_file | quote }}
+      {% endif %}
+    executable: /usr/bin/python3
+  register: memattr_survey_baseline_raw
+  # The append is the experiment's own ledger, not a change to the node under test: this
+  # role reports `changed` only for things that could affect a boot, and a file written
+  # under /var/lib long after the kernel parsed the table cannot.
+  changed_when: false
+
+- name: Parse the baseline record
+  ansible.builtin.set_fact:
+    memattr_survey_baseline: "{{ memattr_survey_baseline_raw.stdout | from_json }}"
+
+- name: Assert the journal is PERSISTENT
+  # The end-of-run harvest is the authoritative record precisely because it re-derives
+  # every boot from the journal. On a volatile journal (/run/log/journal only) each
+  # reboot erases the evidence and the harvest would silently return one boot — a survey
+  # that looks complete and holds nothing. Verified, not assumed.
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_baseline.journal_storage.persistent | bool
+    fail_msg: >-
+      {{ inventory_hostname }} has NO persistent journal (/var/log/journal missing).
+      Every reboot would erase the record this survey depends on. Refusing to run.
+    quiet: true
+
+- name: Report the starting state
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: idle, prod, persistent journal.
+      kernel {{ memattr_survey_baseline.kernel_release | default('?') }},
+      installed {{ (memattr_survey_baseline.smbios_installed_kb | default(0) | int / 1048576) | round(1) }} GiB.
+      baseline boot MEMATTR={{ memattr_survey_baseline.memattr_address | default('none') }}
+      kernel_read={{ memattr_survey_baseline.kernel_read | default('?') }}
+      verdict={{ memattr_survey_baseline.verdict | default('?') | upper }}.
+      resuming from round {{ memattr_survey_status.completed_rounds }}
+      of {{ memattr_survey_rounds }}.

infra/ansible/roles/memattr_survey/tasks/round.yml

@@ -0,0 +1,98 @@
+---
+# ONE round on ONE node: reboot, wait for it back, measure the new boot, record it twice.
+# Included in a loop from tasks/main.yml with loop_var `memattr_survey_round`.
+#
+# FAIL-SAFE, PER NODE. The block/rescue below is what keeps one dead node from ending the
+# survey: on any failure this host sets memattr_survey_aborted and every later round is
+# skipped FOR THIS HOST ONLY, while the other nodes keep going (Ansible runs the loop for
+# all hosts in lockstep, so the reboots are genuinely concurrent). The abort is reported
+# loudly here AND again in the final analysis, which counts rounds from the artifacts
+# rather than from Ansible's host state — so a node that vanishes so hard that it is
+# marked unreachable (which rescue cannot catch) still shows up as a short arm.
+
+- name: "Round {{ memattr_survey_round }} — survey step"
+  block:
+    - name: "Round {{ memattr_survey_round }} — reboot and wait for the node to return"
+      ansible.builtin.reboot:
+        reboot_timeout: "{{ memattr_survey_reboot_timeout | int }}"
+        connect_timeout: "{{ memattr_survey_connect_timeout | int }}"
+        post_reboot_delay: "{{ memattr_survey_post_reboot_delay | int }}"
+        msg: "memattr survey round {{ memattr_survey_round }} of {{ memattr_survey_rounds }}"
+
+    - name: "Round {{ memattr_survey_round }} — measure the new boot and append to the ledger"
+      ansible.builtin.script:
+        cmd: >-
+          memattr_probe.py --round {{ memattr_survey_round }}
+          --append {{ memattr_survey_state_file | quote }}
+        executable: /usr/bin/python3
+      register: memattr_survey_round_raw
+      changed_when: false   # see the note in preflight.yml — the ledger is not the node
+
+    - name: "Round {{ memattr_survey_round }} — parse the record"
+      ansible.builtin.set_fact:
+        memattr_survey_record: "{{ memattr_survey_round_raw.stdout | from_json }}"
+
+    - name: "Round {{ memattr_survey_round }} — result"
+      ansible.builtin.debug:
+        msg: >-
+          [{{ inventory_hostname }} r{{ memattr_survey_round }}/{{ memattr_survey_rounds }}]
+          MEMATTR={{ memattr_survey_record.memattr_address | default('none') }}
+          read={{ memattr_survey_record.kernel_read }}
+          {%- if memattr_survey_record.memattr_table_version is not none %}
+          (version {{ memattr_survey_record.memattr_table_version }})
+          {%- endif %}
+          MemTotal={{ (memattr_survey_record.mem_total_kb | default(0) | int / 1048576) | round(2) }}G
+          of {{ (memattr_survey_record.smbios_installed_kb | default(0) | int / 1048576) | round(0) | int }}G
+          => {{ memattr_survey_record.verdict | upper }}
+          boot={{ memattr_survey_record.boot_id | default('?') | truncate(9, true, '') }}
+      when: memattr_survey_show_progress | bool
+
+    # BRACES. The on-node ledger already survives a reboot; re-fetching it every round
+    # means the controller also holds every completed round if the DRIVER dies, without
+    # waiting for the end-of-run harvest. One tiny file per host, per-host destination
+    # paths, so three concurrent fetches cannot collide.
+    - name: "Round {{ memattr_survey_round }} — mirror the ledger to the controller"
+      ansible.builtin.fetch:
+        src: "{{ memattr_survey_state_file }}"
+        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
+        flat: true
+      changed_when: false
+      when: memattr_survey_write_artifacts | bool
+
+  rescue:
+    - name: "Round {{ memattr_survey_round }} — ABORT this node, continue the others"
+      ansible.builtin.set_fact:
+        memattr_survey_aborted: true
+        memattr_survey_abort_reason: >-
+          round {{ memattr_survey_round }} failed:
+          {{ ansible_failed_result.msg | default('unknown error') | trim }}
+
+    # The abort must survive the run, not live only in terminal scrollback. Without
+    # this the report can see that an arm is SHORT but not WHY, and "0 rounds" reads
+    # like a targeting mistake instead of a node that failed to come back.
+    - name: "Round {{ memattr_survey_round }} — record the abort as an artifact"
+      ansible.builtin.copy:
+        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
+        mode: "0644"
+        content: "{{ memattr_survey_abort_record | to_nice_json }}"
+      vars:
+        memattr_survey_abort_record:
+          schema: "memattr-survey/1"
+          host: "{{ inventory_hostname }}"
+          aborted_at_round: "{{ memattr_survey_round }}"
+          target_rounds: "{{ memattr_survey_rounds }}"
+          reason: "{{ memattr_survey_abort_reason | trim }}"
+          failed_task: "{{ ansible_failed_task.name | default('unknown') }}"
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      when: memattr_survey_write_artifacts | bool
+
+    - name: "Round {{ memattr_survey_round }} — announce the abort"
+      ansible.builtin.debug:
+        msg: >-
+          *** SURVEY ABORTED ON {{ inventory_hostname }} at round
+          {{ memattr_survey_round }}/{{ memattr_survey_rounds }}:
+          {{ memattr_survey_abort_reason }}
+          *** The other nodes continue. This node's arm is SHORT — the analysis will
+          say so, and its p(bad) interval will be correspondingly wider.

infra/ansible/roles/memattr_survey/tasks/rounds.yml

@@ -0,0 +1,40 @@
+---
+# The loop itself, kept separate from round.yml so the loop bounds (which encode the
+# RESUME logic) are readable in one place.
+#
+# RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a
+# re-run after an interruption continues where it stopped and can never lose or duplicate
+# an earlier round. Already at or past the target count => nothing to do, and the run
+# collapses to a harvest.
+#
+# NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which
+# makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest,
+# analysis) still runs, so a dry run exercises the entire pipeline without touching a
+# single machine.
+
+- name: Announce the plan
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}:
+      {%- if not (memattr_survey_reboots_authorised | bool) %}
+      DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm).
+      {%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %}
+      already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do.
+      {%- else %}
+      rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }}
+      to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots).
+      {%- endif %}
+
+- name: Survey round
+  ansible.builtin.include_tasks: round.yml
+  loop: >-
+    {{ range(memattr_survey_status.completed_rounds | int + 1,
+             memattr_survey_rounds | int + 1) | list }}
+  loop_control:
+    loop_var: memattr_survey_round
+    label: "{{ inventory_hostname }} round {{ memattr_survey_round }}"
+  # Evaluated per iteration, so the abort flag set by a rescue in one round skips every
+  # round after it on THIS host and no others.
+  when:
+    - memattr_survey_reboots_authorised | bool
+    - not (memattr_survey_aborted | bool)

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -18,22 +18,170 @@ node_baseline_packages:
   # node from first baseline — package installation is the role's job, never a manual step
   # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
   - smartmontools
+# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
+# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
+# baseline — not a one-off `apt install` someone has to remember on the next node.
+#
+# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
+# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
+# from its contents, so an unlucky boot reserves everything above the table and the node
+# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
+# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
+# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
+# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
+# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
+# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+#
+# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
+# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
+# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
+# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
+# encode the environment into the ROLE, would silently miss a fourth prod node named
+# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
+# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
+# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
+#
+# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
+# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
+# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
+# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
+#
+# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
+# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
+# delete this block. Nothing else in the repo depends on it.
+node_baseline_backports_kernel: false
+# The suite, mirror and components the source line is built from. Split out so a mirror
+# change is a variable, not an edit to a heredoc in tasks/main.yml.
+node_baseline_backports_suite: trixie-backports
+node_baseline_backports_mirror: "http://deb.debian.org/debian"
+# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
+# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
+# surface beyond what the node already trusts is a separate decision nobody has taken.
+node_baseline_backports_components:
+  - main
+  - non-free-firmware
+# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
+# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
+# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
+# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
+# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
+# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
+# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
+node_baseline_backports_kernel_packages:
+  - linux-image-amd64
+# Provenance/rollback note written on the node. An operator standing in front of a prod box
+# with no repo checkout must be able to answer "why is this machine not on the distro
+# kernel, and how do I undo it?" from the machine itself.
+node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
+
 node_baseline_kernel_modules:
   - overlay
   - br_netfilter
   - iscsi_tcp
   - dm_crypt
   - nfs
+
+# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
+# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
+# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
+#
+#   node_baseline_longhorn_source:
+#     device           dedicated block device at node_baseline_longhorn_device (DEV: the
+#                      tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
+#     claim-mountpoint re-purpose the partition the node ALREADY has mounted at
+#                      node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
+#                      path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
+#                      and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
+#                      is correct and a hardcoded one would eventually mkfs the wrong
+#                      partition on a future node. Destructive; heavily guarded — read
+#                      tasks/disks.yml before changing anything here.
+#
+#   node_baseline_etcd_source:
+#     device                dedicated block device at node_baseline_etcd_device (DEV:
+#                           /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
+#                           isolated from Longhorn rebuild I/O after dev was burned by HDD
+#                           contention).
+#     root-filesystem       etcd lives on the OS filesystem, with NO dedicated device and
+#                           NO claimed data partition under it. Kept as a valid, supported
+#                           shape; NOT what prod uses any more (see below).
+#     shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
+#                           CLAIMED data partition (PROD). No dedicated device, but also NOT
+#                           the OS filesystem: the whole point of the 2026-09-04 layout
+#                           revision is that /var (11.8 GiB) can hold neither the container
+#                           image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
+#
+# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
+# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
+# var, which still fails the `disks` assert.
+node_baseline_longhorn_source: device
 node_baseline_longhorn_device: /dev/vdb
-node_baseline_longhorn_mount: /var/lib/longhorn
-node_baseline_longhorn_fs_label: longhorn
+# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
+# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
+# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
+# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
+# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
+#
+# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
+# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
+# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
+# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
+# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
+node_baseline_data_mount: /var/lib/longhorn
+node_baseline_data_fs_label: longhorn
+# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
+# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
+# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
+# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
+# same fact, and the two are checked against each other by nothing but this comment.
+node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
+# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
+# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
+# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
+# install flags, this role reads the same value to create the directory and to place etcd,
+# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
+# value (higher precedence than any role default) is what actually resolves here, even
+# though roles/k3s_server is not in the node-baseline play; the default() keeps
+# node-baseline.yml runnable standalone.
+node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
+# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
+# default so a dev host cannot enter the claim path even by accident.
+node_baseline_longhorn_claim_mountpoint: ""
+# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
+# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
+# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
+# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
+# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
+# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
+# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
+node_baseline_data_retired_fs_label: ""
+node_baseline_data_retired_mount: ""
+# Sanity floor for the claim path: a data partition smaller than this is not "a small
+# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
+node_baseline_longhorn_min_size_gb: 100
 # Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
 # so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
 # See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
-# device renames (matches the Longhorn LABEL= approach).
+# device renames (matches the data partition's LABEL= approach).
+node_baseline_etcd_source: device
 node_baseline_etcd_device: /dev/vdc
-node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
+# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
+# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
+# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
+# to /data/k3s/server/db, inside the claimed partition.
+node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
 node_baseline_etcd_fs_label: k3s-etcd
+# Mounts whose backing devices the claim path must NEVER touch. Used to build the
+# protected-device set from the node's own gathered facts (plus live swap), so a node
+# partitioned differently from the fleet is judged on its own layout rather than against
+# a hardcoded list of partition names.
+node_baseline_protected_mounts:
+  - /
+  - /boot
+  - /boot/efi
+  - /usr
+  - /var
+  - /home
+  - /tmp
 
 # --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
 # These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml

infra/ansible/roles/node_baseline/tasks/disks.yml

@@ -0,0 +1,642 @@
+---
+# =====================================================================================
+# node_baseline — DATA DISKS.  Every task here carries the `disks` tag (applied once by
+# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
+# Split into its own file rather than living in main.yml because it is the one part of
+# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
+#
+# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
+#
+#   dev  (group_vars/all.yml, role defaults)
+#     node_baseline_longhorn_source: device          -> mkfs /dev/vdb, mount LABEL=longhorn
+#                                                       at /var/lib/longhorn (== dataPath)
+#     node_baseline_etcd_source:     device          -> mkfs /dev/vdc, mount LABEL=k3s-etcd
+#     k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
+#     THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
+#     exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
+#
+#   prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
+#     node_baseline_longhorn_source: claim-mountpoint  (claim mountpoint /srv)
+#     node_baseline_data_mount:      /data            (LABEL=k3s-data)
+#     node_baseline_etcd_source:     shared-data-partition
+#     k3s_data_dir:                  /data/k3s
+#     ONE physical disk per node, already fully partitioned by the Debian installer. The
+#     ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
+#     between TWO tenants:  /data/longhorn (Longhorn replicas)  +  /data/k3s (the whole k3s
+#     data dir: etcd AND the containerd image store).
+#
+# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
+# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
+# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
+# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
+# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
+# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
+# partition is the fix; the partition therefore serves two tenants, and neither its
+# mountpoint nor its filesystem label may name only one of them.
+#
+# WHY /data AND LABEL k3s-data.
+#   * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
+#     Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
+#     at, and the idempotency contract is that /srv is gone from /proc/mounts and
+#     /etc/fstab once the claim has happened. Re-using it as the destination would make
+#     "claimed" and "not yet claimed" indistinguishable.
+#   * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
+#     accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
+#     `data` as an example of somebody else's named volume. Labelling ours `data` would make
+#     a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
+#     cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
+#     convention in this repo.
+#
+# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
+# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
+# empty string, and the second one must never silently succeed. `root-filesystem` and
+# `shared-data-partition` are explicit, greppable statements of intent that the assert
+# below accepts; an unset or placeholder device is still a hard failure. The two differ in
+# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
+# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
+# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
+# than printing a claim it never checked (see the etcd section).
+#
+# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
+# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
+# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
+# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
+# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
+# point at the wrong partition on some future node and mkfs it. So the role asks the NODE
+# which block device is mounted at the claim mountpoint, and then proves that answer is
+# safe (see the assert wall below) before touching it. The inventory declares INTENT
+# ("the partition currently mounted at /srv"); the node supplies the FACT.
+# =====================================================================================
+
+# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
+# its data-disk layout. Two failure shapes are caught:
+#   * an undeclared/typo'd source enum, and
+#   * source: device with a device path that is a PLACEHOLDER rather than a real device.
+# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
+# vars now that prod uses neither: if anyone ever flips a source back to `device` without
+# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
+# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
+# stays in the token list because that was the pre-decision placeholder and may still be
+# carried by other environments/branches.
+- name: Assert this environment's data-disk layout has actually been declared
+  ansible.builtin.assert:
+    that:
+      - node_baseline_longhorn_source in ['device', 'claim-mountpoint']
+      - node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
+      # `shared-data-partition` is only coherent when there IS a claimed data partition to
+      # share, and only when the k3s data dir actually sits INSIDE it. Catch the
+      # contradiction here, in inventory terms, rather than three sections later when etcd
+      # has silently landed back on the OS filesystem.
+      - >-
+        node_baseline_etcd_source != 'shared-data-partition'
+        or node_baseline_longhorn_source == 'claim-mountpoint'
+      - >-
+        node_baseline_etcd_source != 'shared-data-partition'
+        or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
+      # The transition knobs are a matched PAIR: a retired label with no retired mountpoint
+      # (or vice versa) cannot describe a state to migrate from.
+      - >-
+        (node_baseline_data_retired_fs_label | length > 0)
+        == (node_baseline_data_retired_mount | length > 0)
+      - >-
+        node_baseline_longhorn_source != 'device'
+        or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
+      - >-
+        node_baseline_etcd_source != 'device'
+        or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
+      - >-
+        node_baseline_longhorn_source != 'claim-mountpoint'
+        or (node_baseline_longhorn_claim_mountpoint | length) > 0
+    fail_msg: >-
+      This host's data-disk layout is not declared.
+      longhorn_source={{ node_baseline_longhorn_source }}
+      (device={{ node_baseline_longhorn_device }},
+      claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
+      etcd_source={{ node_baseline_etcd_source }}
+      (device={{ node_baseline_etcd_device }}).
+      Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
+      `claim-mountpoint` (re-purpose the partition already mounted at
+      node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
+      `device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
+      it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
+      data partition — prod's shape since 2026-09-04, which additionally requires
+      longhorn_source `claim-mountpoint` and a k3s data dir under
+      node_baseline_data_mount={{ node_baseline_data_mount }};
+      k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
+      operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
+      Declare it in the environment's group_vars and record the reasoning in
+      §2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
+      `--tags packages`, which skips every disk task including this assert.
+    quiet: true
+
+# =====================================================================================
+# THE DATA PARTITION  (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
+# =====================================================================================
+
+# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
+# rather than a reformat. Once the claim has happened the partition carries the ext4
+# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
+# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
+# mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
+# "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
+# path means converged, rc != 0 means the claim has not been made. Everything destructive
+# below is gated on rc != 0.
+- name: "Data partition: look for a filesystem already carrying the data-partition label"
+  ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
+  register: node_baseline_lh_labelled
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+# --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
+# WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
+# label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
+# takes over — which then fails closed at "the claim mountpoint must actually be a
+# mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
+# behaviour but it is not convergence, and the prod nodes are in exactly that state today:
+# claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
+# So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
+#
+# WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
+# partition with the new label. It never touches the DATA and never reformats: e2label is a
+# metadata write, exactly as in the claim path below. The common mount task then mounts
+# LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
+# `blkid -L <new label>` succeeds and none of this is reachable.
+#
+# THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
+# volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
+# unmounting it would orphan every one of them:
+#   1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
+#      group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
+#   2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
+#      the same structural gate that keeps dev out of the claim path keeps it out of this one.
+#   3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
+#      replica directories on it fails closed and nothing is unmounted.
+# Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
+- name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
+  ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
+  register: node_baseline_lh_retired_labelled
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when:
+    - node_baseline_longhorn_source == 'claim-mountpoint'
+    - node_baseline_data_retired_fs_label | length > 0
+    - node_baseline_lh_labelled.rc != 0
+
+- name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
+  ansible.builtin.set_fact:
+    # A node claimed under the RETIRED label. Mutually exclusive with the claim below.
+    node_baseline_lh_transition_needed: >-
+      {{ node_baseline_longhorn_source == 'claim-mountpoint'
+         and node_baseline_data_retired_fs_label | length > 0
+         and node_baseline_lh_labelled.rc != 0
+         and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
+    # True only for a claim-mountpoint environment that has NOT been claimed under EITHER
+    # label. A `device` environment (dev) can never enter the claim path at all — that is
+    # the structural guarantee that no generalisation of prod's behaviour can reach the
+    # LIVE dev nodes and their /dev/vdb Longhorn replicas.
+    node_baseline_lh_claim_needed: >-
+      {{ node_baseline_longhorn_source == 'claim-mountpoint'
+         and node_baseline_lh_labelled.rc != 0
+         and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
+
+- name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_data_retired_mount }}
+  register: node_baseline_lh_retired_findmnt
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_transition_needed | bool
+
+# Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
+# not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
+# an unmount of a populated Longhorn store would orphan every replica on the node.
+- name: "Data partition (transition): read the contents of the retired mountpoint"
+  ansible.builtin.find:
+    paths: "{{ node_baseline_data_retired_mount }}"
+    file_type: any
+    hidden: true
+    recurse: false
+    excludes: ['lost+found']
+  register: node_baseline_lh_retired_contents
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): prove this node is safe to re-label"
+  ansible.builtin.assert:
+    that:
+      # 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
+      #    retired label names. The only state this role knows how to transition is the one
+      #    the previous version of it produced; anything else is an operator decision.
+      - (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
+      - >-
+        (node_baseline_lh_retired_findmnt.stdout | default('') | trim)
+        == (node_baseline_lh_retired_labelled.stdout | default('') | trim)
+      # 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
+      #    not the node we think it is, or a tenant has already written data — in both cases
+      #    unmounting and re-labelling under it is destructive and the role must stop.
+      - (node_baseline_lh_retired_contents.files | length) == 0
+    fail_msg: >-
+      REFUSING to transition this node's data partition from
+      LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
+      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
+      device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
+      device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
+      (findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
+      non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
+      {{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
+      If the partition holds DATA, this transition would orphan it — Longhorn replicas are
+      not recoverable from an unmounted, re-labelled filesystem. Migrate the data
+      deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
+      Do NOT relax this assert.
+    quiet: true
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): report exactly what is about to change"
+  ansible.builtin.debug:
+    msg: >-
+      Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
+      from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
+      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
+      unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
+      No filesystem is created and no data is moved.
+  when: node_baseline_lh_transition_needed | bool
+
+# e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
+# breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
+- name: "Data partition (transition): unmount the retired mountpoint"
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_retired_mount }}"
+    state: unmounted
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_retired_mount }}"
+    state: absent_from_fstab
+    backup: true
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): re-stamp the partition with the new filesystem label"
+  ansible.builtin.command: >-
+    e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
+    {{ node_baseline_data_fs_label }}
+  changed_when: true
+  when: node_baseline_lh_transition_needed | bool
+
+# --- The claim (prod, first run only) -------------------------------------------------
+# Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
+# mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
+# returning the device behind `/` — which is precisely the mistake that would hand the
+# root partition to e2label.
+- name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
+  register: node_baseline_lh_claim_findmnt
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
+  ansible.builtin.assert:
+    that:
+      - node_baseline_lh_claim_findmnt.rc == 0
+      - (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
+    fail_msg: >-
+      node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
+      yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
+      this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
+      stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
+      The role will NOT go looking for "a big partition" to claim instead. Either the
+      node is partitioned differently from the fleet — in which case fix the node or give
+      this host its own declaration — or the claim already happened and the label was
+      removed by hand.
+    quiet: true
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): inspect the candidate partition"
+  ansible.builtin.command: >-
+    lsblk --json --nodeps --bytes
+    --output PATH,TYPE,FSTYPE,LABEL,SIZE
+    {{ node_baseline_lh_claim_findmnt.stdout | trim }}
+  register: node_baseline_lh_claim_lsblk
+  changed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+# The devices this role must never, under any circumstances, hand to e2label. Built from
+# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
+# NODE's account of what is load-bearing, not a hardcoded list of partition names.
+- name: "Longhorn (claim): list the devices currently backing active swap"
+  ansible.builtin.command: swapon --noheadings --show=NAME
+  register: node_baseline_lh_swapdevs
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+# Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
+# ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
+- name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
+  ansible.builtin.find:
+    paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    file_type: any
+    hidden: true
+    recurse: false
+    excludes: ['lost+found']
+  register: node_baseline_lh_claim_contents
+  when: node_baseline_lh_claim_needed | bool
+
+# THE GUARD WALL. Five independent conditions, every one of which must hold before the
+# role modifies a filesystem it did not create. Any single failure stops the play with
+# the evidence in the message — nothing is unmounted, relabelled or mounted first.
+- name: "Longhorn (claim): prove the candidate partition is safe to claim"
+  ansible.builtin.assert:
+    that:
+      # 1. It is a real block device — a partition or a whole disk. Catches a claim
+      #    mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
+      #    which would have produced a non-/dev SOURCE or an lsblk row of another type.
+      - node_baseline_lh_candidate.type in ['part', 'disk']
+      # 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
+      #    so a differently-partitioned node is caught by its own facts.
+      - node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
+      # 3. It already holds ext4. The claim path RELABELS rather than reformats (see
+      #    below), which is only meaningful for the filesystem type we are about to
+      #    declare in fstab. Anything else is an operator decision, not a role's.
+      - node_baseline_lh_candidate.fstype == 'ext4'
+      # 4. It carries no filesystem label, or already carries ours. A partition labelled
+      #    `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
+      #    it would make it unfindable by its own fstab entry.
+      - (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
+      # 5. It is big enough to be the partition we meant. A data partition of a few GB is
+      #    not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
+      - (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
+      # 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
+      #    ext4 5% root reserve; this is where that premise is verified per node instead
+      #    of trusted. If any node has real data on it, the play stops here — the role
+      #    never wipes data it cannot account for.
+      - (node_baseline_lh_claim_contents.files | length) == 0
+    fail_msg: >-
+      REFUSING to claim {{ node_baseline_lh_candidate.path }}
+      (mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
+      type={{ node_baseline_lh_candidate.type }},
+      fstype={{ node_baseline_lh_candidate.fstype }},
+      label='{{ node_baseline_lh_candidate.label | default('', true) }}',
+      size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
+      (minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
+      non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
+      {{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
+      devices this node needs and that must never be claimed:
+      {{ node_baseline_lh_protected_devices }}.
+      This is the destructive step of the role and it fails CLOSED. If the partition
+      holds data, back it up or pick a different one; do NOT relax this assert.
+    quiet: true
+  vars:
+    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
+    # The node's OWN account of what it cannot afford to lose: the devices behind the
+    # mounts in node_baseline_protected_mounts (from gathered facts, so a node
+    # partitioned differently is judged on its own layout) plus every active swap device.
+    # Built here rather than hardcoded so no partition-name list can go stale.
+    node_baseline_lh_protected_devices: >-
+      {{ (ansible_facts.mounts | default([])
+          | selectattr('mount', 'in', node_baseline_protected_mounts)
+          | map(attribute='device') | list)
+         + (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): report exactly what is about to be claimed"
+  ansible.builtin.debug:
+    msg: >-
+      Claiming {{ node_baseline_lh_candidate.path }}
+      ({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
+      currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
+      as the data partition: unmount -> drop its fstab entry ->
+      relabel LABEL={{ node_baseline_data_fs_label }} ->
+      mount at {{ node_baseline_data_mount }}.
+  vars:
+    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
+  when: node_baseline_lh_claim_needed | bool
+
+# --- Why the role must unmount and relabel, not just point fstab at the device ---------
+# The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
+# /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
+# path does not. The installer-created /srv filesystem has NO label (fstab pins it by
+# UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
+# the unmount first — and the fstab entry must go in the same breath, or the next boot
+# re-mounts the same partition at /srv underneath Longhorn.
+#
+# RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
+# mkfs would buy nothing and would destroy anything the emptiness check got wrong.
+# e2label is a metadata write: if some assert above was too permissive, the worst case is
+# a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
+# cannot do this job: it declines to touch a device that already has a filesystem, so it
+# would silently no-op and then the LABEL= mount would fail with no useful message.)
+- name: "Longhorn (claim): unmount the claim mountpoint"
+  ansible.posix.mount:
+    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    state: unmounted
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
+  ansible.posix.mount:
+    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    state: absent_from_fstab
+    backup: true
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
+  ansible.builtin.command: >-
+    e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
+    {{ node_baseline_data_fs_label }}
+  changed_when: true
+  when: node_baseline_lh_claim_needed | bool
+
+# --- The dedicated-device path (dev) --------------------------------------------------
+# Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
+# never reformats a device that already carries a filesystem.
+- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_longhorn_device }}"
+    opts: "-L {{ node_baseline_data_fs_label }}"
+  when: node_baseline_longhorn_source == 'device'
+
+# --- Common to ALL paths (device, claim, transition) ----------------------------------
+# Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
+# mount task. Idempotent: on a converged node this reports ok, which is what makes a
+# re-run a no-op rather than a reformat. It is also what completes the transition above:
+# the retired mountpoint is gone from fstab and the partition now answers to the new label,
+# so this task mounts it at the new path.
+- name: Mount the data partition
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_mount }}"
+    src: "LABEL={{ node_baseline_data_fs_label }}"
+    fstype: ext4
+    opts: "defaults,nofail,x-systemd.device-timeout=5s"
+    state: mounted
+
+# --- The per-tenant subdirectories ---------------------------------------------------
+# WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
+# tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
+# subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
+# /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
+# either there would at best be a no-op and at worst re-mode the running dev cluster's data
+# dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
+# mounted — i.e. only the directories it is actually responsible for. That test is derived
+# from the paths themselves, so no environment has to remember to switch it off.
+#
+# MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
+# the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
+# etcd database, the node token and the containerd image store, and k3s would tighten it to
+# 0700 itself on first start — doing it here means it is never briefly world-readable.
+- name: Create the per-tenant data directories on the data partition
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
+    - { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
+  loop_control:
+    label: "{{ item.path }} ({{ item.tenant }})"
+  when: item.path.startswith(node_baseline_data_mount ~ '/')
+
+# =====================================================================================
+# ETCD DATA DIR
+# =====================================================================================
+
+# --- The dedicated-device path (dev) --------------------------------------------------
+# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
+# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
+# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
+# a disk that already carries a filesystem, so re-running against an already-migrated
+# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
+# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
+# role would mount over a populated directory.
+- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_etcd_device }}"
+    opts: "-L {{ node_baseline_etcd_fs_label }}"
+  when: node_baseline_etcd_source == 'device'
+
+- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
+  ansible.posix.mount:
+    path: "{{ node_baseline_etcd_mount }}"
+    src: "LABEL={{ node_baseline_etcd_fs_label }}"
+    fstype: ext4
+    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
+    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
+    opts: "defaults,noatime"
+    dump: "0"
+    passno: "2"
+    state: mounted
+  when: node_baseline_etcd_source == 'device'
+
+# --- The no-dedicated-device paths (prod) ---------------------------------------------
+# Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
+# populated — so etcd gets no device of its own. That is an operator decision
+# (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
+# UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
+# on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
+# latency rather than collapsing it. Accepted to be MEASURED, not assumed.
+#
+# WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
+# filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
+# 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
+# 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
+# could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
+# partition and etcd moved with it: `shared-data-partition`. Both enum values are still
+# supported, and the role now PROVES which one is true instead of printing a claim.
+#
+# There is nothing to CREATE here for either shape — the data-partition section above
+# already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
+# role's job is to verify the assumption rather than act on it.
+- name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_etcd_mount }}
+  register: node_baseline_etcd_existing_mount
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+- name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
+  ansible.builtin.assert:
+    that:
+      - node_baseline_etcd_existing_mount.rc != 0
+    fail_msg: >-
+      node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
+      NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
+      ALREADY a separate mount, backed by
+      {{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
+      contradiction the role will not resolve on its own: either this node was previously
+      built with a dedicated etcd device (in which case unmount it deliberately, after
+      accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
+      the node in a state where the declared layout and the real one disagree.
+    quiet: true
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+# Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
+# WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
+# existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
+# that does exist — the same filesystem the data dir will be created on. TARGET is captured
+# separately from the human-readable line because the assert below judges on it.
+- name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
+  ansible.builtin.shell: |
+    set -eu
+    p={{ node_baseline_etcd_mount | quote }}
+    while [ ! -d "$p" ]; do p=$(dirname "$p"); done
+    findmnt --noheadings --first-only --output TARGET --target "$p"
+    findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
+  register: node_baseline_etcd_backing_fs
+  changed_when: false
+  check_mode: false
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+# THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
+# `shared-data-partition` is a distinct enum value rather than a comment. Declaring
+# k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
+# above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
+# of container images would land back on the OS filesystem — the exact failure this change
+# exists to prevent, and completely invisible until the disk filled. So the role checks that
+# the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
+# run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
+# which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
+# whole record for that shape.
+- name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
+  ansible.builtin.assert:
+    that:
+      - >-
+        (node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
+        == node_baseline_data_mount
+    fail_msg: >-
+      node_baseline_etcd_source is `shared-data-partition`, which asserts that
+      {{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
+      {{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
+      '{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
+      (full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
+      The data partition is therefore NOT mounted where this layout requires, and k3s would
+      put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
+      filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
+      image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
+      this node until the mount is correct.
+    quiet: true
+  when: node_baseline_etcd_source == 'shared-data-partition'
+
+- name: "etcd (no dedicated device): show which filesystem etcd will live on"
+  ansible.builtin.debug:
+    msg: >-
+      etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
+      (source={{ node_baseline_etcd_source }}); it lands on
+      {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
+      (SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
+      measured risk — §2.
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1,6 +1,6 @@
 ---
-# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
-# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
+# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
+# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
 # and the DATA-DISK work have different readiness: packages are safe on any node from
 # first contact, while the disk tasks need block devices that a new environment may not
 # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -12,6 +12,10 @@
 # mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
 # at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
 #   ansible-playbook playbooks/node-baseline.yml --tags dns
+# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
+# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
+# it must be runnable without touching a disk decision that is still open:
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
 
 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
   ansible.builtin.apt:
@@ -21,6 +25,237 @@
     cache_valid_time: 3600
   tags: [packages]
 
+# =====================================================================================
+# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
+# =====================================================================================
+# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
+# FUTURE prod node gets it from provisioning instead of from someone remembering.
+#
+# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
+# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
+# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
+# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
+# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
+# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
+# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
+# (first released in v6.13) validates the size and rejects the table instead. It is absent
+# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
+# stable feed will ever pick it up. There is NO kernel cmdline workaround —
+# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
+# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
+# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
+#
+# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
+# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
+# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
+# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
+# change for no benefit. See defaults/main.yml for why this is a variable rather than a
+# `k3s-prod-*` hostname test.
+#
+# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
+# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
+# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
+# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
+# before touching the next, never node 2 first — and a role cannot enforce that. The last
+# task here reports the pending reboot loudly instead of letting a green play imply the node
+# is already fixed.
+- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
+  when: node_baseline_backports_kernel | bool
+  tags: [kernel]
+  block:
+
+    # The suite name and the running release must agree, or apt resolves no candidate and
+    # `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
+    # a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
+    # dist-upgraded to forky while group_vars still says trixie-backports).
+    - name: Assert the configured backports suite matches this node's Debian release
+      ansible.builtin.assert:
+        that:
+          - ansible_facts['distribution'] == 'Debian'
+          - node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
+        fail_msg: >-
+          node_baseline_backports_kernel is enabled but node_baseline_backports_suite
+          ({{ node_baseline_backports_suite }}) is not the backports suite for this node's
+          release ({{ ansible_facts['distribution'] }}
+          {{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
+          candidate and this play would report success while leaving the node on the
+          vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
+        quiet: true
+
+    # deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
+    # prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
+    # rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
+    # idempotent in a way `apt_repository` — which rewrites and reformats — is not.
+    #
+    # THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
+    # backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
+    # apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
+    # 500 for a package that was explicitly installed from there so it keeps getting updates.
+    # Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
+    # HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
+    # be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
+    - name: Enable the Debian backports apt source (deb822)
+      ansible.builtin.copy:
+        dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
+          # A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
+          # machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
+          # Table its firmware publishes (upstream 8fbe4c49c0cc). See
+          # infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
+          # on this node.
+          #
+          # Backports is NotAutomatic, so nothing is taken from here unless asked for by name
+          # with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
+          # ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
+          # Pin-Priority would switch that off.
+          Types: deb
+          URIs: {{ node_baseline_backports_mirror }}
+          Suites: {{ node_baseline_backports_suite }}
+          Components: {{ node_baseline_backports_components | join(' ') }}
+          Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
+      register: node_baseline_backports_source
+
+    - name: Refresh the apt cache after enabling backports
+      ansible.builtin.apt:
+        update_cache: true
+      when: node_baseline_backports_source is changed
+
+    # `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
+    # nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
+    # nothing at all — the single most likely way to write this task and have it silently fix
+    # nothing forever. `latest` + `default_release` resolves the candidate inside backports
+    # and upgrades the metapackage to it. Once the node is current a re-run is a genuine
+    # no-op: no change, no churn, no spurious reboot flag. Scope is exactly
+    # node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
+    # is dragged out of the main archive.
+    - name: Install the fixed kernel from backports (>= 6.13)
+      ansible.builtin.apt:
+        name: "{{ node_baseline_backports_kernel_packages }}"
+        state: latest
+        default_release: "{{ node_baseline_backports_suite }}"
+        update_cache: true
+        cache_valid_time: 3600
+      register: node_baseline_backports_kernel_apt
+
+    # WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
+    # metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
+    # answer anybody can roll back from six months later.
+    - name: Read back the installed backports kernel metapackage version
+      ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
+      register: node_baseline_backports_kernel_version
+      changed_when: false
+      check_mode: false
+
+    # `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
+    # neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
+    # list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
+    # GRUB's "Advanced options" without uninstalling anything.
+    - name: List every kernel image installed on this node, newest first
+      ansible.builtin.shell:
+        cmd: set -o pipefail; linux-version list | linux-version sort --reverse
+        executable: /bin/bash
+      register: node_baseline_installed_kernels
+      changed_when: false
+      check_mode: false
+
+    # The node must be able to explain itself to an operator standing in front of it with no
+    # repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
+    # role is a real no-op rather than a permanent daily "changed".
+    - name: Record the deviation, the exact version, and the rollback path on the node
+      ansible.builtin.copy:
+        dest: "{{ node_baseline_backports_kernel_stamp }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
+          #
+          # THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
+          #
+          # WHY
+          #   This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
+          #   Linux < 6.13 reserves memory from it without validating its size, and an unlucky
+          #   boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
+          #   k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
+          #   is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
+          #   v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
+          #   Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
+          #
+          # PROOF THE FIX IS ACTIVE (not merely a lucky boot)
+          #   BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
+          #   >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
+          #     zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
+          #   (roles/node_health does the version check for you; see the health report's
+          #   firmware.efi_memattr finding.)
+          #
+          #   FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
+          #   and each prints a DIFFERENT line — all three are rejections:
+          #     tbl->version > 2         'Unexpected EFI Memory Attributes table version %d'
+          #                              (NOTE: printed with %d, so garbage may read NEGATIVE)
+          #     bad desc_size            'Unexpected EFI Memory Attributes descriptor size ...'
+          #     num_entries > 64K        '[Firmware Bug]: Corrupted EFI Memory Attributes
+          #                               Table detected! (version == ..., num_entries == ...)'
+          #   Which one appears depends on WHICH field this boot's garbage corrupted, so do
+          #   NOT treat the absence of the FW_BUG line as the fix being absent — all three
+          #   prod nodes printed the *version* line on 2026-09-04.
+          #   The one line that means the fix is NOT active:
+          #     'Failed to map EFI Memory Attributes table @ 0x...'  — the kernel trusted the
+          #     corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
+          #     escalate, do not shrug.
+          #
+          # INSTALLED
+          #   source   /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
+          #   package  {{ node_baseline_backports_kernel_packages | join(' ') }}
+          #   version  {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
+          #   kernels installed on this node, newest first:
+          {% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
+          #     {{ kv }}
+          {% endfor %}
+          #
+          # SECURITY COST
+          #   Packages from *-backports are provided AS IS. They are NOT covered by the Debian
+          #   Security Team's guarantees for the main archive: no security-tracker commitment,
+          #   no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
+          #
+          # ROLLBACK — no repo checkout needed
+          #   1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
+          #      and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
+          #      GNU/Linux". That answers "is the new kernel the problem?" without
+          #      uninstalling anything. Note it reinstates the memattr bug for that boot.
+          #   2. PERMANENT. Set node_baseline_backports_kernel: false in
+          #      inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
+          #      puts the backports kernel straight back — then, on this node:
+          #        apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
+          #        apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
+          #        rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
+          #        apt-get update && update-grub && reboot
+          #
+          # EXIT CONDITION
+          #   If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
+          #   rebase), this deviation can be removed entirely: flip the variable to false,
+          #   roll back as above, and delete the `kernel` block from roles/node_baseline.
+
+    # Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
+    # it means the fixed kernel is on disk. See the block header for why the role will not
+    # reboot on its own.
+    - name: Report whether this node is running the kernel it now has installed
+      ansible.builtin.debug:
+        msg: >-
+          {{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
+              ~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
+              prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
+              node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
+              with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
+             if newest != ansible_facts['kernel']
+             else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
+                   ~ '); no reboot pending for the memattr fix.') }}
+      vars:
+        newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
+
 - name: Persist kernel modules for k3s/Cilium/Longhorn
   ansible.builtin.copy:
     dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
@@ -79,70 +314,32 @@
     state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
   tags: [system]
 
-# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
-# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
-# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
-# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
-# which do not exist on bare metal either — a confusing failure that looks like a wiring
-# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
-# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
-- name: Assert this environment's data-disk devices have actually been decided
-  ansible.builtin.assert:
-    that:
-      - "'BLOCKED' not in node_baseline_longhorn_device"
-      - "'BLOCKED' not in node_baseline_etcd_device"
-    fail_msg: >-
-      node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
-      placeholders for this host's environment
-      (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
-      The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
-      and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
-      the SMART read that decision waits on can be taken) re-run this play with
-      `--tags packages`, which skips every disk task including this assert.
-    quiet: true
-  tags: [disks]
-
-- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
-  community.general.filesystem:
-    fstype: ext4
-    dev: "{{ node_baseline_longhorn_device }}"
-    opts: "-L {{ node_baseline_longhorn_fs_label }}"
-  tags: [disks]
-
-- name: Mount the Longhorn data disk
-  ansible.posix.mount:
-    path: "{{ node_baseline_longhorn_mount }}"
-    src: "LABEL={{ node_baseline_longhorn_fs_label }}"
-    fstype: ext4
-    opts: "defaults,nofail,x-systemd.device-timeout=5s"
-    state: mounted
-  tags: [disks]
-
-# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
-# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
-# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
-# a disk that already carries a filesystem, so re-running against an already-migrated
-# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
-# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
-# role would mount over a populated directory.
-- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
-  community.general.filesystem:
-    fstype: ext4
-    dev: "{{ node_baseline_etcd_device }}"
-    opts: "-L {{ node_baseline_etcd_fs_label }}"
-  tags: [disks]
-
-- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
-  ansible.posix.mount:
-    path: "{{ node_baseline_etcd_mount }}"
-    src: "LABEL={{ node_baseline_etcd_fs_label }}"
-    fstype: ext4
-    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
-    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
-    opts: "defaults,noatime"
-    dump: "0"
-    passno: "2"
-    state: mounted
+# =====================================================================================
+# DATA DISKS — tag `disks`, in tasks/disks.yml
+# =====================================================================================
+# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
+# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
+# should be readable end-to-end on its own; and both environments' storage shapes — dev's
+# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
+# the OS filesystem — are declared there as an explicit enum rather than inferred from
+# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
+#
+# The `disks` tag is applied ONCE here, in BOTH places it is needed:
+#   * `tags:` on the include itself, so `--tags packages` never even reaches the include
+#     and a placeholder device path is not so much as templated; and
+#   * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
+#     optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
+#     its own tags to its children the way `import_tasks` does, so without `apply` the
+#     included tasks are untagged and `--tags disks` selects the include and then runs
+#     NOTHING inside it. That failure is silent and reports success, which is the worst
+#     shape a guard can have.
+# A FULL run against a node whose layout is undeclared stops on the assert at the top of
+# that file instead of failing obscurely at mkfs time.
+- name: Provision the data disks (Longhorn store, etcd data dir)
+  ansible.builtin.include_tasks:
+    file: disks.yml
+    apply:
+      tags: [disks]
   tags: [disks]
 
 - name: Disable swap now

infra/ansible/roles/node_health/README.md

@@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI,
   EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified
   against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM).
-- **Never `become`s unless it must.** Only SMART needs root.
+- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring
+  buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM
+  inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and
+  the privileged second pass re-runs the *same* collector with `--privileged` so the
+  parsing has one home. Without escalation those checks SKIP with the exact reason.
 
 ## What it checks
 
@@ -44,15 +48,99 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
 | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
+| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
 | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
+| `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
 
 Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
 in task logic.
 
+
+## The EFI memattr check
+
+`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
+
+HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
+does not validate it before reserving memory from its contents. **Two different kernel
+functions matter, and the check used to conflate them:**
+
+- `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both
+  fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so
+  **catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through
+  `phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything
+  above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and
+  costs nothing measurable.
+- `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints
+  "Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or
+  merely large.
+
+So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not*
+prove memory was lost: that depends on the garbage **value** at that address on that boot,
+not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03)
+booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on
+measured memory, never on the log line alone:**
+
+| kernel log | measured shortfall | verdict |
+|---|---|---|
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic |
+| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working |
+| no memattr line, log reaches back to boot | (not consulted) | PASS |
+| no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
+| `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
+
+The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt
+table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt
+table is not passing a health check even when it kept its RAM.
+
+### The FW_BUG line is proof, but its ABSENCE proves nothing
+
+`efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG
+line:
+
+| gate | message | reachable when |
+|---|---|---|
+| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
+| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not |
+| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not |
+
+**All three are rejections.** Which one a boot prints is decided by *which field* that boot's
+garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very
+often prints the *first* message, and looking only for the FW_BUG line will make a working
+fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed
+the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version
+alongside the log line, and why the check must never demand the FW_BUG string as its
+evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in
+the kernel image itself, not a line in `dmesg`.
+
+**The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged
+*negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that
+until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected
+a corrupt table was reported as one that "accepted it without complaint" — a false PASS on
+the exact bug this check exists to find.
+
+**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
+`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
+workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
+rebooting/CMOS/BIOS/USB changes only move where the table lands.
+
+### The two memory checks cannot contradict each other
+
+`memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine
+lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus
+the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in
+`tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads
+it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of
+its own, so there is exactly one verdict on lost memory per host and both checks quote it.
+A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a
+future memory loss with a different cause is still caught.
+
 ## Output
 
 1. A human summary printed in the play output.

infra/ansible/roles/node_health/defaults/main.yml

@@ -45,6 +45,33 @@ node_health_thresholds:
   # Memory (percentage of MemTotal still available)
   mem_available_pct_warn: 15
   mem_available_pct_fail: 5
+  # --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ----------------------------
+  # Two independent measurements of the same question, both fed by memory.physical_total:
+  #
+  #   unmanaged  /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own
+  #              memory map says exists, that never reached the allocator. Needs no root
+  #              and no tools, works identically on metal and in a VM.
+  #   shortfall  MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is
+  #              installed. Needs root, and only bare metal reports it. Independent of the
+  #              kernel's memory map entirely, so it also catches RAM the firmware never
+  #              handed the kernel in the first place.
+  #
+  # MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and
+  # not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2%
+  # k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state.
+  # The healthy figure is the kernel image, memmap and firmware reservations; it is a
+  # near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10%
+  # keeps clearance for that. There is no plausible tuning of a threshold in this band
+  # that changes any of those five verdicts.
+  #
+  # The DIMM shortfall runs a little higher than the unmanaged figure on the same node
+  # (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and
+  # the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the
+  # kernel's to manage. Same bands: the signal being caught is 83%, not 3%.
+  mem_unmanaged_pct_warn: 10
+  mem_unmanaged_pct_fail: 25
+  mem_dimm_shortfall_pct_warn: 10
+  mem_dimm_shortfall_pct_fail: 25
   # Temperatures (deg C, any sensor)
   temp_warn_c: 70
   temp_fail_c: 85

infra/ansible/roles/node_health/files/node_health_collect.py

@@ -97,10 +97,22 @@ def read_kv(path, sep=":"):
     return out
 
 
+def which(tool):
+    """Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits.
+
+    `dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the
+    unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed
+    but unreachable would be a false reason on a SKIP, which is the one thing this
+    collector may never do."""
+    return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+
+
 def run(argv, timeout=20):
     """Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent."""
-    if shutil.which(argv[0]) is None:
+    exe = which(argv[0])
+    if exe is None:
         return None, "", "%s not installed" % argv[0]
+    argv = [exe] + list(argv[1:])
     try:
         proc = subprocess.run(
             argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout
@@ -284,9 +296,300 @@ def collect_memory():
         edac["ce_total"] = ce_total
         edac["ue_total"] = ue_total
     mem["edac"] = edac
+    # Physically present vs actually usable — the cause-independent memory-loss signal.
+    mem["zones"] = collect_zoneinfo(mem["mem_total_kb"])
+    # Firmware's own inventory of what is physically installed. Root-only; attempted
+    # here anyway so the unprivileged pass records the exact reason it could not read.
+    mem["devices"] = collect_memory_devices()
     return mem
 
 
+# --- Memory the kernel has but never hands out --------------------------------------
+# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
+# no DMI. Per zone it reports:
+#   spanned  pages in the zone's address range, holes included
+#   present  spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says
+#            physically exists here
+#   managed  present MINUS everything memblock reserved before the buddy allocator was
+#            handed the rest (kernel image, memmap, initrd, crashkernel, firmware
+#            reservations). sum(managed) IS MemTotal, exactly, on every node measured.
+# So `present - managed` is, by construction, "RAM this machine has that the OS will
+# never allocate", whatever the reason. On a healthy node it is the couple of hundred
+# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and
+# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% —
+# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude
+# separation, which is why this needs no clever tuning to be trustworthy.
+#
+# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if
+# a future firmware, kernel or hypervisor lost the memory for an entirely different
+# reason, and it needs neither root nor a parseable kernel log.
+def collect_zoneinfo(mem_total_kb):
+    raw = read_text("/proc/zoneinfo", optional=False)
+    page_kb = os.sysconf("SC_PAGE_SIZE") // 1024
+    out = {
+        "available": raw is not None,
+        "reason": None if raw is not None else "/proc/zoneinfo could not be read",
+        "page_size_kb": page_kb,
+        "zones": [],
+        "present_kb": None,
+        "managed_kb": None,
+        "unmanaged_kb": None,
+        "unmanaged_pct": None,
+    }
+    if raw is None:
+        return out
+    node = zone = None
+    cur = {}
+    present_pages = managed_pages = 0
+    for line in raw.splitlines():
+        fields = line.split()
+        if line.startswith("Node ") and "zone" in line:
+            # "Node 0, zone   Normal"
+            if cur:
+                out["zones"].append(cur)
+            node, zone = fields[1].rstrip(","), fields[-1]
+            cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None}
+        elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"):
+            try:
+                cur[fields[0]] = int(fields[1])
+            except ValueError:
+                cur[fields[0]] = None
+    if cur:
+        out["zones"].append(cur)
+    # An empty parse is a PARSE failure, not "this node has no memory" — say so and stay
+    # null rather than emitting a confident 0.
+    if not out["zones"]:
+        out["available"] = False
+        out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it"
+        return out
+    for z in out["zones"]:
+        present_pages += z["present"] or 0
+        managed_pages += z["managed"] or 0
+    out["present_kb"] = present_pages * page_kb
+    out["managed_kb"] = managed_pages * page_kb
+    out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"]
+    if out["present_kb"] > 0:
+        out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1)
+    # Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the
+    # two readings disagree and the discrepancy itself is worth showing.
+    out["matches_mem_total"] = (
+        None if (mem_total_kb is None or out["managed_kb"] is None)
+        else abs(out["managed_kb"] - mem_total_kb) <= page_kb
+    )
+    return out
+
+
+# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) -----------------------------------
+# The independent answer to "how much RAM is physically in this box", from the firmware's
+# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads
+# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that
+# failure is recorded as the SKIP reason — never as zero DIMMs.
+#
+# On a VM there is usually either no type-17 structure at all or a single synthetic entry
+# that merely restates the configured guest RAM; both are handled by reporting what was
+# found and letting the check SKIP when nothing populated was reported.
+DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def collect_memory_devices():
+    out = {
+        "available": False,
+        "reason": None,
+        "devices": [],
+        "slots": 0,
+        "populated_slots": 0,
+        "installed_kb": None,
+    }
+    if which("dmidecode") is None:
+        out["reason"] = ("dmidecode is not installed on this node; it ships with "
+                         "roles/node_baseline (node_baseline_packages). "
+                         "This role never installs packages.")
+        return out
+    rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30)
+    if rc != 0:
+        out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % (
+            rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output")
+        return out
+    # Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which
+    # may or may not hold a module ("Size: No Module Installed").
+    total_kb = 0
+    saw_size = False
+    for block in stdout.split("\n\n"):
+        if "Memory Device" not in block:
+            continue
+        fields = {}
+        for line in block.splitlines():
+            key, sep, value = line.partition(":")
+            if sep:
+                fields[key.strip()] = value.strip()
+        if "Size" not in fields:
+            continue
+        out["slots"] += 1
+        size_kb = None
+        parts = fields["Size"].split()
+        if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS:
+            try:
+                size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()]
+            except ValueError:
+                size_kb = None
+        if size_kb:
+            saw_size = True
+            total_kb += size_kb
+            out["populated_slots"] += 1
+        out["devices"].append(
+            {
+                "locator": fields.get("Locator"),
+                "bank_locator": fields.get("Bank Locator"),
+                # Verbatim, so an unparsed spelling is visible rather than silently null.
+                "size_raw": fields["Size"],
+                "size_kb": size_kb,
+                "type": fields.get("Type"),
+                "speed": fields.get("Speed"),
+                "configured_speed": fields.get("Configured Memory Speed"),
+                "manufacturer": fields.get("Manufacturer"),
+                "part_number": fields.get("Part Number"),
+                "rank": fields.get("Rank"),
+                # Serial deliberately NOT collected: the artifact already carries drive
+                # serials and a DIMM serial adds identifying data no check reads.
+            }
+        )
+    out["available"] = saw_size
+    if not saw_size:
+        out["reason"] = ("dmidecode ran but reported no populated memory device "
+                         "(%d slot structure(s) found) — typical of a VM with no SMBIOS "
+                         "type 17 inventory" % out["slots"])
+    else:
+        out["installed_kb"] = total_kb
+    return out
+
+
+# --- Kernel ring buffer: the EFI Memory Attributes Table ----------------------------
+# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative
+# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
+# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
+# the product). The kernel prints one of two lines:
+#
+#   "Failed to map EFI Memory Attributes table @ 0x..."   the garbage version read <= 2, so
+#       the kernel TRUSTED the table and reserved memory from it. This is emitted later, by
+#       efi_memattr_apply_permissions()'s memremap(), and it fails for ANY sufficiently
+#       wrong tbl_size. So it proves the table was corrupt and trusted — it does NOT prove
+#       memory was lost: only a NEGATIVE tbl_size (bit 31 set, sign-extended and clamped to
+#       PHYS_ADDR_MAX) reserves everything above the table, while a garbage-but-positive
+#       tbl_size costs nothing measurable. Whether memory actually went missing is decided
+#       by MEASUREMENT (collect_zoneinfo below), not by this line: k3s-prod-2 lost 13.3 GiB
+#       twice at 0xbeeb0018 and then booted with all 15.1 GiB at the same address, same line.
+#   "Unexpected EFI Memory Attributes table version <N>"  the garbage version read > 2, so
+#       the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
+#       firmware is still publishing a corrupt table and the outcome is decided afresh by
+#       what the garbage reads on every boot. That is luck, not immunity.
+#
+# This collector reports the LINES and the addresses; it never rules on memory. Classifying
+# is roles/node_health/tasks/assess.yml, which rules on the measured shortfall.
+#
+# It is boot-to-boot NON-DETERMINISTIC, which is precisely why it is detected on every
+# health run instead of diagnosed once.
+#
+# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
+# every node here). The unprivileged pass therefore records WHY it could not read, and
+# roles/node_health re-runs this collector with `--privileged` when escalation works.
+BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:")
+EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
+MEMATTR_FAILED_RE = re.compile(
+    r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
+# NOTE the `-?`. The kernel prints this version with %d — SIGNED — so a garbage field with
+# bit 31 set is logged NEGATIVE (k3s-prod-2, 2026-09-04: "version -1569980416"). Without the
+# minus sign this regex missed those lines entirely and the state fell through to "clean",
+# i.e. a node whose kernel had just REJECTED a corrupt table was reported as one whose
+# kernel "accepted it without complaint" — a false PASS on the exact bug this check exists
+# to find. Anchor future edits on that: the field is signed, and every diagnostic that
+# quotes it must survive a negative value.
+MEMATTR_VERSION_RE = re.compile(
+    r"Unexpected EFI Memory Attributes table version\s+(-?\d+)", re.I)
+MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
+# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
+# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
+# captured separately, above, and quoting it as evidence would bury the one line that
+# matters in 200 characters of unrelated table addresses.
+MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I)
+# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line
+# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict.
+MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available")
+MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved")
+
+
+def collect_klog():
+    out = {
+        "available": False,
+        "reason": None,
+        "boot_covered": None,
+        "memattr_state": None,
+        "memattr_published": None,
+        "memattr_table_address": None,
+        "memattr_version": None,
+        "memattr_lines": [],
+        "memory_line": None,
+    }
+    rc, stdout, stderr = run(["dmesg"], timeout=30)
+    if rc is None:
+        out["reason"] = "dmesg is not installed on this node"
+        return out
+    if rc != 0:
+        detail = (stderr or stdout).strip().splitlines()
+        out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % (
+            rc, os.geteuid(), detail[-1] if detail else "no output",
+            " — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG"
+            if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "")
+        return out
+
+    out["available"] = True
+    # Does the buffer still reach back to early boot? The ring buffer WRAPS, and every
+    # memattr line is printed in the first fraction of a second. If the boot header is
+    # gone then so may be a memattr line that was printed — absence would prove nothing,
+    # and the check must SKIP rather than PASS. This is the difference between
+    # "unaffected" and "could not tell", and the role is not allowed to blur it.
+    out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout))
+
+    published = EFI_CONFIG_RE.search(stdout)
+    if published:
+        out["memattr_published"] = True
+        out["memattr_table_address"] = published.group(1)
+    elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M):
+        # UEFI booted and the config-table line is present, but it lists no MEMATTR=.
+        out["memattr_published"] = False
+    # else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None.
+
+    for line in stdout.splitlines():
+        if MEMATTR_LINE_RE.search(line):
+            out["memattr_lines"].append(line.strip())
+    joined = "\n".join(out["memattr_lines"])
+    if MEMATTR_CORRUPT_RE.search(joined):
+        # Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected.
+        out["memattr_state"] = "corrupt_rejected"
+    elif MEMATTR_FAILED_RE.search(joined):
+        out["memattr_state"] = "failed_to_map"
+        addr = MEMATTR_FAILED_RE.search(joined).group(1)
+        if addr:
+            out["memattr_table_address"] = addr
+    elif MEMATTR_VERSION_RE.search(joined):
+        out["memattr_state"] = "bad_version"
+        out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1))
+    else:
+        out["memattr_state"] = "clean"
+
+    mem_line = MEMORY_LINE_RE.search(stdout)
+    if mem_line:
+        full = [l for l in stdout.splitlines() if mem_line.group(0) in l]
+        reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None
+        out["memory_line"] = {
+            "available_kb": int(mem_line.group(1)),
+            "total_kb": int(mem_line.group(2)),
+            "reserved_kb": int(reserved.group(1)) if reserved else None,
+            "raw": full[0].strip() if full else None,
+        }
+    return out
+
+
 # --- Thermals -----------------------------------------------------------------------
 def collect_thermal():
     zones = []
@@ -708,15 +1011,34 @@ def collect_os():
 def collect_tools():
     tools = {}
     for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
-        tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+        tools[tool] = which(tool)
     return tools
 
 
 def main():
+    # --privileged: re-collect ONLY the fields that genuinely need root, for the second
+    # pass roles/node_health makes when (and only when) escalation is proven to work
+    # (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place —
+    # re-implementing the kernel-log grep in Jinja would be a second source of truth for
+    # the same fact, and the two would drift.
+    if "--privileged" in sys.argv[1:]:
+        doc = {
+            "schema": SCHEMA,
+            "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "collector_uid": os.geteuid(),
+            "klog": collect_klog(),
+            "memory": {"devices": collect_memory_devices()},
+        }
+        doc["unreadable"] = UNREADABLE
+        json.dump(doc, sys.stdout, indent=2, sort_keys=True)
+        sys.stdout.write("\n")
+        return
+
     doc = {
         "schema": SCHEMA,
         "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
         "collector_uid": os.geteuid(),
+        "klog": collect_klog(),
         "os": collect_os(),
         "dmi": collect_dmi(),
         "cpu": collect_cpu(),

infra/ansible/roles/node_health/tasks/assess.yml

@@ -387,6 +387,118 @@
                ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
                ~ avail_pct | string ~ '%)') }}
 
+# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
+# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
+# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
+# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
+# from the presence of a kernel-log line — and they duly contradicted each other on the same
+# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
+# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
+# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
+# measurement was right; the inference from the log line was wrong — the memattr block below
+# has the refined mechanism.
+#
+# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
+# place and both checks read this fact: neither derives a shortfall of its own, and the
+# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
+# disagree about whether memory was lost — there is only one verdict on that question, and
+# it is this one. Thresholds and their measured baselines: defaults/main.yml.
+- name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
+  ansible.builtin.set_fact:
+    node_health_mem_shortfall: "{{ {
+      'zones_pct': z_pct, 'zones_status': z_status,
+      'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
+      'status': worst_status } }}"
+  vars:
+    ansible_become: false
+    th: "{{ node_health_thresholds }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+
+    # 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
+    #    actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
+    #    tools, identical on metal and in a VM — so this half is available almost always.
+    z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
+    z_status: >-
+      {{ 'SKIP' if z_pct is not number
+         else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
+               else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
+
+    # 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
+    #    VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
+    #    never as a zero DIMM count. Independent of the kernel's memory map entirely.
+    inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
+    d_pct: >-
+      {{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
+         if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
+         else none }}
+    d_status: >-
+      {{ 'SKIP' if d_pct is not number
+         else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
+               else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
+
+    # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
+    # SKIPs must never add up to a PASS.
+    worst_status: >-
+      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
+         | default('SKIP', true) }}
+
+# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
+# firmware.efi_memattr names the one cause we have actually hit here; this one asks the
+# blunter question — "is this machine using the RAM it has?" — and answers it from two
+# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
+# losing memory for some other reason still shows up.
+- name: Assess — usable memory vs the memory this machine physically has
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'memory.physical_total',
+      'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
+      'status': phys_status,
+      'detail': phys_detail }] }}"
+  vars:
+    ansible_become: false
+    ms: "{{ node_health_mem_shortfall }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+    # Straight from the shared measurement — this check re-derives nothing.
+    z_pct: "{{ ms.zones_pct }}"
+    inst_kb: "{{ ms.installed_kb }}"
+    d_pct: "{{ ms.dimm_pct }}"
+    phys_status: "{{ ms.status }}"
+    # What the kernel log says, IF it was read — quoted here only as attribution for the
+    # measured shortfall. The verdict above is the shared measurement and never depends on
+    # the log; firmware.efi_memattr reads that same verdict, so the two always agree.
+    phys_cause: >-
+      {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
+           firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
+           if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
+         else ('. The kernel log was NOT read here, so this shortfall is unattributed —
+                see check firmware.efi_memattr')
+           if not (node_health_facts.klog.available | bool)
+         else '' }}
+    phys_detail: >-
+      {{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
+           ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
+           ~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
+             was never handed to the allocator (/proc/zoneinfo)')
+          if z_pct is number
+          else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
+         ~ '. '
+         ~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
+             ~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
+             ~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
+             ~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
+             ~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
+            if d_pct is number
+            else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
+         ~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
+              reservations and (on this platform) 64 MiB of graphics stolen memory.
+              A shortfall of this size is memory the machine HAS and cannot use')
+            if phys_status in ['WARN', 'FAIL'] else '')
+         ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
+
 - name: Assess — swap (kubelet requires it off)
   ansible.builtin.set_fact:
     node_health_checks: "{{ node_health_checks + [{
@@ -674,3 +786,245 @@
                ~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
                ~ (' — serial numbers not read (root-only DMI fields)'
                   if d.product_serial is none else '')) }}
+
+# --- EFI Memory Attributes Table ------------------------------------------------------
+# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
+# what follows is only what this check needs to be read correctly.
+#
+# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
+# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
+# are involved, and conflating them is what made this check wrong until 2026-09-03:
+#
+#   efi_memattr_init()              tbl_size = sizeof(*tbl) + num_entries * desc_size, then
+#                                   memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
+#                                   is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
+#                                   NEGATIVE: a garbage product with bit 31 set sign-extends
+#                                   through phys_addr_t and memblock_cap_size() clamps it to
+#                                   PHYS_ADDR_MAX — everything above the table is reserved.
+#                                   A garbage-but-POSITIVE tbl_size reserves a modest, wrong
+#                                   amount and costs nothing measurable.
+#   efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
+#                                   THIS is what prints "Failed to map". It fails for ANY
+#                                   sufficiently wrong tbl_size — negative OR merely large.
+#
+# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
+# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
+# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
+# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
+# is the counter-example that falsified the old inference.
+#
+# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
+#
+#   "Failed to map ... @ 0x..."   the bug FIRED: corrupt table, trusted by the kernel.
+#       FAIL when the shared measurement (node_health_mem_shortfall) says memory is
+#       catastrophically missing — memory is lost NOW.
+#       WARN otherwise — this boot drew a survivable garbage value; the next may not.
+#   "Unexpected ... version <N>"  version read > 2, so the kernel bailed out before
+#       reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
+#       safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
+#       gates and no draw can reach the fatal arithmetic any more. Same line, opposite
+#       meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
+#   FW_BUG "Corrupted EFI Memory Attributes Table detected!"  the >= 6.13 THIRD gate
+#       (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
+#       must not be treated as such: it is only reachable on a boot whose garbage leaves the
+#       version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
+#       line instead. All three prod nodes did exactly that on 2026-09-04.
+#
+# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
+# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
+# measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
+# firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
+# the draw still happens but every outcome is a rejection, so the same lines mean the
+# validation is working.
+#
+# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
+# so it cannot be diagnosed once. It has to be measured on every health run.
+#
+# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
+# node_health_mem_shortfall, computed once further up this file; this check does not measure
+# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
+# one verdict on any host and both checks quote it.
+#
+# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
+# ONLY if the kernel log was actually read AND still reaches back to early boot — every
+# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
+# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
+# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
+# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
+# log at all, so a node whose log is unreadable is still not unmeasured.
+- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'firmware.efi_memattr',
+      'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
+      'status': memattr_status,
+      'detail': memattr_detail }] }}"
+  vars:
+    ansible_become: false
+    k: "{{ node_health_facts.klog }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    # THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
+    # near memory.physical_total. This check contributes the kernel-log facts only.
+    ms: "{{ node_health_mem_shortfall }}"
+    ma_state: "{{ k.memattr_state | default(none) }}"
+    # DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
+    # carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
+    # THREE checks before it computes tbl_size at all:
+    #     tbl->version > 2                       -> "Unexpected ... table version %d"
+    #     desc_size implausible                  -> "Unexpected ... descriptor size %u ..."
+    #     num_entries > SZ_64K                   -> FW_BUG "Corrupted EFI Memory Attributes
+    #                                               Table detected! ..."
+    # ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
+    # given boot produces is decided by WHICH field that boot's garbage lands in — it is not
+    # a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
+    # version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
+    # memblock_reserve(). That is the whole difference, and it is why the verdict for
+    # `bad_version` below is opposite on the two kernels: same log line, completely different
+    # meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
+    # something the `version` test can parse.
+    memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
+    memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
+    # `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
+    # COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
+    # this check and memory.physical_total can no longer disagree about lost memory.
+    # `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
+    # boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
+    # WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
+    # forever would make the check unable to ever show the fix landing, which is exactly what
+    # it is for.
+    memattr_status: >-
+      {{ 'SKIP' if not (k.available | bool)
+         else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
+               else ('WARN' if ma_state == 'failed_to_map'
+                     else ('PASS' if ma_state == 'corrupt_rejected'
+                           else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
+                                 if ma_state == 'bad_version'
+                                 else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
+    # One remediation, quoted by both findings. NOTE what it must NOT say: there is no
+    # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
+    # could take effect. Suggesting one would be a fix that quietly does nothing.
+    memattr_fix: >-
+      FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
+      upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
+      bogus", first released in v6.13 and absent from v6.12 and from the current
+      linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
+      There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
+      bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
+      unplugging USB only change where the table lands and may APPEAR to fix it — they are
+      a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+    # The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
+    # to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
+    # kernel log has no part in; the verdict word comes from ms.status, so it always agrees
+    # with memory.physical_total. When the measurement is missing it says so, and says why.
+    memattr_figures: >-
+      {{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+          ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
+          ~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
+            allocator (/proc/zoneinfo)')
+         if ms.zones_pct is number
+         else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
+                reports as installed (/proc/zoneinfo NOT read — '
+                ~ (z.reason | default('no reason recorded', true)) ~ ')')
+               if ms.dimm_pct is number else '') }}
+    memattr_corroboration: >-
+      {{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
+          ~ ms.status ~ '): ' ~ memattr_figures
+          ~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
+             else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
+                   else ' — the normal few percent (kernel image, memmap, firmware
+                        reservations, graphics stolen memory). NO measurable memory was lost
+                        on this boot.')))
+         if (memattr_figures | trim | length) > 0
+         else (' Whether memory was lost on this boot could NOT be measured here: neither
+                present-vs-managed pages nor installed DIMM capacity was read — see check
+                memory.physical_total for the reasons.') }}
+    # The kernel log lines themselves, verbatim. A verdict the operator cannot check
+    # against the evidence is not much better than a wrong one.
+    memattr_evidence: >-
+      {{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
+          if (k.memattr_lines | default([]) | length) > 0 else '')
+         ~ ((' At boot the kernel reported '
+             ~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+             ~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
+             ~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
+             ~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
+            if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
+            else '') }}
+    memattr_detail: >-
+      {%- if not (k.available | bool) -%}
+      the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
+      Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
+      lives only in dmesg.{{ memattr_corroboration }}
+      {%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
+      THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
+      line. The kernel failed to map the EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }}, which proves the
+      table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
+      from its contents; the measurement below shows that reservation was the catastrophic
+      kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
+      through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
+      table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'failed_to_map' -%}
+      THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
+      loss either. The kernel failed to map the EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }}: the table was
+      corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
+      WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
+      the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
+      with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
+      tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
+      measurable and still fails the later memremap() that printed this line. This boot drew a survivable
+      value; the draw is repeated on EVERY boot, at the same address — this platform has
+      been seen to lose 13.3 GiB and to keep its full memory at one identical table address
+      (infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
+      {{ memattr_fix }}
+      {%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
+      the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
+      kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
+      VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
+      boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
+      outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
+      efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
+      WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
+      Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
+      (num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
+      version field <= 2. Both messages are rejections, and which one a boot prints is the
+      lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
+      path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
+      PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
+      {{ memattr_corroboration }}
+      {%- elif ma_state == 'bad_version' -%}
+      EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
+      CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
+      version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
+      kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
+      EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
+      tbl_size the garbage produces — nothing measurable if that value is positive,
+      everything above the table if it is negative.{{ memattr_evidence }}
+      {{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'corrupt_rejected' -%}
+      the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
+      validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
+      cannot cost this node memory on this or any future boot; no
+      action.{{ memattr_evidence }}{{ memattr_corroboration }}
+      {%- elif not (k.boot_covered | bool) -%}
+      NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
+      back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
+      the ring buffer, which wraps). Every memattr diagnostic is printed within the first
+      fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
+      pass. Read /var/log/kern.log for this boot, or re-run after a
+      reboot.{{ memattr_corroboration }}
+      {%- elif k.memattr_published is sameas true -%}
+      the firmware publishes an EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }} and the kernel
+      accepted it without complaint, in a log that still reaches back to early boot. Not
+      affected on this boot.
+      {%- elif k.memattr_published is sameas false -%}
+      this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
+      lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
+      guest.
+      {%- else -%}
+      no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
+      so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
+      boot, so that absence is measured, not assumed.
+      {%- endif -%}

infra/ansible/roles/node_health/tasks/collect_privileged.yml

@@ -0,0 +1,62 @@
+---
+# The SECOND pass of the collector, run with escalation, for the two facts that are
+# genuinely root-gated:
+#
+#   * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an
+#     unprivileged read gets EPERM. This is where the EFI Memory Attributes Table
+#     diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference
+#     between a node that has lost 13.3 GiB and one that has not.
+#   * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed
+#     DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem.
+#
+# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a
+# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line-
+# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND
+# source of truth for the same fact, and the two would drift. `--privileged` makes the
+# collector emit exactly those subtrees and nothing else, so the parsing stays in one
+# place and is identical whichever pass produced it.
+#
+# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass
+# actually failed to read one of them (main.yml checks both). On a node where
+# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule
+# is that everything readable WITHOUT root is read without root.
+#
+# If escalation is unavailable the checks that consume these SKIP with the exact reason
+# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line"
+# and "could not read the kernel log" are different answers and the role says which.
+
+- name: Re-collect the root-only facts with escalation
+  ansible.builtin.script:
+    cmd: node_health_collect.py --privileged
+    executable: /usr/bin/python3
+  become: true
+  vars:
+    ansible_become: true
+  register: node_health_privileged_raw
+  changed_when: false   # strictly read-only; it can never change the node
+
+- name: Assert the privileged payload is the schema this role understands
+  ansible.builtin.assert:
+    that:
+      - (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1'
+    fail_msg: >-
+      Privileged collector pass returned an unexpected schema; role and collector are
+      out of step.
+    quiet: true
+  vars:
+    ansible_become: false
+
+# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive
+# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine
+# would REPLACE it and throw away the reasons the first pass recorded.
+- name: Merge the privileged facts over the unprivileged ones
+  ansible.builtin.set_fact:
+    node_health_facts: >-
+      {{ node_health_facts | combine({
+           'klog': priv.klog,
+           'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}),
+           'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])),
+         }) }}
+  vars:
+    ansible_become: false
+    priv: "{{ node_health_privileged_raw.stdout | from_json }}"

infra/ansible/roles/node_health/tasks/main.yml

@@ -4,9 +4,11 @@
 # ORDER OF OPERATIONS
 #   1. Collect everything readable WITHOUT root (one script, one round trip).
 #   2. Probe whether privilege escalation actually works on this host.
-#   3. Only if it does AND smartmontools is installed: read SMART.
-#   4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
-#   5. Render the human summary + write the JSON artifact.
+#   3. Only if it does, and only for what the first pass could NOT read: re-run the same
+#      collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
+#   4. Only if it does AND smartmontools is installed: read SMART.
+#   5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#   6. Render the human summary + write the JSON artifact.
 #
 # BECOME HANDLING — the subtle bit, read before editing.
 #   `ansible_become` set as an inventory/group var (which is exactly what
@@ -43,6 +45,16 @@
   ansible.builtin.include_tasks: probe_privilege.yml
   when: node_health_try_privileged | bool
 
+# Only when the unprivileged pass actually failed to read one of them. On a node with
+# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
+# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
+- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
+  ansible.builtin.include_tasks: collect_privileged.yml
+  when:
+    - node_health_root_available | bool
+    - (not (node_health_facts.klog.available | bool))
+      or (not (node_health_facts.memory.devices.available | bool))
+
 - name: Read SMART data (privileged)
   ansible.builtin.include_tasks: collect_smart.yml
   when:

infra/ansible/roles/node_health/templates/summary.txt.j2

@@ -12,6 +12,17 @@
 {% macro gib(n) -%}
 {{ (n | int / 1073741824) | round(1) }} GiB
 {%- endmacro %}
+{# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not
+   report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of
+   its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised
+   "object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it
+   as `n/a` (the same token this template already uses for unreported readings) rather than
+   defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The
+   collector already guards this — tasks/collect_smart.yml uses selectattr('thresh',
+   'defined') and `a.thresh | default(0)` — so only the template was exposed. #}
+{% macro thr(a) -%}
+{{ a.thresh | default('n/a') }}
+{%- endmacro %}
 ================================================================================
  NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
  collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
@@ -45,6 +56,15 @@
 {% endif %}
  Memory       {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
  ECC/EDAC     {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
+{% set mz = node_health_facts.memory.zones %}
+{% set md = node_health_facts.memory.devices %}
+ Present RAM  {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }}
+ DIMMs        {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }}
+{% for dimm in md.devices | default([]) if dimm.size_kb %}
+              {{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }}
+{% endfor %}
+{% set klog = node_health_facts.klog %}
+ EFI memattr  {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
  Load         {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
 
 --------------------------------------------------------------------------------
@@ -64,19 +84,19 @@
               lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }}  temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
               pending(197)={{ sm.pending_sectors | default('?') }}  offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }}  reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }}  crc(199)={{ sm.crc_errors | default('?') }}
 {% if sm.failing_now or sm.failed_past %}
-              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %}
+              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %}
 
 {% elif sm.thresholds_present %}
-              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %}
+              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %}
 
 {% endif %}
               vendor context (NOT used for the verdict — raw units are vendor-defined):
               reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }}  grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }}  erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }}  program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }}  unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
 {% if sm.realloc_attr %}
-              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged)
+              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged)
 {% endif %}
 {% for a in sm.attributes | default([]) %}
-              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
+              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
 
 {% endfor %}
 {% else %}

platform/kustomization.yaml

@@ -12,7 +12,15 @@ kind: Kustomization
 resources:
   # ---- Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress ----
   - kyverno            # admission enforcer FIRST — policies present before other components reconcile
-  - cilium             # CNI (adopts the running Ansible-installed release; parity verified)
+  #      cilium (the CNI) is DELIBERATELY NOT here: it now reconciles in its OWN Flux Kustomization
+  #      (clusters/base/cilium.yaml, dependsOn flux-system, wait:true), and `platform` dependsOn IT.
+  #      Not the headlamp/oauth2-proxy rationale (a leaf that must not gate) but its mirror image:
+  #      the CNI is the one component that SHOULD gate everything, and inside this shared tier its
+  #      health was indistinguishable from eleven unrelated components, the only emergency valve was
+  #      freezing the whole tier, and the tier's 10m/5m interval/timeout are wrong for a 3-node
+  #      DaemonSet rollout. Gating is preserved via the explicit `platform.dependsOn: [cilium]` edge.
+  #      NOTE: platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system in
+  #      this tree — that is why the move required a prune-disabled transition. See DECISIONS-LOG.
   - longhorn           # replicated block storage / CSI
   - external-snapshotter  # CSI snapshot CRDs + snapshot-controller (VolumeSnapshot API; CDI smart-clone)
   - cert-manager       # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token`

platform/longhorn/README.md

@@ -25,7 +25,7 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
 | Requirement | Summary | How this component serves it |
 |---|---|---|
 | **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
-| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
+| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the node's data disk at `${LONGHORN_DATA_PATH}` (a `cluster-vars` knob: dev `/var/lib/longhorn`, a dedicated disk; prod `/data/longhorn`, a subdirectory of the partition it shares with k3s's data dir). `replicaAutoBalance: best-effort` re-spreads on node return. |
 | **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
 
 ## Dev-scoped choices (change for prod)
@@ -48,9 +48,19 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
 
 ## Node prerequisites (already handled, informational)
 
-Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
-`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
-`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
+Each node presents ext4 storage at `${LONGHORN_DATA_PATH}` and installs `open-iscsi` +
+`nfs-common` + kernel modules — all via `roles/node_baseline` (`infra/cloud-init/` is a README
+now; provisioning became API-only). This component assumes that floor. The shape differs per
+environment, which is why the path is a knob:
+
+| | |
+|---|---|
+| dev | a **dedicated** disk (`/dev/vdb`, `LABEL=longhorn`) mounted at `/var/lib/longhorn` — the mount root *is* the dataPath |
+| prod | `/data/longhorn`, a **subdirectory** of the ~424 GB partition (`LABEL=k3s-data`) mounted at `/data`, whose sibling `/data/k3s` holds k3s's entire data dir (etcd + the containerd image store, moved off the 11.8 GiB `/var` — `infra/PROD-PROVISIONING.md` §2) |
+
+`[parity]`: the value for an environment MUST equal `node_baseline_longhorn_data_path` in that
+environment's Ansible group_vars. Ansible creates and mounts the directory; this component tells
+Longhorn to use it.
 
 ## Contract (consumed by everyone with state)
 

platform/longhorn/helmrelease.yaml

@@ -42,10 +42,22 @@ spec:
       defaultClass: false
 
     defaultSettings:
-      # --- data path (R-STATE-02) ---
-      # Dedicated Longhorn data disk (/dev/vdb → ext4) is mounted here by cloud-init on every node
-      # (infra/cloud-init/user-data.yaml.tftpl). Longhorn stores replicas on this disk only.
-      defaultDataPath: "/var/lib/longhorn"
+      # --- data path (R-STATE-02, R-ENV-02) ---
+      # The on-node directory every replica is written into. A cluster-vars KNOB, not a literal,
+      # because the two environments mount their data disk differently and a hardcoded path would
+      # put one of them on the OS filesystem:
+      #   dev  /var/lib/longhorn  — a dedicated ext4 disk (/dev/vdb) mounted there by
+      #                             roles/node_baseline. The mount root IS the dataPath: one disk,
+      #                             one tenant. 18 live volumes; this value must never change.
+      #   prod /data/longhorn     — a SUBDIRECTORY of the claimed ~424 GB partition mounted at
+      #                             /data, whose sibling /data/k3s holds k3s's entire data dir
+      #                             (etcd + the containerd image store, moved off the 11.8 GiB
+      #                             /var). Two tenants, so Longhorn gets a subdirectory rather
+      #                             than the mount root. infra/PROD-PROVISIONING.md §2.
+      # [parity] MUST equal roles/node_baseline's node_baseline_longhorn_data_path for the same
+      # environment — Ansible creates and mounts the directory, Helm tells Longhorn to use it.
+      # Set in BOTH overlays (clusters/dev, clusters/prod); base default is the dev value.
+      defaultDataPath: "${LONGHORN_DATA_PATH}"
 
       # --- replication (R-STATE-02) ---
       # Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly

platform/longhorn/values.yaml

@@ -8,7 +8,9 @@
 #   defaultClass: false                              # we own StorageClasses (storageclasses.yaml)
 #
 # defaultSettings:
-#   defaultDataPath: /var/lib/longhorn               # dedicated data disk (infra/cloud-init)
+#   defaultDataPath: ${LONGHORN_DATA_PATH}           # cluster-vars knob: dev /var/lib/longhorn
+#                                                    # (dedicated /dev/vdb), prod /data/longhorn
+#                                                    # (a subdirectory of the shared data partition)
 #   defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT}   # overlay knob (=3 in dev)
 #   replicaSoftAntiAffinity: false                   # HARD anti-affinity: one replica per node
 #   replicaZoneSoftAntiAffinity: true                # single zone (3 mini-PCs, no failure zones)