docs(prod): node 1's DIMMs are not a matched pair

Surfaced by the new node_health DIMM reporting, unrelated to the memattr bug and NOT a fault:
DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667).
Both 16 GB so capacity is right, but different rank organisation and rated speed mean the pair
runs at the slower 2667.

Recorded in §0 where the symmetry decision gets made, because §2 requires the three nodes to end
up with identical layouts and the previous '2x16 GB Micron' note hid the mismatch.

Serves: R-ENV-03. Task: T-7.1.

DECISIONS-LOG.md

@@ -1694,3 +1694,67 @@ blast radius (2 of 3 agents down at once), not a missed detection.
   `${...}` that can silently drift from the real HelmRelease.
 - `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
   out while a legitimate Longhorn upgrade is still running.
+
+---
+
+## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
+
+All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
+
+### Landed and verified on the live cluster
+
+| Change | Verified |
+|---|---|
+| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
+| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
+| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
+| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
+| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
+
+### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
+
+Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
+`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
+
+```sh
+[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"   # LAST line of the hook
+```
+
+On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
+the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
+
+**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
+long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
+execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
+**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
+dead code is not a safe no-op — it is the first execution of never-tested code, and should be
+treated with the caution of new code, not of a one-character escape.
+
+Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
+on both paths, but only by accident of which statement happened to be last — made explicit there too
+so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
+Kustomizations Ready, both OIDC providers still registered.
+
+### STILL TO DO — strict substitution is committed but NOT ACTIVE
+
+`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
+`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
+ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
+controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
+standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
+~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
+
+**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
+With the gate on but the ordering edge absent, the same race would turn a transient empty render
+into a transient hard Kustomization failure.
+
+### Prod consequences recorded this session
+
+- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
+  soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
+  first" safety net that the last two days have leaned on heavily, twice.
+- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
+  currently points at Gitea running on the dev cluster.
+- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
+  with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
+  Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.

apps/wave-0/gitea.yaml

@@ -128,7 +128,7 @@ spec:
               - /bin/sh
               - -c
               - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
-                \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n"
+                \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:

apps/wave-1/nextcloud.yaml

@@ -127,7 +127,7 @@ spec:
               - -c
               - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
                 [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
-                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
+                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:

catalog/gitea.yaml

@@ -162,6 +162,12 @@ lifecycle:
               gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
             fi
           fi
+          # A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
+          # it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
+          # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
+          # the moment its OIDC guard was fixed and the block actually ran for the first time.
+          exit 0
 exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/nextcloud.yaml

@@ -203,7 +203,17 @@ lifecycle:
             [ "$missing" = 0 ] && break
             p=$((p + 1)); sleep 2
           done
-          [ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          if [ "$p" -ge 15 ]; then
+            nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          fi
+          # A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
+          # the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
+          # SUCCESS path the test returned 1 and became the script's exit status. That was latent for
+          # as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
+          # exposed it immediately. Every failure path here logs loudly and writes a marker file and
+          # deliberately does NOT crash the pod — the break-glass local admin must stay usable.
+          exit 0
 exposure: public                # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
 storage:

clusters/base/cilium.yaml

@@ -0,0 +1,60 @@
+---
+# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
+# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
+#
+# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
+#
+# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
+# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
+# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
+# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
+#
+#   * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
+#   * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
+#     tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
+#     `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
+#     the valve the 2026-09-01 substitution incident wanted and did not have.
+#   * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
+#     `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
+#
+# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
+# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
+# expressed as an explicit edge instead of an accident of co-tenancy.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: cilium
+  namespace: flux-system
+spec:
+  # A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
+  # kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
+  interval: 1h
+  retryInterval: 2m
+  # 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
+  # rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
+  timeout: 10m
+  prune: true
+  # TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
+  wait: true
+  # Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
+  # by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
+  # is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
+  # revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
+  # `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/cilium
+  # MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
+  # kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
+  # this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
+  # renders empty). That is the intended fail-closed behaviour — but it means the stanza is
+  # load-bearing for the CNI coming up at all. Do not remove it.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
+  # oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).

clusters/base/kustomization.yaml

@@ -5,6 +5,13 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - flux-system.yaml     # GitRepository (reconciliation source)
+  # Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
+  # of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
+  # Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
+  # `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
+  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
+  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
+  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).

clusters/base/platform.yaml

@@ -11,7 +11,18 @@ spec:
   interval: 10m
   retryInterval: 2m
   timeout: 5m
-  prune: true          # drift correction: delete resources removed from git (R-GITOPS-03)
+  # Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
+  # on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
+  # live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
+  # `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
+  # any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
+  #
+  # LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
+  # Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
+  # made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
+  # Cilium. Before moving ANY platform/* component in future, check whether it is the sole
+  # declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
+  prune: true
   wait: true           # block `apps` until platform is healthy
   # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
   #
@@ -40,6 +51,13 @@ spec:
   # health-gates its own children (see clusters/dev/sync.yaml).
   dependsOn:
     - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+    # THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
+    # (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
+    # suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
+    # nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
+    # is healthy. It is also revision-gated, which is what makes the prune-safety argument above
+    # deterministic rather than a race.
+    - name: cilium
   sourceRef:
     kind: GitRepository
     name: flux-system

conformance/.gitignore

@@ -8,3 +8,22 @@ _artifacts/
 
 # Python bytecode from lib/ (status.py imports coverage.py); never committed.
 __pycache__/
+
+# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
+# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
+# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
+# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
+# fact, so it must survive in git.
+#
+# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
+# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
+# the `conformance/runs/` directory itself, so re-including the files here works (git
+# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
+# live under _artifacts/).
+#
+# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
+# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
+# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
+!runs/
+!runs/**
+!runs/**/*.junit.xml

conformance/README.md

@@ -22,6 +22,7 @@ behavioral / L3 fault-injection** (this harness).
 ./run.sh --lint                       # structural-validate test defs (no cluster)
 ./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
 ./run.sh --require-all                # phase-exit gate: every requirement must be covered
+./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
 ```
 
 **Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -105,13 +106,89 @@ lib/
   common.sh                  tagging convention + test discovery/selection helpers
   coverage.py                requirement-coverage generator (stdlib only)
   status.py                  requirement-STATUS generator (stdlib only; --selftest)
+  record_run.py              run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
   emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
+runs/
+  ledger.jsonl               TRACKED append-only proof-of-pass ledger (one row per
+                             testcase per run) — the only source status.py trusts
+  <UTC>-<short-sha>/         TRACKED per-run evidence: run.log + report.junit.xml
 tests/
   l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
 incluster/
   chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)
 ```
 
+## Proof of pass — the run ledger (`runs/`)
+
+`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
+proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
+root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
+only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
+asked for — **a committed, machine-readable run ledger**.
+
+```bash
+./run.sh --record                     # runs normally, and captures the evidence
+# -> conformance/runs/<UTC>-<short-sha>/run.log            (tee'd, carries the sentinels)
+# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml   (that run's JUnit)
+# -> conformance/runs/ledger.jsonl                         (one appended row per testcase)
+python3 lib/record_run.py --run-dir runs/<id> [--dry-run]  # or record an existing run dir
+python3 lib/record_run.py --selftest                       # deterministic, no cluster
+```
+
+**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
+that silently records itself is how a bad run becomes a permanent false proof. **Commit
+the run dir and the ledger** — an uncommitted ledger proves nothing.
+
+### Why a committed JUnit alone would be WORSE than nothing
+
+The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
+`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
+and `exit 0`, and their `check:` accepts either outcome —
+
+```yaml
+(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+ || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+```
+
+So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
+A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
+R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
+
+**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
+`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
+`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
+verdict token), so the table below is never hand-maintained:
+
+| Test | pass token (proof) | skip token (never proof) |
+|---|---|---|
+| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
+| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
+| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
+| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
+| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
+
+A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
+token present in that test's log lines, and **no** skip token anywhere in them. A skip
+token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
+`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
+non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
+skips as JUnit `<skipped/>`, never as a green pass.
+
+### Staleness — a proof is a claim about a tree, not a badge
+
+`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
+last commit touching that requirement's implementing manifests or its verifying test**.
+Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
+dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
+it never counts. Re-run with `--record` to refresh.
+
+A requirement reads `proven` only when **every** verifying test does; one skipped, failed
+or unrecorded test drags the whole requirement down. With an empty ledger `--status`
+behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
+
 ## How CI invokes it
 CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
 

conformance/lib/record_run.py

@@ -0,0 +1,949 @@
+#!/usr/bin/env python3
+"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
+
+WHY THIS EXISTS
+---------------
+`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
+fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
+`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
+`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
+that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
+and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
+tracked (see conformance/.gitignore for the negations that let it survive).
+
+THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
+------------------------------------------------------------------
+The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
+StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
+`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
+injection/dr-restore:
+
+    (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+     || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+
+So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
+(verified against the committed reports). Therefore:
+
+  *** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
+
+AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
+------------------------------------------------------
+Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
+it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
+the log of every run of that test. A section-blind grep therefore reads a skip token out
+of the source of every skip-gated test and can never record `proven` for any of them: the
+ledger fails closed so completely that it records nothing and proves nothing. Tokens are
+consequently read from SECTIONS (see classify_sections): a pass token only from
+`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
+pass — within genuine output, which is the only place either was ever evidence.
+
+That is the fake-green this project refuses, and it would be worse than no ledger at all
+because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
+PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
+files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
+unless the log carries that test's own pass token and carries NO skip token.
+
+HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
+--------------------------------------------------------
+Hard-coding a token table would rot the moment a test is edited. Instead every token is
+parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
+`check:` block is a declared sentinel. Classification is by the project's naming convention
+(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
+— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
+and this tool follows automatically; delete them and the test simply becomes unprovable
+rather than silently proven.
+
+  node-loss   pass NODE_LOSS_VOLUME_AVAILABLE_OK              skip NODE_LOSS_SKIPPED_OK
+  dr-restore  pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK     skip DR_RESTORE_SKIPPED_OK
+              (+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
+
+WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
+-----------------------------------------------------------
+The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
+dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
+any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
+pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
+can express "skipped" as "passed", and only they require a sentinel — see
+SENTINEL_REQUIRED_LAYERS.
+
+LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
+  ts_utc            run-dir timestamp, else the JUnit's mtime      (when)
+  git_sha           `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
+                    modified — a proof captured from a dirty tree is NOT honoured by
+                    status.py, because what ran is not what is committed  (what)
+  env               --env / CONF_ENV (cluster overlay the run targeted)
+  chainsaw_version  CHAINSAW_VERSION from versions.env (the pin IS the truth)
+  kube_version      `kubectl version` server gitVersion, else "unknown"
+  layer             L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
+  test              testcase name (== the test dir basename, asserted by convention)
+  requirements[]    the `conformance.home.k3s/R-*` labels on the test — the SAME
+                    authoritative source coverage.py uses, not the JUnit classname
+  junit_result      passed | failed | error | skipped
+  sentinel          the DECIDING token grepped from run.log, or null
+  proof             proven | skipped | failed | junit-skipped
+                    | unknown-no-sentinel-declared | unknown-sentinel-absent
+  run_id            the runs/<dir> name (provenance: which log backs this row)
+  note              one line saying WHY this row landed on that proof state
+
+`proven` is emitted on exactly one path (see decide()); every other path is non-green.
+
+RECORDING IS EXPLICIT, NEVER IMPLICIT
+-------------------------------------
+Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
+how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
+ledger row at all.
+
+EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
+  0  rows appended.
+  1  a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
+  3  the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
+     NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
+
+Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
+status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+_LIB = Path(__file__).resolve().parent
+if str(_LIB) not in sys.path:
+    sys.path.insert(0, str(_LIB))
+# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
+# tree, and the lint stage scans untracked files. Same guard as status.py.
+sys.dont_write_bytecode = True
+import coverage as cov  # noqa: E402  — reuse the registry/test parsers, don't fork them
+
+CONF_ROOT = _LIB.parent
+REPO_ROOT = CONF_ROOT.parent
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
+# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
+# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
+# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
+# by JUnit alone.
+SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
+
+# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
+# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
+# harmless for its own report, but here a mis-read layer decides whether a sentinel is
+# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
+_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
+                     ("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
+                     ("/meta/", "META"))
+
+
+def layer_of(path: Path) -> str:
+    rel = str(path)
+    for frag, layer in _LAYER_BY_SUBTREE:
+        if frag in rel:
+            return layer
+    return ""
+
+# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
+# Both quote styles are accepted; the repo uses single quotes throughout.
+_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
+
+# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
+# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
+# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
+# mis-classifying a SKIP token as a pass would manufacture one.
+_SKIP_TOKEN_RE = re.compile(r"_SKIP")
+
+# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
+# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
+#     l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
+# Field 2 is the test name; it is how output is attributed back to a testcase. The
+# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
+_LOG_HEADER_RE = re.compile(
+    r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
+    r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
+
+# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
+# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
+
+
+def _strip_noise(line: str) -> str:
+    return _ANSI_RE.sub("", line).replace("\b", "")
+
+
+# ===================================================================== output sections
+# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
+# -----------------------------------------------------------------------------
+# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
+# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
+# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
+# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
+# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
+# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
+# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
+# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
+# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
+#
+# So tokens are attributed to SECTIONS, never to raw lines:
+#     l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |     <- header, no section
+#         === COMMAND                                            <- marker (indent 8)
+#         /usr/bin/sh -c set -euo pipefail                       <- source, indent >= 8
+#           echo NODE_LOSS_SKIPPED_OK                            <- SOURCE, not output
+#     l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |    <- indent 4 => section ends
+#         === STDOUT                                             <- marker
+#         NODE_LOSS_VOLUME_AVAILABLE_OK                          <- the REAL emission
+#
+# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
+# (`=== RUN   chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
+# after the verb, so they match neither the indent nor the end-of-line anchor; they are
+# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
+# section that would swallow the whole log.
+_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
+_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
+
+# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
+# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
+# never echoes script source, so there is nothing to be confused by and the legacy
+# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
+# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
+# "" and are evidence for nothing.
+UNSECTIONED = "*"
+
+# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
+# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
+# never manufacture a proof.
+PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
+
+# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
+# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
+# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
+# one. Unknown section names therefore fail closed in both directions.
+SOURCE_SECTIONS = frozenset({"COMMAND"})
+
+
+def classify_sections(log_text: str) -> list[tuple[str, str]]:
+    """[(section, line)] for every line of a run log.
+
+    `section` is the enclosing `=== NAME` block, "" for a line inside no block, or
+    UNSECTIONED for every line when the log declares no sections at all. A block runs from
+    its marker until the first non-blank line indented LESS than the marker (blank lines do
+    not end it) or until the next marker; the marker line itself is content of nothing.
+    """
+    rows: list[tuple[str, str]] = []
+    cur, cur_indent, sectioned = "", 0, False
+    for raw in log_text.splitlines():
+        line = _strip_noise(raw)
+        m = _SECTION_MARKER_RE.match(line)
+        if m and m.group(2) not in _GO_TEST_MARKERS:
+            sectioned = True
+            cur, cur_indent = m.group(2), len(m.group(1))
+            rows.append(("", line))
+            continue
+        if cur:
+            if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
+                rows.append((cur, line))
+                continue
+            cur = ""
+        rows.append(("", line))
+    if not sectioned:
+        return [(UNSECTIONED, line) for _, line in rows]
+    return rows
+
+
+def scan_texts(lines) -> tuple[str, str]:
+    """(text a PASS token may be read from, text a SKIP token may be read from).
+
+    Accepts either [(section, line)] pairs from classify_sections() or a plain list of
+    strings, which is treated as an unsectioned log (legacy whole-log scanning).
+    """
+    pass_lines, skip_lines = [], []
+    for item in lines:
+        section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
+        if section in PASS_SECTIONS:
+            pass_lines.append(line)
+        if section and section not in SOURCE_SECTIONS:
+            skip_lines.append(line)
+    return "\n".join(pass_lines), "\n".join(skip_lines)
+
+_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
+
+
+# ========================================================================= sentinels
+def parse_sentinels(test_yaml: Path) -> dict:
+    """Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
+
+    Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
+    the LAST non-skip token in file order — the test's verdict token, i.e. the one its
+    final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
+    A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
+    unprovable rather than proven; see decide().
+    """
+    try:
+        text = test_yaml.read_text()
+    except (OSError, UnicodeDecodeError):
+        return {"pass": [], "skip": [], "terminal_pass": None}
+    toks = _SENTINEL_RE.findall(text)
+    passes, skips = [], []
+    for t in toks:
+        (skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
+    # De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
+    def uniq(xs: list[str]) -> list[str]:
+        return list(dict.fromkeys(xs))
+    passes, skips = uniq(passes), uniq(skips)
+    return {"pass": passes, "skip": skips,
+            "terminal_pass": passes[-1] if passes else None}
+
+
+def sentinel_map(tests_dir: Path) -> dict[str, dict]:
+    """{test-name: sentinels} for every test in the tree.
+
+    Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
+    name=...>`. This repo keeps that equal to the test's directory basename; the mapping
+    is by name, so it stays correct if that ever drifts.
+    """
+    out: dict[str, dict] = {}
+    for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
+        meta = cov.parse_test(f)
+        name = meta["name"] or f.parent.name
+        s = parse_sentinels(f)
+        s["dir"] = str(f.parent)
+        s["layer"] = layer_of(f) or meta["layer"]
+        s["requirements"] = meta["labels"]
+        out[name] = s
+    return out
+
+
+# =============================================================================== log
+def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
+                                                            bool]:
+    """Split a run log into per-testcase buckets of (section, line) pairs.
+
+    Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
+    script's sections underneath that header, so the test name is carried by a state
+    machine: a header naming a KNOWN test switches the current bucket, every following line
+    lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
+    chatter (and any other pipe-shaped line) from opening a bogus bucket.
+
+    Section labels come from classify_sections() and travel WITH the lines, so a bucket
+    still knows which of its lines were script source and which were real output.
+
+    Returns (buckets, attributed) where `attributed` is False when the log contained no
+    recognisable header at all — the signal for the degraded global scan in decide().
+    """
+    buckets: dict[str, list[tuple[str, str]]] = {}
+    cur: str | None = None
+    attributed = False
+    for section, line in classify_sections(log_text):
+        m = _LOG_HEADER_RE.match(line)
+        if m and m.group(1) in known:
+            cur = m.group(1)
+            attributed = True
+        if cur is not None:
+            buckets.setdefault(cur, []).append((section, line))
+    return buckets, attributed
+
+
+def unique_tokens(smap: dict[str, dict]) -> set[str]:
+    """Tokens claimed by EXACTLY ONE test in the tree.
+
+    Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
+    tests, so in an unattributable log it can never be evidence for either; but
+    `NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
+    """
+    owners: dict[str, set[str]] = {}
+    for name, s in smap.items():
+        for t in s["pass"] + s["skip"]:
+            owners.setdefault(t, set()).add(name)
+    return {t for t, o in owners.items() if len(o) == 1}
+
+
+# ============================================================================ decide
+def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
+           scope: str) -> tuple[str | None, str, str]:
+    """The ONE place a testcase becomes `proven` — or, far more often, does not.
+
+    Returns (sentinel, proof, note).
+
+    Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
+      1. JUnit failure/error            -> failed        (a red run is recorded as red)
+      2. JUnit <skipped/>               -> junit-skipped  (never green; ci/lib.sh rule)
+      3. ANY declared skip token seen   -> skipped        (BEATS a green JUnit and beats
+           IN REAL OUTPUT                                  the pass token: if any gated
+                                                           step self-skipped, the run did
+                                                           not prove the requirement)
+      4. no sentinel declared           -> proven only for L0/L1, whose skips are already
+                                           JUnit skips; unknown-no-sentinel-declared for
+                                           the Chainsaw layers
+      5. EVERY declared pass token seen -> proven
+      6. anything else                  -> unknown-sentinel-absent
+    Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
+    asserts `pass || skip`, so a missing pass token means a step did not reach its proven
+    branch, whatever the JUnit says.
+
+    Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
+    only in `=== STDOUT`, a skip token counts in any section that is not script source.
+    That narrows WHERE a token is read from and changes nothing about what it means — skip
+    still outranks pass, and it outranks it within genuine output, which is the only place
+    either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
+    strings (an unsectioned log; see scan_texts / UNSECTIONED).
+    """
+    if junit_result in ("failed", "error"):
+        return (None, "failed", f"JUnit recorded {junit_result}")
+    if junit_result == "skipped":
+        return (None, "junit-skipped",
+                "JUnit <skipped/> — the test did not execute; a skip is never green")
+
+    pass_text, skip_text = scan_texts(lines)
+    seen_skip = [t for t in sent["skip"] if t in skip_text]
+    if seen_skip:
+        return (seen_skip[0], "skipped",
+                f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
+                f"exited 0 without exercising the behaviour, so this is NOT proof")
+
+    if not sent["pass"] and not sent["skip"]:
+        if layer not in SENTINEL_EXEMPT_LAYERS:
+            return (None, "unknown-no-sentinel-declared",
+                    f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
+                    f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
+                    f"refusing to call it proven")
+        return (None, "proven",
+                f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
+                f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
+
+    missing = [t for t in sent["pass"] if t not in pass_text]
+    if not missing:
+        return (sent["terminal_pass"], "proven",
+                f"run log OUTPUT carries every declared pass token "
+                f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
+    return (None, "unknown-sentinel-absent",
+            f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
+            f"[{scope}] — refusing to infer proof from the JUnit alone")
+
+
+# ============================================================================= junit
+def junit_cases(path: Path) -> list[dict]:
+    """[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
+    root = ET.parse(path).getroot()
+    cases: list[dict] = []
+    for c in root.iter("testcase"):
+        if c.find("failure") is not None:
+            result = "failed"
+        elif c.find("error") is not None:
+            result = "error"
+        elif c.find("skipped") is not None:
+            result = "skipped"
+        else:
+            result = "passed"
+        props = {p.get("name"): (p.get("value") or "")
+                 for p in c.iter("property") if p.get("name")}
+        reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
+        layer = (props.get("layer") or "").upper()
+        if not layer:
+            layer = (c.get("classname") or "").split(":", 1)[0].upper()
+        cases.append({"name": c.get("name") or "?", "layer": layer,
+                      "requirements": reqs, "result": result})
+    return cases
+
+
+# =============================================================================== git
+def git_sha(repo: Path) -> str:
+    """HEAD, suffixed `-dirty` when TRACKED files differ from it.
+
+    Untracked files are deliberately ignored, matching status.py's `tracked_files()`
+    posture (untracked scratch is not repo state) — and because a `--record` run creates
+    untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
+    A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
+    the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
+    """
+    try:
+        sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
+                             capture_output=True, text=True, check=True).stdout.strip()
+    except (subprocess.CalledProcessError, OSError):
+        return "unknown"
+    dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--"],
+                           capture_output=True).returncode != 0
+    return f"{sha}-dirty" if dirty else sha
+
+
+def pinned_chainsaw_version(conf_root: Path) -> str:
+    """CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
+    f = conf_root / "versions.env"
+    if f.is_file():
+        m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
+        if m:
+            return m.group(1)
+    return "unknown"
+
+
+def server_kube_version() -> str:
+    """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
+    for args in (["kubectl", "version", "-o", "json"],
+                 ["kubectl", "version", "--output=json"]):
+        try:
+            p = subprocess.run(args, capture_output=True, text=True, timeout=15)
+        except (OSError, subprocess.TimeoutExpired):
+            continue
+        if p.returncode != 0 or not p.stdout.strip():
+            continue
+        try:
+            v = json.loads(p.stdout).get("serverVersion") or {}
+        except json.JSONDecodeError:
+            continue
+        if v.get("gitVersion"):
+            return v["gitVersion"]
+    return "unknown"
+
+
+def run_timestamp(run_dir: Path, junit: Path) -> str:
+    """UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
+    m = _RUN_ID_TS_RE.match(run_dir.name)
+    if m:
+        t = m.group(1)
+        return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
+    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
+
+
+# ============================================================================ record
+def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
+               kube_version: str, sha: str) -> list[dict]:
+    junit = run_dir / "report.junit.xml"
+    log = run_dir / "run.log"
+    smap = sentinel_map(tests_dir)
+    log_text = log.read_text(errors="replace")
+    buckets, attributed = attribute_log(log_text, set(smap))
+    uniq = unique_tokens(smap)
+    all_pairs = classify_sections(log_text)
+    sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
+    section_note = ("tokens read from real-output sections only"
+                    if sectioned else "log declares no === sections; whole-log scan")
+    ts = run_timestamp(run_dir, junit)
+
+    rows: list[dict] = []
+    for case in junit_cases(junit):
+        name = case["name"]
+        sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
+                               "layer": case["layer"], "requirements": []})
+        layer = sent.get("layer") or case["layer"] or "?"
+        # Requirements come from the test's LABELS (coverage.py's authoritative source);
+        # the JUnit property is the fallback for a testcase with no matching test file.
+        reqs = sent.get("requirements") or case["requirements"]
+
+        if name in buckets:
+            lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
+        elif not attributed:
+            # Degraded mode: the log had no recognisable Chainsaw header (a redirected or
+            # reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
+            # exactly one test, so a token another test emitted can never prove this one.
+            lines = all_pairs
+            sent = dict(sent,
+                        **{"pass": [t for t in sent["pass"] if t in uniq],
+                           "skip": [t for t in sent["skip"] if t in uniq]})
+            sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
+            scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
+                     + section_note)
+        else:
+            lines, scope = [], "no log lines attributed to this testcase"
+
+        sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
+        rows.append({
+            "ts_utc": ts,
+            "git_sha": sha,
+            "env": env,
+            "chainsaw_version": chainsaw_version,
+            "kube_version": kube_version,
+            "layer": layer,
+            "test": name,
+            "requirements": reqs,
+            "junit_result": case["result"],
+            "sentinel": sentinel,
+            "proof": proof,
+            "run_id": run_dir.name,
+            "note": note,
+        })
+    return rows
+
+
+def append_rows(ledger: Path, rows: list[dict]) -> None:
+    """Append-only, one compact JSON object per line. Never rewrites existing rows."""
+    ledger.parent.mkdir(parents=True, exist_ok=True)
+    with ledger.open("a", encoding="utf-8") as fh:
+        for r in rows:
+            fh.write(json.dumps(r, sort_keys=True) + "\n")
+
+
+# ========================================================================== selftest
+def _selftest() -> int:
+    """Deterministic, no-cluster self-test of every decision rule (house pattern:
+    `status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
+
+    The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
+    assertion standing between this ledger and a permanent false proof of R-STATE-03.
+    """
+    import tempfile
+    ok = True
+
+    def check(label: str, cond: bool) -> None:
+        nonlocal ok
+        if not cond:
+            sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
+            ok = False
+
+    # --- token derivation, against the dr-restore/node-loss shape ---------------------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
+            "  name: dr-restore\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-03: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-03\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo OFF_CLUSTER_TARGET_OK'\n"
+            "        check:\n"
+            "          (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
+            "    - script:\n        content: 'echo x'\n        check:\n"
+            "          (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
+        s = parse_sentinels(tdir / "chainsaw-test.yaml")
+        check("pass tokens derived from the test file",
+              s["pass"] == ["OFF_CLUSTER_TARGET_OK",
+                            "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
+        check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
+              s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
+        check("terminal pass token is the LAST non-skip token",
+              s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
+
+        smap = sentinel_map(root / "tests")
+        check("sentinel_map keys on metadata.name and carries layer + labels",
+              smap["dr-restore"]["layer"] == "L3"
+              and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
+
+    # --- decide(): the SKIP-is-never-proven contract ----------------------------------
+    l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
+          "skip": ["DR_RESTORE_SKIPPED_OK"],
+          "terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
+
+    # *** THE CASE THIS WHOLE FILE EXISTS FOR ***
+    sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
+    check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
+          proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
+                          "DR_RESTORE_SKIPPED_OK"], "t")
+    check("a skip token BEATS a pass token in the same log (worst wins)",
+          proof == "skipped")
+
+    _, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
+    check("green JUnit with no sentinel in the log is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "passed", l3, [], "t")
+    check("green JUnit with NO log lines attributed is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "failed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("a failing JUnit outranks a pass token in the log", proof == "failed")
+
+    _, proof, _ = decide("L3", "skipped", l3, [], "t")
+    check("JUnit <skipped/> records junit-skipped, never green",
+          proof == "junit-skipped")
+
+    multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
+    _, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
+    check("every declared pass token must be present, not just the terminal one",
+          proof == "unknown-sentinel-absent")
+    _, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
+    check("all pass tokens present == proven", proof == "proven")
+
+    none = {"pass": [], "skip": [], "terminal_pass": None}
+    _, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
+    check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("L1", "passed", none, [], "t")
+    check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
+          proof == "proven")
+    _, proof, _ = decide("L0", "skipped", none, [], "t")
+    check("an L0 JUnit skip is still never green", proof == "junit-skipped")
+    _, proof, _ = decide("META", "passed", none, [], "t")
+    check("META (a Chainsaw layer) needs a sentinel like L2/L3",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("", "passed", none, [], "t")
+    check("an unknown/blank layer fails CLOSED — sentinel required",
+          proof == "unknown-no-sentinel-declared")
+
+    # --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
+    # Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
+    # `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
+    # inside `=== COMMAND` on every run of a skip-gated test.
+    nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
+          "skip": ["NODE_LOSS_SKIPPED_OK"],
+          "terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+    _CMD = ("    l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |\n"
+            "        === COMMAND\n"
+            "        /usr/bin/sh -c set -euo pipefail\n"
+            "        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
+            "          echo NODE_LOSS_SKIPPED_OK\n"
+            "\n"
+            "          exit 0\n"
+            "        fi\n"
+            "        echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
+
+    ran = classify_sections(_CMD + "        === STDOUT\n"
+                            "        ready nodes: 3\n"
+                            "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
+    sent, proof, _ = decide("L3", "passed", nl, ran, "t")
+    check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
+          proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+    selfskipped = classify_sections(_CMD + "        === STDOUT\n"
+                                    "        SKIP: 'longhorn' StorageClass absent\n"
+                                    "        NODE_LOSS_SKIPPED_OK\n")
+    sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
+    check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
+          proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
+
+    neither = classify_sections(_CMD + "        === STDOUT\n        ready nodes: 3\n")
+    _, proof, _ = decide("L3", "passed", nl, neither, "t")
+    check("both tokens in COMMAND and neither in STDOUT is NOT proven",
+          proof == "unknown-sentinel-absent")
+
+    both_out = classify_sections("        === STDOUT\n"
+                                 "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                                 "        NODE_LOSS_SKIPPED_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, both_out, "t")
+    check("skip still BEATS pass when both are in genuine output", proof == "skipped")
+
+    stderr_skip = classify_sections("        === STDERR\n        NODE_LOSS_SKIPPED_OK\n"
+                                    "        === STDOUT\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
+    check("a skip token in a non-source section (STDERR) still costs the proof",
+          proof == "skipped")
+
+    stderr_pass = classify_sections("        === STDERR\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
+    check("a pass token OUTSIDE stdout cannot prove anything",
+          proof == "unknown-sentinel-absent")
+
+    # --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
+    gorows = classify_sections("=== RUN   chainsaw\n=== PAUSE chainsaw\n"
+                               "=== CONT  chainsaw/node-loss\n"
+                               "        === COMMAND\n        echo NODE_LOSS_SKIPPED_OK\n"
+                               "    l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
+                               "--- PASS: chainsaw/node-loss (131.09s)\n")
+    secs = [sec for sec, _ in gorows]
+    check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
+          secs[:3] == ["", "", ""])
+    check("a section covers its indented body and ends at a less-indented line",
+          secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
+    check("a log with no === markers stays wholly scannable (UNSECTIONED)",
+          {sec for sec, _ in classify_sections("plain line\n  another\n")} == {UNSECTIONED})
+
+    # --- the preserved run itself, as a fixture ----------------------------------------
+    # runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
+    # (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
+    fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
+    if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
+        rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
+        row = next((r for r in rows if r["test"] == "node-loss"), None)
+        check("preserved real L3 run records node-loss/R-STATE-02 as proven",
+              row is not None and row["proof"] == "proven"
+              and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
+              and row["requirements"] == ["R-STATE-02"])
+        check("preserved real L3 run attributes (no DEGRADED fallback needed)",
+              row is not None and "DEGRADED" not in row["note"])
+        pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
+        secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
+        check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
+              secs == {"COMMAND"})
+        secs = {sec for sec, line in pairs
+                if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+        check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
+
+    # --- log attribution --------------------------------------------------------------
+    log = ("[run.sh] Coverage report first\n"
+           "    | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
+           "        NODE_LOSS_SKIPPED_OK\n"
+           "    | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
+           "        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
+    b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
+    check("log lines are attributed to the testcase whose header preceded them",
+          attributed
+          and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
+          and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
+    check("a pass token does NOT leak across testcase boundaries",
+          not any("DR_RESTORED" in l for _, l in b["node-loss"]))
+    check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
+          all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
+
+    # Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
+    # regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
+    real_hdr = ("    l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
+                "\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
+                "        === STDOUT\n        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
+    check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
+          attributed_real and "node-loss" in b2
+          and ("STDOUT", "        NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
+    _, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
+    check("an unattributable log is reported as such (degraded mode)", not attributed2)
+
+    smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
+             "b": {"pass": ["DNS_OK"], "skip": []}}
+    check("only tokens owned by exactly one test may be used in degraded mode",
+          unique_tokens(smap2) == {"A_ONLY_OK"})
+
+    # --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "node-loss"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "kind: Test\nmetadata:\n  name: node-loss\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-02: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-02\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo hi'\n        check:\n"
+            "          (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
+            " || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
+        rd = root / "runs" / "20260901T120000Z-abc1234"
+        rd.mkdir(parents=True)
+        (rd / "report.junit.xml").write_text(
+            '<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
+            'name="node-loss"><properties>'
+            '<property name="requirements" value="R-STATE-02"/>'
+            '<property name="layer" value="L3"/></properties></testcase>'
+            "</testsuite></testsuites>")
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      SKIP: 'longhorn' StorageClass absent\n"
+            "      NODE_LOSS_SKIPPED_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
+              len(rows) == 1 and rows[0]["junit_result"] == "passed"
+              and rows[0]["proof"] == "skipped"
+              and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
+        check("end-to-end: row carries ts from the run-dir name and the R-ID label",
+              rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
+              and rows[0]["requirements"] == ["R-STATE-02"])
+
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
+              rows[0]["proof"] == "proven"
+              and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+        ledger = root / "ledger.jsonl"
+        append_rows(ledger, rows)
+        append_rows(ledger, rows)
+        check("ledger is append-only JSONL (one parsable object per line)",
+              len(ledger.read_text().strip().splitlines()) == 2
+              and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
+
+    if not ok:
+        return 1
+    sys.stderr.write("record_run.py selftest: PASS\n")
+    return 0
+
+
+# ============================================================================== main
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--run-dir", type=Path,
+                    help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"append-only ledger (default {DEFAULT_LEDGER})")
+    ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
+    ap.add_argument("--repo", type=Path, default=REPO_ROOT)
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
+    ap.add_argument("--chainsaw-version", default=None,
+                    help="override the versions.env pin (recorded verbatim)")
+    ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
+                    help="override the `kubectl version` probe (recorded verbatim)")
+    ap.add_argument("--dry-run", action="store_true",
+                    help="print the rows that WOULD be appended; write nothing")
+    ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
+    args = ap.parse_args()
+
+    if args.selftest:
+        return _selftest()
+    if args.run_dir is None:
+        sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
+        return 1
+
+    run_dir = args.run_dir.resolve()
+    junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
+    # BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
+    # recording from it alone is exactly the false proof this tool exists to prevent.
+    missing = [p.name for p in (junit, log) if not p.is_file()]
+    if missing:
+        sys.stderr.write(
+            f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
+            f"The JUnit alone cannot distinguish a self-skip from a real pass "
+            f"(see module docstring). SKIPPED (not green).\n")
+        return 3
+
+    try:
+        rows = build_rows(
+            run_dir, args.tests, args.env,
+            args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
+            args.kube_version or server_kube_version(),
+            git_sha(args.repo))
+    except ET.ParseError as e:
+        sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
+        return 1
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
+        return 1
+
+    if not rows:
+        sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
+                         f"SKIPPED (not green).\n")
+        return 3
+
+    for r in rows:
+        print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
+              f"({','.join(r['requirements']) or 'untagged'}) "
+              f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
+    counts: dict[str, int] = {}
+    for r in rows:
+        counts[r["proof"]] = counts.get(r["proof"], 0) + 1
+    print(f"[record] {len(rows)} testcase(s): "
+          + ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
+
+    if args.dry_run:
+        print("[record] --dry-run: ledger NOT written.")
+        return 0
+    try:
+        append_rows(args.ledger, rows)
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
+        return 1
+    print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
+    print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/lib/status.py

@@ -15,26 +15,42 @@ This generator derives, from the REPOSITORY ALONE, three orthogonal axes per req
                        from a Flux Kustomization entrypoint (i.e. actually reconciled)?
   2. VERIFICATION    — is there a requirement-tagged test, at which layer, and is it an
                        EXECUTABLE test or a TODO scaffold?
-  3. PROOF           — has that test actually passed?  *** DELIBERATELY NOT DERIVED. ***
+  3. PROOF           — has that test actually passed?  Derived ONLY from the committed
+                       run ledger, and ONLY as far as that ledger's evidence reaches.
 
-WHAT THIS TOOL REFUSES TO DERIVE (and why)
+PROOF — WHAT CHANGED, AND WHAT DID NOT
-------------------------------------------
+--------------------------------------
-PROOF-OF-PASS IS NOT DERIVABLE FROM REPO STATE. Nothing tracked in git records a test
+This report used to say `not-derivable` for every requirement, because nothing tracked in
-result: `conformance/_artifacts/` and `results/` are git-ignored, and the only prose
+git recorded a test result and the only prose record of a green run lived in
-record of a green run lives in DECISIONS-LOG.md — hand-maintained, unstructured, and
+DECISIONS-LOG.md — hand-maintained, unstructured, exactly the source the operator ruled
-exactly the kind of source the operator has ruled out. The project's own culture is that
+out. The fix this docstring asked for now exists: `conformance/runs/ledger.jsonl`, an
-"merged" is never "done" (agents/definition-of-done.md, R-TEST-03), so this report will
+append-only ledger written by `lib/record_run.py` from a run's JUnit **and its run log**
-NOT infer, guess, or default a requirement to green. Every requirement's proof state is
+(see conformance/.gitignore for the negations that keep both in git).
-reported as `not-derivable`, and the headline says so. If an untracked JUnit artifact
-happens to be lying in the workspace it is surfaced ONLY as a quarantined advisory with
-its mtime — never counted, never mixed into a status.
 
-The honest fix is a committed, machine-readable run ledger; until that exists, the most
+THE REFUSAL POSTURE IS UNCHANGED. Evidence replaced "not derivable" only where evidence
-this report can say is "wired and testable", never "proven".
+exists; nowhere is optimism substituted for it:
+
+  * A ledger row reaches `proven` only if `record_run.py` saw that test's own PASS
+    sentinel token in the run log. The L2/L3 tests SELF-SKIP GREEN — Chainsaw records
+    `passed` on `DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries no
+    stdout — so a skip token maps to `skipped` and can never become proof.
+  * A proof is honoured only while it still describes the current code: if any of the
+    requirement's implementing manifests or verifying tests changed in a commit that is
+    NOT an ancestor of the run's `git_sha`, the row degrades to `stale-proof`, which is
+    not green. A run recorded from a dirty tree (`<sha>-dirty`) is never honoured.
+  * A requirement with no ledger row, or whose verifying tests did not ALL report proven,
+    stays `not-derivable` / `unknown` — never rounded up.
+  * The untracked `_artifacts/report.junit.xml` is STILL only a quarantined advisory with
+    its mtime. It is not the ledger, it is not evidence, and it feeds no status.
+
+With an empty or absent ledger this report behaves exactly as it did before: every
+requirement `not-derivable`, headline PROVEN 0, "the most this report can say is wired and
+testable, never proven".
 
 READS (all read-only)
   requirements/registry.yaml     the authoritative R-* mirror  (via lib/coverage.py)
   tests/**/chainsaw-test.yaml    requirement tags + layer + executable annotation
+  runs/ledger.jsonl              COMMITTED proof-of-pass rows (via lib/record_run.py)
   clusters/<env>/               Flux entrypoint -> the reconciliation graph
   `git ls-files`                 TRACKED files only; untracked scratch is not repo state
   ../home-network-k3s/docs/00-vision-and-scope.md   OPTIONAL registry<->spec parity check
@@ -92,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io"
 #   conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there.
 #   ci/tests          — gate unit-test fixtures embed example R-IDs as DATA.
 #   registry.yaml     — the requirement mirror itself, not an implementation of anything.
-IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/")
+# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the
+# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as
+# implementing evidence makes `last_change` move every time proof is recorded, so a row's
+# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is
+# committed — recording the proof destroys it. The ledger is the proof record; it must not
+# participate in the freshness computation about itself. (Found live 2026-09-01 on the first
+# PROVEN row.)
+IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/")
 IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)
 
 
@@ -343,6 +366,169 @@ def spec_requirement_ids(spec_root: Path) -> set[str] | None:
     return set(RID_RE.findall(doc.read_text()))
 
 
+# ========================================================================== ledger
+# PROOF axis. Everything here reads `conformance/runs/ledger.jsonl` (written ONLY by an
+# explicit `lib/record_run.py` / `run.sh --record` invocation) and downgrades relentlessly.
+# Nothing in this section may ever turn absent, ambiguous or aged evidence into green.
+
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# Severity ranking, WORST FIRST. A requirement's proof is the MINIMUM over its verifying
+# tests, so one failed/skipped/missing test drags the whole requirement down and a single
+# proven test can never carry the others. `stale-proof` ranks below `proven` on purpose:
+# it is a real past pass that no longer describes the current code, and it is NOT green.
+PROOF_ORDER = {
+    "failed": 0,              # a ledger row recorded a JUnit failure/error
+    "unknown": 1,             # ran, but the log could not evidence pass vs self-skip
+    "not-derivable": 2,       # no ledger row for this test at all (the pre-ledger default)
+    "junit-skipped": 3,       # JUnit <skipped/> — did not execute
+    "skipped": 4,             # SELF-SKIPPED green: exited 0 without exercising the behaviour
+    "stale-proof": 5,         # proven at a sha the implementation/test has since moved past
+    "proven": 6,              # the ONLY green state
+}
+# record_run.py's two "we cannot tell" states collapse to one reported state here; the
+# per-test detail in status.json keeps the distinction for whoever has to fix it.
+_PROOF_ALIASES = {
+    "unknown-no-sentinel-declared": "unknown",
+    "unknown-sentinel-absent": "unknown",
+}
+
+
+def read_ledger(path: Path) -> list[dict]:
+    """Parse the append-only JSONL ledger. A malformed line is SKIPPED, never guessed at.
+
+    Absent or empty file -> [] , which reproduces this report's pre-ledger behaviour
+    exactly (every requirement `not-derivable`, headline PROVEN 0).
+    """
+    if not path.is_file():
+        return []
+    rows: list[dict] = []
+    for line in path.read_text(errors="replace").splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        try:
+            obj = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(obj, dict) and obj.get("test"):
+            rows.append(obj)
+    return rows
+
+
+def latest_by_test(rows: list[dict]) -> dict[str, dict]:
+    """Most recent ledger row per test name (ts_utc, then file order as the tiebreak).
+
+    Latest wins rather than best-ever: a test that has since gone back to self-skipping
+    must stop reading as proven.
+    """
+    latest: dict[str, dict] = {}
+    for i, r in enumerate(rows):
+        key = r["test"]
+        prev = latest.get(key)
+        if prev is None or (r.get("ts_utc", ""), i) >= (prev.get("ts_utc", ""),
+                                                       prev["_i"]):
+            latest[key] = {**r, "_i": i}
+    return latest
+
+
+class _GitOracle:
+    """Small cached wrapper for the two git questions the staleness rule asks."""
+
+    def __init__(self, repo: Path):
+        self.repo = repo
+        self._last: dict[tuple[str, ...], str | None] = {}
+        self._anc: dict[tuple[str, str], bool] = {}
+
+    def _git(self, *args: str) -> subprocess.CompletedProcess:
+        return subprocess.run(["git", "-C", str(self.repo), *args],
+                              capture_output=True, text=True)
+
+    def last_change(self, paths: tuple[str, ...]) -> str | None:
+        """Commit that last touched any of `paths`; None if git knows none of them."""
+        if not paths:
+            return None
+        if paths in self._last:
+            return self._last[paths]
+        p = self._git("log", "-1", "--format=%H", "--", *paths)
+        sha = p.stdout.strip() if p.returncode == 0 else ""
+        self._last[paths] = sha or None
+        return self._last[paths]
+
+    def is_ancestor(self, older: str, newer: str) -> bool:
+        """True iff `older` is an ancestor of (or equal to) `newer`.
+
+        False whenever either rev is unresolvable — an unknown or `-dirty` sha (a run
+        recorded from a modified working tree) can never satisfy the freshness test, which
+        is the intended fail-closed direction.
+        """
+        key = (older, newer)
+        if key in self._anc:
+            return self._anc[key]
+        ok = self._git("merge-base", "--is-ancestor", older, newer).returncode == 0
+        self._anc[key] = ok
+        return ok
+
+
+def derive_proof(git: _GitOracle, rid: str, verifiers: list[dict],
+                 impl_files: list[dict], latest: dict[str, dict]) -> dict:
+    """PROOF for one requirement, from the ledger, with the staleness rule applied.
+
+    STALENESS RULE — a proof is a claim about a specific tree, not a permanent badge:
+      relevant paths = every implementing artifact this report found for the requirement
+                       + every verifying test's directory
+      last_change    = `git log -1 -- <relevant paths>`
+      A row's `proven` is honoured only if `last_change` is an ANCESTOR of the row's
+      `git_sha`, i.e. the run happened at or after the newest change to the code it
+      claims to prove. Otherwise it degrades to `stale-proof` (not green). A `-dirty`
+      sha, a sha this clone does not have (rebased/dropped), or a path git has never
+      seen (an uncommitted test) all fail the ancestry test and degrade the same way.
+
+    The requirement's state is the MINIMUM over its verifying tests: every verifying test
+    must be proven and fresh before the requirement reads `proven`. A test with no ledger
+    row contributes `not-derivable`, so a partially-recorded run cannot manufacture green.
+    """
+    paths = sorted({f["path"] for f in impl_files}
+                   | {os.path.dirname(v["path"]) for v in verifiers if v.get("path")})
+    last_change = git.last_change(tuple(paths))
+
+    per_test: list[dict] = []
+    for v in verifiers:
+        row = latest.get(v["name"])
+        if row is None:
+            per_test.append({"test": v["name"], "state": "not-derivable",
+                             "reason": "no ledger row — this test has never been "
+                                       "recorded by run.sh --record / record_run.py"})
+            continue
+        base = _PROOF_ALIASES.get(row.get("proof", ""), row.get("proof", "unknown"))
+        if base not in PROOF_ORDER:
+            base = "unknown"
+        state, reason = base, row.get("note", "")
+        if base == "proven":
+            sha = row.get("git_sha", "")
+            if last_change and not git.is_ancestor(last_change, sha):
+                state = "stale-proof"
+                reason = (f"proven at {sha[:12]}, but the implementing manifests / test "
+                          f"changed in {last_change[:12]}, which that run predates")
+        per_test.append({"test": v["name"], "state": state, "reason": reason,
+                         "run_id": row.get("run_id"), "ts_utc": row.get("ts_utc"),
+                         "git_sha": row.get("git_sha"), "env": row.get("env"),
+                         "sentinel": row.get("sentinel"),
+                         "junit_result": row.get("junit_result")})
+
+    if not per_test:
+        state = "not-derivable"
+        reason = "no requirement-tagged test exists to be proven"
+    else:
+        worst = min(per_test, key=lambda d: PROOF_ORDER[d["state"]])
+        state, reason = worst["state"], worst["reason"]
+        if state == "not-derivable" and not reason:
+            reason = "no committed ledger row for this requirement's verifying test(s)"
+
+    return {"state": state, "reason": reason, "last_change_sha": last_change,
+            "evidence_paths": paths, "per_test": per_test}
+
+
 # ========================================================================== build
 IMPL_ORDER = {"none": 0, "off-cluster": 1, "unwired": 2, "reconciled-partial": 3,
               "reconciled": 4}
@@ -369,7 +555,13 @@ LABELS = {
 
 def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, list[dict]],
           applied: set[str], gitops_roots: set[str], spec_ids: set[str] | None,
-          env: str) -> dict:
+          env: str, ledger_rows: list[dict] | None = None) -> dict:
+    # PROOF axis inputs. `ledger_rows=None` / `[]` reproduces the pre-ledger behaviour:
+    # `latest` is empty, so every requirement derives `not-derivable` exactly as before.
+    ledger_rows = ledger_rows or []
+    latest = latest_by_test(ledger_rows)
+    git = _GitOracle(repo)
+
     classified = [classify_test(t) for t in tests]
     by_req_tests: dict[str, list[dict]] = {r["id"]: [] for r in registry}
     for t, c in zip(tests, classified):
@@ -439,11 +631,9 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
                 "blocked_by": sorted({t["blocked_by"] for t in ts if t["blocked_by"]}),
                 "tests": ts,
             },
-            # NEVER derived. See the module docstring.
+            # Derived ONLY from the committed ledger, and only as far as it reaches.
-            "proof": {"state": "not-derivable",
+            # With no ledger row this stays `not-derivable`, as it always was.
-                      "reason": "no committed, machine-readable record of a test run "
+            "proof": derive_proof(git, rid, ts, ev, latest),
-                                "exists in the repo (_artifacts/ and results/ are "
-                                "git-ignored); DECISIONS-LOG.md prose is not a signal"},
             "status": LABELS[(impl_state, verify_state)],
         }
 
@@ -492,8 +682,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
             and v["verification"]["state"] == "executable"),
         "safety_critical_total": count(lambda v: v["safety_critical"]),
         "safety_critical_without_executable_l3": len(safety_scaffold),
-        "proven": 0,
+        # PROVEN is now a real count — but only of requirements whose EVERY verifying
-        "proven_derivable": False,
+        # test has a committed ledger row that saw its pass sentinel, at a sha the
+        # implementation has not moved past. `stale-proof` and `skipped` are counted
+        # separately and are NOT green.
+        "proven": count(lambda v: v["proof"]["state"] == "proven"),
+        "proof_stale": count(lambda v: v["proof"]["state"] == "stale-proof"),
+        "proof_skipped": count(lambda v: v["proof"]["state"] == "skipped"),
+        "proof_failed": count(lambda v: v["proof"]["state"] == "failed"),
+        "proof_unknown": count(lambda v: v["proof"]["state"] == "unknown"),
+        "proof_not_derivable": count(
+            lambda v: v["proof"]["state"] in ("not-derivable", "junit-skipped")),
+        "ledger_rows": len(ledger_rows),
+        "proven_derivable": bool(ledger_rows),
         "coverage_style_covered": count(lambda v: v["verification"]["state"] != "none"),
     }
     summary["uncited_by_any_reconciled_manifest"] = count(
@@ -552,16 +753,39 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append(f"| Safety-critical requirements whose L3 test is a scaffold | "
              f"**{s['safety_critical_without_executable_l3']}** of "
              f"{s['safety_critical_total']} |")
+    if s["proven_derivable"]:
+        o.append(f"| **PROVEN (every verifying test passed, evidenced)** | "
+                 f"**{s['proven']}** |")
+        o.append(f"| Proof exists but is STALE (code changed after the run) | "
+                 f"{s['proof_stale']} |")
+        o.append(f"| Verifying test SELF-SKIPPED green (not proof) | {s['proof_skipped']} |")
+        o.append(f"| Verifying test FAILED in the recorded run | {s['proof_failed']} |")
+        o.append(f"| Ran, but the log could not evidence pass vs self-skip | "
+                 f"{s['proof_unknown']} |")
+        o.append(f"| No committed ledger row yet | {s['proof_not_derivable']} |")
+        o.append("")
+        o.append(f"> **Proof comes from `conformance/runs/ledger.jsonl`** "
+                 f"({s['ledger_rows']} row(s)), written only by an explicit "
+                 f"`run.sh --record` / `lib/record_run.py` invocation. A row reaches "
+                 f"`proven` ONLY when the run log carried that test's own pass sentinel "
+                 f"token — the L2/L3 tests self-skip GREEN (Chainsaw records `passed` on "
+                 f"`DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries "
+                 f"no stdout), so a skip token maps to `skipped` and can never become "
+                 f"proof. A proof is honoured only while the implementing manifests and "
+                 f"the test itself have not changed since the recorded `git_sha`; "
+                 f"otherwise it degrades to `stale-proof`, which is not green.\n")
+    else:
         o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
         o.append("")
-    o.append("> **Proof is deliberately not derived.** Nothing tracked in git records a "
+        o.append("> **No proof is derivable: `conformance/runs/ledger.jsonl` is absent or "
-             "test result: `conformance/_artifacts/` and `results/` are git-ignored, and "
+                 "empty.** The ledger is the committed, machine-readable run record; until "
-             "the only record of a green run is DECISIONS-LOG.md prose — hand-maintained, "
+                 "a run is captured with `run.sh --record`, nothing tracked in git records "
-             "which is the very thing this report replaces. Per "
+                 "a test result (`conformance/_artifacts/` and `results/` are git-ignored, "
-             "`agents/definition-of-done.md` a requirement is done only when its tagged "
+                 "and DECISIONS-LOG.md prose is hand-maintained — the very thing this "
-             "test **passes**, so the strongest claim this report can make is "
+                 "report replaces). Per `agents/definition-of-done.md` a requirement is "
-             "*wired and testable*. Closing the gap needs a committed, machine-readable "
+                 "done only when its tagged test **passes**, so the strongest claim this "
-             "run ledger; until then, treat every row below as unproven.\n")
+                 "report can make is *wired and testable*; treat every row below as "
+                 "unproven.\n")
     o.append(f"> For contrast, `coverage.md` counts **{s['coverage_style_covered']}** "
              f"requirements as covered — that count includes the "
              f"{s['verify_scaffold_only']} whose only verifying test is a TODO scaffold.\n")
@@ -612,7 +836,15 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
             if ver["blocked_by"]:
                 vcell += f" · blocked-by: {'; '.join(ver['blocked_by'])}"
         sc = " ⚠safety" if v["safety_critical"] else ""
-        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | not-derivable |")
+        pr = v["proof"]
+        if pr["state"] == "not-derivable":
+            pcell = "not-derivable"
+        else:
+            runs = sorted({d["run_id"] for d in pr["per_test"] if d.get("run_id")})
+            pcell = f"**{pr['state']}**" if pr["state"] == "proven" else pr["state"]
+            if runs:
+                pcell += f" ({', '.join(runs)})"
+        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | {pcell} |")
 
     if rep["safety_critical_without_executable_l3"]:
         o.append("\n## Safety-critical requirements whose L3 fault test is a SCAFFOLD "
@@ -701,22 +933,76 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append("| executable vs scaffold | TODO markers in the test body outrank the "
              "`executable` annotation; a real body without the annotation still counts "
              "and is flagged | conservative — ambiguity never upgrades |")
-    o.append("| **test actually passed** | **NOT DERIVED** — nothing in git records a "
+    o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
-             "run result | refused |")
+             "every verifying test: JUnit result **plus** that test's own pass sentinel "
+             "token grepped from the run log | exact where a run was recorded, refused "
+             "everywhere else — a self-skip token or a missing/unrecognised sentinel "
+             "NEVER yields `proven` |")
+    o.append("| proof still current | the row's `git_sha` must be a descendant of the "
+             "last commit touching the requirement's implementing manifests or its test "
+             "| conservative — an unresolvable or `-dirty` sha degrades to `stale-proof` |")
     o.append("| cluster liveness (pods/CRs healthy) | **NOT DERIVED** — no cluster is "
              "contacted; `reconciled` means Flux *would* apply it, not that it is "
              "healthy | refused |")
     o.append("| phase/board status | **NOT DERIVED** — `agents/task-board.md` is stale "
              "and is not a signal | refused |")
 
+    o.append("\n## Run ledger — the committed proof-of-pass evidence\n")
+    if not s["proven_derivable"]:
+        o.append("- `conformance/runs/ledger.jsonl` holds no rows. Capture a run with "
+                 "`./run.sh --record` (recording is never implicit: a run that silently "
+                 "records itself is how a bad run becomes a permanent false proof), then "
+                 "commit `conformance/runs/<run-id>/` **with its `run.log`** and the "
+                 "ledger. The log is the load-bearing half — Chainsaw's JUnit carries no "
+                 "stdout, so without it a self-skip is indistinguishable from a pass.")
+    else:
+        o.append(f"- {s['ledger_rows']} row(s) in `conformance/runs/ledger.jsonl`. "
+                 f"Per-test evidence for every non-`not-derivable` requirement below; "
+                 f"full rows (env, chainsaw/kube versions, sentinel, note) are in "
+                 f"`status.json` under `by_requirement.<R-ID>.proof.per_test`.\n")
+        o.append("| Requirement | Proof | Test | Ledger state | Sentinel | Run |")
+        o.append("|---|---|---|---|---|---|")
+        for rid, v in rep["by_requirement"].items():
+            pr = v["proof"]
+            if pr["state"] == "not-derivable":
+                continue
+            for d in pr["per_test"]:
+                o.append(f"| {rid} | {pr['state']} | `{d['test']}` | {d['state']} | "
+                         f"`{d.get('sentinel') or '—'}` | {d.get('run_id') or '—'} |")
+        stale = [(rid, v) for rid, v in rep["by_requirement"].items()
+                 if v["proof"]["state"] == "stale-proof"]
+        if stale:
+            o.append("\n### Stale proofs — a real pass that no longer describes this tree\n")
+            o.append("These requirements DID pass, but their implementing manifests or "
+                     "their verifying test changed in a commit the recorded run predates. "
+                     "A stale proof is not green; re-run with `--record`.\n")
+            for rid, v in stale:
+                pr = v["proof"]
+                o.append(f"- **{rid}** — {pr['reason']}")
+        skipped = [(rid, v) for rid, v in rep["by_requirement"].items()
+                   if v["proof"]["state"] == "skipped"]
+        if skipped:
+            o.append("\n### SELF-SKIPPED green — recorded, and deliberately NOT proof\n")
+            o.append("Chainsaw reported `passed` for these, because the test exited 0 "
+                     "after emitting its skip token: a precondition (Longhorn "
+                     "StorageClass, Velero CRDs, >=3 Ready nodes) was absent, so the "
+                     "behaviour was never exercised. This is exactly the row a "
+                     "JUnit-only ledger would have recorded as green.\n")
+            for rid, v in skipped:
+                toks = ", ".join(sorted({d.get("sentinel") or "?"
+                                         for d in v["proof"]["per_test"]
+                                         if d["state"] == "skipped"}))
+                o.append(f"- **{rid}** — skip token(s): `{toks}`")
+
     o.append("\n## Advisory — untracked local run artifact (NOT counted)\n")
     if advisory:
         o.append(f"- `{advisory['path']}` (untracked, git-ignored), last written "
                  f"{advisory['mtime']}: {advisory['tests']} testcase(s), "
                  f"{advisory['failures']} failure(s), {advisory['skipped']} skipped.")
         o.append(f"- requirement IDs it names: {', '.join(advisory['requirements']) or '—'}")
-        o.append("- This is one developer's local workspace, not repo state. It is shown "
+        o.append("- This is one developer's local workspace, not repo state, and it is NOT "
-                 "for orientation only and contributes to NO status above.")
+                 "the ledger. It is shown for orientation only and contributes to NO "
+                 "status above — a JUnit alone cannot tell a self-skip from a pass.")
     else:
         o.append("- none present.")
     return "\n".join(o) + "\n"
@@ -841,6 +1127,122 @@ def _selftest() -> int:
               classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
               == "scaffold")
 
+    # ---------------------------------------------------------------- PROOF / ledger
+    # These are the assertions that stand between this report and a fake green.
+    check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
+          read_ledger(Path("/nonexistent/ledger.jsonl")) == [])
+
+    def row(**kw):
+        base = {"ts_utc": "2026-09-01T12:00:00Z", "git_sha": "deadbeef", "env": "dev",
+                "chainsaw_version": "v0.2.12", "kube_version": "v1.36.4", "layer": "L3",
+                "test": "dr-restore", "requirements": ["R-STATE-03"],
+                "junit_result": "passed", "sentinel": None, "proof": "skipped",
+                "run_id": "20260901T120000Z-deadbee", "note": ""}
+        base.update(kw)
+        return base
+
+    verifiers = [{"name": "dr-restore",
+                  "path": "conformance/tests/l3-fault-injection/dr-restore/"
+                          "chainsaw-test.yaml"}]
+
+    with tempfile.TemporaryDirectory() as td:
+        gr = Path(td)
+        env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
+               "GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
+
+        def git(*a):
+            return subprocess.run(["git", "-C", str(gr), *a], capture_output=True,
+                                  text=True, env=env)
+
+        git("init", "-q", "-b", "main")
+        (gr / "platform").mkdir()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03\n")
+        git("add", "-A"); git("commit", "-qm", "one")
+        sha_a = git("rev-parse", "HEAD").stdout.strip()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03 v2\n")
+        git("add", "-A"); git("commit", "-qm", "two")
+        sha_b = git("rev-parse", "HEAD").stdout.strip()
+
+        oracle = _GitOracle(gr)
+        impl = [{"path": "platform/velero.yaml"}]
+
+        # *** THE CASE THIS WHOLE FEATURE EXISTS FOR ***
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="skipped", junit_result="passed",
+                                              sentinel="DR_RESTORE_SKIPPED_OK",
+                                              git_sha=sha_b)]))
+        check("a SELF-SKIP row on a GREEN JUnit is 'skipped', NEVER 'proven'",
+              pr["state"] == "skipped")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a proven row at the CURRENT sha is honoured", pr["state"] == "proven")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_a,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("STALENESS: proven at a sha the implementation has moved past degrades "
+              "to stale-proof", pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=f"{sha_b}-dirty",
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a run recorded from a DIRTY tree is never honoured as proof",
+              pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha="0" * 40)]))
+        check("a git_sha this clone cannot resolve degrades, never upgrades",
+              pr["state"] == "stale-proof")
+
+        for bad, expect in (("failed", "failed"),
+                            ("junit-skipped", "junit-skipped"),
+                            ("unknown-sentinel-absent", "unknown"),
+                            ("unknown-no-sentinel-declared", "unknown"),
+                            ("something-new-from-the-future", "unknown")):
+            pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                              latest_by_test([row(proof=bad, git_sha=sha_b)]))
+            check(f"ledger state {bad!r} maps to {expect!r}, never to proven",
+                  pr["state"] == expect)
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl, {})
+        check("no ledger row at all stays not-derivable",
+              pr["state"] == "not-derivable")
+
+        # Two verifying tests, only one proven: the requirement must NOT read proven.
+        two = verifiers + [{"name": "data-survival",
+                            "path": "conformance/tests/l3-fault-injection/"
+                                    "data-survival/chainsaw-test.yaml"}]
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b)]))
+        check("a requirement is proven only when EVERY verifying test is",
+              pr["state"] == "not-derivable")
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b),
+                                          row(test="data-survival", proof="skipped",
+                                              git_sha=sha_b)]))
+        check("worst-of wins across verifying tests (skip drags proven down)",
+              pr["state"] == "skipped")
+
+        # Latest row wins, not best-ever: a test that regressed to self-skipping must
+        # stop reading as proven.
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(ts_utc="2026-09-01T10:00:00Z",
+                                              proof="proven", git_sha=sha_b),
+                                          row(ts_utc="2026-09-02T10:00:00Z",
+                                              proof="skipped", git_sha=sha_b)]))
+        check("the LATEST ledger row wins, not the best one ever recorded",
+              pr["state"] == "skipped")
+
+        # A malformed ledger line is skipped, never guessed at.
+        lf = gr / "ledger.jsonl"
+        lf.write_text('{"test":"a","proof":"proven"}\nnot json at all\n\n'
+                      '{"proof":"proven"}\n')
+        rows = read_ledger(lf)
+        check("malformed / test-less ledger lines are dropped, not guessed at",
+              len(rows) == 1 and rows[0]["test"] == "a")
+
     if not ok:
         return 1
     sys.stderr.write("status.py selftest: PASS\n")
@@ -865,6 +1267,10 @@ def main() -> int:
                     help="exit 1 on defects this tool owns: registry/spec drift, "
                          "executable-annotation conflicts, or a safety-critical "
                          "requirement whose only L3 test is a scaffold")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"committed proof-of-pass ledger written by lib/record_run.py "
+                         f"(default {DEFAULT_LEDGER}). Absent or empty => every "
+                         f"requirement stays `not-derivable`, as before the ledger existed")
     ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
     args = ap.parse_args()
 
@@ -895,7 +1301,12 @@ def main() -> int:
     applied, nodes = reconciled_files(repo, entry)
     gitops_roots = {n["resolved"].split("/", 1)[0] for n in nodes if n["exists"]}
     impl = scan_implementation(repo, tracked_files(repo))
-    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env)
+    ledger_rows = read_ledger(args.ledger)
+    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env,
+                ledger_rows)
+    rep["ledger"] = {"path": os.path.relpath(args.ledger, REPO_ROOT),
+                     "rows": len(ledger_rows),
+                     "present": args.ledger.is_file()}
     rep["flux_graph"] = nodes
     rep["gitops_roots"] = sorted(gitops_roots)
     rep["reconciled_file_count"] = len(applied)
@@ -915,7 +1326,11 @@ def main() -> int:
           f"(+{s['wired_or_partial_and_testable'] - s['wired_and_testable']} partial) | "
           f"scaffold-only {s['verify_scaffold_only']} | "
           f"safety-critical w/o executable L3 "
-          f"{s['safety_critical_without_executable_l3']} | PROVEN: not derivable")
+          f"{s['safety_critical_without_executable_l3']} | "
+          + (f"PROVEN: {s['proven']} (stale {s['proof_stale']}, self-skipped "
+             f"{s['proof_skipped']}, failed {s['proof_failed']}, unknown "
+             f"{s['proof_unknown']}) from {s['ledger_rows']} ledger row(s)"
+             if s["proven_derivable"] else "PROVEN: not derivable (empty run ledger)"))
     print(f"[status] wrote {args.out/'status.md'} and {args.out/'status.json'}")
 
     rc = 0

conformance/run.sh

@@ -14,12 +14,33 @@
 #   ./run.sh --status                 # DERIVED STATUS report only (no cluster): per
 #                                     #   requirement, is it implemented, is it actually
 #                                     #   reconciled by Flux, is its test executable or a
-#                                     #   scaffold. Proof-of-pass is deliberately NOT
+#                                     #   scaffold, and whether the committed run ledger
-#                                     #   derived — see lib/status.py.
+#                                     #   (runs/ledger.jsonl) actually proves it — see
+#                                     #   lib/status.py. No ledger row => not-derivable.
 #   ./run.sh --lint                   # structural-validate test defs (no cluster)
 #   ./run.sh --strict-coverage        # hygiene gate: fail on tag/annotation drift
 #   ./run.sh --requirements R-NET-01  # ALSO asserts those IDs are covered (merge gate)
 #   ./run.sh --require-all            # phase-exit: every requirement must be covered
+#   ./run.sh --record                 # RUN AND CAPTURE: tee the whole run to
+#                                     #   runs/<UTC>-<short-sha>/run.log, copy the JUnit
+#                                     #   beside it, and append one ledger row per
+#                                     #   testcase to runs/ledger.jsonl (lib/record_run.py)
+#
+# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
+# --------------------------------------------------------------------------
+# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
+# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
+# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
+# that — `conformance/runs/` (see conformance/.gitignore for the negations).
+#
+# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
+# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
+#
+# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
+# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
+# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
+# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
+# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
 #
 # Artifacts (always written under conformance/_artifacts/, gitignored):
 #   _artifacts/report.junit.xml        JUnit (Chainsaw output, or skipped-scaffold shim)
@@ -28,6 +49,11 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Run ledger (TRACKED — commit these; --record only):
+#   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
+#   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
+#   runs/ledger.jsonl                         append-only, one row per testcase per run
+#
 # Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
 # execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
 # green — the harness is proven reproducible while later phases fill in behavior.
@@ -47,12 +73,17 @@ STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
 PYTHON="${PYTHON:-python3}"
 
+# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
+# under `tee`. Captured BEFORE parsing so nothing is normalized away.
+ORIG_ARGS=("$@")
+
 LAYER="all"
 REQS=""
 MODE="run"          # run | coverage-only | status | lint
 STRICT_COVERAGE=0
 REQUIRE_ALL=0
 FORCE_NO_CLUSTER=0
+RECORD=0
 
 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
 # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
@@ -83,7 +114,9 @@ FORCE_NO_CLUSTER=0
 # longer a gap and has been removed from this list; the coverage report counts it as covered.
 DOCUMENTED_GAPS=""
 
-usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
+# Print the header block (line 2 through the closing `####` rule, which is dropped) as
+# help. Bounded by the rule rather than a fixed line number so the header can grow.
+usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
 
 while [ $# -gt 0 ]; do
   case "$1" in
@@ -95,6 +128,7 @@ while [ $# -gt 0 ]; do
     --strict-coverage) STRICT_COVERAGE=1; shift ;;
     --require-all)    REQUIRE_ALL=1; shift ;;
     --no-cluster)     FORCE_NO_CLUSTER=1; shift ;;
+    --record)         RECORD=1; shift ;;
     -h|--help)        usage; exit 0 ;;
     *) echo "unknown arg: $1" >&2; usage; exit 2 ;;
   esac
@@ -104,6 +138,62 @@ mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
 
 log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
 
+###############################################################################
+# --record: capture this run as committed evidence, then hand it to the ledger.
+#
+# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
+# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
+# operator-visible output — no second, differently-behaving "recording mode" to drift.
+# CONF_RECORD_CHILD guards against recursion.
+#
+# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
+# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
+# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
+###############################################################################
+if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
+  case "${MODE}" in
+    run) ;;
+    # --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
+    # test, so a ledger row from one of them would be a row about nothing.
+    *) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
+  esac
+  RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
+  RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
+  RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
+  mkdir -p "${RUN_DIR}"
+  # Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
+  # ever capture a report THIS run produced. Copying a stale artifact would pair someone
+  # else's result with this run's log — a fabricated pairing, which is precisely the class
+  # of false proof the ledger exists to prevent.
+  rm -f "${JUNIT_OUT}"
+  # Strip --record from the argv handed to the child; everything else passes through.
+  CHILD_ARGS=()
+  for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
+    [ "$a" = "--record" ] || CHILD_ARGS+=("$a")
+  done
+  log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
+  set +e
+  CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
+    | tee "${RUN_DIR}/run.log"
+  RUN_RC=${PIPESTATUS[0]}
+  set -e
+  if [ -f "${JUNIT_OUT}" ]; then
+    cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
+  else
+    log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
+  fi
+  # Explicit, separate step: read the JUnit + the log and append one row per testcase.
+  # rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
+  # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
+  # to the real ledger; operators should never set it.
+  "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
+    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
+    --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
+  log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
+  log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
+  exit "${RUN_RC}"
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no

conformance/runs/20260901T235117Z-e9a4b66/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="131.090121" />
+  </testsuite>
+</testsuites>

conformance/runs/20260901T235117Z-e9a4b66/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 23:51:18 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1788306678898809444
+        victim node (currently hosts the attached volume): k3s-dev-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-dev-2 labeled
+        node/k3s-dev-2 cordoned
+        cordoned + labelled victim node: k3s-dev-2
+        pod "writer" deleted from chainsaw-fond-marlin namespace
+        evicted writer off the victim node; volume detached from k3s-dev-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
+        recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-dev-2 uncordoned
+        node/k3s-dev-2 unlabeled
+        restored node to service: k3s-dev-2
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-fond-marlin namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:53:29 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (131.09s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260902T001907Z-fdc3265/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="149.053701" />
+  </testsuite>
+</testsuites>

conformance/runs/20260902T001907Z-fdc3265/run.log

@@ -0,0 +1,671 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 00:19:07 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true http://192.168.1.9:9000 velero
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=velero  endpoint=http://192.168.1.9:9000
+        backup endpoint host: 192.168.1.9
+        backup endpoint resolves to: 192.168.1.9
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 81)
+        endpoint is not any Pod IP (checked 186)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=192.168.1.9 bucket=velero
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
+        namespace/chainsaw-cunning-hen-dr-src created
+        namespace/chainsaw-cunning-hen-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1788308352223263220
+        original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
+        backup.velero.io/chainsaw-cunning-hen-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=Completed
+        backup chainsaw-cunning-hen-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-cunning-hen-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-cunning-hen-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
+        recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
+        velero objects for chainsaw-cunning-hen cleaned up (ns velero)
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
+        released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:21:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (149.05s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -0,0 +1,2 @@
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}

infra/PROD-PROVISIONING.md

@@ -25,20 +25,187 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
 |---|---|---|---|
 | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
 | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
-| Model | HP EliteDesk 800 G5 Desktop Mini | | |
+| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | |
-| CPU | i7-9700T, 8c/8t, VT-x | | |
+| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | |
-| RAM | 31 GiB | | |
+| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | |
-| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | |
+| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | |
-| OS | Debian 13 (trixie), kernel 6.12 | | |
+| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | |
-| Firmware | HP R21 Ver. 02.26.00 | | |
+| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | |
-| `/dev/kvm` | present | | |
+| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | |
+| `/dev/kvm` | present | present | |
+| Serial | MXL1052NGN | MXL0073949 | |
+
+**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check;
+unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**),
+`DIMM3` a Micron **16**ATF2G64HZ-**2G6**E1 (DDR4-**2667**). Both are 16 GB so capacity is correct,
+but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded
+only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch.
+
+**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a
+healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica
+placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's
+NVMe is the drive node 1 should have had.
 
 Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).
 
 ---
 
+## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01)
+
+> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory
+> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine.
+> Keeping the correction visible because the wrong framing nearly cost money and a board RMA.
+
+`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware
+publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before
+reserving memory based on its contents.** Fixed upstream in **6.13**.
+
+### The evidence that breaks it open
+
+The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full
+15.9 GiB two boots earlier**:
+
+| Boot | `Memory: … available` | reserved | memattr line |
+|---|---|---|---|
+| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` |
+| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
+| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
+
+The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in
+that shape, and none is fixed by a reboot.
+
+### Mechanism (verified against kernel source)
+
+`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`:
+
+```c
+if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; }
+tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
+memblock_reserve(efi_mem_attr_table, tbl_size);
+```
+
+**`num_entries` and `desc_size` are never validated.** With a corrupt table:
+- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)*
+- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s
+  the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)*
+
+That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` =
+DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero.
+
+### BOTH prod nodes are affected — node 1 has only been lucky
+
+| Node | memattr version field | Outcome |
+|---|---|---|
+| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** |
+| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB |
+| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB |
+
+Node 1 is one unlucky boot away from the identical fault. **Fix both.**
+
+### Mechanism, traced to the byte (second, independent confirmation)
+
+A second research pass reached the same root cause by a different route and pinned the arithmetic:
+
+- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The
+  kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that
+  `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it.
+- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and
+  when bit 31 lands the result is **negative**.
+- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to
+  ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical
+  address above `0xbeeb0018` is reserved.**
+- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages`
+  (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`.
+- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by
+  `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is
+  therefore positive proof the reservation already happened** — it is a consequence, not the cause.
+
+The cleanest A/B tell between the two nodes is one line of dmesg:
+
+| | `NODE_DATA(0) allocated` | meaning |
+|---|---|---|
+| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation |
+| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement |
+
+Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
+node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.
+
+### THERE IS NO KERNEL CMDLINE WORKAROUND
+
+`efi_memattr_init()` is called from `efi_config_parse_tables()` under only
+`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()`
+**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap`
+**do not prevent the reservation.** Booting a fixed kernel is the only real fix.
+
+### The fix
+
+**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is
+clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message
+describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable
+branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints
+`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory.
+
+The hardening landed exactly one release too late for Debian 13: the bug has existed since the
+feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up
+`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim
+*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably
+lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the
+release lifetime, and the fix is small and already upstream.
+
+Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382)
+(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the
+table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook
+Studio G5**, 16 GiB → 1.5 GiB).
+
+### What NOT to do
+
+- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is
+  a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be
+  installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6.
+- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full
+  R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries.
+- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket:
+  they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They
+  may appear to "fix" it. They are not fixes.
+
+### Hypotheses killed, and why — so they are not re-run
+
+- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by
+  an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP
+  documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control.
+- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by
+  `hp-bioscfg` on this platform.
+- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB.
+- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by
+  measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`.
+  Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never
+  `memblock_add()`ed — which would have left `present` at ~0, not 3340288.
+- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`.
+- **MTRR trim** — MTRRs near-identical to node 1, no trim warning.
+- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal.
+- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly
+  explained: everything from `0xbeeb0018` upward.
+
 ## 1. Firmware / BIOS (per node, before OS install)
 
+> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes
+> **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a
+> `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes",
+> which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
+> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye.
+>
+> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4,
+> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is
+> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means
+> F10 on the box.
+>
+> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is
+> achievable today with reads alone, and would have caught node 2's drift on its first health run.
+> Converging it automatically additionally requires provisioning a BIOS Setup Password on every
+> node, which is a real security decision (a stored credential that can rewrite firmware) and is
+> the operator's call, not a default.
+
+
 - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
       boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
 - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
@@ -50,6 +217,80 @@ Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data di
       nodes match.
 - [ ] **Firmware version recorded** in §0 and consistent across all three nodes.
 
+### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01
+
+**There is no default BIOS password, and no factory default exists to look up.** HP business
+desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we
+observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5
+describes both purely as things the owner configures in Computer Setup.
+
+**Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents
+clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once).
+**Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely,
+turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The
+permissive default is the right setting for a homelab.
+
+#### CORRECTION — error `0x4` is NOT an authorisation failure
+
+I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That
+was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI
+`SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a
+different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have
+unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting`
+returns:
+
+| Code | Meaning |
+|---|---|
+| 0 | Success |
+| 1 | Not Supported |
+| **4** | **Failed** (generic) |
+| 6 | Access Denied |
+
+So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and
+the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither
+of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up
+in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`.
+
+**Most likely actual cause (inferred, testable):** with an empty password the driver emits an
+*untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when
+blank (`Password = "<utf-16/>"`). The driver only prepends that prefix for a *non-empty* string. So
+setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would
+work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this
+firmware already makes the driver log `Prerequisites size value exceeded the maximum number of
+elements supported or data may be malformed` at boot.
+
+**`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and
+`new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM;
+nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read.
+
+#### Recommendation
+
+1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff
+   `current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV
+   win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10.
+2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision
+   one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to
+   `current_password`, retry one benign attribute. Return `0` confirms the `<utf-16/>` theory; `6`
+   would mean a password mismatch; another `4` means the driver is incompatible with R21 and the
+   fallback is `hp-repsetup`.
+3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup
+   Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it
+   needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**.
+   Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a
+   security downgrade.
+4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in
+   principle — the private key never leaves the workstation, each change is individually signed, with
+   anti-replay — but provisioning and every signature require Windows-only tooling. For three
+   machines whose firmware settings change roughly never, the operational cost far exceeds the risk
+   reduction. BCU is Windows/WinPE-only.
+5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full
+   pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) —
+   functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via
+   `hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is
+   persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the
+   password in cleartext on the command line — it would land in process listings and shell history
+   unless wrapped. Store it in the Ansible vault **and** offline.
+
 ## 2. Disk layout — **OPEN DECISION, blocks the build**
 
 `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went

infra/ansible/inventory/hosts.ini

@@ -87,11 +87,13 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
 
 [k3s_prod_secondary]
-# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md
+# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
-# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so
+# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
-# no play can accidentally target a host that isn't there; uncomment each line only once
+# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
-# that node is actually on the bench and answering SSH.
+# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
-# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
+# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
+# node 3 remains commented until it is racked.
+k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
 # k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
 
 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]

infra/ansible/roles/node_health/README.md

@@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI,
   EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified
   against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM).
-- **Never `become`s unless it must.** Only SMART needs root.
+- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring
+  buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM
+  inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and
+  the privileged second pass re-runs the *same* collector with `--privileged` so the
+  parsing has one home. Without escalation those checks SKIP with the exact reason.
 
 ## What it checks
 
@@ -44,15 +48,51 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
 | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
+| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
 | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
+| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
 
 Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
 in task logic.
 
+
+## The EFI memattr check
+
+`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
+
+HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
+does not validate it before reserving memory from its contents —
+`drivers/firmware/efi/memattr.c` multiplies two never-validated fields
+(`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the
+table address to the top of RAM. Which of two kernel-log lines appears decides whether the
+node keeps its memory, and it is **decided afresh on every boot**:
+
+| kernel log | meaning | verdict |
+|---|---|---|
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table |
+| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working |
+| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS |
+| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected |
+| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected |
+
+The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and
+differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before
+it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing.
+
+**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
+`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
+workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
+rebooting/CMOS/BIOS/USB changes only move where the table lands.
+
+`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node
+whose kernel log cannot be read is still not unmeasured — and a future memory loss with a
+different cause is still caught.
+
 ## Output
 
 1. A human summary printed in the play output.

infra/ansible/roles/node_health/defaults/main.yml

@@ -45,6 +45,33 @@ node_health_thresholds:
   # Memory (percentage of MemTotal still available)
   mem_available_pct_warn: 15
   mem_available_pct_fail: 5
+  # --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ----------------------------
+  # Two independent measurements of the same question, both fed by memory.physical_total:
+  #
+  #   unmanaged  /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own
+  #              memory map says exists, that never reached the allocator. Needs no root
+  #              and no tools, works identically on metal and in a VM.
+  #   shortfall  MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is
+  #              installed. Needs root, and only bare metal reports it. Independent of the
+  #              kernel's memory map entirely, so it also catches RAM the firmware never
+  #              handed the kernel in the first place.
+  #
+  # MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and
+  # not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2%
+  # k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state.
+  # The healthy figure is the kernel image, memmap and firmware reservations; it is a
+  # near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10%
+  # keeps clearance for that. There is no plausible tuning of a threshold in this band
+  # that changes any of those five verdicts.
+  #
+  # The DIMM shortfall runs a little higher than the unmanaged figure on the same node
+  # (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and
+  # the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the
+  # kernel's to manage. Same bands: the signal being caught is 83%, not 3%.
+  mem_unmanaged_pct_warn: 10
+  mem_unmanaged_pct_fail: 25
+  mem_dimm_shortfall_pct_warn: 10
+  mem_dimm_shortfall_pct_fail: 25
   # Temperatures (deg C, any sensor)
   temp_warn_c: 70
   temp_fail_c: 85

infra/ansible/roles/node_health/files/node_health_collect.py

@@ -97,10 +97,22 @@ def read_kv(path, sep=":"):
     return out
 
 
+def which(tool):
+    """Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits.
+
+    `dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the
+    unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed
+    but unreachable would be a false reason on a SKIP, which is the one thing this
+    collector may never do."""
+    return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+
+
 def run(argv, timeout=20):
     """Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent."""
-    if shutil.which(argv[0]) is None:
+    exe = which(argv[0])
+    if exe is None:
         return None, "", "%s not installed" % argv[0]
+    argv = [exe] + list(argv[1:])
     try:
         proc = subprocess.run(
             argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout
@@ -284,9 +296,285 @@ def collect_memory():
         edac["ce_total"] = ce_total
         edac["ue_total"] = ue_total
     mem["edac"] = edac
+    # Physically present vs actually usable — the cause-independent memory-loss signal.
+    mem["zones"] = collect_zoneinfo(mem["mem_total_kb"])
+    # Firmware's own inventory of what is physically installed. Root-only; attempted
+    # here anyway so the unprivileged pass records the exact reason it could not read.
+    mem["devices"] = collect_memory_devices()
     return mem
 
 
+# --- Memory the kernel has but never hands out --------------------------------------
+# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
+# no DMI. Per zone it reports:
+#   spanned  pages in the zone's address range, holes included
+#   present  spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says
+#            physically exists here
+#   managed  present MINUS everything memblock reserved before the buddy allocator was
+#            handed the rest (kernel image, memmap, initrd, crashkernel, firmware
+#            reservations). sum(managed) IS MemTotal, exactly, on every node measured.
+# So `present - managed` is, by construction, "RAM this machine has that the OS will
+# never allocate", whatever the reason. On a healthy node it is the couple of hundred
+# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and
+# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% —
+# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude
+# separation, which is why this needs no clever tuning to be trustworthy.
+#
+# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if
+# a future firmware, kernel or hypervisor lost the memory for an entirely different
+# reason, and it needs neither root nor a parseable kernel log.
+def collect_zoneinfo(mem_total_kb):
+    raw = read_text("/proc/zoneinfo", optional=False)
+    page_kb = os.sysconf("SC_PAGE_SIZE") // 1024
+    out = {
+        "available": raw is not None,
+        "reason": None if raw is not None else "/proc/zoneinfo could not be read",
+        "page_size_kb": page_kb,
+        "zones": [],
+        "present_kb": None,
+        "managed_kb": None,
+        "unmanaged_kb": None,
+        "unmanaged_pct": None,
+    }
+    if raw is None:
+        return out
+    node = zone = None
+    cur = {}
+    present_pages = managed_pages = 0
+    for line in raw.splitlines():
+        fields = line.split()
+        if line.startswith("Node ") and "zone" in line:
+            # "Node 0, zone   Normal"
+            if cur:
+                out["zones"].append(cur)
+            node, zone = fields[1].rstrip(","), fields[-1]
+            cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None}
+        elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"):
+            try:
+                cur[fields[0]] = int(fields[1])
+            except ValueError:
+                cur[fields[0]] = None
+    if cur:
+        out["zones"].append(cur)
+    # An empty parse is a PARSE failure, not "this node has no memory" — say so and stay
+    # null rather than emitting a confident 0.
+    if not out["zones"]:
+        out["available"] = False
+        out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it"
+        return out
+    for z in out["zones"]:
+        present_pages += z["present"] or 0
+        managed_pages += z["managed"] or 0
+    out["present_kb"] = present_pages * page_kb
+    out["managed_kb"] = managed_pages * page_kb
+    out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"]
+    if out["present_kb"] > 0:
+        out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1)
+    # Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the
+    # two readings disagree and the discrepancy itself is worth showing.
+    out["matches_mem_total"] = (
+        None if (mem_total_kb is None or out["managed_kb"] is None)
+        else abs(out["managed_kb"] - mem_total_kb) <= page_kb
+    )
+    return out
+
+
+# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) -----------------------------------
+# The independent answer to "how much RAM is physically in this box", from the firmware's
+# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads
+# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that
+# failure is recorded as the SKIP reason — never as zero DIMMs.
+#
+# On a VM there is usually either no type-17 structure at all or a single synthetic entry
+# that merely restates the configured guest RAM; both are handled by reporting what was
+# found and letting the check SKIP when nothing populated was reported.
+DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def collect_memory_devices():
+    out = {
+        "available": False,
+        "reason": None,
+        "devices": [],
+        "slots": 0,
+        "populated_slots": 0,
+        "installed_kb": None,
+    }
+    if which("dmidecode") is None:
+        out["reason"] = ("dmidecode is not installed on this node; it ships with "
+                         "roles/node_baseline (node_baseline_packages). "
+                         "This role never installs packages.")
+        return out
+    rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30)
+    if rc != 0:
+        out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % (
+            rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output")
+        return out
+    # Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which
+    # may or may not hold a module ("Size: No Module Installed").
+    total_kb = 0
+    saw_size = False
+    for block in stdout.split("\n\n"):
+        if "Memory Device" not in block:
+            continue
+        fields = {}
+        for line in block.splitlines():
+            key, sep, value = line.partition(":")
+            if sep:
+                fields[key.strip()] = value.strip()
+        if "Size" not in fields:
+            continue
+        out["slots"] += 1
+        size_kb = None
+        parts = fields["Size"].split()
+        if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS:
+            try:
+                size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()]
+            except ValueError:
+                size_kb = None
+        if size_kb:
+            saw_size = True
+            total_kb += size_kb
+            out["populated_slots"] += 1
+        out["devices"].append(
+            {
+                "locator": fields.get("Locator"),
+                "bank_locator": fields.get("Bank Locator"),
+                # Verbatim, so an unparsed spelling is visible rather than silently null.
+                "size_raw": fields["Size"],
+                "size_kb": size_kb,
+                "type": fields.get("Type"),
+                "speed": fields.get("Speed"),
+                "configured_speed": fields.get("Configured Memory Speed"),
+                "manufacturer": fields.get("Manufacturer"),
+                "part_number": fields.get("Part Number"),
+                "rank": fields.get("Rank"),
+                # Serial deliberately NOT collected: the artifact already carries drive
+                # serials and a DIMM serial adds identifying data no check reads.
+            }
+        )
+    out["available"] = saw_size
+    if not saw_size:
+        out["reason"] = ("dmidecode ran but reported no populated memory device "
+                         "(%d slot structure(s) found) — typical of a VM with no SMBIOS "
+                         "type 17 inventory" % out["slots"])
+    else:
+        out["installed_kb"] = total_kb
+    return out
+
+
+# --- Kernel ring buffer: the EFI Memory Attributes Table ----------------------------
+# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative
+# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
+# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
+# the product). The kernel prints exactly one of two lines, and WHICH one decides whether
+# the node keeps its RAM:
+#
+#   "Failed to map EFI Memory Attributes table @ 0x..."   the table's garbage version read
+#       <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW.
+#   "Unexpected EFI Memory Attributes table version <N>"  the garbage version read > 2, so
+#       the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
+#       firmware is still publishing a corrupt table and the outcome is decided afresh by
+#       where the table lands on every boot. That is luck, not immunity.
+#
+# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots
+# before it lost 13.3 GiB), which is precisely why it is detected on every health run
+# instead of diagnosed once.
+#
+# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
+# every node here). The unprivileged pass therefore records WHY it could not read, and
+# roles/node_health re-runs this collector with `--privileged` when escalation works.
+BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:")
+EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
+MEMATTR_FAILED_RE = re.compile(
+    r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
+MEMATTR_VERSION_RE = re.compile(
+    r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I)
+MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
+# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
+# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
+# captured separately, above, and quoting it as evidence would bury the one line that
+# matters in 200 characters of unrelated table addresses.
+MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I)
+# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line
+# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict.
+MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available")
+MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved")
+
+
+def collect_klog():
+    out = {
+        "available": False,
+        "reason": None,
+        "boot_covered": None,
+        "memattr_state": None,
+        "memattr_published": None,
+        "memattr_table_address": None,
+        "memattr_version": None,
+        "memattr_lines": [],
+        "memory_line": None,
+    }
+    rc, stdout, stderr = run(["dmesg"], timeout=30)
+    if rc is None:
+        out["reason"] = "dmesg is not installed on this node"
+        return out
+    if rc != 0:
+        detail = (stderr or stdout).strip().splitlines()
+        out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % (
+            rc, os.geteuid(), detail[-1] if detail else "no output",
+            " — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG"
+            if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "")
+        return out
+
+    out["available"] = True
+    # Does the buffer still reach back to early boot? The ring buffer WRAPS, and every
+    # memattr line is printed in the first fraction of a second. If the boot header is
+    # gone then so may be a memattr line that was printed — absence would prove nothing,
+    # and the check must SKIP rather than PASS. This is the difference between
+    # "unaffected" and "could not tell", and the role is not allowed to blur it.
+    out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout))
+
+    published = EFI_CONFIG_RE.search(stdout)
+    if published:
+        out["memattr_published"] = True
+        out["memattr_table_address"] = published.group(1)
+    elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M):
+        # UEFI booted and the config-table line is present, but it lists no MEMATTR=.
+        out["memattr_published"] = False
+    # else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None.
+
+    for line in stdout.splitlines():
+        if MEMATTR_LINE_RE.search(line):
+            out["memattr_lines"].append(line.strip())
+    joined = "\n".join(out["memattr_lines"])
+    if MEMATTR_CORRUPT_RE.search(joined):
+        # Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected.
+        out["memattr_state"] = "corrupt_rejected"
+    elif MEMATTR_FAILED_RE.search(joined):
+        out["memattr_state"] = "failed_to_map"
+        addr = MEMATTR_FAILED_RE.search(joined).group(1)
+        if addr:
+            out["memattr_table_address"] = addr
+    elif MEMATTR_VERSION_RE.search(joined):
+        out["memattr_state"] = "bad_version"
+        out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1))
+    else:
+        out["memattr_state"] = "clean"
+
+    mem_line = MEMORY_LINE_RE.search(stdout)
+    if mem_line:
+        full = [l for l in stdout.splitlines() if mem_line.group(0) in l]
+        reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None
+        out["memory_line"] = {
+            "available_kb": int(mem_line.group(1)),
+            "total_kb": int(mem_line.group(2)),
+            "reserved_kb": int(reserved.group(1)) if reserved else None,
+            "raw": full[0].strip() if full else None,
+        }
+    return out
+
+
 # --- Thermals -----------------------------------------------------------------------
 def collect_thermal():
     zones = []
@@ -708,15 +996,34 @@ def collect_os():
 def collect_tools():
     tools = {}
     for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
-        tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+        tools[tool] = which(tool)
     return tools
 
 
 def main():
+    # --privileged: re-collect ONLY the fields that genuinely need root, for the second
+    # pass roles/node_health makes when (and only when) escalation is proven to work
+    # (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place —
+    # re-implementing the kernel-log grep in Jinja would be a second source of truth for
+    # the same fact, and the two would drift.
+    if "--privileged" in sys.argv[1:]:
         doc = {
             "schema": SCHEMA,
             "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
             "collector_uid": os.geteuid(),
+            "klog": collect_klog(),
+            "memory": {"devices": collect_memory_devices()},
+        }
+        doc["unreadable"] = UNREADABLE
+        json.dump(doc, sys.stdout, indent=2, sort_keys=True)
+        sys.stdout.write("\n")
+        return
+
+    doc = {
+        "schema": SCHEMA,
+        "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+        "collector_uid": os.geteuid(),
+        "klog": collect_klog(),
         "os": collect_os(),
         "dmi": collect_dmi(),
         "cpu": collect_cpu(),

infra/ansible/roles/node_health/tasks/assess.yml

@@ -387,6 +387,83 @@
                ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
                ~ avail_pct | string ~ '%)') }}
 
+# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the
+# one cause we have actually hit here; this one asks the blunter question — "is this
+# machine using the RAM it has?" — and answers it from two sources that share NOTHING
+# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some
+# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml.
+- name: Assess — usable memory vs the memory this machine physically has
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'memory.physical_total',
+      'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
+      'status': phys_status,
+      'detail': phys_detail }] }}"
+  vars:
+    ansible_become: false
+    th: "{{ node_health_thresholds }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+
+    # 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
+    #    actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
+    #    tools, identical on metal and in a VM — so this half is available almost always.
+    z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
+    z_status: >-
+      {{ 'SKIP' if z_pct is not number
+         else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
+               else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
+
+    # 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
+    #    VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
+    #    never as a zero DIMM count. Independent of the kernel's memory map entirely.
+    inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
+    d_pct: >-
+      {{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
+         if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
+         else none }}
+    d_status: >-
+      {{ 'SKIP' if d_pct is not number
+         else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
+               else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
+
+    # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
+    # SKIPs must never add up to a PASS.
+    phys_status: >-
+      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
+         | default('SKIP', true) }}
+    # What the kernel log says, IF it was read — quoted here only as attribution for a
+    # shortfall this check measured on its own. The verdict above never depends on it.
+    phys_cause: >-
+      {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
+           firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
+           if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
+         else ('. The kernel log was NOT read here, so this shortfall is unattributed —
+                see check firmware.efi_memattr')
+           if not (node_health_facts.klog.available | bool)
+         else '' }}
+    phys_detail: >-
+      {{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
+           ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
+           ~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
+             was never handed to the allocator (/proc/zoneinfo)')
+          if z_pct is number
+          else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
+         ~ '. '
+         ~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
+             ~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
+             ~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
+             ~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
+             ~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
+            if d_pct is number
+            else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
+         ~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
+              reservations and (on this platform) 64 MiB of graphics stolen memory.
+              A shortfall of this size is memory the machine HAS and cannot use')
+            if phys_status in ['WARN', 'FAIL'] else '')
+         ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
+
 - name: Assess — swap (kubelet requires it off)
   ansible.builtin.set_fact:
     node_health_checks: "{{ node_health_checks + [{
@@ -674,3 +751,127 @@
                ~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
                ~ (' — serial numbers not read (root-only DMI fields)'
                   if d.product_serial is none else '')) }}
+
+# --- EFI Memory Attributes Table ------------------------------------------------------
+# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
+# what follows is only what this check needs to be read correctly.
+#
+# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
+# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c
+# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s
+# the product — everything from the table address to the top of RAM. The kernel prints
+# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field
+# happens to read > 2:
+#
+#   "Failed to map EFI Memory Attributes table @ 0x..."  version read <= 2, the kernel
+#       TRUSTED the table. The node has lost the memory. FAIL.
+#   "Unexpected EFI Memory Attributes table version <N>"  version read > 2, the kernel
+#       bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a
+#       corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS:
+#       k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself
+#       booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one
+#       unlucky boot from losing 83% of its RAM is not passing a health check.
+#
+# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
+# so it cannot be diagnosed once. It has to be measured on every health run.
+#
+# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
+# ONLY if the kernel log was actually read AND still reaches back to early boot — every
+# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
+# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
+# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
+# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
+# log at all, so a node whose log is unreadable is still not unmeasured.
+- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'firmware.efi_memattr',
+      'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
+      'status': memattr_status,
+      'detail': memattr_detail }] }}"
+  vars:
+    ansible_become: false
+    k: "{{ node_health_facts.klog }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    ma_state: "{{ k.memattr_state | default(none) }}"
+    memattr_status: >-
+      {{ 'SKIP' if not (k.available | bool)
+         else ('FAIL' if ma_state == 'failed_to_map'
+               else ('WARN' if ma_state == 'bad_version'
+                     else ('PASS' if ma_state == 'corrupt_rejected'
+                           else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
+    # One remediation, quoted by both findings. NOTE what it must NOT say: there is no
+    # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
+    # could take effect. Suggesting one would be a fix that quietly does nothing.
+    memattr_fix: >-
+      FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
+      upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
+      bogus", first released in v6.13 and absent from v6.12 and from the current
+      linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
+      There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
+      bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
+      unplugging USB only change where the table lands and may APPEAR to fix it — they are
+      a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+    # The symptom, measured independently of this log (see memory.physical_total).
+    memattr_corroboration: >-
+      {{ (' Measured independently of this log: MemTotal '
+          ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+          ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
+          ~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away
+            from the allocator (/proc/zoneinfo) — check memory.physical_total.')
+         if ((z.available | bool) and z.unmanaged_pct is number) else '' }}
+    # The kernel log lines themselves, verbatim. A verdict the operator cannot check
+    # against the evidence is not much better than a wrong one.
+    memattr_evidence: >-
+      {{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
+          if (k.memattr_lines | default([]) | length) > 0 else '')
+         ~ ((' At boot the kernel reported '
+             ~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+             ~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
+             ~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
+             ~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
+            if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
+            else '') }}
+    memattr_detail: >-
+      {%- if not (k.available | bool) -%}
+      the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
+      Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
+      lives only in dmesg.{{ memattr_corroboration }}
+      {%- elif ma_state == 'failed_to_map' -%}
+      THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory
+      Attributes table at {{ k.memattr_table_address | default('an unreported address') }}
+      and reserved memory from the corrupt table anyway — everything from that address
+      upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'bad_version' -%}
+      EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table
+      here too; this boot survived only because its garbage version field read
+      {{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out
+      before reserving anything. The outcome is re-decided on EVERY boot: if that field
+      reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right
+      now and this node is one unlucky boot from the failure state.{{ memattr_evidence }}
+      {{ memattr_fix }}
+      {%- elif ma_state == 'corrupt_rejected' -%}
+      the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
+      validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is
+      intact and stays intact across reboots; no action.{{ memattr_evidence }}
+      {%- elif not (k.boot_covered | bool) -%}
+      NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
+      back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
+      the ring buffer, which wraps). Every memattr diagnostic is printed within the first
+      fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
+      pass. Read /var/log/kern.log for this boot, or re-run after a
+      reboot.{{ memattr_corroboration }}
+      {%- elif k.memattr_published is sameas true -%}
+      the firmware publishes an EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }} and the kernel
+      accepted it without complaint, in a log that still reaches back to early boot. Not
+      affected on this boot.
+      {%- elif k.memattr_published is sameas false -%}
+      this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
+      lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
+      guest.
+      {%- else -%}
+      no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
+      so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
+      boot, so that absence is measured, not assumed.
+      {%- endif -%}

infra/ansible/roles/node_health/tasks/collect_privileged.yml

@@ -0,0 +1,62 @@
+---
+# The SECOND pass of the collector, run with escalation, for the two facts that are
+# genuinely root-gated:
+#
+#   * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an
+#     unprivileged read gets EPERM. This is where the EFI Memory Attributes Table
+#     diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference
+#     between a node that has lost 13.3 GiB and one that has not.
+#   * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed
+#     DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem.
+#
+# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a
+# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line-
+# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND
+# source of truth for the same fact, and the two would drift. `--privileged` makes the
+# collector emit exactly those subtrees and nothing else, so the parsing stays in one
+# place and is identical whichever pass produced it.
+#
+# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass
+# actually failed to read one of them (main.yml checks both). On a node where
+# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule
+# is that everything readable WITHOUT root is read without root.
+#
+# If escalation is unavailable the checks that consume these SKIP with the exact reason
+# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line"
+# and "could not read the kernel log" are different answers and the role says which.
+
+- name: Re-collect the root-only facts with escalation
+  ansible.builtin.script:
+    cmd: node_health_collect.py --privileged
+    executable: /usr/bin/python3
+  become: true
+  vars:
+    ansible_become: true
+  register: node_health_privileged_raw
+  changed_when: false   # strictly read-only; it can never change the node
+
+- name: Assert the privileged payload is the schema this role understands
+  ansible.builtin.assert:
+    that:
+      - (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1'
+    fail_msg: >-
+      Privileged collector pass returned an unexpected schema; role and collector are
+      out of step.
+    quiet: true
+  vars:
+    ansible_become: false
+
+# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive
+# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine
+# would REPLACE it and throw away the reasons the first pass recorded.
+- name: Merge the privileged facts over the unprivileged ones
+  ansible.builtin.set_fact:
+    node_health_facts: >-
+      {{ node_health_facts | combine({
+           'klog': priv.klog,
+           'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}),
+           'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])),
+         }) }}
+  vars:
+    ansible_become: false
+    priv: "{{ node_health_privileged_raw.stdout | from_json }}"

infra/ansible/roles/node_health/tasks/main.yml

@@ -4,9 +4,11 @@
 # ORDER OF OPERATIONS
 #   1. Collect everything readable WITHOUT root (one script, one round trip).
 #   2. Probe whether privilege escalation actually works on this host.
-#   3. Only if it does AND smartmontools is installed: read SMART.
+#   3. Only if it does, and only for what the first pass could NOT read: re-run the same
-#   4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#      collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
-#   5. Render the human summary + write the JSON artifact.
+#   4. Only if it does AND smartmontools is installed: read SMART.
+#   5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#   6. Render the human summary + write the JSON artifact.
 #
 # BECOME HANDLING — the subtle bit, read before editing.
 #   `ansible_become` set as an inventory/group var (which is exactly what
@@ -43,6 +45,16 @@
   ansible.builtin.include_tasks: probe_privilege.yml
   when: node_health_try_privileged | bool
 
+# Only when the unprivileged pass actually failed to read one of them. On a node with
+# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
+# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
+- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
+  ansible.builtin.include_tasks: collect_privileged.yml
+  when:
+    - node_health_root_available | bool
+    - (not (node_health_facts.klog.available | bool))
+      or (not (node_health_facts.memory.devices.available | bool))
+
 - name: Read SMART data (privileged)
   ansible.builtin.include_tasks: collect_smart.yml
   when:

infra/ansible/roles/node_health/templates/summary.txt.j2

@@ -45,6 +45,15 @@
 {% endif %}
  Memory       {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
  ECC/EDAC     {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
+{% set mz = node_health_facts.memory.zones %}
+{% set md = node_health_facts.memory.devices %}
+ Present RAM  {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }}
+ DIMMs        {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }}
+{% for dimm in md.devices | default([]) if dimm.size_kb %}
+              {{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }}
+{% endfor %}
+{% set klog = node_health_facts.klog %}
+ EFI memattr  {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
  Load         {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
 
 --------------------------------------------------------------------------------

platform/kustomization.yaml

@@ -12,7 +12,15 @@ kind: Kustomization
 resources:
   # ---- Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress ----
   - kyverno            # admission enforcer FIRST — policies present before other components reconcile
-  - cilium             # CNI (adopts the running Ansible-installed release; parity verified)
+  #      cilium (the CNI) is DELIBERATELY NOT here: it now reconciles in its OWN Flux Kustomization
+  #      (clusters/base/cilium.yaml, dependsOn flux-system, wait:true), and `platform` dependsOn IT.
+  #      Not the headlamp/oauth2-proxy rationale (a leaf that must not gate) but its mirror image:
+  #      the CNI is the one component that SHOULD gate everything, and inside this shared tier its
+  #      health was indistinguishable from eleven unrelated components, the only emergency valve was
+  #      freezing the whole tier, and the tier's 10m/5m interval/timeout are wrong for a 3-node
+  #      DaemonSet rollout. Gating is preserved via the explicit `platform.dependsOn: [cilium]` edge.
+  #      NOTE: platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system in
+  #      this tree — that is why the move required a prune-disabled transition. See DECISIONS-LOG.
   - longhorn           # replicated block storage / CSI
   - external-snapshotter  # CSI snapshot CRDs + snapshot-controller (VolumeSnapshot API; CDI smart-clone)
   - cert-manager       # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token`