feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex

*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***

Operator simplified the architecture, inverting c8c1d29:
  dev  -> *.dev.home.keithjackson.dev        (was the apex)
  prod -> *.home.keithjackson.dev            (the apex; no prod. subdomain, no CNAME layer)

WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.

Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.

VALUES
                BASE_DOMAIN                  TXT_OWNER_ID   DOMAIN_FILTER
  base          dev.home.keithjackson.dev    sentinel       keithjackson.dev
  dev           dev.home.keithjackson.dev    dev            keithjackson.dev
  prod          home.keithjackson.dev        prod           keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.

dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.

DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.

VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.

THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.

Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.

Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.

RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.

ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.

The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.

Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/vms/unifi/30-virtualmachine.yaml

@@ -102,7 +102,7 @@ spec:
             userData: |-
               #cloud-config
               hostname: unifi
-              fqdn: unifi.home.keithjackson.dev
+              fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
               packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]

apps/vms/unifi/50-service-l4.yaml

@@ -46,7 +46,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
     # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
     # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
-    external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

apps/wave-0/gitea.yaml

@@ -90,9 +90,9 @@ spec:
               name: gitea-bootstrap
               key: secret-key
         - name: GITEA__server__DOMAIN
-          value: gitea.home.keithjackson.dev
+          value: gitea.${BASE_DOMAIN}
         - name: GITEA__server__ROOT_URL
-          value: https://gitea.home.keithjackson.dev/
+          value: https://gitea.${BASE_DOMAIN}/
         - name: GITEA__actions__ENABLED
           value: 'true'
         - name: GITEA__actions__DEFAULT_ACTIONS_URL
@@ -127,7 +127,7 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
                 \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
@@ -207,7 +207,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - gitea.home.keithjackson.dev
+  - gitea.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/jackett.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:9117
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - jackett.home.keithjackson.dev
+  - jackett.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/pihole.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:80
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -229,7 +229,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - pihole.home.keithjackson.dev
+  - pihole.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/plex.yaml

@@ -183,7 +183,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - plex.home.keithjackson.dev
+  - plex.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/qbittorrent.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - qbittorrent.home.keithjackson.dev
+  - qbittorrent.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/gitea-actions.yaml

@@ -165,7 +165,7 @@ spec:
               exec /sbin/tini -- /opt/act/run.sh
           env:
             - name: GITEA_INSTANCE_URL
-              value: https://gitea.home.keithjackson.dev
+              value: https://gitea.${BASE_DOMAIN}
             - name: GITEA_RUNNER_REGISTRATION_TOKEN
               valueFrom:
                 secretKeyRef:

apps/wave-1/nextcloud.yaml

@@ -104,13 +104,13 @@ spec:
               name: nextcloud-bootstrap
               key: admin-password
         - name: NEXTCLOUD_TRUSTED_DOMAINS
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEHOST
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEPROTOCOL
           value: https
         - name: OVERWRITECLIURL
-          value: https://nextcloud.home.keithjackson.dev
+          value: https://nextcloud.${BASE_DOMAIN}
         - name: NEXTCLOUD_OIDC_CLIENT_SECRET
           valueFrom:
             secretKeyRef:
@@ -125,9 +125,9 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
-                [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
-                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc]\
+                \ client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
+                ); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -228,7 +228,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - nextcloud.home.keithjackson.dev
+  - nextcloud.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/radarr.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:7878
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - radarr.home.keithjackson.dev
+  - radarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/semaphore.yaml

@@ -110,7 +110,7 @@ spec:
         - name: SEMAPHORE_ADMIN_NAME
           value: Semaphore Admin
         - name: SEMAPHORE_ADMIN_EMAIL
-          value: admin@home.keithjackson.dev
+          value: admin@${BASE_DOMAIN}
         - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
           valueFrom:
             secretKeyRef:
@@ -231,7 +231,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - semaphore.home.keithjackson.dev
+  - semaphore.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/sonarr.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - sonarr.home.keithjackson.dev
+  - sonarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-2/seerr.yaml

@@ -223,7 +223,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - seerr.home.keithjackson.dev
+  - seerr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

bootstrap/flux/bootstrap.sh

@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #
-# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
+# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
+# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
+# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
+# same script bootstraps dev and prod with no fork (R-ENV-02).
 #
 # Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
 # =============================================================================
@@ -33,7 +36,7 @@
 #   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
+#   3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
 set -euo pipefail
 
 # --- config (env-overridable) ------------------------------------------------
-FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"     # dev is the ONLY Phase-1 build target
+# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
+# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
+# every existing caller — override it, never edit it.
+FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
 FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
 SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
 # The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,7 +68,7 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
 
-echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
+echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
 
 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
 for bin in flux kubectl; do
@@ -134,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
   exit 1
 fi
 
-# --- 3. apply the authored dev entrypoint ------------------------------------
+# --- 3. apply the authored environment entrypoint ----------------------------
 echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
 

bootstrap/gitea/bootstrap-gitea.sh

@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 # bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
-# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
-# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
-# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
-# CONTRIBUTING.md).
+# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
+# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
+# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
+# using it now.
 #
-# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
+# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
+# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
+#
+#   PROD (bare metal, current use):  HOST_NAT_IP=192.168.2.132
+#       The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
+#       NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
+#       three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
+#       apply to prod, so do not "fix" prod by opening zones.
+#
+#   DEV (libvirt VMs, HISTORICAL):   HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
+#       *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
+#       bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
+#       recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
+#
+# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
+# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
+# author-only work and must never be fired off casually or "to see if it works": it starts a
+# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
+# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
+# it will then reconcile. Know which HEAD you are publishing before you run it.
 set -euo pipefail
 ENGINE="${ENGINE:-podman}"
 GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}"   # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
 # subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
 DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
 DATA_DIR="podman volume ${DATA_VOLUME}"   # display label only (data lives in the named volume)
-# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
+# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
 # clone/redirect URLs are correct from inside the cluster.
-# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
+#
+# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
+# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
+# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
+# Pass it explicitly, e.g. for prod:   HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
+#
+# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
+# VM->host:3000 by default and had to be opened once with
 #   sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
-HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
+# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
+if [ -z "${HOST_NAT_IP:-}" ]; then
+  echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
+  echo "       prod: HOST_NAT_IP=192.168.2.132   (same LAN as the nodes, no NAT, no firewall change)" >&2
+  echo "       There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
+  exit 1
+fi
 ORG="${GITEA_ORG:-Home-Network}"
 REPO="${GITEA_REPO:-home-network-k3s-impl}"
 
@@ -75,8 +107,17 @@ cat <<EOF
 
 Gitea bootstrap source ready (${DATA_DIR}).
   UI (host):                     ${BASE}/${ORG}/${REPO}
-  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git   (clusters/dev)
+  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
+                                 (patch this url into the overlay being bootstrapped:
+                                  clusters/prod today; clusters/dev has already cut over)
   Admin creds (host, 0600):      ${ADMIN_CRED_FILE}
-Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
+Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
+firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
+must allow VM->host:${HTTP_PORT} —
   ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
+REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
+at this source will reconcile THAT revision.
+TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
+any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
+procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
 EOF

catalog/gitea.yaml

@@ -24,9 +24,9 @@ env:
         name: gitea-bootstrap
         key: secret-key
   - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.home.keithjackson.dev
+    value: gitea.${BASE_DOMAIN}
   - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
   - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
     value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
   - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,7 +104,7 @@ lifecycle:
           # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
           # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
           # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           gitea_oidc_fail() {
             echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
@@ -168,7 +168,7 @@ lifecycle:
           # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
           # the moment its OIDC guard was fixed and the block actually ran for the first time.
           exit 0
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: data
@@ -178,7 +178,7 @@ storage:
                                 # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
     port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)

catalog/jackett.yaml

@@ -17,7 +17,7 @@ name: jackett
 kind: container
 source:
   image: lscr.io/linuxserver/jackett:latest
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
@@ -25,7 +25,7 @@ storage:
     class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
     port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)

catalog/keycloak.yaml

@@ -18,7 +18,7 @@ dependencies:
     port: 5432
     protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
     port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
   mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.

catalog/nextcloud.yaml

@@ -50,13 +50,13 @@ env:
   #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
   #     though it speaks plain HTTP behind the proxy. ---
   - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEPROTOCOL
     value: https
   - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
   # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
   #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
   #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
           # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
           # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
           OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           nextcloud_oidc_fail() {
             echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -228,7 +228,7 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
     port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/pihole.yaml

@@ -14,7 +14,7 @@ storage:
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)

catalog/plex.yaml

@@ -72,7 +72,7 @@ storage:
                               # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
     port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
   mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/qbittorrent.yaml

@@ -32,7 +32,7 @@ name: qbittorrent
 kind: container
 source:
   image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
@@ -40,7 +40,7 @@ storage:
     class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
     port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)

catalog/radarr.yaml

@@ -9,7 +9,7 @@ name: radarr
 kind: container
 source:
   image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)

catalog/render.py

@@ -44,7 +44,16 @@ except ImportError:  # pragma: no cover - explicit, actionable failure
 
 # --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
 LBL = "homelab.keithjackson.dev"
-ZONE = "home.keithjackson.dev"
+# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
+# Every rendered manifest lands under a Flux path whose Kustomization declares
+# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
+# reconcile time: dev renders dev.home.keithjackson.dev, prod renders home.keithjackson.dev (the
+# APEX — operator decision 2026-09-04; prod IS the apex, so there is no `prod.` label and no CNAMEs).
+# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
+# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
+# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
+# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
+ZONE = "${BASE_DOMAIN}"
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
@@ -598,7 +607,7 @@ def r_internet_egress(s: dict) -> dict:
 def r_httproutes(s: dict) -> list:
     """One HTTPRoute per ingressRoute, attaching host -> service port on the shared Gateway.
 
-    TLS is the Gateway's wildcard *.home.keithjackson.dev cert (no per-service cert step);
+    TLS is the Gateway's wildcard *.${BASE_DOMAIN} cert (no per-service cert step);
     ExternalDNS manages the record from the HTTPRoute (R-TLS/R-DNS, renderer-contract).
     """
     out = []

catalog/seerr.yaml

@@ -57,7 +57,7 @@ dependencies:
     port: 32400
     protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
     port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/semaphore.yaml

@@ -58,7 +58,7 @@ env:
   - name: SEMAPHORE_ADMIN_NAME
     value: Semaphore Admin
   - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
   # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
   #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
   - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -110,7 +110,7 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
     port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/sonarr.yaml

@@ -9,7 +9,7 @@ name: sonarr
 kind: container
 source:
   image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)

clusters/base/cluster-vars.yaml

@@ -17,15 +17,103 @@ data:
   CLUSTER_ENV: "base"                       # dev | prod  (overlays set this)
 
   # --- DNS / TLS ---
-  BASE_DOMAIN: "home.keithjackson.dev"      # platform hostname suffix (same both envs; split-horizon)
-  DNS_ZONE: "keithjackson.dev"              # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
-                                            # ExternalDNS domain-filter must be the zone, not the subdomain
+  # THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
+  # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
+  # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
+  #
+  # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
+  # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
+  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
+  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
+  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
+  # suffix, so the two clusters can no longer describe the same name.
+  #
+  #   dev   -> dev.home.keithjackson.dev        (a subdomain; = this base default)
+  #   prod  -> home.keithjackson.dev            (THE APEX — prod IS the apex, no `prod.` label)
+  #
+  # OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
+  # had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
+  # whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
+  # DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
+  # default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
+  # string dev holds.
+  #
+  # DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
+  # now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
+  # URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
+  # push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
+  # Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
+  # cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
+  # infra/PROD-PROVISIONING.md §10.
+  #
+  # DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
+  # `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
+  BASE_DOMAIN: "dev.home.keithjackson.dev"
+
+  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
+  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
+  # so it cannot be created, updated or deleted regardless of TXT ownership.
+  #
+  # BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
+  # environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
+  #
+  # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
+  # SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+  # matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
+  # record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
+  # narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
+  # it — the change is not defence in depth, it is a total outage of DNS publication.
+  #
+  # WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
+  #   (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
+  #   (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
+  # That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
+  # both environments live in the one zone either way.
+  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
+
+  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
+  # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
+  # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
+  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
+  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
+  #
+  #   dev -> `dev`      prod -> `prod`      base default -> a non-colliding sentinel (below)
+  #
+  # DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
+  # this migration dev HAD to render `default`, because its LIVE apex records were already registered
+  # in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
+  # THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
+  # *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
+  # is now free to be chosen on its merits. Two reasons to take `dev`:
+  #   - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
+  #     clusters silently shared it, which IS the incident). An environment whose identity is the
+  #     forget-value can never be told apart from a misconfigured newcomer. No environment should
+  #     hold it.
+  #   - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
+  #     owned by `default`, which no environment now claims, so nothing deletes them implicitly on
+  #     some future reconcile — they are retired deliberately, and prod's publish is gated on that
+  #     retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
+  #     race between two reconcile loops.
+  #
+  # PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
+  # dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
+  # the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
+  # would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
+  # roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
+  # environment's id. A new environment that forgets this knob must not silently inherit another's
+  # identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
+  # can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
+  # instead of destroying a live environment's DNS.
+  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
+
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
   GATEWAY_VIP: "192.168.1.53"                # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
                                             # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
                                             # takes the pool's first free IP, and coredns-custom answers
-                                            # *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
+                                            # *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
                                             # hairpins through the Gateway (correct SNI -> wildcard cert).
                                             # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
                                             # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
@@ -74,6 +162,17 @@ data:
 
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
+  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
+  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
+  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
+  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
+  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
+  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
+  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
+  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
+  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
+  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
+  LONGHORN_DATA_PATH: "/var/lib/longhorn"
 
   # --- observability sizing (docs/10) ---
   MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long

clusters/dev/kustomization.yaml

@@ -5,6 +5,20 @@
 # reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
 # It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
 # on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
+#
+# ##############################################################################################
+# ###  *** DO NOT MERGE THIS FILE TO `main` UNATTENDED (as of 2026-09-04) ***                 ###
+# ###                                                                                        ###
+# ###  This overlay now carries dev's DNS MIGRATION target: BASE_DOMAIN moves from the apex   ###
+# ###  `home.keithjackson.dev` to `dev.home.keithjackson.dev`, and the GitRepository url       ###
+# ###  moves with it. sync.yaml reconciles ./clusters/dev from `main` EVERY 10 MINUTES, so a   ###
+# ###  merge to main IS the cutover — it executes on the live dev cluster with nobody present. ###
+# ###  Dev's Gitea is both its GitOps source and the push target: if the new host is not up    ###
+# ###  with a Ready cert first, dev stops reconciling and pushing cannot repair it.            ###
+# ###                                                                                        ###
+# ###  Run the ordered runbook: infra/PROD-PROVISIONING.md §10 (steps D1-D8), operator        ###
+# ###  present. Merging is step D3, not step one.                                             ###
+# ##############################################################################################
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
@@ -131,7 +145,24 @@ patches:
         # via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
         # secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
         # revert this url (docs/05).
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        #
+        # *** THE STRAND. THIS URL IS THE MOST DANGEROUS LINE IN THE REPOSITORY. ***
+        # It is BOTH dev's GitOps source AND the operator's push target, and it is NOT tokenized —
+        # it cannot be: the `flux-system` root Kustomization that renders ./clusters/dev declares NO
+        # postBuild.substituteFrom, so `gitea.${BASE_DOMAIN}` here would ship the LITERAL token and
+        # break the source outright (the L0 flux-postbuild-substitution check asserts exactly this).
+        # So it is a hand-maintained literal that must be moved in lockstep with BASE_DOMAIN below.
+        #
+        # It has been moved to dev's NEW suffix as part of the 2026-09-04 migration authoring. That
+        # makes this file the TARGET state, NOT a safe unattended merge: dev's root Kustomization
+        # (sync.yaml) reconciles ./clusters/dev from `main` every 10m, so MERGING THIS TO main WHILE
+        # DEV IS LIVE MIGRATES DEV UNATTENDED. If gitea.dev.home.keithjackson.dev does not already
+        # resolve and serve a valid cert at that instant, dev stops reconciling and CANNOT BE FIXED
+        # BY PUSHING — the broken thing is the path Flux uses to read pushes.
+        # ORDERING THAT AVOIDS IT (infra/PROD-PROVISIONING.md §10, steps D1-D3): suspend dev's root
+        # Kustomization or park this url on the bootstrap Gitea (http://192.168.2.132:3000/...) for
+        # the migration window, and only restore the in-cluster url after the new host is proven.
+        url: https://gitea.dev.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   # --- dev-scoped cluster-vars (R-ENV-02) ---
@@ -148,6 +179,60 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # ==========================================================================================
+      # DEV DNS IDENTITY (R-DNS-01/02/03) — DEV MOVES OFF THE APEX. **NOT YET APPLIED.**
+      # ==========================================================================================
+      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day:
+      #
+      #   dev   -> dev.home.keithjackson.dev      (this knob; = the base default, restated)
+      #   prod  -> home.keithjackson.dev          (THE APEX — prod IS the apex, no `prod.` label,
+      #                                            and therefore no apex CNAME layer at all)
+      #
+      # *** THIS VALUE IS AHEAD OF THE LIVE CLUSTER. *** The dev cluster is serving the APEX right
+      # now. Merging this file does not perform the migration — it DEFINES the target. The cutover is
+      # an ORDERED, OPERATOR-PRESENT runbook: infra/PROD-PROVISIONING.md §10.
+      #
+      # WHY IT CANNOT BE A PLAIN VARIABLE FLIP — THE STRAND. `spec.url` of the GitRepository ~20
+      # lines above is https://gitea.home.keithjackson.dev/... . Dev's Gitea is BOTH dev's GitOps
+      # source AND the operator's push target. The moment BASE_DOMAIN moves, dev's Gitea HTTPRoute,
+      # DNS record and wildcard cert all move to gitea.dev.home.keithjackson.dev. If that host does
+      # not ALREADY resolve with a valid, Ready cert, dev stops reconciling — and it CANNOT BE FIXED
+      # BY PUSHING, because the thing that is broken is the path Flux uses to read pushes. Recovery
+      # is then hands-on (point the source at the bootstrap Gitea on 192.168.2.132:3000, or repair
+      # in-cluster by hand). §10 states the ordering that avoids it: publish and verify the new host
+      # FIRST, or park the source on the bootstrap Gitea for the window.
+      #
+      # ALSO MOVED BY THIS ONE LINE, all on a live cluster: the `*.` wildcard Certificate (a fresh
+      # ACME order), Keycloak's issuer URL and ~12 OIDC clients' redirect URIs, kubectl SSO (a k3s
+      # server flag => a rolling restart + every user re-issuing their kubeconfig), coredns-custom's
+      # split-horizon zone, Homepage's links, and 6 L2 chainsaw tests that hardcode the apex.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "dev.home.keithjackson.dev"
+      # ExternalDNS scope. MUST stay the Cloudflare ZONE — NOT dev.home.keithjackson.dev. FINDING
+      # F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this for ZONE SELECTION, so a
+      # strict-subdomain filter matches no zone and publishes NOTHING ("no hosted zone matching
+      # record DNS Name"). Dev moving to a subdomain does NOT make narrowing safe — it makes it
+      # newly tempting, which is why this comment is here. Same value the retired DNS_ZONE knob held.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        value: "keithjackson.dev"
+      # OWNER ID: `dev` (was `default` until 2026-09-04). The old value was FORCED — dev's live apex
+      # records were already in Cloudflare's TXT registry under owner `default`, so renaming would
+      # have orphaned every one of them. THE MIGRATION DISSOLVES THAT CONSTRAINT: on the new suffix
+      # dev publishes a brand-new record set with no prior registry entries to recognise, so the id
+      # is chosen on its merits, and `default` has none — it is the value external-dns uses when you
+      # FORGET the flag, which is exactly how both clusters came to share it. Consequences, both
+      # deliberate:
+      #   - dev's OLD apex records remain owned by `default`, which no environment now claims, so
+      #     nothing deletes them implicitly. Retiring them is an explicit §10 step, and prod's first
+      #     publish is gated on that retirement being CONFIRMED — not on two reconcile loops racing.
+      #   - prod stays `prod` and must NEVER be given `default`, or prod would own the live dev
+      #     cluster's apex records and sync-delete them. Prod inherits the apex NAMES, not its OWNER.
+      # dev != prod is the load-bearing invariant; the base default stays a non-colliding sentinel.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "dev"
       # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
       # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
       # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -156,7 +241,7 @@ patches:
       #
       # Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
       # (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
-      # *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
+      # *.dev.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
       - op: replace
         path: /data/GATEWAY_VIP
         value: "192.168.1.53"
@@ -190,6 +275,16 @@ patches:
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
+      # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
+      # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
+      # else would orphan every one of them (they are not recoverable from the old path by
+      # rescheduling). Restated explicitly even though it equals the base default, exactly like the
+      # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
+      # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/var/lib/longhorn"
       # smaller retention / volumes to keep the dev workstation footprint light.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

clusters/prod/kustomization.yaml

@@ -115,24 +115,78 @@ patches:
             name: sops-age
   # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
   #
-  # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
+  # ############################################################################################
+  # ###                                                                                      ###
+  # ###   *** T E M P O R A R Y   B O O T S T R A P   S O U R C E  —  NOT PERMANENT ***       ###
+  # ###                                                                                      ###
+  # ###   The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside   ###
+  # ###   any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST   ###
+  # ###   be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this  ###
+  # ###   and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is  ###
+  # ###   a live single point of failure, not a settled design. See "CUTOVER" below.          ###
+  # ###                                                                                      ###
+  # ############################################################################################
   #
-  # Two independent reasons, both consequences of the split-horizon design:
-  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
-  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
-  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
-  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
-  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
-  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
-  #      gitea.home.keithjackson.dev to itself and never reach a repository.
+  # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
+  # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
+  # REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
+  # which changes WHY the blocker holds but not THAT it holds:
+  #   1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
+  #      pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
+  #      inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
+  #      BEFORE dev is retired, or prod loses its source of truth.
+  #      NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
+  #      gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
+  #      the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
+  #      cutover below a one-line change once prod actually runs Gitea.
+  #   2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
+  #      easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
+  #      ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
+  #      prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
+  #      listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
+  #      to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
+  #      remains is purely reason 1 — the Gitea is not there YET.
   #
-  # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
-  # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
-  # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
+  # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
+  #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
+  #   operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
+  #   reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
+  #   change was needed. This is the same "local bootstrap source" pattern dev itself used before
+  #   its Phase-6 cutover (docs/05 "Source-of-truth").
+  #
+  # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
+  # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
+  # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
+  # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
+  #
+  # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
+  # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
+  #   curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
+  #   git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git  -> refs/heads/main
+  # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
+  # THAT FIRST: a private repo here fails exactly that way.
+  #
+  # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
+  #   WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
+  #         through prod's own Gateway VIP with a publicly-trusted cert.
+  #   DO:   1. push/mirror this repo into the prod in-cluster Gitea;
+  #         2. change the url below to
+  #            https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+  #            and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
+  #            clusters/dev/kustomization.yaml, is what "done" looks like);
+  #         3. confirm `flux get sources git -n flux-system` reconciles the new revision;
+  #         4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
+  #            exactly as dev retained its bootstrap Gitea.
+  #   UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
+  #   powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
+  #   it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
   #
   # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
   # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
   # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
+  # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
+  # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
+  # is the revision you intend prod to run (`git ls-remote` above, against local `main`).
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository
@@ -140,7 +194,9 @@ patches:
         name: flux-system
         namespace: flux-system
       spec:
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
+        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
+        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   - target:
@@ -154,9 +210,96 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # ==========================================================================================
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
+      # ==========================================================================================
+      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
+      # (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
+      #
+      #   dev   -> dev.home.keithjackson.dev     (dev carries the label; clusters/dev)
+      #   prod  -> home.keithjackson.dev         (THE APEX — this knob)
+      #
+      # There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
+      # name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
+      # Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
+      #
+      # BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
+      # HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
+      # EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
+      # consequences, one of which already happened:
+      #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
+      #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
+      #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
+      #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
+      #      have raced each other's `_acme-challenge` TXT records on that one zone.
+      #
+      # *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
+      # Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
+      # therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
+      # tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
+      #   (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
+      #   (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
+      #       (verified against Cloudflare directly — a pfSense override is NOT evidence about public
+      #       DNS; that mistake was made on 2026-09-04), AND
+      #   (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
+      #       that same name set while dev's still exists counts against LE's duplicate-certificate
+      #       limit (5 per week per identical name set).
+      # Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
+      # the 2026-09-04 incident with the roles reversed.
+      #
+      # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
+      # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
+      # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
+      # trusts a different Keycloak than the one prod actually runs.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "home.keithjackson.dev"
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        # MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
+        # like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
+        # F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
+        # zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
+        # record DNS Name", so prod publishes NOTHING. An agent narrowed this to
+        # prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
+        # Isolation from dev does NOT come from this filter — it comes from
+        # (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
+        # (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
+        # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
+        value: "keithjackson.dev"
+      # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
+      # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
+      # its own and, under policy: sync, deleted the records behind them.
+      #
+      # PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
+      # NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
+      # the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
+      # prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
+      # never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
+      # invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
+      # simply fails to publish until those records are retired, which is the correct behaviour.
+      # dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
+      # non-colliding sentinel. All three differ.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "prod"
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
+      # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
+      # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
+      # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
+      # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
+      # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
+      #
+      # [parity] MUST equal node_baseline_longhorn_data_path in
+      # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
+      # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
+      # path on the OS filesystem instead of the data partition and /var fills up.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/data/longhorn"
       # longer retention / larger volumes on prod NVMe.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

conformance/.gitignore

@@ -8,3 +8,22 @@ _artifacts/
 
 # Python bytecode from lib/ (status.py imports coverage.py); never committed.
 __pycache__/
+
+# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
+# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
+# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
+# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
+# fact, so it must survive in git.
+#
+# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
+# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
+# the `conformance/runs/` directory itself, so re-including the files here works (git
+# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
+# live under _artifacts/).
+#
+# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
+# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
+# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
+!runs/
+!runs/**
+!runs/**/*.junit.xml

conformance/README.md

@@ -22,6 +22,7 @@ behavioral / L3 fault-injection** (this harness).
 ./run.sh --lint                       # structural-validate test defs (no cluster)
 ./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
 ./run.sh --require-all                # phase-exit gate: every requirement must be covered
+./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
 ```
 
 **Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -105,13 +106,89 @@ lib/
   common.sh                  tagging convention + test discovery/selection helpers
   coverage.py                requirement-coverage generator (stdlib only)
   status.py                  requirement-STATUS generator (stdlib only; --selftest)
+  record_run.py              run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
   emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
+runs/
+  ledger.jsonl               TRACKED append-only proof-of-pass ledger (one row per
+                             testcase per run) — the only source status.py trusts
+  <UTC>-<short-sha>/         TRACKED per-run evidence: run.log + report.junit.xml
 tests/
   l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
 incluster/
   chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)
 ```
 
+## Proof of pass — the run ledger (`runs/`)
+
+`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
+proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
+root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
+only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
+asked for — **a committed, machine-readable run ledger**.
+
+```bash
+./run.sh --record                     # runs normally, and captures the evidence
+# -> conformance/runs/<UTC>-<short-sha>/run.log            (tee'd, carries the sentinels)
+# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml   (that run's JUnit)
+# -> conformance/runs/ledger.jsonl                         (one appended row per testcase)
+python3 lib/record_run.py --run-dir runs/<id> [--dry-run]  # or record an existing run dir
+python3 lib/record_run.py --selftest                       # deterministic, no cluster
+```
+
+**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
+that silently records itself is how a bad run becomes a permanent false proof. **Commit
+the run dir and the ledger** — an uncommitted ledger proves nothing.
+
+### Why a committed JUnit alone would be WORSE than nothing
+
+The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
+`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
+and `exit 0`, and their `check:` accepts either outcome —
+
+```yaml
+(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+ || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+```
+
+So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
+A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
+R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
+
+**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
+`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
+`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
+verdict token), so the table below is never hand-maintained:
+
+| Test | pass token (proof) | skip token (never proof) |
+|---|---|---|
+| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
+| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
+| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
+| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
+| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
+
+A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
+token present in that test's log lines, and **no** skip token anywhere in them. A skip
+token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
+`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
+non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
+skips as JUnit `<skipped/>`, never as a green pass.
+
+### Staleness — a proof is a claim about a tree, not a badge
+
+`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
+last commit touching that requirement's implementing manifests or its verifying test**.
+Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
+dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
+it never counts. Re-run with `--record` to refresh.
+
+A requirement reads `proven` only when **every** verifying test does; one skipped, failed
+or unrecorded test drags the whole requirement down. With an empty ledger `--status`
+behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
+
 ## How CI invokes it
 CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
 

conformance/lib/record_run.py

@@ -0,0 +1,949 @@
+#!/usr/bin/env python3
+"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
+
+WHY THIS EXISTS
+---------------
+`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
+fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
+`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
+`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
+that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
+and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
+tracked (see conformance/.gitignore for the negations that let it survive).
+
+THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
+------------------------------------------------------------------
+The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
+StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
+`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
+injection/dr-restore:
+
+    (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+     || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+
+So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
+(verified against the committed reports). Therefore:
+
+  *** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
+
+AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
+------------------------------------------------------
+Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
+it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
+the log of every run of that test. A section-blind grep therefore reads a skip token out
+of the source of every skip-gated test and can never record `proven` for any of them: the
+ledger fails closed so completely that it records nothing and proves nothing. Tokens are
+consequently read from SECTIONS (see classify_sections): a pass token only from
+`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
+pass — within genuine output, which is the only place either was ever evidence.
+
+That is the fake-green this project refuses, and it would be worse than no ledger at all
+because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
+PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
+files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
+unless the log carries that test's own pass token and carries NO skip token.
+
+HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
+--------------------------------------------------------
+Hard-coding a token table would rot the moment a test is edited. Instead every token is
+parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
+`check:` block is a declared sentinel. Classification is by the project's naming convention
+(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
+— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
+and this tool follows automatically; delete them and the test simply becomes unprovable
+rather than silently proven.
+
+  node-loss   pass NODE_LOSS_VOLUME_AVAILABLE_OK              skip NODE_LOSS_SKIPPED_OK
+  dr-restore  pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK     skip DR_RESTORE_SKIPPED_OK
+              (+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
+
+WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
+-----------------------------------------------------------
+The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
+dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
+any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
+pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
+can express "skipped" as "passed", and only they require a sentinel — see
+SENTINEL_REQUIRED_LAYERS.
+
+LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
+  ts_utc            run-dir timestamp, else the JUnit's mtime      (when)
+  git_sha           `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
+                    modified — a proof captured from a dirty tree is NOT honoured by
+                    status.py, because what ran is not what is committed  (what)
+  env               --env / CONF_ENV (cluster overlay the run targeted)
+  chainsaw_version  CHAINSAW_VERSION from versions.env (the pin IS the truth)
+  kube_version      `kubectl version` server gitVersion, else "unknown"
+  layer             L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
+  test              testcase name (== the test dir basename, asserted by convention)
+  requirements[]    the `conformance.home.k3s/R-*` labels on the test — the SAME
+                    authoritative source coverage.py uses, not the JUnit classname
+  junit_result      passed | failed | error | skipped
+  sentinel          the DECIDING token grepped from run.log, or null
+  proof             proven | skipped | failed | junit-skipped
+                    | unknown-no-sentinel-declared | unknown-sentinel-absent
+  run_id            the runs/<dir> name (provenance: which log backs this row)
+  note              one line saying WHY this row landed on that proof state
+
+`proven` is emitted on exactly one path (see decide()); every other path is non-green.
+
+RECORDING IS EXPLICIT, NEVER IMPLICIT
+-------------------------------------
+Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
+how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
+ledger row at all.
+
+EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
+  0  rows appended.
+  1  a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
+  3  the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
+     NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
+
+Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
+status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+_LIB = Path(__file__).resolve().parent
+if str(_LIB) not in sys.path:
+    sys.path.insert(0, str(_LIB))
+# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
+# tree, and the lint stage scans untracked files. Same guard as status.py.
+sys.dont_write_bytecode = True
+import coverage as cov  # noqa: E402  — reuse the registry/test parsers, don't fork them
+
+CONF_ROOT = _LIB.parent
+REPO_ROOT = CONF_ROOT.parent
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
+# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
+# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
+# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
+# by JUnit alone.
+SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
+
+# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
+# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
+# harmless for its own report, but here a mis-read layer decides whether a sentinel is
+# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
+_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
+                     ("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
+                     ("/meta/", "META"))
+
+
+def layer_of(path: Path) -> str:
+    rel = str(path)
+    for frag, layer in _LAYER_BY_SUBTREE:
+        if frag in rel:
+            return layer
+    return ""
+
+# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
+# Both quote styles are accepted; the repo uses single quotes throughout.
+_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
+
+# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
+# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
+# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
+# mis-classifying a SKIP token as a pass would manufacture one.
+_SKIP_TOKEN_RE = re.compile(r"_SKIP")
+
+# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
+# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
+#     l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
+# Field 2 is the test name; it is how output is attributed back to a testcase. The
+# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
+_LOG_HEADER_RE = re.compile(
+    r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
+    r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
+
+# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
+# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
+
+
+def _strip_noise(line: str) -> str:
+    return _ANSI_RE.sub("", line).replace("\b", "")
+
+
+# ===================================================================== output sections
+# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
+# -----------------------------------------------------------------------------
+# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
+# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
+# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
+# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
+# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
+# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
+# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
+# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
+# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
+#
+# So tokens are attributed to SECTIONS, never to raw lines:
+#     l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |     <- header, no section
+#         === COMMAND                                            <- marker (indent 8)
+#         /usr/bin/sh -c set -euo pipefail                       <- source, indent >= 8
+#           echo NODE_LOSS_SKIPPED_OK                            <- SOURCE, not output
+#     l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |    <- indent 4 => section ends
+#         === STDOUT                                             <- marker
+#         NODE_LOSS_VOLUME_AVAILABLE_OK                          <- the REAL emission
+#
+# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
+# (`=== RUN   chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
+# after the verb, so they match neither the indent nor the end-of-line anchor; they are
+# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
+# section that would swallow the whole log.
+_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
+_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
+
+# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
+# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
+# never echoes script source, so there is nothing to be confused by and the legacy
+# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
+# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
+# "" and are evidence for nothing.
+UNSECTIONED = "*"
+
+# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
+# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
+# never manufacture a proof.
+PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
+
+# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
+# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
+# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
+# one. Unknown section names therefore fail closed in both directions.
+SOURCE_SECTIONS = frozenset({"COMMAND"})
+
+
+def classify_sections(log_text: str) -> list[tuple[str, str]]:
+    """[(section, line)] for every line of a run log.
+
+    `section` is the enclosing `=== NAME` block, "" for a line inside no block, or
+    UNSECTIONED for every line when the log declares no sections at all. A block runs from
+    its marker until the first non-blank line indented LESS than the marker (blank lines do
+    not end it) or until the next marker; the marker line itself is content of nothing.
+    """
+    rows: list[tuple[str, str]] = []
+    cur, cur_indent, sectioned = "", 0, False
+    for raw in log_text.splitlines():
+        line = _strip_noise(raw)
+        m = _SECTION_MARKER_RE.match(line)
+        if m and m.group(2) not in _GO_TEST_MARKERS:
+            sectioned = True
+            cur, cur_indent = m.group(2), len(m.group(1))
+            rows.append(("", line))
+            continue
+        if cur:
+            if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
+                rows.append((cur, line))
+                continue
+            cur = ""
+        rows.append(("", line))
+    if not sectioned:
+        return [(UNSECTIONED, line) for _, line in rows]
+    return rows
+
+
+def scan_texts(lines) -> tuple[str, str]:
+    """(text a PASS token may be read from, text a SKIP token may be read from).
+
+    Accepts either [(section, line)] pairs from classify_sections() or a plain list of
+    strings, which is treated as an unsectioned log (legacy whole-log scanning).
+    """
+    pass_lines, skip_lines = [], []
+    for item in lines:
+        section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
+        if section in PASS_SECTIONS:
+            pass_lines.append(line)
+        if section and section not in SOURCE_SECTIONS:
+            skip_lines.append(line)
+    return "\n".join(pass_lines), "\n".join(skip_lines)
+
+_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
+
+
+# ========================================================================= sentinels
+def parse_sentinels(test_yaml: Path) -> dict:
+    """Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
+
+    Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
+    the LAST non-skip token in file order — the test's verdict token, i.e. the one its
+    final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
+    A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
+    unprovable rather than proven; see decide().
+    """
+    try:
+        text = test_yaml.read_text()
+    except (OSError, UnicodeDecodeError):
+        return {"pass": [], "skip": [], "terminal_pass": None}
+    toks = _SENTINEL_RE.findall(text)
+    passes, skips = [], []
+    for t in toks:
+        (skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
+    # De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
+    def uniq(xs: list[str]) -> list[str]:
+        return list(dict.fromkeys(xs))
+    passes, skips = uniq(passes), uniq(skips)
+    return {"pass": passes, "skip": skips,
+            "terminal_pass": passes[-1] if passes else None}
+
+
+def sentinel_map(tests_dir: Path) -> dict[str, dict]:
+    """{test-name: sentinels} for every test in the tree.
+
+    Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
+    name=...>`. This repo keeps that equal to the test's directory basename; the mapping
+    is by name, so it stays correct if that ever drifts.
+    """
+    out: dict[str, dict] = {}
+    for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
+        meta = cov.parse_test(f)
+        name = meta["name"] or f.parent.name
+        s = parse_sentinels(f)
+        s["dir"] = str(f.parent)
+        s["layer"] = layer_of(f) or meta["layer"]
+        s["requirements"] = meta["labels"]
+        out[name] = s
+    return out
+
+
+# =============================================================================== log
+def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
+                                                            bool]:
+    """Split a run log into per-testcase buckets of (section, line) pairs.
+
+    Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
+    script's sections underneath that header, so the test name is carried by a state
+    machine: a header naming a KNOWN test switches the current bucket, every following line
+    lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
+    chatter (and any other pipe-shaped line) from opening a bogus bucket.
+
+    Section labels come from classify_sections() and travel WITH the lines, so a bucket
+    still knows which of its lines were script source and which were real output.
+
+    Returns (buckets, attributed) where `attributed` is False when the log contained no
+    recognisable header at all — the signal for the degraded global scan in decide().
+    """
+    buckets: dict[str, list[tuple[str, str]]] = {}
+    cur: str | None = None
+    attributed = False
+    for section, line in classify_sections(log_text):
+        m = _LOG_HEADER_RE.match(line)
+        if m and m.group(1) in known:
+            cur = m.group(1)
+            attributed = True
+        if cur is not None:
+            buckets.setdefault(cur, []).append((section, line))
+    return buckets, attributed
+
+
+def unique_tokens(smap: dict[str, dict]) -> set[str]:
+    """Tokens claimed by EXACTLY ONE test in the tree.
+
+    Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
+    tests, so in an unattributable log it can never be evidence for either; but
+    `NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
+    """
+    owners: dict[str, set[str]] = {}
+    for name, s in smap.items():
+        for t in s["pass"] + s["skip"]:
+            owners.setdefault(t, set()).add(name)
+    return {t for t, o in owners.items() if len(o) == 1}
+
+
+# ============================================================================ decide
+def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
+           scope: str) -> tuple[str | None, str, str]:
+    """The ONE place a testcase becomes `proven` — or, far more often, does not.
+
+    Returns (sentinel, proof, note).
+
+    Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
+      1. JUnit failure/error            -> failed        (a red run is recorded as red)
+      2. JUnit <skipped/>               -> junit-skipped  (never green; ci/lib.sh rule)
+      3. ANY declared skip token seen   -> skipped        (BEATS a green JUnit and beats
+           IN REAL OUTPUT                                  the pass token: if any gated
+                                                           step self-skipped, the run did
+                                                           not prove the requirement)
+      4. no sentinel declared           -> proven only for L0/L1, whose skips are already
+                                           JUnit skips; unknown-no-sentinel-declared for
+                                           the Chainsaw layers
+      5. EVERY declared pass token seen -> proven
+      6. anything else                  -> unknown-sentinel-absent
+    Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
+    asserts `pass || skip`, so a missing pass token means a step did not reach its proven
+    branch, whatever the JUnit says.
+
+    Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
+    only in `=== STDOUT`, a skip token counts in any section that is not script source.
+    That narrows WHERE a token is read from and changes nothing about what it means — skip
+    still outranks pass, and it outranks it within genuine output, which is the only place
+    either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
+    strings (an unsectioned log; see scan_texts / UNSECTIONED).
+    """
+    if junit_result in ("failed", "error"):
+        return (None, "failed", f"JUnit recorded {junit_result}")
+    if junit_result == "skipped":
+        return (None, "junit-skipped",
+                "JUnit <skipped/> — the test did not execute; a skip is never green")
+
+    pass_text, skip_text = scan_texts(lines)
+    seen_skip = [t for t in sent["skip"] if t in skip_text]
+    if seen_skip:
+        return (seen_skip[0], "skipped",
+                f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
+                f"exited 0 without exercising the behaviour, so this is NOT proof")
+
+    if not sent["pass"] and not sent["skip"]:
+        if layer not in SENTINEL_EXEMPT_LAYERS:
+            return (None, "unknown-no-sentinel-declared",
+                    f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
+                    f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
+                    f"refusing to call it proven")
+        return (None, "proven",
+                f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
+                f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
+
+    missing = [t for t in sent["pass"] if t not in pass_text]
+    if not missing:
+        return (sent["terminal_pass"], "proven",
+                f"run log OUTPUT carries every declared pass token "
+                f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
+    return (None, "unknown-sentinel-absent",
+            f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
+            f"[{scope}] — refusing to infer proof from the JUnit alone")
+
+
+# ============================================================================= junit
+def junit_cases(path: Path) -> list[dict]:
+    """[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
+    root = ET.parse(path).getroot()
+    cases: list[dict] = []
+    for c in root.iter("testcase"):
+        if c.find("failure") is not None:
+            result = "failed"
+        elif c.find("error") is not None:
+            result = "error"
+        elif c.find("skipped") is not None:
+            result = "skipped"
+        else:
+            result = "passed"
+        props = {p.get("name"): (p.get("value") or "")
+                 for p in c.iter("property") if p.get("name")}
+        reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
+        layer = (props.get("layer") or "").upper()
+        if not layer:
+            layer = (c.get("classname") or "").split(":", 1)[0].upper()
+        cases.append({"name": c.get("name") or "?", "layer": layer,
+                      "requirements": reqs, "result": result})
+    return cases
+
+
+# =============================================================================== git
+def git_sha(repo: Path) -> str:
+    """HEAD, suffixed `-dirty` when TRACKED files differ from it.
+
+    Untracked files are deliberately ignored, matching status.py's `tracked_files()`
+    posture (untracked scratch is not repo state) — and because a `--record` run creates
+    untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
+    A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
+    the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
+    """
+    try:
+        sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
+                             capture_output=True, text=True, check=True).stdout.strip()
+    except (subprocess.CalledProcessError, OSError):
+        return "unknown"
+    dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--"],
+                           capture_output=True).returncode != 0
+    return f"{sha}-dirty" if dirty else sha
+
+
+def pinned_chainsaw_version(conf_root: Path) -> str:
+    """CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
+    f = conf_root / "versions.env"
+    if f.is_file():
+        m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
+        if m:
+            return m.group(1)
+    return "unknown"
+
+
+def server_kube_version() -> str:
+    """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
+    for args in (["kubectl", "version", "-o", "json"],
+                 ["kubectl", "version", "--output=json"]):
+        try:
+            p = subprocess.run(args, capture_output=True, text=True, timeout=15)
+        except (OSError, subprocess.TimeoutExpired):
+            continue
+        if p.returncode != 0 or not p.stdout.strip():
+            continue
+        try:
+            v = json.loads(p.stdout).get("serverVersion") or {}
+        except json.JSONDecodeError:
+            continue
+        if v.get("gitVersion"):
+            return v["gitVersion"]
+    return "unknown"
+
+
+def run_timestamp(run_dir: Path, junit: Path) -> str:
+    """UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
+    m = _RUN_ID_TS_RE.match(run_dir.name)
+    if m:
+        t = m.group(1)
+        return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
+    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
+
+
+# ============================================================================ record
+def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
+               kube_version: str, sha: str) -> list[dict]:
+    junit = run_dir / "report.junit.xml"
+    log = run_dir / "run.log"
+    smap = sentinel_map(tests_dir)
+    log_text = log.read_text(errors="replace")
+    buckets, attributed = attribute_log(log_text, set(smap))
+    uniq = unique_tokens(smap)
+    all_pairs = classify_sections(log_text)
+    sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
+    section_note = ("tokens read from real-output sections only"
+                    if sectioned else "log declares no === sections; whole-log scan")
+    ts = run_timestamp(run_dir, junit)
+
+    rows: list[dict] = []
+    for case in junit_cases(junit):
+        name = case["name"]
+        sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
+                               "layer": case["layer"], "requirements": []})
+        layer = sent.get("layer") or case["layer"] or "?"
+        # Requirements come from the test's LABELS (coverage.py's authoritative source);
+        # the JUnit property is the fallback for a testcase with no matching test file.
+        reqs = sent.get("requirements") or case["requirements"]
+
+        if name in buckets:
+            lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
+        elif not attributed:
+            # Degraded mode: the log had no recognisable Chainsaw header (a redirected or
+            # reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
+            # exactly one test, so a token another test emitted can never prove this one.
+            lines = all_pairs
+            sent = dict(sent,
+                        **{"pass": [t for t in sent["pass"] if t in uniq],
+                           "skip": [t for t in sent["skip"] if t in uniq]})
+            sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
+            scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
+                     + section_note)
+        else:
+            lines, scope = [], "no log lines attributed to this testcase"
+
+        sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
+        rows.append({
+            "ts_utc": ts,
+            "git_sha": sha,
+            "env": env,
+            "chainsaw_version": chainsaw_version,
+            "kube_version": kube_version,
+            "layer": layer,
+            "test": name,
+            "requirements": reqs,
+            "junit_result": case["result"],
+            "sentinel": sentinel,
+            "proof": proof,
+            "run_id": run_dir.name,
+            "note": note,
+        })
+    return rows
+
+
+def append_rows(ledger: Path, rows: list[dict]) -> None:
+    """Append-only, one compact JSON object per line. Never rewrites existing rows."""
+    ledger.parent.mkdir(parents=True, exist_ok=True)
+    with ledger.open("a", encoding="utf-8") as fh:
+        for r in rows:
+            fh.write(json.dumps(r, sort_keys=True) + "\n")
+
+
+# ========================================================================== selftest
+def _selftest() -> int:
+    """Deterministic, no-cluster self-test of every decision rule (house pattern:
+    `status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
+
+    The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
+    assertion standing between this ledger and a permanent false proof of R-STATE-03.
+    """
+    import tempfile
+    ok = True
+
+    def check(label: str, cond: bool) -> None:
+        nonlocal ok
+        if not cond:
+            sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
+            ok = False
+
+    # --- token derivation, against the dr-restore/node-loss shape ---------------------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
+            "  name: dr-restore\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-03: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-03\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo OFF_CLUSTER_TARGET_OK'\n"
+            "        check:\n"
+            "          (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
+            "    - script:\n        content: 'echo x'\n        check:\n"
+            "          (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
+        s = parse_sentinels(tdir / "chainsaw-test.yaml")
+        check("pass tokens derived from the test file",
+              s["pass"] == ["OFF_CLUSTER_TARGET_OK",
+                            "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
+        check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
+              s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
+        check("terminal pass token is the LAST non-skip token",
+              s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
+
+        smap = sentinel_map(root / "tests")
+        check("sentinel_map keys on metadata.name and carries layer + labels",
+              smap["dr-restore"]["layer"] == "L3"
+              and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
+
+    # --- decide(): the SKIP-is-never-proven contract ----------------------------------
+    l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
+          "skip": ["DR_RESTORE_SKIPPED_OK"],
+          "terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
+
+    # *** THE CASE THIS WHOLE FILE EXISTS FOR ***
+    sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
+    check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
+          proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
+                          "DR_RESTORE_SKIPPED_OK"], "t")
+    check("a skip token BEATS a pass token in the same log (worst wins)",
+          proof == "skipped")
+
+    _, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
+    check("green JUnit with no sentinel in the log is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "passed", l3, [], "t")
+    check("green JUnit with NO log lines attributed is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "failed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("a failing JUnit outranks a pass token in the log", proof == "failed")
+
+    _, proof, _ = decide("L3", "skipped", l3, [], "t")
+    check("JUnit <skipped/> records junit-skipped, never green",
+          proof == "junit-skipped")
+
+    multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
+    _, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
+    check("every declared pass token must be present, not just the terminal one",
+          proof == "unknown-sentinel-absent")
+    _, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
+    check("all pass tokens present == proven", proof == "proven")
+
+    none = {"pass": [], "skip": [], "terminal_pass": None}
+    _, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
+    check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("L1", "passed", none, [], "t")
+    check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
+          proof == "proven")
+    _, proof, _ = decide("L0", "skipped", none, [], "t")
+    check("an L0 JUnit skip is still never green", proof == "junit-skipped")
+    _, proof, _ = decide("META", "passed", none, [], "t")
+    check("META (a Chainsaw layer) needs a sentinel like L2/L3",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("", "passed", none, [], "t")
+    check("an unknown/blank layer fails CLOSED — sentinel required",
+          proof == "unknown-no-sentinel-declared")
+
+    # --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
+    # Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
+    # `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
+    # inside `=== COMMAND` on every run of a skip-gated test.
+    nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
+          "skip": ["NODE_LOSS_SKIPPED_OK"],
+          "terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+    _CMD = ("    l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |\n"
+            "        === COMMAND\n"
+            "        /usr/bin/sh -c set -euo pipefail\n"
+            "        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
+            "          echo NODE_LOSS_SKIPPED_OK\n"
+            "\n"
+            "          exit 0\n"
+            "        fi\n"
+            "        echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
+
+    ran = classify_sections(_CMD + "        === STDOUT\n"
+                            "        ready nodes: 3\n"
+                            "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
+    sent, proof, _ = decide("L3", "passed", nl, ran, "t")
+    check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
+          proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+    selfskipped = classify_sections(_CMD + "        === STDOUT\n"
+                                    "        SKIP: 'longhorn' StorageClass absent\n"
+                                    "        NODE_LOSS_SKIPPED_OK\n")
+    sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
+    check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
+          proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
+
+    neither = classify_sections(_CMD + "        === STDOUT\n        ready nodes: 3\n")
+    _, proof, _ = decide("L3", "passed", nl, neither, "t")
+    check("both tokens in COMMAND and neither in STDOUT is NOT proven",
+          proof == "unknown-sentinel-absent")
+
+    both_out = classify_sections("        === STDOUT\n"
+                                 "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                                 "        NODE_LOSS_SKIPPED_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, both_out, "t")
+    check("skip still BEATS pass when both are in genuine output", proof == "skipped")
+
+    stderr_skip = classify_sections("        === STDERR\n        NODE_LOSS_SKIPPED_OK\n"
+                                    "        === STDOUT\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
+    check("a skip token in a non-source section (STDERR) still costs the proof",
+          proof == "skipped")
+
+    stderr_pass = classify_sections("        === STDERR\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
+    check("a pass token OUTSIDE stdout cannot prove anything",
+          proof == "unknown-sentinel-absent")
+
+    # --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
+    gorows = classify_sections("=== RUN   chainsaw\n=== PAUSE chainsaw\n"
+                               "=== CONT  chainsaw/node-loss\n"
+                               "        === COMMAND\n        echo NODE_LOSS_SKIPPED_OK\n"
+                               "    l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
+                               "--- PASS: chainsaw/node-loss (131.09s)\n")
+    secs = [sec for sec, _ in gorows]
+    check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
+          secs[:3] == ["", "", ""])
+    check("a section covers its indented body and ends at a less-indented line",
+          secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
+    check("a log with no === markers stays wholly scannable (UNSECTIONED)",
+          {sec for sec, _ in classify_sections("plain line\n  another\n")} == {UNSECTIONED})
+
+    # --- the preserved run itself, as a fixture ----------------------------------------
+    # runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
+    # (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
+    fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
+    if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
+        rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
+        row = next((r for r in rows if r["test"] == "node-loss"), None)
+        check("preserved real L3 run records node-loss/R-STATE-02 as proven",
+              row is not None and row["proof"] == "proven"
+              and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
+              and row["requirements"] == ["R-STATE-02"])
+        check("preserved real L3 run attributes (no DEGRADED fallback needed)",
+              row is not None and "DEGRADED" not in row["note"])
+        pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
+        secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
+        check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
+              secs == {"COMMAND"})
+        secs = {sec for sec, line in pairs
+                if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+        check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
+
+    # --- log attribution --------------------------------------------------------------
+    log = ("[run.sh] Coverage report first\n"
+           "    | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
+           "        NODE_LOSS_SKIPPED_OK\n"
+           "    | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
+           "        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
+    b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
+    check("log lines are attributed to the testcase whose header preceded them",
+          attributed
+          and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
+          and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
+    check("a pass token does NOT leak across testcase boundaries",
+          not any("DR_RESTORED" in l for _, l in b["node-loss"]))
+    check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
+          all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
+
+    # Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
+    # regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
+    real_hdr = ("    l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
+                "\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
+                "        === STDOUT\n        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
+    check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
+          attributed_real and "node-loss" in b2
+          and ("STDOUT", "        NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
+    _, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
+    check("an unattributable log is reported as such (degraded mode)", not attributed2)
+
+    smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
+             "b": {"pass": ["DNS_OK"], "skip": []}}
+    check("only tokens owned by exactly one test may be used in degraded mode",
+          unique_tokens(smap2) == {"A_ONLY_OK"})
+
+    # --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "node-loss"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "kind: Test\nmetadata:\n  name: node-loss\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-02: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-02\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo hi'\n        check:\n"
+            "          (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
+            " || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
+        rd = root / "runs" / "20260901T120000Z-abc1234"
+        rd.mkdir(parents=True)
+        (rd / "report.junit.xml").write_text(
+            '<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
+            'name="node-loss"><properties>'
+            '<property name="requirements" value="R-STATE-02"/>'
+            '<property name="layer" value="L3"/></properties></testcase>'
+            "</testsuite></testsuites>")
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      SKIP: 'longhorn' StorageClass absent\n"
+            "      NODE_LOSS_SKIPPED_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
+              len(rows) == 1 and rows[0]["junit_result"] == "passed"
+              and rows[0]["proof"] == "skipped"
+              and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
+        check("end-to-end: row carries ts from the run-dir name and the R-ID label",
+              rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
+              and rows[0]["requirements"] == ["R-STATE-02"])
+
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
+              rows[0]["proof"] == "proven"
+              and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+        ledger = root / "ledger.jsonl"
+        append_rows(ledger, rows)
+        append_rows(ledger, rows)
+        check("ledger is append-only JSONL (one parsable object per line)",
+              len(ledger.read_text().strip().splitlines()) == 2
+              and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
+
+    if not ok:
+        return 1
+    sys.stderr.write("record_run.py selftest: PASS\n")
+    return 0
+
+
+# ============================================================================== main
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--run-dir", type=Path,
+                    help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"append-only ledger (default {DEFAULT_LEDGER})")
+    ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
+    ap.add_argument("--repo", type=Path, default=REPO_ROOT)
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
+    ap.add_argument("--chainsaw-version", default=None,
+                    help="override the versions.env pin (recorded verbatim)")
+    ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
+                    help="override the `kubectl version` probe (recorded verbatim)")
+    ap.add_argument("--dry-run", action="store_true",
+                    help="print the rows that WOULD be appended; write nothing")
+    ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
+    args = ap.parse_args()
+
+    if args.selftest:
+        return _selftest()
+    if args.run_dir is None:
+        sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
+        return 1
+
+    run_dir = args.run_dir.resolve()
+    junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
+    # BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
+    # recording from it alone is exactly the false proof this tool exists to prevent.
+    missing = [p.name for p in (junit, log) if not p.is_file()]
+    if missing:
+        sys.stderr.write(
+            f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
+            f"The JUnit alone cannot distinguish a self-skip from a real pass "
+            f"(see module docstring). SKIPPED (not green).\n")
+        return 3
+
+    try:
+        rows = build_rows(
+            run_dir, args.tests, args.env,
+            args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
+            args.kube_version or server_kube_version(),
+            git_sha(args.repo))
+    except ET.ParseError as e:
+        sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
+        return 1
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
+        return 1
+
+    if not rows:
+        sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
+                         f"SKIPPED (not green).\n")
+        return 3
+
+    for r in rows:
+        print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
+              f"({','.join(r['requirements']) or 'untagged'}) "
+              f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
+    counts: dict[str, int] = {}
+    for r in rows:
+        counts[r["proof"]] = counts.get(r["proof"], 0) + 1
+    print(f"[record] {len(rows)} testcase(s): "
+          + ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
+
+    if args.dry_run:
+        print("[record] --dry-run: ledger NOT written.")
+        return 0
+    try:
+        append_rows(args.ledger, rows)
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
+        return 1
+    print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
+    print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/lib/status.py

@@ -15,26 +15,42 @@ This generator derives, from the REPOSITORY ALONE, three orthogonal axes per req
                        from a Flux Kustomization entrypoint (i.e. actually reconciled)?
   2. VERIFICATION    — is there a requirement-tagged test, at which layer, and is it an
                        EXECUTABLE test or a TODO scaffold?
-  3. PROOF           — has that test actually passed?  *** DELIBERATELY NOT DERIVED. ***
+  3. PROOF           — has that test actually passed?  Derived ONLY from the committed
+                       run ledger, and ONLY as far as that ledger's evidence reaches.
 
-WHAT THIS TOOL REFUSES TO DERIVE (and why)
-------------------------------------------
-PROOF-OF-PASS IS NOT DERIVABLE FROM REPO STATE. Nothing tracked in git records a test
-result: `conformance/_artifacts/` and `results/` are git-ignored, and the only prose
-record of a green run lives in DECISIONS-LOG.md — hand-maintained, unstructured, and
-exactly the kind of source the operator has ruled out. The project's own culture is that
-"merged" is never "done" (agents/definition-of-done.md, R-TEST-03), so this report will
-NOT infer, guess, or default a requirement to green. Every requirement's proof state is
-reported as `not-derivable`, and the headline says so. If an untracked JUnit artifact
-happens to be lying in the workspace it is surfaced ONLY as a quarantined advisory with
-its mtime — never counted, never mixed into a status.
+PROOF — WHAT CHANGED, AND WHAT DID NOT
+--------------------------------------
+This report used to say `not-derivable` for every requirement, because nothing tracked in
+git recorded a test result and the only prose record of a green run lived in
+DECISIONS-LOG.md — hand-maintained, unstructured, exactly the source the operator ruled
+out. The fix this docstring asked for now exists: `conformance/runs/ledger.jsonl`, an
+append-only ledger written by `lib/record_run.py` from a run's JUnit **and its run log**
+(see conformance/.gitignore for the negations that keep both in git).
 
-The honest fix is a committed, machine-readable run ledger; until that exists, the most
-this report can say is "wired and testable", never "proven".
+THE REFUSAL POSTURE IS UNCHANGED. Evidence replaced "not derivable" only where evidence
+exists; nowhere is optimism substituted for it:
+
+  * A ledger row reaches `proven` only if `record_run.py` saw that test's own PASS
+    sentinel token in the run log. The L2/L3 tests SELF-SKIP GREEN — Chainsaw records
+    `passed` on `DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries no
+    stdout — so a skip token maps to `skipped` and can never become proof.
+  * A proof is honoured only while it still describes the current code: if any of the
+    requirement's implementing manifests or verifying tests changed in a commit that is
+    NOT an ancestor of the run's `git_sha`, the row degrades to `stale-proof`, which is
+    not green. A run recorded from a dirty tree (`<sha>-dirty`) is never honoured.
+  * A requirement with no ledger row, or whose verifying tests did not ALL report proven,
+    stays `not-derivable` / `unknown` — never rounded up.
+  * The untracked `_artifacts/report.junit.xml` is STILL only a quarantined advisory with
+    its mtime. It is not the ledger, it is not evidence, and it feeds no status.
+
+With an empty or absent ledger this report behaves exactly as it did before: every
+requirement `not-derivable`, headline PROVEN 0, "the most this report can say is wired and
+testable, never proven".
 
 READS (all read-only)
   requirements/registry.yaml     the authoritative R-* mirror  (via lib/coverage.py)
   tests/**/chainsaw-test.yaml    requirement tags + layer + executable annotation
+  runs/ledger.jsonl              COMMITTED proof-of-pass rows (via lib/record_run.py)
   clusters/<env>/               Flux entrypoint -> the reconciliation graph
   `git ls-files`                 TRACKED files only; untracked scratch is not repo state
   ../home-network-k3s/docs/00-vision-and-scope.md   OPTIONAL registry<->spec parity check
@@ -92,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io"
 #   conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there.
 #   ci/tests          — gate unit-test fixtures embed example R-IDs as DATA.
 #   registry.yaml     — the requirement mirror itself, not an implementation of anything.
-IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/")
+# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the
+# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as
+# implementing evidence makes `last_change` move every time proof is recorded, so a row's
+# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is
+# committed — recording the proof destroys it. The ledger is the proof record; it must not
+# participate in the freshness computation about itself. (Found live 2026-09-01 on the first
+# PROVEN row.)
+IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/")
 IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)
 
 
@@ -343,6 +366,169 @@ def spec_requirement_ids(spec_root: Path) -> set[str] | None:
     return set(RID_RE.findall(doc.read_text()))
 
 
+# ========================================================================== ledger
+# PROOF axis. Everything here reads `conformance/runs/ledger.jsonl` (written ONLY by an
+# explicit `lib/record_run.py` / `run.sh --record` invocation) and downgrades relentlessly.
+# Nothing in this section may ever turn absent, ambiguous or aged evidence into green.
+
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# Severity ranking, WORST FIRST. A requirement's proof is the MINIMUM over its verifying
+# tests, so one failed/skipped/missing test drags the whole requirement down and a single
+# proven test can never carry the others. `stale-proof` ranks below `proven` on purpose:
+# it is a real past pass that no longer describes the current code, and it is NOT green.
+PROOF_ORDER = {
+    "failed": 0,              # a ledger row recorded a JUnit failure/error
+    "unknown": 1,             # ran, but the log could not evidence pass vs self-skip
+    "not-derivable": 2,       # no ledger row for this test at all (the pre-ledger default)
+    "junit-skipped": 3,       # JUnit <skipped/> — did not execute
+    "skipped": 4,             # SELF-SKIPPED green: exited 0 without exercising the behaviour
+    "stale-proof": 5,         # proven at a sha the implementation/test has since moved past
+    "proven": 6,              # the ONLY green state
+}
+# record_run.py's two "we cannot tell" states collapse to one reported state here; the
+# per-test detail in status.json keeps the distinction for whoever has to fix it.
+_PROOF_ALIASES = {
+    "unknown-no-sentinel-declared": "unknown",
+    "unknown-sentinel-absent": "unknown",
+}
+
+
+def read_ledger(path: Path) -> list[dict]:
+    """Parse the append-only JSONL ledger. A malformed line is SKIPPED, never guessed at.
+
+    Absent or empty file -> [] , which reproduces this report's pre-ledger behaviour
+    exactly (every requirement `not-derivable`, headline PROVEN 0).
+    """
+    if not path.is_file():
+        return []
+    rows: list[dict] = []
+    for line in path.read_text(errors="replace").splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        try:
+            obj = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(obj, dict) and obj.get("test"):
+            rows.append(obj)
+    return rows
+
+
+def latest_by_test(rows: list[dict]) -> dict[str, dict]:
+    """Most recent ledger row per test name (ts_utc, then file order as the tiebreak).
+
+    Latest wins rather than best-ever: a test that has since gone back to self-skipping
+    must stop reading as proven.
+    """
+    latest: dict[str, dict] = {}
+    for i, r in enumerate(rows):
+        key = r["test"]
+        prev = latest.get(key)
+        if prev is None or (r.get("ts_utc", ""), i) >= (prev.get("ts_utc", ""),
+                                                       prev["_i"]):
+            latest[key] = {**r, "_i": i}
+    return latest
+
+
+class _GitOracle:
+    """Small cached wrapper for the two git questions the staleness rule asks."""
+
+    def __init__(self, repo: Path):
+        self.repo = repo
+        self._last: dict[tuple[str, ...], str | None] = {}
+        self._anc: dict[tuple[str, str], bool] = {}
+
+    def _git(self, *args: str) -> subprocess.CompletedProcess:
+        return subprocess.run(["git", "-C", str(self.repo), *args],
+                              capture_output=True, text=True)
+
+    def last_change(self, paths: tuple[str, ...]) -> str | None:
+        """Commit that last touched any of `paths`; None if git knows none of them."""
+        if not paths:
+            return None
+        if paths in self._last:
+            return self._last[paths]
+        p = self._git("log", "-1", "--format=%H", "--", *paths)
+        sha = p.stdout.strip() if p.returncode == 0 else ""
+        self._last[paths] = sha or None
+        return self._last[paths]
+
+    def is_ancestor(self, older: str, newer: str) -> bool:
+        """True iff `older` is an ancestor of (or equal to) `newer`.
+
+        False whenever either rev is unresolvable — an unknown or `-dirty` sha (a run
+        recorded from a modified working tree) can never satisfy the freshness test, which
+        is the intended fail-closed direction.
+        """
+        key = (older, newer)
+        if key in self._anc:
+            return self._anc[key]
+        ok = self._git("merge-base", "--is-ancestor", older, newer).returncode == 0
+        self._anc[key] = ok
+        return ok
+
+
+def derive_proof(git: _GitOracle, rid: str, verifiers: list[dict],
+                 impl_files: list[dict], latest: dict[str, dict]) -> dict:
+    """PROOF for one requirement, from the ledger, with the staleness rule applied.
+
+    STALENESS RULE — a proof is a claim about a specific tree, not a permanent badge:
+      relevant paths = every implementing artifact this report found for the requirement
+                       + every verifying test's directory
+      last_change    = `git log -1 -- <relevant paths>`
+      A row's `proven` is honoured only if `last_change` is an ANCESTOR of the row's
+      `git_sha`, i.e. the run happened at or after the newest change to the code it
+      claims to prove. Otherwise it degrades to `stale-proof` (not green). A `-dirty`
+      sha, a sha this clone does not have (rebased/dropped), or a path git has never
+      seen (an uncommitted test) all fail the ancestry test and degrade the same way.
+
+    The requirement's state is the MINIMUM over its verifying tests: every verifying test
+    must be proven and fresh before the requirement reads `proven`. A test with no ledger
+    row contributes `not-derivable`, so a partially-recorded run cannot manufacture green.
+    """
+    paths = sorted({f["path"] for f in impl_files}
+                   | {os.path.dirname(v["path"]) for v in verifiers if v.get("path")})
+    last_change = git.last_change(tuple(paths))
+
+    per_test: list[dict] = []
+    for v in verifiers:
+        row = latest.get(v["name"])
+        if row is None:
+            per_test.append({"test": v["name"], "state": "not-derivable",
+                             "reason": "no ledger row — this test has never been "
+                                       "recorded by run.sh --record / record_run.py"})
+            continue
+        base = _PROOF_ALIASES.get(row.get("proof", ""), row.get("proof", "unknown"))
+        if base not in PROOF_ORDER:
+            base = "unknown"
+        state, reason = base, row.get("note", "")
+        if base == "proven":
+            sha = row.get("git_sha", "")
+            if last_change and not git.is_ancestor(last_change, sha):
+                state = "stale-proof"
+                reason = (f"proven at {sha[:12]}, but the implementing manifests / test "
+                          f"changed in {last_change[:12]}, which that run predates")
+        per_test.append({"test": v["name"], "state": state, "reason": reason,
+                         "run_id": row.get("run_id"), "ts_utc": row.get("ts_utc"),
+                         "git_sha": row.get("git_sha"), "env": row.get("env"),
+                         "sentinel": row.get("sentinel"),
+                         "junit_result": row.get("junit_result")})
+
+    if not per_test:
+        state = "not-derivable"
+        reason = "no requirement-tagged test exists to be proven"
+    else:
+        worst = min(per_test, key=lambda d: PROOF_ORDER[d["state"]])
+        state, reason = worst["state"], worst["reason"]
+        if state == "not-derivable" and not reason:
+            reason = "no committed ledger row for this requirement's verifying test(s)"
+
+    return {"state": state, "reason": reason, "last_change_sha": last_change,
+            "evidence_paths": paths, "per_test": per_test}
+
+
 # ========================================================================== build
 IMPL_ORDER = {"none": 0, "off-cluster": 1, "unwired": 2, "reconciled-partial": 3,
               "reconciled": 4}
@@ -369,7 +555,13 @@ LABELS = {
 
 def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, list[dict]],
           applied: set[str], gitops_roots: set[str], spec_ids: set[str] | None,
-          env: str) -> dict:
+          env: str, ledger_rows: list[dict] | None = None) -> dict:
+    # PROOF axis inputs. `ledger_rows=None` / `[]` reproduces the pre-ledger behaviour:
+    # `latest` is empty, so every requirement derives `not-derivable` exactly as before.
+    ledger_rows = ledger_rows or []
+    latest = latest_by_test(ledger_rows)
+    git = _GitOracle(repo)
+
     classified = [classify_test(t) for t in tests]
     by_req_tests: dict[str, list[dict]] = {r["id"]: [] for r in registry}
     for t, c in zip(tests, classified):
@@ -439,11 +631,9 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
                 "blocked_by": sorted({t["blocked_by"] for t in ts if t["blocked_by"]}),
                 "tests": ts,
             },
-            # NEVER derived. See the module docstring.
-            "proof": {"state": "not-derivable",
-                      "reason": "no committed, machine-readable record of a test run "
-                                "exists in the repo (_artifacts/ and results/ are "
-                                "git-ignored); DECISIONS-LOG.md prose is not a signal"},
+            # Derived ONLY from the committed ledger, and only as far as it reaches.
+            # With no ledger row this stays `not-derivable`, as it always was.
+            "proof": derive_proof(git, rid, ts, ev, latest),
             "status": LABELS[(impl_state, verify_state)],
         }
 
@@ -492,8 +682,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
             and v["verification"]["state"] == "executable"),
         "safety_critical_total": count(lambda v: v["safety_critical"]),
         "safety_critical_without_executable_l3": len(safety_scaffold),
-        "proven": 0,
-        "proven_derivable": False,
+        # PROVEN is now a real count — but only of requirements whose EVERY verifying
+        # test has a committed ledger row that saw its pass sentinel, at a sha the
+        # implementation has not moved past. `stale-proof` and `skipped` are counted
+        # separately and are NOT green.
+        "proven": count(lambda v: v["proof"]["state"] == "proven"),
+        "proof_stale": count(lambda v: v["proof"]["state"] == "stale-proof"),
+        "proof_skipped": count(lambda v: v["proof"]["state"] == "skipped"),
+        "proof_failed": count(lambda v: v["proof"]["state"] == "failed"),
+        "proof_unknown": count(lambda v: v["proof"]["state"] == "unknown"),
+        "proof_not_derivable": count(
+            lambda v: v["proof"]["state"] in ("not-derivable", "junit-skipped")),
+        "ledger_rows": len(ledger_rows),
+        "proven_derivable": bool(ledger_rows),
         "coverage_style_covered": count(lambda v: v["verification"]["state"] != "none"),
     }
     summary["uncited_by_any_reconciled_manifest"] = count(
@@ -552,16 +753,39 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append(f"| Safety-critical requirements whose L3 test is a scaffold | "
              f"**{s['safety_critical_without_executable_l3']}** of "
              f"{s['safety_critical_total']} |")
-    o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
-    o.append("")
-    o.append("> **Proof is deliberately not derived.** Nothing tracked in git records a "
-             "test result: `conformance/_artifacts/` and `results/` are git-ignored, and "
-             "the only record of a green run is DECISIONS-LOG.md prose — hand-maintained, "
-             "which is the very thing this report replaces. Per "
-             "`agents/definition-of-done.md` a requirement is done only when its tagged "
-             "test **passes**, so the strongest claim this report can make is "
-             "*wired and testable*. Closing the gap needs a committed, machine-readable "
-             "run ledger; until then, treat every row below as unproven.\n")
+    if s["proven_derivable"]:
+        o.append(f"| **PROVEN (every verifying test passed, evidenced)** | "
+                 f"**{s['proven']}** |")
+        o.append(f"| Proof exists but is STALE (code changed after the run) | "
+                 f"{s['proof_stale']} |")
+        o.append(f"| Verifying test SELF-SKIPPED green (not proof) | {s['proof_skipped']} |")
+        o.append(f"| Verifying test FAILED in the recorded run | {s['proof_failed']} |")
+        o.append(f"| Ran, but the log could not evidence pass vs self-skip | "
+                 f"{s['proof_unknown']} |")
+        o.append(f"| No committed ledger row yet | {s['proof_not_derivable']} |")
+        o.append("")
+        o.append(f"> **Proof comes from `conformance/runs/ledger.jsonl`** "
+                 f"({s['ledger_rows']} row(s)), written only by an explicit "
+                 f"`run.sh --record` / `lib/record_run.py` invocation. A row reaches "
+                 f"`proven` ONLY when the run log carried that test's own pass sentinel "
+                 f"token — the L2/L3 tests self-skip GREEN (Chainsaw records `passed` on "
+                 f"`DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries "
+                 f"no stdout), so a skip token maps to `skipped` and can never become "
+                 f"proof. A proof is honoured only while the implementing manifests and "
+                 f"the test itself have not changed since the recorded `git_sha`; "
+                 f"otherwise it degrades to `stale-proof`, which is not green.\n")
+    else:
+        o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
+        o.append("")
+        o.append("> **No proof is derivable: `conformance/runs/ledger.jsonl` is absent or "
+                 "empty.** The ledger is the committed, machine-readable run record; until "
+                 "a run is captured with `run.sh --record`, nothing tracked in git records "
+                 "a test result (`conformance/_artifacts/` and `results/` are git-ignored, "
+                 "and DECISIONS-LOG.md prose is hand-maintained — the very thing this "
+                 "report replaces). Per `agents/definition-of-done.md` a requirement is "
+                 "done only when its tagged test **passes**, so the strongest claim this "
+                 "report can make is *wired and testable*; treat every row below as "
+                 "unproven.\n")
     o.append(f"> For contrast, `coverage.md` counts **{s['coverage_style_covered']}** "
              f"requirements as covered — that count includes the "
              f"{s['verify_scaffold_only']} whose only verifying test is a TODO scaffold.\n")
@@ -612,7 +836,15 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
             if ver["blocked_by"]:
                 vcell += f" · blocked-by: {'; '.join(ver['blocked_by'])}"
         sc = " ⚠safety" if v["safety_critical"] else ""
-        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | not-derivable |")
+        pr = v["proof"]
+        if pr["state"] == "not-derivable":
+            pcell = "not-derivable"
+        else:
+            runs = sorted({d["run_id"] for d in pr["per_test"] if d.get("run_id")})
+            pcell = f"**{pr['state']}**" if pr["state"] == "proven" else pr["state"]
+            if runs:
+                pcell += f" ({', '.join(runs)})"
+        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | {pcell} |")
 
     if rep["safety_critical_without_executable_l3"]:
         o.append("\n## Safety-critical requirements whose L3 fault test is a SCAFFOLD "
@@ -701,22 +933,76 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append("| executable vs scaffold | TODO markers in the test body outrank the "
              "`executable` annotation; a real body without the annotation still counts "
              "and is flagged | conservative — ambiguity never upgrades |")
-    o.append("| **test actually passed** | **NOT DERIVED** — nothing in git records a "
-             "run result | refused |")
+    o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
+             "every verifying test: JUnit result **plus** that test's own pass sentinel "
+             "token grepped from the run log | exact where a run was recorded, refused "
+             "everywhere else — a self-skip token or a missing/unrecognised sentinel "
+             "NEVER yields `proven` |")
+    o.append("| proof still current | the row's `git_sha` must be a descendant of the "
+             "last commit touching the requirement's implementing manifests or its test "
+             "| conservative — an unresolvable or `-dirty` sha degrades to `stale-proof` |")
     o.append("| cluster liveness (pods/CRs healthy) | **NOT DERIVED** — no cluster is "
              "contacted; `reconciled` means Flux *would* apply it, not that it is "
              "healthy | refused |")
     o.append("| phase/board status | **NOT DERIVED** — `agents/task-board.md` is stale "
              "and is not a signal | refused |")
 
+    o.append("\n## Run ledger — the committed proof-of-pass evidence\n")
+    if not s["proven_derivable"]:
+        o.append("- `conformance/runs/ledger.jsonl` holds no rows. Capture a run with "
+                 "`./run.sh --record` (recording is never implicit: a run that silently "
+                 "records itself is how a bad run becomes a permanent false proof), then "
+                 "commit `conformance/runs/<run-id>/` **with its `run.log`** and the "
+                 "ledger. The log is the load-bearing half — Chainsaw's JUnit carries no "
+                 "stdout, so without it a self-skip is indistinguishable from a pass.")
+    else:
+        o.append(f"- {s['ledger_rows']} row(s) in `conformance/runs/ledger.jsonl`. "
+                 f"Per-test evidence for every non-`not-derivable` requirement below; "
+                 f"full rows (env, chainsaw/kube versions, sentinel, note) are in "
+                 f"`status.json` under `by_requirement.<R-ID>.proof.per_test`.\n")
+        o.append("| Requirement | Proof | Test | Ledger state | Sentinel | Run |")
+        o.append("|---|---|---|---|---|---|")
+        for rid, v in rep["by_requirement"].items():
+            pr = v["proof"]
+            if pr["state"] == "not-derivable":
+                continue
+            for d in pr["per_test"]:
+                o.append(f"| {rid} | {pr['state']} | `{d['test']}` | {d['state']} | "
+                         f"`{d.get('sentinel') or '—'}` | {d.get('run_id') or '—'} |")
+        stale = [(rid, v) for rid, v in rep["by_requirement"].items()
+                 if v["proof"]["state"] == "stale-proof"]
+        if stale:
+            o.append("\n### Stale proofs — a real pass that no longer describes this tree\n")
+            o.append("These requirements DID pass, but their implementing manifests or "
+                     "their verifying test changed in a commit the recorded run predates. "
+                     "A stale proof is not green; re-run with `--record`.\n")
+            for rid, v in stale:
+                pr = v["proof"]
+                o.append(f"- **{rid}** — {pr['reason']}")
+        skipped = [(rid, v) for rid, v in rep["by_requirement"].items()
+                   if v["proof"]["state"] == "skipped"]
+        if skipped:
+            o.append("\n### SELF-SKIPPED green — recorded, and deliberately NOT proof\n")
+            o.append("Chainsaw reported `passed` for these, because the test exited 0 "
+                     "after emitting its skip token: a precondition (Longhorn "
+                     "StorageClass, Velero CRDs, >=3 Ready nodes) was absent, so the "
+                     "behaviour was never exercised. This is exactly the row a "
+                     "JUnit-only ledger would have recorded as green.\n")
+            for rid, v in skipped:
+                toks = ", ".join(sorted({d.get("sentinel") or "?"
+                                         for d in v["proof"]["per_test"]
+                                         if d["state"] == "skipped"}))
+                o.append(f"- **{rid}** — skip token(s): `{toks}`")
+
     o.append("\n## Advisory — untracked local run artifact (NOT counted)\n")
     if advisory:
         o.append(f"- `{advisory['path']}` (untracked, git-ignored), last written "
                  f"{advisory['mtime']}: {advisory['tests']} testcase(s), "
                  f"{advisory['failures']} failure(s), {advisory['skipped']} skipped.")
         o.append(f"- requirement IDs it names: {', '.join(advisory['requirements']) or '—'}")
-        o.append("- This is one developer's local workspace, not repo state. It is shown "
-                 "for orientation only and contributes to NO status above.")
+        o.append("- This is one developer's local workspace, not repo state, and it is NOT "
+                 "the ledger. It is shown for orientation only and contributes to NO "
+                 "status above — a JUnit alone cannot tell a self-skip from a pass.")
     else:
         o.append("- none present.")
     return "\n".join(o) + "\n"
@@ -841,6 +1127,122 @@ def _selftest() -> int:
               classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
               == "scaffold")
 
+    # ---------------------------------------------------------------- PROOF / ledger
+    # These are the assertions that stand between this report and a fake green.
+    check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
+          read_ledger(Path("/nonexistent/ledger.jsonl")) == [])
+
+    def row(**kw):
+        base = {"ts_utc": "2026-09-01T12:00:00Z", "git_sha": "deadbeef", "env": "dev",
+                "chainsaw_version": "v0.2.12", "kube_version": "v1.36.4", "layer": "L3",
+                "test": "dr-restore", "requirements": ["R-STATE-03"],
+                "junit_result": "passed", "sentinel": None, "proof": "skipped",
+                "run_id": "20260901T120000Z-deadbee", "note": ""}
+        base.update(kw)
+        return base
+
+    verifiers = [{"name": "dr-restore",
+                  "path": "conformance/tests/l3-fault-injection/dr-restore/"
+                          "chainsaw-test.yaml"}]
+
+    with tempfile.TemporaryDirectory() as td:
+        gr = Path(td)
+        env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
+               "GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
+
+        def git(*a):
+            return subprocess.run(["git", "-C", str(gr), *a], capture_output=True,
+                                  text=True, env=env)
+
+        git("init", "-q", "-b", "main")
+        (gr / "platform").mkdir()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03\n")
+        git("add", "-A"); git("commit", "-qm", "one")
+        sha_a = git("rev-parse", "HEAD").stdout.strip()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03 v2\n")
+        git("add", "-A"); git("commit", "-qm", "two")
+        sha_b = git("rev-parse", "HEAD").stdout.strip()
+
+        oracle = _GitOracle(gr)
+        impl = [{"path": "platform/velero.yaml"}]
+
+        # *** THE CASE THIS WHOLE FEATURE EXISTS FOR ***
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="skipped", junit_result="passed",
+                                              sentinel="DR_RESTORE_SKIPPED_OK",
+                                              git_sha=sha_b)]))
+        check("a SELF-SKIP row on a GREEN JUnit is 'skipped', NEVER 'proven'",
+              pr["state"] == "skipped")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a proven row at the CURRENT sha is honoured", pr["state"] == "proven")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_a,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("STALENESS: proven at a sha the implementation has moved past degrades "
+              "to stale-proof", pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=f"{sha_b}-dirty",
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a run recorded from a DIRTY tree is never honoured as proof",
+              pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha="0" * 40)]))
+        check("a git_sha this clone cannot resolve degrades, never upgrades",
+              pr["state"] == "stale-proof")
+
+        for bad, expect in (("failed", "failed"),
+                            ("junit-skipped", "junit-skipped"),
+                            ("unknown-sentinel-absent", "unknown"),
+                            ("unknown-no-sentinel-declared", "unknown"),
+                            ("something-new-from-the-future", "unknown")):
+            pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                              latest_by_test([row(proof=bad, git_sha=sha_b)]))
+            check(f"ledger state {bad!r} maps to {expect!r}, never to proven",
+                  pr["state"] == expect)
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl, {})
+        check("no ledger row at all stays not-derivable",
+              pr["state"] == "not-derivable")
+
+        # Two verifying tests, only one proven: the requirement must NOT read proven.
+        two = verifiers + [{"name": "data-survival",
+                            "path": "conformance/tests/l3-fault-injection/"
+                                    "data-survival/chainsaw-test.yaml"}]
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b)]))
+        check("a requirement is proven only when EVERY verifying test is",
+              pr["state"] == "not-derivable")
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b),
+                                          row(test="data-survival", proof="skipped",
+                                              git_sha=sha_b)]))
+        check("worst-of wins across verifying tests (skip drags proven down)",
+              pr["state"] == "skipped")
+
+        # Latest row wins, not best-ever: a test that regressed to self-skipping must
+        # stop reading as proven.
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(ts_utc="2026-09-01T10:00:00Z",
+                                              proof="proven", git_sha=sha_b),
+                                          row(ts_utc="2026-09-02T10:00:00Z",
+                                              proof="skipped", git_sha=sha_b)]))
+        check("the LATEST ledger row wins, not the best one ever recorded",
+              pr["state"] == "skipped")
+
+        # A malformed ledger line is skipped, never guessed at.
+        lf = gr / "ledger.jsonl"
+        lf.write_text('{"test":"a","proof":"proven"}\nnot json at all\n\n'
+                      '{"proof":"proven"}\n')
+        rows = read_ledger(lf)
+        check("malformed / test-less ledger lines are dropped, not guessed at",
+              len(rows) == 1 and rows[0]["test"] == "a")
+
     if not ok:
         return 1
     sys.stderr.write("status.py selftest: PASS\n")
@@ -865,6 +1267,10 @@ def main() -> int:
                     help="exit 1 on defects this tool owns: registry/spec drift, "
                          "executable-annotation conflicts, or a safety-critical "
                          "requirement whose only L3 test is a scaffold")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"committed proof-of-pass ledger written by lib/record_run.py "
+                         f"(default {DEFAULT_LEDGER}). Absent or empty => every "
+                         f"requirement stays `not-derivable`, as before the ledger existed")
     ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
     args = ap.parse_args()
 
@@ -895,7 +1301,12 @@ def main() -> int:
     applied, nodes = reconciled_files(repo, entry)
     gitops_roots = {n["resolved"].split("/", 1)[0] for n in nodes if n["exists"]}
     impl = scan_implementation(repo, tracked_files(repo))
-    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env)
+    ledger_rows = read_ledger(args.ledger)
+    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env,
+                ledger_rows)
+    rep["ledger"] = {"path": os.path.relpath(args.ledger, REPO_ROOT),
+                     "rows": len(ledger_rows),
+                     "present": args.ledger.is_file()}
     rep["flux_graph"] = nodes
     rep["gitops_roots"] = sorted(gitops_roots)
     rep["reconciled_file_count"] = len(applied)
@@ -915,7 +1326,11 @@ def main() -> int:
           f"(+{s['wired_or_partial_and_testable'] - s['wired_and_testable']} partial) | "
           f"scaffold-only {s['verify_scaffold_only']} | "
           f"safety-critical w/o executable L3 "
-          f"{s['safety_critical_without_executable_l3']} | PROVEN: not derivable")
+          f"{s['safety_critical_without_executable_l3']} | "
+          + (f"PROVEN: {s['proven']} (stale {s['proof_stale']}, self-skipped "
+             f"{s['proof_skipped']}, failed {s['proof_failed']}, unknown "
+             f"{s['proof_unknown']}) from {s['ledger_rows']} ledger row(s)"
+             if s["proven_derivable"] else "PROVEN: not derivable (empty run ledger)"))
     print(f"[status] wrote {args.out/'status.md'} and {args.out/'status.json'}")
 
     rc = 0

conformance/run.sh

@@ -14,12 +14,33 @@
 #   ./run.sh --status                 # DERIVED STATUS report only (no cluster): per
 #                                     #   requirement, is it implemented, is it actually
 #                                     #   reconciled by Flux, is its test executable or a
-#                                     #   scaffold. Proof-of-pass is deliberately NOT
-#                                     #   derived — see lib/status.py.
+#                                     #   scaffold, and whether the committed run ledger
+#                                     #   (runs/ledger.jsonl) actually proves it — see
+#                                     #   lib/status.py. No ledger row => not-derivable.
 #   ./run.sh --lint                   # structural-validate test defs (no cluster)
 #   ./run.sh --strict-coverage        # hygiene gate: fail on tag/annotation drift
 #   ./run.sh --requirements R-NET-01  # ALSO asserts those IDs are covered (merge gate)
 #   ./run.sh --require-all            # phase-exit: every requirement must be covered
+#   ./run.sh --record                 # RUN AND CAPTURE: tee the whole run to
+#                                     #   runs/<UTC>-<short-sha>/run.log, copy the JUnit
+#                                     #   beside it, and append one ledger row per
+#                                     #   testcase to runs/ledger.jsonl (lib/record_run.py)
+#
+# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
+# --------------------------------------------------------------------------
+# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
+# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
+# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
+# that — `conformance/runs/` (see conformance/.gitignore for the negations).
+#
+# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
+# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
+#
+# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
+# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
+# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
+# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
+# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
 #
 # Artifacts (always written under conformance/_artifacts/, gitignored):
 #   _artifacts/report.junit.xml        JUnit (Chainsaw output, or skipped-scaffold shim)
@@ -28,6 +49,11 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Run ledger (TRACKED — commit these; --record only):
+#   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
+#   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
+#   runs/ledger.jsonl                         append-only, one row per testcase per run
+#
 # Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
 # execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
 # green — the harness is proven reproducible while later phases fill in behavior.
@@ -47,12 +73,17 @@ STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
 PYTHON="${PYTHON:-python3}"
 
+# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
+# under `tee`. Captured BEFORE parsing so nothing is normalized away.
+ORIG_ARGS=("$@")
+
 LAYER="all"
 REQS=""
 MODE="run"          # run | coverage-only | status | lint
 STRICT_COVERAGE=0
 REQUIRE_ALL=0
 FORCE_NO_CLUSTER=0
+RECORD=0
 
 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
 # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
@@ -83,7 +114,9 @@ FORCE_NO_CLUSTER=0
 # longer a gap and has been removed from this list; the coverage report counts it as covered.
 DOCUMENTED_GAPS=""
 
-usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
+# Print the header block (line 2 through the closing `####` rule, which is dropped) as
+# help. Bounded by the rule rather than a fixed line number so the header can grow.
+usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
 
 while [ $# -gt 0 ]; do
   case "$1" in
@@ -95,6 +128,7 @@ while [ $# -gt 0 ]; do
     --strict-coverage) STRICT_COVERAGE=1; shift ;;
     --require-all)    REQUIRE_ALL=1; shift ;;
     --no-cluster)     FORCE_NO_CLUSTER=1; shift ;;
+    --record)         RECORD=1; shift ;;
     -h|--help)        usage; exit 0 ;;
     *) echo "unknown arg: $1" >&2; usage; exit 2 ;;
   esac
@@ -104,6 +138,62 @@ mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
 
 log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
 
+###############################################################################
+# --record: capture this run as committed evidence, then hand it to the ledger.
+#
+# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
+# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
+# operator-visible output — no second, differently-behaving "recording mode" to drift.
+# CONF_RECORD_CHILD guards against recursion.
+#
+# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
+# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
+# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
+###############################################################################
+if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
+  case "${MODE}" in
+    run) ;;
+    # --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
+    # test, so a ledger row from one of them would be a row about nothing.
+    *) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
+  esac
+  RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
+  RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
+  RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
+  mkdir -p "${RUN_DIR}"
+  # Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
+  # ever capture a report THIS run produced. Copying a stale artifact would pair someone
+  # else's result with this run's log — a fabricated pairing, which is precisely the class
+  # of false proof the ledger exists to prevent.
+  rm -f "${JUNIT_OUT}"
+  # Strip --record from the argv handed to the child; everything else passes through.
+  CHILD_ARGS=()
+  for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
+    [ "$a" = "--record" ] || CHILD_ARGS+=("$a")
+  done
+  log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
+  set +e
+  CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
+    | tee "${RUN_DIR}/run.log"
+  RUN_RC=${PIPESTATUS[0]}
+  set -e
+  if [ -f "${JUNIT_OUT}" ]; then
+    cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
+  else
+    log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
+  fi
+  # Explicit, separate step: read the JUnit + the log and append one row per testcase.
+  # rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
+  # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
+  # to the real ledger; operators should never set it.
+  "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
+    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
+    --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
+  log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
+  log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
+  exit "${RUN_RC}"
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no

conformance/runs/20260901T235117Z-e9a4b66/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="131.090121" />
+  </testsuite>
+</testsuites>

conformance/runs/20260901T235117Z-e9a4b66/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 23:51:18 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1788306678898809444
+        victim node (currently hosts the attached volume): k3s-dev-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-dev-2 labeled
+        node/k3s-dev-2 cordoned
+        cordoned + labelled victim node: k3s-dev-2
+        pod "writer" deleted from chainsaw-fond-marlin namespace
+        evicted writer off the victim node; volume detached from k3s-dev-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
+        recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-dev-2 uncordoned
+        node/k3s-dev-2 unlabeled
+        restored node to service: k3s-dev-2
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-fond-marlin namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:53:29 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (131.09s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260902T001907Z-fdc3265/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="149.053701" />
+  </testsuite>
+</testsuites>

conformance/runs/20260902T001907Z-fdc3265/run.log

@@ -0,0 +1,671 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 00:19:07 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true http://192.168.1.9:9000 velero
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=velero  endpoint=http://192.168.1.9:9000
+        backup endpoint host: 192.168.1.9
+        backup endpoint resolves to: 192.168.1.9
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 81)
+        endpoint is not any Pod IP (checked 186)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=192.168.1.9 bucket=velero
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
+        namespace/chainsaw-cunning-hen-dr-src created
+        namespace/chainsaw-cunning-hen-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1788308352223263220
+        original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
+        backup.velero.io/chainsaw-cunning-hen-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=Completed
+        backup chainsaw-cunning-hen-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-cunning-hen-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-cunning-hen-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
+        recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
+        velero objects for chainsaw-cunning-hen cleaned up (ns velero)
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
+        released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:21:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (149.05s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -0,0 +1,2 @@
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -68,7 +68,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
@@ -76,8 +76,8 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -91,7 +91,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -91,7 +91,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

infra/ansible/README.md

@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOP
 > ```
 > (Both take the same target-group override as the rest — see **Targeting prod** below.)
 
+> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
+> ```
+> ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
+> ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey
+> ```
+> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
+> until `-e memattr_survey_reboots_authorised=true` is passed. See
+> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
+
 ## Re-provisioning a node (tofu -replace / fresh OS)
 A node rebuild wipes **everything on the host filesystem** — including host-level config
 that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
@@ -109,6 +118,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
 `playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
 keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
 bring-up chain.
+`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
+for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
+role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
+can be mistyped; inventory membership cannot.
 
 ```bash
 # dev — completely unchanged, no extra vars

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -87,24 +87,155 @@ kube_vip_address: "192.168.2.9"
 # — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
 cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
 
-# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
-# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
-# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
-# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
-# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
-# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
-# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
+# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
+# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
+# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
+# backports kernel as a side effect of sharing the role), and this line switches it on for
+# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
+# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
+# rather than from anyone remembering.
 #
-# The overrides below are deliberately present-but-broken: leaving them UNSET would
-# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
-# hardware — a confusing failure that looks like a wiring bug. This value fails just
-# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
-# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
-# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
-# updates these two lines. All three nodes must end up with an identical layout
-# (Longhorn replica placement assumes symmetry) before any of this is used for real.
-node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
-node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
+# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
+# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
+# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
+# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
+# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
+# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
+# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
+# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
+# in §3.
+#
+# COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
+# guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
+# the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
+# back to false, roll back per the note the role writes to
+# /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
+# roles/node_baseline.
+#
+# The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
+# FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
+node_baseline_backports_kernel: true
+
+# --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
+# Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
+#
+#   DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
+#                     disk, claimed and mounted at the NEUTRAL mountpoint `/data`
+#                     (LABEL=k3s-data), then split between TWO tenants:
+#                       /data/longhorn   Longhorn's dataPath
+#                       /data/k3s        k3s --data-dir (etcd AND the containerd image store)
+#   ETCD           -> NO dedicated device. It lives inside /data/k3s on that partition.
+#
+# ================================ THE 2026-09-04 REVISION ================================
+# The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
+# /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
+# data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
+# deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
+# would therefore have filled /var during its FIRST image pulls — and with etcd on the same
+# partition, a full /var means etcd cannot write its WAL. That is cluster death, not
+# degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
+# mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
+# tenants. Nothing was installed on prod when this was decided, which is exactly why it was
+# decided then — it is cheap now and very expensive after bring-up.
+#
+# THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
+# were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
+# contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
+# MEASURED, not assumed — exit condition in §2.
+# =========================================================================================
+#
+# Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
+# not as a device path and a blank. That distinction is the point of this block:
+#
+# 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
+#    k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
+#    (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
+#    Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
+#    freezes a fact about today's hardware into inventory, where it goes stale silently the
+#    first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
+#    nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
+#    So inventory declares the INTENT ("the partition currently mounted at /srv") and the
+#    role discovers the device on the node, then proves it is safe to claim (a partition or
+#    disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
+#    unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
+#
+# 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
+#    describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
+#    layout etcd is inside /data/k3s on the claimed data partition, which is neither a
+#    dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
+#    Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
+#    and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
+#    on every run, that the filesystem backing the etcd data dir really is /data. If the
+#    mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
+#    WAL would land back on the OS filesystem, invisibly, until the disk filled. The
+#    `root-filesystem` value makes no such claim and gets no such assert — which is exactly
+#    why the two are distinct enum values rather than one with a comment.
+#
+# 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
+#    the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
+#    the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
+#    this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
+#    k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
+#    Longhorn volumes are untouched.
+node_baseline_longhorn_source: claim-mountpoint
+node_baseline_longhorn_claim_mountpoint: /srv
+node_baseline_etcd_source: shared-data-partition
+
+# --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
+# `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
+# CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
+# /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
+# as the destination would make "claimed" and "not claimed" indistinguishable.
+# `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
+# partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
+# else's named volume — so labelling ours `data` would make a stranger's `data` volume look
+# already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
+# `k3s-etcd` label convention already in this repo.
+node_baseline_data_mount: /data
+node_baseline_data_fs_label: k3s-data
+# Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
+# equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
+# side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
+# default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
+node_baseline_longhorn_data_path: /data/longhorn
+
+# --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
+# The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
+# `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
+# label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
+# would walk into the greenfield claim path — which fails closed at "the claim mountpoint
+# must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
+# is right but it is not convergence, so the transition is DECLARED here: the role unmounts
+# the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
+# the partition is empty (nothing orphaned) and is the device the retired label names.
+# Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
+# SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
+# then skip on the retired-label lookup and nothing else depends on them.
+node_baseline_data_retired_fs_label: longhorn
+node_baseline_data_retired_mount: /var/lib/longhorn
+
+# --- k3s --data-dir (§2) -----------------------------------------------------------
+# THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
+# claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
+# `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
+# which defers to this variable) creates the directory 0700 on the data partition and derives
+# etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
+# Ansible side cannot disagree with itself.
+#
+# Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
+# it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
+# dir would orphan that mount under a running cluster.
+k3s_data_dir: /data/k3s
+
+# Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
+# source uses a device path any more, but leaving these UNSET would let prod silently
+# inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
+# metal) the moment anyone flipped a source back to `device`. These values can never parse
+# as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
+# before mkfs is ever reached.
+node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
+node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
 
 # --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
 # roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
@@ -158,15 +289,70 @@ node_baseline_dns_servers:
   #   LAN clients       -> PiHole. Not a cluster concern.
   #   nodes             -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
   #
-  # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
-  # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
-  # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
-  # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
-  # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
-  # with the dev Gateway VIP.
+  # THE k3s API SERVER is a node process and carries `oidc-issuer-url=...` (built from
+  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
+  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
   #
-  # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
-  # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
-  # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
-  # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
+  # ORDERING HAZARD, NOT YET CLEARED. Prod's issuer is keycloak.home.keithjackson.dev — an APEX
+  # name, because prod IS the apex under the 2026-09-04 inversion (k3s_base_domain below).
+  # That is the name the LIVE DEV CLUSTER answers today, and its public A record currently
+  # points at DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. A prod
+  # node resolving its own issuer through these public upstreams therefore reaches dev until:
+  #   (1) dev has migrated to *.dev.home.keithjackson.dev and RETIRED its apex records, and
+  #   (2) PROD's own external-dns has published keycloak.home.keithjackson.dev at PROD's
+  #       Gateway VIP (192.168.2.20).
+  # Only then do the public answer and the in-cluster split-horizon answer AGREE, and no
+  # /etc/hosts pin is needed. Until then kubectl OIDC SSO on prod cannot work; the cert-based
+  # admin kubeconfig is the break-glass path and always works. Ordering, not a design gap —
+  # the ordered steps are infra/PROD-PROVISIONING.md §10.
 node_baseline_dns_search: []
+
+# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
+# PROD **IS** THE APEX. [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml —
+# the same one-fact-in-two-places contract as k3s_data_dir and k3s_api_host. The Flux side
+# cannot supply this: the kube-apiserver's oidc-issuer-url is a k3s server FLAG written to
+# /etc/rancher/k3s/config.yaml, not a manifest, so postBuild.substituteFrom never sees it.
+# Full rationale in roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url derives.
+#
+# WHY THE SUFFIXES SPLIT (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve
+# the SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
+# deleted every one of dev's public records.
+#
+# WHICH WAY ROUND — INVERTED 2026-09-04 by operator decision, superseding the interim split
+# committed earlier the same day (prod on `prod.home.` behind apex CNAMEs). The simpler target:
+#   prod -> home.keithjackson.dev       (THE APEX; this variable — no `prod.` label, no CNAMEs)
+#   dev  -> dev.home.keithjackson.dev   (roles/k3s_server/defaults/main.yml)
+# So prod's Keycloak issuer is https://keycloak.home.keithjackson.dev/realms/homelab, and the
+# `kubernetes` OIDC client in PROD's OWN Keycloak realm is what prod's kubectl SSO authenticates
+# against.
+#
+# ORDERING HAZARD SPECIFIC TO THIS VARIABLE. keycloak.home.keithjackson.dev is the name the LIVE
+# dev cluster answers today, and prod's nodes resolve the issuer through PUBLIC upstreams (see
+# node_baseline_dns_search above). So until dev has vacated the apex and PROD's external-dns has
+# published that name at PROD's Gateway VIP (192.168.2.20), a prod node resolving its own issuer
+# gets DEV's VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. kubectl OIDC SSO on prod
+# therefore does not work until §10's dev-vacate steps are complete; the cert-based admin
+# kubeconfig is the break-glass path and always works. Ordering, not a design gap.
+k3s_base_domain: "home.keithjackson.dev"
+
+# --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
+# roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
+# and group_vars/all.yml carries no override, so dev keeps resolving exactly that).
+# Prod bootstraps its OWN overlay. Before this seam existed the role could bootstrap
+# clusters/dev ONLY — bootstrap/flux/bootstrap.sh's FLUX_ENV_PATH default was never
+# exported by the role — so `flux bootstrap ... --path=clusters/prod` (§8 step 4 of
+# infra/PROD-PROVISIONING.md) was simply not reachable without editing the role.
+#
+# Same seam as kube_vip_address / k3s_api_host / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel above: set once for the [k3s_prod] umbrella group, so a
+# FOURTH prod node inherits it from group membership with no per-host or per-run flag.
+#
+# WHAT clusters/prod STILL NEEDS before this is safe to run — do NOT invoke the
+# flux_bootstrap role until these are true (PROD-PROVISIONING.md §7):
+#   - the remaining TBD-PROD-* placeholders in clusters/prod/kustomization.yaml are real
+#     (S3_BACKUP_ENDPOINT); they are INVALID on purpose so an early apply fails loudly;
+#   - the GitRepository url there is the TEMPORARY workstation Gitea
+#     (http://192.168.2.132:3000/...) and that container is actually running and reachable
+#     from the prod nodes — it is outside every cluster and can simply be powered off;
+#   - the `sops-age` Secret is installed out of band on prod (bootstrap/sops).
+flux_bootstrap_env_path: "clusters/prod"

infra/ansible/inventory/hosts.ini

@@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 
 # =============================================================================
 # PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
-# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
-# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
-# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
-# nothing has been run against node 1, and the operator has decided the CLUSTER stays
-# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
-# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
-# later member-add dance.
+# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
+# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
+# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
+# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
+# check and the node-baseline PACKAGE floor has been run, and the operator has decided
+# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
+# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
+# single-server-etcd window, no later member-add dance.
 #
 # KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
 # hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
@@ -87,12 +88,23 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
 
 [k3s_prod_secondary]
-# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md
-# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so
-# no play can accidentally target a host that isn't there; uncomment each line only once
-# that node is actually on the bench and answering SSH.
-# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
-# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
+# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
+# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
+# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
+# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
+# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
+# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
+# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
+# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
+# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
+# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
+# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
+# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
+# three-way, not a node-1 outlier.
+# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
+# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
+k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
+k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
 
 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
 # pattern; named to match the group_vars directory so Ansible auto-loads it).

infra/ansible/playbooks/memattr-survey.yml

@@ -0,0 +1,137 @@
+---
+# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
+# R-ENV-03, R-TEST-02).
+#
+# WHAT QUESTION THIS ANSWERS
+#   infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
+#   EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
+#   Linux 6.12 reserves everything from the table address to the top of RAM without
+#   validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
+#   playbook measures them:
+#
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
+#        failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
+#        fleet-wide were at other addresses. If bad outcomes map onto a narrow address
+#        range, the causal chain closes with nothing touched.
+#     2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+#   §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
+#   is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
+#   a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
+#   boots per arm; detecting a mere odds shift needs ~50.
+#
+# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
+# package installs, no hardware moves, no node_baseline run. Its job is to establish real
+# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
+# a Fast Boot flip, a 6.13 kernel) compares against it.
+#
+# HOW TO RUN IT
+#   Dry run first — every stage except the reboots, so the pipeline is proven before 45
+#   minutes of wall clock are committed:
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml'
+#
+#   The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml \
+#           -e memattr_survey_reboots_authorised=true'
+#
+#   Fewer rounds, or one node:
+#     -e memattr_survey_rounds=5        --limit k3s-prod-2
+#
+#   RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
+#   rounds are never lost or double-counted. Analysis alone, over existing artifacts:
+#     ansible-playbook playbooks/memattr-survey.yml --tags analyse
+#
+# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
+# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
+# not a member of memattr_survey_required_group — the dev cluster is live, and a survey
+# that reboots it 20 times because of a mistyped --limit is exactly the accident worth
+# engineering against.
+#
+# HOW TO READ THE OUTPUT
+#   infra/ansible/_artifacts/memattr-survey/  (gitignored, regenerated every run)
+#     <host>.rounds.jsonl  one JSON line per round, mirrored off the node after EVERY round
+#     <host>.boots.json    every boot the persistent journal holds, re-derived in one pass
+#                          at the end — the AUTHORITATIVE record
+#     summary.md           the analysis, also printed at the end of the run
+#   See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
+#   with NO memattr log line is a healthy third state and not a missing measurement.
+
+# =====================================================================================
+# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
+# before a single reboot. That is the right asymmetry: the cost of a false stop is a
+# re-run, the cost of a false start is rebooting a machine that was doing something.
+# =====================================================================================
+- name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  any_errors_fatal: true
+  tags: [survey]
+  tasks:
+    - name: Preflight and baseline probe
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: preflight
+
+# =====================================================================================
+# PLAY 2 — THE SURVEY.
+#
+# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
+# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
+# keep collecting.
+#
+# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
+# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
+# that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
+# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
+# blocks belongs to exactly one host, so the other two sit idle through it and the survey
+# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
+# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
+# appeared in the output. `free` lets each host walk its own task list at its own pace,
+# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
+#
+# Consequences worth knowing before editing: output INTERLEAVES between hosts (every
+# progress line is host-tagged for exactly this reason), and `run_once` is unreliable
+# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
+# directory create on every host instead. Concurrency is also capped by `forks`
+# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
+# than letting a bigger fleet silently serialise again.
+# =====================================================================================
+- name: memattr survey — reboot rounds and journal harvest
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  strategy: free
+  tags: [survey]
+  tasks:
+    - name: Run the reboot rounds
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: rounds
+
+    - name: Harvest every retained boot from the persistent journal
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: harvest
+
+# =====================================================================================
+# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
+# even if every surveyed node died, because the artifacts are already on disk and the
+# analysis counts rounds from THOSE rather than from Ansible's host state. A node that
+# vanished shows up as a short arm with a wider interval, not as a missing report.
+# =====================================================================================
+- name: memattr survey — analyse
+  hosts: localhost
+  gather_facts: false
+  become: false
+  tags: [analyse]
+  tasks:
+    - name: Produce the analysis
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: analyse

infra/ansible/roles/cilium/defaults/main.yml

@@ -15,9 +15,20 @@ cilium_namespace: "kube-system"
 # Cilium 1.20.1 targets Gateway API v1.6.1.
 gateway_api_version: "v1.6.1"
 gateway_api_crd_urls:
+  # ONE URL, deliberately. As of v1.6.0 the STANDARD channel bundle already contains
+  # TLSRoute (plus TCPRoute, UDPRoute, ListenerSet and BackendTLSPolicy), so the separate
+  # experimental TLSRoute manifest this list used to carry is redundant — and worse than
+  # redundant: v1.6 also ships a ValidatingAdmissionPolicy,
+  # `safe-upgrades.gateway.networking.k8s.io`, which REJECTS an experimental-channel CRD
+  # applied over a standard-channel one. Applying the old second URL therefore fails hard:
+  #   ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' ... denied request:
+  #   Installing experimental CRDs on top of standard channel CRDs is prohibited by default.
+  # Verified 2026-09-04 on the prod bootstrap: standard-install.yaml v1.6.1 alone produces the
+  # exact 10 gateway.networking.k8s.io CRDs the LIVE dev cluster is running, tlsroutes included
+  # and annotated channel=standard, bundle-version=v1.6.1 — so this is parity with dev, not a
+  # reduction. Dev needs no reapply; it already has them. If gateway_api_version is ever moved
+  # BACK below v1.6.0, the experimental TLSRoute URL has to come back with it.
   - "https://github.com/kubernetes-sigs/gateway-api/releases/download/{{ gateway_api_version }}/standard-install.yaml"
-  # TLSRoute (experimental) is required by Cilium's Gateway API support.
-  - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
 
 # Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
 # comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev

infra/ansible/roles/flux_bootstrap/defaults/main.yml

@@ -19,5 +19,22 @@ flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
 flux_bootstrap_sops_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/sops/install-age-key.sh"
 
 # Assumed interface of bootstrap/flux (GitOps Engineer, T-1.4):
-#   runs `flux bootstrap ... --path=clusters/dev` against this cluster.
+#   runs `flux install` + `kubectl apply -k <env path>` against this cluster.
 flux_bootstrap_flux_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/flux/bootstrap.sh"
+
+# --- WHICH ENVIRONMENT OVERLAY THIS BOOTSTRAPS (R-ENV-02) -------------------------
+# The one thing that makes this role environment-agnostic. bootstrap/flux/bootstrap.sh
+# reads FLUX_ENV_PATH (default clusters/dev) and ends with `kubectl apply -k <that>`;
+# tasks/main.yml exports this value into that variable, so the role NEVER names an
+# environment and the task graph is identical for dev and prod.
+#
+# THE DEFAULT IS DEV ON PURPOSE — dev is LIVE, and group_vars/all.yml carries no
+# override, so a dev run resolves exactly the value it always had (clusters/dev) and
+# behaves byte-for-byte as before.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same group_vars
+# seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel. No role fork, no per-run extra-var, no second
+# playbook: group membership decides. Before this existed the role could bootstrap
+# ONLY clusters/dev, so prod bring-up was impossible without editing the role.
+flux_bootstrap_env_path: "clusters/dev"

infra/ansible/roles/flux_bootstrap/tasks/main.yml

@@ -4,11 +4,17 @@
 # — it calls the entrypoints the Security + GitOps Engineers author there.
 #
 # Order matters: install the age decryption key FIRST (so Flux can decrypt on its
-# very first reconcile), THEN run `flux bootstrap` at clusters/dev.
+# very first reconcile), THEN bootstrap Flux at the environment's overlay.
 #
 # The two commands are variables (see defaults/main.yml). If the bootstrap workers
 # expose different entrypoints, override flux_bootstrap_sops_cmd / _flux_cmd — the
 # task graph does not change.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): this role names NO environment. Which overlay gets
+# bootstrapped comes from `flux_bootstrap_env_path` (defaults/main.yml, default
+# clusters/dev), exported below as FLUX_ENV_PATH — the variable bootstrap/flux/
+# bootstrap.sh already reads. Prod overrides it in group_vars/k3s_prod/vars.yml, the
+# same seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name.
 
 - name: Assert the bootstrap/ subtree exists (authored by the other worker)
   ansible.builtin.stat:
@@ -29,11 +35,14 @@
   run_once: true
   changed_when: true
 
-- name: Bootstrap Flux against clusters/dev (bootstrap/flux)
+- name: Bootstrap Flux against this environment's overlay (bootstrap/flux)
   ansible.builtin.command:
     cmd: "{{ flux_bootstrap_flux_cmd }}"
   environment:
     KUBECONFIG: "{{ flux_bootstrap_kubeconfig }}"
+    # The ONLY per-environment input. clusters/dev by default (dev unchanged);
+    # clusters/prod for [k3s_prod] hosts via group_vars.
+    FLUX_ENV_PATH: "{{ flux_bootstrap_env_path }}"
   delegate_to: localhost
   become: false
   run_once: true

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
 # the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
 k3s_disable: "traefik,servicelb,local-storage"
 
+# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
+# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
+# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
+# image store (agent/containerd) — which is by far the largest tenant.
+#
+# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
+# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
+# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
+# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
+# running cluster's entire state directory out from under it and orphan that etcd mount.
+#
+# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
+# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
+# node inherits it from group membership rather than from anyone remembering. It is set
+# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
+# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
+# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
+# this same variable). One line moves both.
+#
+# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
+# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
+# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
+# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
+# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
+#
+# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
+# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
+# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
+# Restart handler for a value that is not changing. The install flags are the correct and
+# sufficient place, and prod has no k3s installed yet.
+k3s_data_dir: "/var/lib/rancher/k3s"
+
 # Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
 k3s_server_mode: "secondary"
 
@@ -34,3 +66,44 @@ k3s_tls_sans: >-
   {{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
       + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
      | unique }}
+
+# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
+# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
+#
+# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
+# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
+# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
+# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
+#
+#   Flux side     clusters/base/cluster-vars.yaml   BASE_DOMAIN      (per-overlay)
+#   Ansible side  THIS variable                     k3s_base_domain  (per-inventory-group)
+#
+# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
+# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
+# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
+# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
+# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
+#
+# THE DEFAULT IS THE DEV VALUE — the same convention as k3s_data_dir above and cluster-vars'
+# GATEWAY_VIP / LONGHORN_DATA_PATH: the role default is dev's value and prod overrides it in
+# inventory/group_vars/k3s_prod/vars.yml (the same inventory seam k3s_data_dir / k3s_api_host /
+# kube_vip_address use, so a FOURTH prod server inherits it from group membership rather than
+# from anyone remembering).
+#
+# *** WHICH DEV VALUE — UPDATED 2026-09-04, AND IT IS AHEAD OF THE LIVE DEV CLUSTER. ***
+# The operator inverted the suffix split: PROD is now the apex `home.keithjackson.dev` and DEV
+# carries the label `dev.home.keithjackson.dev`. This default therefore moved from the apex to
+# dev's subdomain, mirroring BASE_DOMAIN in clusters/dev/kustomization.yaml.
+#
+# CONSEQUENCE — this default no longer matches what the live dev nodes are running, so unlike
+# every previous edit to this file A DEV RE-RUN IS NOT A NO-OP. Rendering config.yaml with the
+# new issuer CHANGES the file, FIRES the Restart handler, and rolls k3s on all three dev
+# servers, which also invalidates every user's OIDC kubeconfig (SSO must be re-issued). That is
+# a deliberate step of the dev DNS migration, NOT a side effect to discover during unrelated
+# work: DO NOT run the k3s_server role against dev until step D6 of
+# infra/PROD-PROVISIONING.md §10, and never while the operator is away. The break-glass
+# cert-based admin kubeconfig keeps working throughout, which is what makes the step safe.
+k3s_base_domain: "dev.home.keithjackson.dev"
+
+# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
+k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -22,6 +22,7 @@
       --write-kubeconfig-mode=644
       {% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
       --node-ip={{ k3s_node_ip }}
+      --data-dir={{ k3s_data_dir }}
 
 # On an ALREADY-installed cluster the install task above is idempotent (creates:
 # k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
@@ -74,13 +75,19 @@
       # OIDC (Keycloak SSO for kubectl). The API server trusts ID tokens from the homelab realm;
       # the `kubernetes` PUBLIC client (PKCE) + kubelogin mint them, and RBAC binds the `groups`
       # claim (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins->cluster-admin,
-      # oidc:viewers->view). Nodes resolve keycloak.home.keithjackson.dev to the Cilium gateway VIP
-      # (192.168.1.53) and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
+      # oidc:viewers->view). Nodes resolve keycloak.<k3s_base_domain> to the Cilium gateway VIP
+      # and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
       # prefixes namespace federated identities away from cert / service-account subjects.
       # ADDITIVE + break-glass-safe: the cert-based admin kubeconfig keeps full access with or
       # without Keycloak, so a bad realm/OIDC state can never lock you out of the cluster.
+      #
+      # NOT a Flux ${BASE_DOMAIN} token: this is a k3s SERVER FLAG, not a manifest, so
+      # postBuild.substituteFrom never touches it. k3s_oidc_issuer_url derives from
+      # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
+      # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
+      # config.yaml is unchanged and the Restart handler does not fire.
       kube-apiserver-arg:
-        - "oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab"
+        - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
         - "oidc-client-id=kubernetes"
         - "oidc-username-claim=email"
         - "oidc-username-prefix=oidc:"
@@ -97,15 +104,20 @@
     INSTALL_K3S_VERSION: "{{ k3s_version }}"
   when: k3s_server_mode == 'primary'
 
+# PATH FOLLOWS --data-dir. k3s writes the node-token to <data-dir>/server/token, NOT to
+# the literal /var/lib/rancher/k3s. On dev k3s_data_dir IS /var/lib/rancher/k3s (the role
+# default, restated explicitly above) so this is byte-for-byte the old path; on prod
+# --data-dir is /data/k3s (group_vars/k3s_prod) and the hardcoded path did not exist at all
+# — this task would have timed out after 180s and the two secondaries could never join.
 - name: Wait for the k3s node-token to exist (primary)
   ansible.builtin.wait_for:
-    path: /var/lib/rancher/k3s/server/token
+    path: "{{ k3s_data_dir }}/server/token"
     timeout: 180
   when: k3s_server_mode == 'primary'
 
 - name: Read the k3s node-token (primary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   register: k3s_token_raw
   when: k3s_server_mode == 'primary'
 
@@ -125,9 +137,11 @@
 # "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
 # dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
 # delegates to the PROD primary using this same role, unchanged (R-ENV-02).
+# k3s_data_dir is a CLUSTER-WIDE value (group_vars for prod, role default for dev), so the
+# secondary's own copy of it is also the primary's data dir — correct for the delegated read.
 - name: Fetch the k3s node-token directly from the primary (secondary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   delegate_to: "{{ groups[k3s_primary_group][0] }}"
   register: k3s_primary_token_raw
   when:

infra/ansible/roles/kube_vip/defaults/main.yml

@@ -31,4 +31,14 @@ kube_vip_retry_period: 2
 
 # k3s auto-deploy manifests dir: k3s applies anything here on start, so kube-vip
 # is reproducible and independent of Flux/Helm.
-kube_vip_manifest_path: "/var/lib/rancher/k3s/server/manifests/kube-vip.yaml"
+#
+# THE PATH FOLLOWS --data-dir. k3s reads auto-deploy manifests from
+# <data-dir>/server/manifests, so this must be derived from the cluster-wide
+# k3s_data_dir (roles/k3s_server turns the same variable into --data-dir), not from the
+# literal /var/lib/rancher/k3s. Dev leaves k3s_data_dir unset outside roles/k3s_server's
+# own defaults, so the fallback below reproduces dev's existing path exactly; prod sets
+# k3s_data_dir: /data/k3s in group_vars/k3s_prod, where the old hardcoded path was simply
+# not the directory k3s watches — the manifest would have been written and silently never
+# applied, and the control-plane VIP would never have come up.
+kube_vip_manifest_path: >-
+  {{ k3s_data_dir | default('/var/lib/rancher/k3s') }}/server/manifests/kube-vip.yaml

infra/ansible/roles/kube_vip/tasks/main.yml

@@ -1,13 +1,22 @@
 ---
 # Deploy kube-vip (control-plane HA VIP) via the k3s auto-deploy manifests dir.
-# k3s applies /var/lib/rancher/k3s/server/manifests/*.yaml on start, so this is
-# reproducible and independent of Flux. The same manifest is written on every
+# k3s applies <data-dir>/server/manifests/*.yaml on start, so this is
+# reproducible and independent of Flux (see kube_vip_manifest_path in defaults/main.yml —
+# the dir follows k3s's --data-dir, which prod moves to /data/k3s). The same manifest is written on every
 # server so the DaemonSet survives the loss of any single node.
 
-- name: Detect the LAN interface holding the node's 192.168.1.5x address
+# WHICH INTERFACE kube-vip ARPs THE VIP ON. Detected from this node's OWN k3s_node_ip
+# (inventory, per host) rather than a hardcoded subnet: the previous awk matched the
+# literal 192.168.1.5[0-2], i.e. DEV's node addresses, so on prod (192.168.2.10/.11/.12)
+# it returned nothing and the assert below aborted the play. k3s_node_ip is the same
+# address k3s advertises with --node-ip and is defined for dev and prod alike, so this
+# resolves eno1 on prod and the identical dev interface it always resolved before.
+# `ip addr show to <ip>/32` does the matching in iproute2 instead of a regex over dotted
+# quads (where `.` would match any character).
+- name: Detect the LAN interface holding this node's k3s_node_ip
   ansible.builtin.shell: >-
     set -o pipefail;
-    ip -o -4 addr show | awk '/192\.168\.1\.5[0-2]\//{print $2; exit}'
+    ip -o -4 addr show to {{ k3s_node_ip }}/32 | awk '{print $2; exit}'
   args:
     executable: /bin/bash
   register: kube_vip_iface_detect
@@ -25,7 +34,7 @@
       - kube_vip_vip_interface is defined
       - kube_vip_vip_interface | length > 0
     fail_msg: >-
-      Could not detect the 192.168.1.5x LAN interface for kube-vip on
+      Could not detect the interface holding {{ k3s_node_ip }} for kube-vip on
       {{ inventory_hostname }} — set kube_vip_interface explicitly.
 
 - name: Ensure the k3s auto-deploy manifests directory exists

infra/ansible/roles/memattr_survey/README.md

@@ -0,0 +1,201 @@
+# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery
+
+**Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02
+**Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism
+is settled there and this role does **not** re-derive it.
+
+## The question
+
+§0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes
+Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it,
+and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from
+the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.
+
+Two things §0b leaves unmeasured, and this role measures:
+
+1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed
+   failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes
+   fleet-wide were at other addresses. If bad outcomes map onto a narrow address range,
+   the causal chain closes without touching hardware or firmware.
+2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each.
+
+## Why 20 rounds
+
+Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson
+interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is
+worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a
+mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds`
+is a variable; lowering it below 20 produces an arm that cannot answer the question it was
+built for, and the analysis will say so.
+
+## This is the BASELINE arm
+
+**Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware
+moves, no `node_baseline` run. The role installs nothing and configures nothing; its only
+write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's
+own allocation footprint — initrd size, module set, boot device — is *part of the causal
+chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that
+the kernel release and cmdline were constant on every node, for exactly this reason.
+
+## Running it
+
+```bash
+# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml'
+
+# The real thing. Reboots are OPT-IN; the interlock is deliberate.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml \
+      -e memattr_survey_reboots_authorised=true'
+
+# Re-analyse existing artifacts, no hardware touched.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml --tags analyse'
+```
+
+Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`,
+`-e memattr_survey_hosts=<group>`.
+
+20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are
+independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That
+parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count;
+preflight asserts the latter. See the playbook comment for why the default linear
+strategy silently serialises a looped `include_tasks` (it was measured, not assumed).
+
+## Safety — verified, not assumed
+
+The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That
+premise is re-checked on **every** run, because a document and a machine drift apart:
+
+| Check | Failure mode it prevents |
+|---|---|
+| Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. |
+| `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. |
+| `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. |
+| `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. |
+| `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. |
+
+Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the
+whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a
+false start is a rebooted production machine.
+
+## Fail-safe and resume
+
+* **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node
+  that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the
+  observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that
+  host only** and the others keep collecting. The abort is announced immediately and again
+  in the report — which counts rounds from the **artifacts**, not from Ansible's host
+  state, so even a node that goes fully unreachable (which `rescue` cannot catch) still
+  appears as a short arm with a wider interval, never as a silently missing report.
+* **Resumable.** Round numbers come from the on-node ledger, so re-running after an
+  interruption continues where it stopped and can neither lose nor duplicate a round. The
+  boot found on a resume is *not* re-appended: it is already round N of the interrupted
+  run, and double-counting it would bias the sample toward whichever outcome survived the
+  interruption.
+* **The abort is durable, not scrollback.** `<host>.aborted.json` records the round, the
+  failing task and the reason, so the report can say *why* an arm is short instead of
+  leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every
+  run and re-written only if that run aborts. Its *absence* is not proof of success — a
+  node can be lost abruptly enough that Ansible calls it unreachable before the rescue
+  runs — so completeness is still judged on round count, and the marker only supplies the
+  reason when there is one.
+* **Analysis survives everything.** It is a separate play on `localhost`, so it runs even
+  if every surveyed node died. With a node down, `--tags analyse` is the way in: a full
+  run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host.
+
+## Belt and braces on the data
+
+Two independent records, joined on `boot_id`:
+
+| | written by | authoritative for |
+|---|---|---|
+| `<host>.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** |
+| `<host>.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** |
+| `<host>.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** |
+
+The journal does not care whether the driver was alive: the boot happened, the kernel
+logged it, journald persisted it. So the harvest can only ever be a *superset* of the
+ledger, and a round the driver failed to append still shows up.
+
+## Reading the output
+
+Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as
+`roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never
+committed. `summary.md` is also printed at the end of the run.
+
+### The verdict is measured, never inferred from a log line
+
+This is the subtlest part of the design. There are **three** kernel readings, not two:
+
+| `kernel_read` | What the kernel logged | Memory |
+|---|---|---|
+| `valid` | **nothing at all** — it parsed a table it considered valid | intact |
+| `bail-version` | `Unexpected EFI Memory Attributes table version <garbage>` | intact — the *lucky* path, and it looks alarming |
+| `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** |
+
+Silence is therefore *evidence of health*, not a missing measurement — which is why the
+record carries an explicit `memattr_diagnostic_present: false` rather than an empty field.
+And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`,
+*later* than the `memblock_reserve()` that did the damage: it is a consequence, not the
+cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and
+`kernel_read` is recorded alongside it as an independent observation. §5a of the report
+cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.
+
+Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed
+clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything
+landing between them is reported as `indeterminate` and excluded from `n` — a new third
+thing deserves human eyes, not a coin flip into a bucket.
+
+### What is and is not pooled
+
+Boots the survey did **not cause** are shown in the per-round table for context and
+excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot
+each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so
+pooling it would double-count prior evidence into a run whose point is fresh trials).
+
+### The statistics
+
+`files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is.
+
+* **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the
+  Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly
+  instead of needing a special case. A bare fraction is what this exercise exists to stop
+  reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
+* **The address↔outcome question is tested three ways**, because the tempting version is
+  circular:
+  * **4b determinism** — was any single address seen with *both* outcomes? One such
+    address falsifies "the address determines the outcome" outright.
+  * **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed
+    up as a p-value: the bad set is defined after seeing the data.
+  * **4d the pre-registered prediction** — §0b, written before this run, names
+    `0xbeeb0018`. Fisher exact against this run's boots only. This is the one
+    non-circular test available.
+  * **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a
+    run produces no bad boot, `verdict` has no variance to explain, but `kernel_read`
+    does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in
+    §0b's mechanism. An address that reproduces the same *parse* boot after boot is the
+    address doing causal work, even when none of those parses was catastrophic.
+
+## Files
+
+| Path | What it is |
+|---|---|
+| `tasks/preflight.yml` | Safety gate, resume point, baseline probe |
+| `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock |
+| `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue |
+| `tasks/harvest.yml` | The authoritative journal pass |
+| `tasks/analyse.yml` | Controller-side analysis |
+| `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable |
+| `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report |
+
+## What this cannot tell you
+
+It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute
+the difference between nodes, because node 2 is simultaneously the only single-DIMM node
+and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled
+rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS
+settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this
+role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all
+three nodes, and that is not in dispute.

infra/ansible/roles/memattr_survey/defaults/main.yml

@@ -0,0 +1,92 @@
+---
+# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
+# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
+# settled and is NOT re-derived here.
+#
+# THE QUESTION THIS ROLE ANSWERS
+#   §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
+#   Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
+#        observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
+#        successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
+#        address range, the causal chain is closed with no hardware or firmware touched.
+#     2. Is the fault node-2-specific, or does any node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+# WHY 20 ROUNDS AND NOT FIVE
+#   Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
+#   interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
+#   boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
+#   needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
+#   not a round number.
+#
+# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
+#   No BIOS writes, no kernel change, no package installs, no hardware moves. The role
+#   installs nothing and configures nothing; its only write to a node is its own append-
+#   only ledger under /var/lib. Anything that alters the boot chain's allocation
+#   footprint — and per §0b that includes the initrd, the module set and the boot device —
+#   changes the variable under test and voids the arm.
+
+# --- Sample size --------------------------------------------------------------------
+# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
+# machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
+memattr_survey_rounds: 20
+
+# --- Safety interlock ----------------------------------------------------------------
+# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
+# preflight, a baseline probe of the current boot, the journal harvest and the analysis.
+# That is a genuinely useful dry run and it is also how you prove the pipeline works
+# before committing 45 minutes of wall clock to it.
+memattr_survey_reboots_authorised: false
+
+# The role refuses to run against a host that is not in this group. The prod boxes carry
+# no k3s and nothing is running on them, which is the entire reason rebooting them is
+# sanctioned; the dev cluster is live and must never be caught by a stray --limit.
+memattr_survey_required_group: k3s_prod
+
+# Unit names whose presence or activity means something IS running on this node. Any hit
+# aborts the whole playbook before the first reboot (see tasks/preflight.yml).
+memattr_survey_forbidden_units:
+  - k3s
+  - k3s-agent
+  - kubelet
+  - containerd
+  - docker
+# Paths whose existence means k3s has been installed here, even if not currently running.
+memattr_survey_forbidden_paths:
+  - /etc/rancher/k3s
+  - /var/lib/rancher/k3s
+  - /usr/local/bin/k3s
+
+# --- Reboot behaviour -----------------------------------------------------------------
+# A node that does not answer within this is declared lost: the role stops rebooting THAT
+# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
+# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
+# slow boot.
+memattr_survey_reboot_timeout: 600
+memattr_survey_connect_timeout: 20
+# Let journald settle before reading the new boot's kernel log. The kernel log itself is
+# complete long before SSH is up; this is purely margin.
+memattr_survey_post_reboot_delay: 15
+
+# --- On-node ledger (belt) -------------------------------------------------------------
+# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
+# the controller dies. This is what makes the survey resumable: re-running skips rounds
+# already recorded here rather than starting over.
+memattr_survey_state_dir: /var/lib/memattr-survey
+memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
+
+# --- Controller artifacts (braces) ------------------------------------------------------
+# Same convention as roles/node_health and conformance/_artifacts/: generated output under
+# a `_artifacts/` directory that .gitignore excludes, never committed.
+#   <host>.rounds.jsonl  the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
+#                        mid-survey still leaves every completed round on the controller
+#   <host>.boots.json    the journal harvest — every retained boot re-derived in one
+#                        pass. THIS is the authoritative record; the ledger is the backup.
+#   summary.md           the analysis (also printed)
+memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
+memattr_survey_write_artifacts: true
+
+# Print each round's one-line result as it happens. A 45-minute unattended run with no
+# output is indistinguishable from a hung one.
+memattr_survey_show_progress: true

infra/ansible/roles/memattr_survey/files/memattr_analyse.py

@@ -0,0 +1,521 @@
+#!/usr/bin/env python3
+"""memattr_analyse.py — turn a memattr survey's artifacts into the answer.
+
+Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads
+  <host>.boots.json    the journal harvest — authoritative, one entry per retained boot
+  <host>.rounds.jsonl  the per-round ledger — authoritative for ROUND NUMBERS
+and joins them on boot_id.
+
+WHAT IT ANSWERS, in order:
+
+  1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated
+     loudly and up front, because every number below it is weaker than it looks.
+
+  2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing
+     this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and
+     completely different evidence. Clopper-Pearson is the exact (conservative) binomial
+     interval — no normal approximation, correct at 0/n and n/n where the approximation is
+     degenerate and where a clean arm is most likely to land.
+
+  3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three
+     ways, deliberately, because the tempting version of this test is circular:
+       (a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such
+           address falsifies "the address determines the outcome" outright, and no amount
+           of correlation elsewhere repairs it.
+       (b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported
+           as sorted min/max per outcome plus the count of good addresses falling inside
+           the bad range. This is descriptive; it is NOT dressed up as a p-value, because
+           the bad set was chosen after seeing the data.
+       (c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts
+           that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots
+           this run observed, which is the one non-circular test available.
+
+  4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory
+     ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell.
+
+Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import math
+import os
+import sys
+from collections import Counter, defaultdict
+
+# §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad
+# both times, while all five successes fleet-wide were at other addresses.
+PREREGISTERED_BAD_ADDRESS = "0xbeeb0018"
+
+
+# --- Exact binomial interval ----------------------------------------------------------
+def binom_tail_ge(n: int, k: int, p: float) -> float:
+    """P(X >= k) for X ~ Bin(n, p)."""
+    if k <= 0:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1))
+
+
+def binom_tail_le(n: int, k: int, p: float) -> float:
+    """P(X <= k) for X ~ Bin(n, p)."""
+    if k >= n:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1))
+
+
+def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
+    """Exact 95% (by default) binomial confidence interval for k successes in n trials.
+
+    Solved by bisection on the exact tail probabilities rather than via the Beta/F
+    quantile identity, so it needs nothing outside the standard library and the
+    boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing:
+      lower bound solves P(X >= k | p) = alpha/2,  and is 0 when k = 0
+      upper bound solves P(X <= k | p) = alpha/2,  and is 1 when k = n
+    """
+    if n == 0:
+        return (0.0, 1.0)
+
+    def bisect(fn, target, lo, hi):
+        for _ in range(200):
+            mid = (lo + hi) / 2
+            if fn(mid) < target:
+                lo = mid
+            else:
+                hi = mid
+        return (lo + hi) / 2
+
+    lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0)
+    upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0)
+    return (lower, upper)
+
+
+def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float:
+    """Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only.
+
+    Used ONLY for the pre-registered prediction (address == the §0b address vs outcome),
+    never for a bad set defined after the fact — that would be a garden of forking paths
+    with a p-value stapled on.
+    """
+    n = a + b + c + d
+    if n == 0:
+        return 1.0
+    row1, col1 = a + b, a + c
+
+    def prob(x):
+        return (
+            math.comb(row1, x)
+            * math.comb(n - row1, col1 - x)
+            / math.comb(n, col1)
+        )
+
+    observed = prob(a)
+    lo = max(0, col1 - (n - row1))
+    hi = min(row1, col1)
+    # 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating
+    # float noise in the equality case.
+    return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9)))
+
+
+# --- Loading ---------------------------------------------------------------------------
+def load_host(art_dir: str, host: str) -> dict:
+    """Harvest (authoritative outcomes) joined to ledger (authoritative round numbers)."""
+    with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle:
+        harvest = json.load(handle)
+
+    rounds_by_boot: dict[str, int] = {}
+    ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl")
+    ledger: list[dict] = []
+    if os.path.exists(ledger_path):
+        with open(ledger_path, encoding="utf-8") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                try:
+                    record = json.loads(line)
+                except json.JSONDecodeError:
+                    continue
+                ledger.append(record)
+                if record.get("boot_id") and isinstance(record.get("round"), int):
+                    rounds_by_boot[record["boot_id"]] = record["round"]
+
+    boots = []
+    for boot in harvest.get("boots", []):
+        entry = dict(boot)
+        entry["round"] = rounds_by_boot.get(boot.get("boot_id"))
+        boots.append(entry)
+    boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0)))
+
+    # Written by the per-round rescue when a node stops coming back. Its absence is not
+    # proof of success — a node can be lost so abruptly that Ansible calls it unreachable
+    # before the rescue runs — so completeness is still judged on round COUNT, and this
+    # only supplies the reason when one exists.
+    abort = None
+    abort_path = os.path.join(art_dir, f"{host}.aborted.json")
+    if os.path.exists(abort_path):
+        try:
+            with open(abort_path, encoding="utf-8") as handle:
+                abort = json.load(handle)
+        except (OSError, json.JSONDecodeError):
+            abort = {"reason": "abort marker present but unreadable"}
+
+    return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort}
+
+
+def discover_hosts(art_dir: str) -> list[str]:
+    return sorted(
+        os.path.basename(p)[: -len(".boots.json")]
+        for p in glob.glob(os.path.join(art_dir, "*.boots.json"))
+    )
+
+
+# --- Rendering ---------------------------------------------------------------------------
+def gib(kb) -> str:
+    return "?" if kb is None else f"{kb / 1048576:.2f}"
+
+
+def pct(x) -> str:
+    return "?" if x is None else f"{100 * x:.0f}%"
+
+
+def render(hosts: list[dict], expected_rounds: int) -> str:
+    out: list[str] = []
+    w = out.append
+
+    all_boots = [b for h in hosts for b in h["boots"]]
+    # Only boots this survey CAUSED count toward any statistic. Excluded, deliberately:
+    #   * pre-existing journal boots (round None) — observed under an unknown selection
+    #     process, and already counted once in §0b's table;
+    #   * round 0, the boot each node was found sitting on — it is a boot nobody chose,
+    #     but for node 2 it is literally §0b's "boot 0" row, so pooling it would double-
+    #     count prior evidence into a run whose entire point is fresh, independent trials.
+    # Both are still SHOWN in the per-round table, because context is not contamination.
+    surveyed = [b for b in all_boots if b.get("round") not in (None, 0)]
+
+    w("# EFI Memory Attributes Table — baseline reboot survey")
+    w("")
+    w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no")
+    w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this")
+    w("measures p(bad) per node and tests whether the outcome is predicted by the address")
+    w("the firmware published the table at.")
+    w("")
+
+    # --- 1. Completeness -----------------------------------------------------------------
+    w("## 1. Completeness — read this before any number below it")
+    w("")
+    w("| node | rounds recorded | target | journal boots harvested | short? |")
+    w("|---|---:|---:|---:|---|")
+    short_arms = []
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        is_short = n < expected_rounds
+        if is_short:
+            short_arms.append((h["host"], n))
+        w(
+            f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | "
+            f"{'**YES — arm incomplete**' if is_short else 'no'} |"
+        )
+    w("")
+    if short_arms:
+        w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms))
+        w("> Every interval below is correspondingly wider than planned.")
+        w("")
+        for h in hosts:
+            if h.get("abort"):
+                a = h["abort"]
+                w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of "
+                  f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}")
+                w(f"> (failing task: {a.get('failed_task', 'unknown')})")
+                w("")
+
+    # --- 2. Per-round table ---------------------------------------------------------------
+    w("## 2. Per-round results")
+    w("")
+    for h in hosts:
+        installed = h["harvest"].get("smbios_installed_kb")
+        w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)")
+        w("")
+        w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |")
+        w("|---:|---|---|---|---:|---:|---:|---|")
+        for b in h["boots"]:
+            rnd = b.get("round")
+            label = "pre" if rnd is None else str(rnd)
+            usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb")
+            w(
+                f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | "
+                f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | "
+                f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | "
+                f"{gib(usable)} | {pct(b.get('usable_fraction'))} | "
+                f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |"
+            )
+        w("")
+        w("`round` = survey round. `pre` = a boot already in the journal before the survey;")
+        w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED")
+        w("from every statistic below. `kernel read`: `valid` = table")
+        w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage")
+        w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =")
+        w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from")
+        w("the presence of a log line.")
+        w("")
+
+    # --- 3. p(bad) -------------------------------------------------------------------------
+    w("## 3. p(bad) per node — 95% Clopper-Pearson")
+    w("")
+    w("| node | bad | n | p(bad) | 95% CI | reading |")
+    w("|---|---:|---:|---:|---|---|")
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        bad = sum(1 for b in rounds if b.get("verdict") == "bad")
+        lo, hi = clopper_pearson(bad, n)
+        if n == 0:
+            reading = "no data"
+        elif bad == 0:
+            reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}"
+        elif bad == n:
+            reading = f"bad every time in {n}; p(bad) at least {lo:.2f}"
+        else:
+            reading = f"p(bad) between {lo:.2f} and {hi:.2f}"
+        w(
+            f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | "
+            f"{lo:.2f} – {hi:.2f} | {reading} |"
+        )
+    w("")
+    indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")]
+    if indet:
+        w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are")
+        w("> reported as `indeterminate` rather than forced into a bucket. They are")
+        w("> excluded from n above. Inspect them by hand.")
+        w("")
+
+    # --- 4. Address vs outcome -----------------------------------------------------------
+    w("## 4. Is the outcome predicted by where the table landed?")
+    w("")
+    by_addr: dict[str, Counter] = defaultdict(Counter)
+    addr_hosts: dict[str, set] = defaultdict(set)
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        by_addr[addr][b.get("verdict")] += 1
+        addr_hosts[addr].add(b.get("host") or "?")
+
+    w("### 4a. Every address observed in the survey")
+    w("")
+    w("| MEMATTR | good | bad | other | nodes | verdict for this address |")
+    w("|---|---:|---:|---:|---|---|")
+    ambiguous = []
+    for addr in sorted(by_addr, key=lambda a: (a == "none", a)):
+        counts = by_addr[addr]
+        good, bad = counts.get("good", 0), counts.get("bad", 0)
+        other = sum(v for k, v in counts.items() if k not in ("good", "bad"))
+        if good and bad:
+            note = "**BOTH — address does not determine outcome**"
+            ambiguous.append(addr)
+        elif bad:
+            note = "always bad"
+        elif good:
+            note = "always good"
+        else:
+            note = "—"
+        w(
+            f"| `{addr}` | {good} | {bad} | {other} | "
+            f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |"
+        )
+    w("")
+
+    w("### 4b. Determinism")
+    w("")
+    if not by_addr:
+        w("No surveyed boots — nothing to test.")
+    elif ambiguous:
+        w(f"**FALSIFIED for {len(ambiguous)} address(es):** "
+          + ", ".join(f"`{a}`" for a in ambiguous)
+          + " was seen with BOTH outcomes. The address alone cannot determine the outcome.")
+    else:
+        w("**Not falsified.** No address in this survey was ever seen with both outcomes —")
+        w("consistent with the address determining the outcome, though with this many")
+        w("distinct addresses most were seen only once, which makes the test weak.")
+        singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1)
+        w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)")
+    w("")
+
+    w("### 4c. Separation of the address ranges")
+    w("")
+    bad_addrs = sorted({b["memattr_address"] for b in surveyed
+                        if b.get("verdict") == "bad" and b.get("memattr_address")})
+    good_addrs = sorted({b["memattr_address"] for b in surveyed
+                         if b.get("verdict") == "good" and b.get("memattr_address")})
+    if bad_addrs and good_addrs:
+        blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16)
+        inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi]
+        w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`")
+        w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`")
+        w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**"
+          + (" — the ranges interleave, so 'a narrow bad range' does not hold"
+             if inside else " — the ranges are disjoint"))
+        w("")
+        w("Descriptive only. The bad set was defined after seeing the data, so this is a")
+        w("picture, not a test.")
+    elif not bad_addrs:
+        w("No bad boot was observed in this survey, so there is no bad range to separate.")
+    else:
+        w("No good boot was observed in this survey.")
+    w("")
+
+    w("### 4d. The pre-registered prediction")
+    w("")
+    w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the")
+    w("bad address (both prior failures were there; all five prior successes elsewhere).")
+    w("Tested against this survey's boots only:")
+    w("")
+    a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    w(f"| | bad | good |")
+    w(f"|---|---:|---:|")
+    w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |")
+    w(f"| elsewhere | {c} | {d} |")
+    w("")
+    if a + b_ == 0:
+        w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the")
+        w("prediction had no opportunity to be right or wrong. That is itself informative:")
+        w("the address moves every boot, so a specific address is not a reproducible handle.")
+    else:
+        p = fisher_exact_2x2(a, b_, c, d)
+        w(f"Fisher exact (two-sided) p = **{p:.4g}**"
+          + ("  — the prediction holds." if p < 0.05 and a and not b_
+             else "  — not significant at 0.05." if p >= 0.05
+             else "  — significant, but read the cell counts before believing it."))
+    w("")
+
+    w("### 4e. Does the address predict how the KERNEL READ the table?")
+    w("")
+    w("The strongest test this data can support when no bad boot occurred. `verdict` has")
+    w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both")
+    w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:")
+    w("the kernel reads whatever bytes happen to be at the address the firmware published.")
+    w("If the address determines the reading, the address is doing causal work, one step")
+    w("short of the catastrophic path.")
+    w("")
+    reads_by_addr: dict[str, set] = defaultdict(set)
+    n_by_addr: Counter = Counter()
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        reads_by_addr[addr].add(b.get("kernel_read"))
+        n_by_addr[addr] += 1
+    repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1}
+    inconsistent = {a: r for a, r in repeated.items() if len(r) > 1}
+    distinct_reads = {r for rs in reads_by_addr.values() for r in rs}
+    w(f"- distinct addresses: **{len(reads_by_addr)}**, of which "
+      f"**{len(repeated)}** were observed more than once")
+    w(f"- distinct kernel readings across the survey: "
+      + ", ".join(f"`{r}`" for r in sorted(distinct_reads)))
+    w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**")
+    w("")
+    if len(distinct_reads) < 2:
+        w("Only one reading occurred in the whole survey, so there is nothing to predict.")
+    elif inconsistent:
+        w("**The address does NOT determine the reading:** "
+          + "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items()))
+    elif not repeated:
+        w("No address recurred, so the test could not be run.")
+    else:
+        w(f"**The address determines the reading, on every address observed more than once**")
+        w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both")
+        w("readings present in the sample). This is direct evidence for the causal chain")
+        w("§0b describes: the same address reproduces the same parse, boot after boot.")
+        w("It does NOT establish which addresses are catastrophic — no bad boot occurred —")
+        w("only that WHERE the table lands is what decides what the kernel finds there.")
+    w("")
+    w("| MEMATTR | boots | kernel reading(s) |")
+    w("|---|---:|---|")
+    for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)):
+        marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else ""
+        w(f"| `{addr}` | {n_by_addr[addr]} | "
+          f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |")
+    w("")
+
+    # --- 5. Corroboration -------------------------------------------------------------------
+    w("## 5. Corroborating evidence")
+    w("")
+    w("### 5a. Kernel's own reading vs the measured outcome")
+    w("")
+    cross: dict[tuple, int] = Counter()
+    for b in surveyed:
+        cross[(b.get("kernel_read"), b.get("verdict"))] += 1
+    w("| kernel read | verdict | boots |")
+    w("|---|---|---:|")
+    for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))):
+        w(f"| {read} | {verdict} | {count} |")
+    w("")
+    disagreements = [
+        (read, verdict, count)
+        for (read, verdict), count in cross.items()
+        if (read == "failed-map") != (verdict == "bad")
+    ]
+    if disagreements:
+        w("> **The log line and the measurement DISAGREE** in "
+          f"{sum(c for _, _, c in disagreements)} boot(s): "
+          + "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements)
+          + ". That is a finding in its own right — §0b's model says `failed-map` and only")
+        w("> `failed-map` costs memory.")
+    else:
+        w("`failed-map` and only `failed-map` cost memory, in every boot observed —")
+        w("exactly as §0b's mechanism predicts, and independent of the address question.")
+    w("")
+
+    w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)")
+    w("")
+    w("| verdict | NODE_DATA end address | boots |")
+    w("|---|---|---:|")
+    nd: Counter = Counter()
+    for b in surveyed:
+        line = b.get("node_data_line") or ""
+        end = line.split("-")[-1].rstrip("]") if "-" in line else "?"
+        nd[(b.get("verdict"), end)] += 1
+    for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
+        w(f"| {verdict} | `{end}` | {count} |")
+    w("")
+
+    # --- 6. Invariants ------------------------------------------------------------------
+    w("## 6. Nothing under test changed")
+    w("")
+    w("| node | kernels seen | distinct cmdlines | SMBIOS installed |")
+    w("|---|---|---:|---|")
+    for h in hosts:
+        kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")})
+        cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")}
+        w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | "
+          f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |")
+    w("")
+    w("More than one kernel or more than one cmdline on a node would void that node's arm.")
+    w("")
+    return "\n".join(out) + "\n"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--artifact-dir", required=True)
+    parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted")
+    parser.add_argument("--out", default=None, help="also write the report here")
+    args = parser.parse_args()
+
+    names = discover_hosts(args.artifact_dir)
+    if not names:
+        print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr)
+        return 1
+    hosts = [load_host(args.artifact_dir, name) for name in names]
+    report = render(hosts, args.rounds)
+    if args.out:
+        with open(args.out, "w", encoding="utf-8") as handle:
+            handle.write(report)
+    sys.stdout.write(report)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

infra/ansible/roles/memattr_survey/files/memattr_probe.py

@@ -0,0 +1,489 @@
+#!/usr/bin/env python3
+"""memattr_probe.py — read one boot's EFI Memory Attributes Table outcome off a node.
+
+Part of roles/memattr_survey (infra/ansible). See that role's README.md for the
+question this answers; see infra/PROD-PROVISIONING.md §0b for the settled mechanism.
+
+WHY A SCRIPT AND NOT A HANDFUL OF `command:` TASKS
+  The record for one boot needs six sources cross-referenced (kernel log config-table
+  line, kernel log memattr diagnostic, kernel log `Memory:` line, kernel log
+  `NODE_DATA` line, /proc/meminfo, SMBIOS) and one derived verdict. Doing that in
+  Jinja across six registered results is unreadable and would put the classification
+  logic in a template instead of in one testable place. It also has to be exactly the
+  same logic for a live boot and for a boot harvested out of the journal, which a
+  single --all-boots mode gives for free.
+
+MODES
+  (default)      one JSON record for the CURRENT boot, on stdout
+  --all-boots    one JSON object with a `boots` list: EVERY boot the persistent
+                 journal still holds, re-derived from scratch. This is the
+                 AUTHORITATIVE record — it survives the driver script dying mid-run.
+  --status       {"completed_rounds": N, "records": M} from the on-node ledger, so the
+                 playbook can resume without re-deriving anything.
+
+  --round N      stamp the record with the survey round number
+  --append PATH  ALSO append the record to PATH as one JSON line (the on-node ledger),
+                 created 0644 with parents. Appended with O_APPEND in one write() so a
+                 concurrent reader never sees a half line.
+
+VERDICT IS MEASURED, NOT INFERRED FROM A LOG LINE
+  A node whose kernel reads a VALID table logs no memattr diagnostic at all and is
+  perfectly healthy; a node whose kernel bails out on a garbage version > 2 logs a
+  scary warning and is ALSO perfectly healthy. Only the third state — "Failed to map"
+  — costs memory, and even that message is emitted by a LATER function than the one
+  that did the damage. So the verdict comes from usable RAM versus SMBIOS installed
+  RAM, and `kernel_read` is recorded ALONGSIDE it as an independent observation. If
+  the two ever disagree, that disagreement is itself the finding.
+
+EXIT STATUS is 0 whenever a record could be produced, even a mostly-null one: a field
+this script could not read comes back as null with a reason in `notes`, never as an
+empty string and never as a hard failure. A survey that aborts on one unreadable field
+loses the other 19 rounds.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+
+SCHEMA = "memattr-survey/1"
+
+# --- Verdict thresholds -------------------------------------------------------------
+# Observed on this fleet: a good boot yields usable/installed of 0.95-0.98 (kernel code,
+# reserved firmware regions and struct page overhead account for the rest); the bad path
+# yields 0.15-0.17 (everything above the table address is reserved). Two orders of
+# separation, so the thresholds are nowhere near either cluster. Anything landing in the
+# gap is deliberately NOT called: it is a new third thing and wants human eyes.
+VERDICT_GOOD_MIN_FRACTION = 0.85
+VERDICT_BAD_MAX_FRACTION = 0.60
+
+# `Memory: 2568544K/16546088K available (... 13958980K reserved ...)`
+RE_MEMORY_LINE = re.compile(
+    r"^Memory:\s+(?P<available>\d+)K/(?P<present>\d+)K available\b.*?"
+    r"(?:(?P<reserved>\d+)K reserved)?"
+)
+# `efi: ACPI=0x... SMBIOS=0x... MEMATTR=0xbeeb0018 MOKvar=0x... INITRD=0x...`
+RE_MEMATTR_ADDR = re.compile(r"\bMEMATTR=(?P<addr>0x[0-9a-fA-F]+)")
+# `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749`
+RE_TABLE_VERSION = re.compile(r"table version\s+(?P<version>-?\d+)")
+# `NODE_DATA(0) allocated [mem 0xbee85680-0xbeeaffff]` — per §0b the cleanest A/B tell:
+# on the bad path pgdat is forced immediately BELOW the table address instead of landing
+# at the top of RAM. Captured because it is free and it is independent evidence.
+RE_NODE_DATA = re.compile(r"^NODE_DATA\(\d+\) allocated \[mem (?P<range>[0-9a-fx]+-[0-9a-fx]+)\]")
+RE_CMDLINE = re.compile(r"^Command line:\s*(?P<cmdline>.*)$")
+# dmidecode type 17 `Size: 16 GB` / `Size: 8192 MB` / `Size: No Module Installed`
+RE_DMI_SIZE = re.compile(r"^\s*Size:\s+(?P<n>\d+)\s+(?P<unit>[kKmMgGtT]B)\s*$")
+
+DMI_UNIT_KB = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def run(argv: list[str], timeout: int = 60) -> tuple[int, str, str]:
+    """Run a command, never raise. Returns (rc, stdout, stderr); rc 127 if absent."""
+    try:
+        proc = subprocess.run(
+            argv, capture_output=True, text=True, timeout=timeout, check=False
+        )
+    except FileNotFoundError:
+        return 127, "", f"{argv[0]}: not found"
+    except subprocess.TimeoutExpired:
+        return 124, "", f"{' '.join(argv)}: timed out after {timeout}s"
+    except OSError as exc:  # pragma: no cover - defensive
+        return 126, "", f"{' '.join(argv)}: {exc}"
+    return proc.returncode, proc.stdout, proc.stderr
+
+
+def read_text(path: str) -> str | None:
+    try:
+        with open(path, "r", encoding="utf-8", errors="replace") as handle:
+            return handle.read()
+    except OSError:
+        return None
+
+
+def kernel_log(boot: str) -> tuple[list[str], str | None]:
+    """Kernel ring buffer for `boot` ('0' = current, '-1' = previous, or a boot ID).
+
+    `-o cat` is deliberate: it yields the message VERBATIM with no host/facility
+    decoration, which is what the survey wants to record, and it is stable across
+    systemd versions. `-k` restricts to _TRANSPORT=kernel.
+    """
+    rc, out, err = run(["journalctl", "-k", "-b", boot, "-o", "cat", "--no-pager"], timeout=180)
+    if rc != 0:
+        return [], f"journalctl -k -b {boot} failed (rc={rc}): {err.strip()[:200]}"
+    return out.splitlines(), None
+
+
+def list_boots() -> tuple[list[dict], str | None]:
+    """Every boot the journal still holds, oldest first.
+
+    Prefers `--list-boots -o json` (systemd >= 250, so always on Debian 13) and falls
+    back to parsing the table, because a survey that cannot enumerate boots has lost
+    its authoritative record and must say so rather than silently reporting one boot.
+    """
+    rc, out, err = run(["journalctl", "--list-boots", "-o", "json", "--no-pager"], timeout=120)
+    if rc == 0:
+        try:
+            parsed = json.loads(out)
+        except json.JSONDecodeError as exc:
+            return [], f"--list-boots -o json unparseable: {exc}"
+        boots = [
+            {
+                "index": int(entry.get("index", 0)),
+                "boot_id": entry.get("boot_id"),
+                "first_entry_usec": entry.get("first_entry"),
+                "last_entry_usec": entry.get("last_entry"),
+            }
+            for entry in parsed
+        ]
+        return sorted(boots, key=lambda b: b["index"]), None
+
+    rc, out, err = run(["journalctl", "--list-boots", "--no-pager"], timeout=120)
+    if rc != 0:
+        return [], f"journalctl --list-boots failed (rc={rc}): {err.strip()[:200]}"
+    boots = []
+    for line in out.splitlines():
+        fields = line.split()
+        if len(fields) < 2 or not re.fullmatch(r"-?\d+", fields[0]):
+            continue  # header row
+        boots.append(
+            {
+                "index": int(fields[0]),
+                "boot_id": fields[1],
+                "first_entry_usec": None,
+                "last_entry_usec": None,
+            }
+        )
+    return sorted(boots, key=lambda b: b["index"]), None
+
+
+def journal_storage() -> dict:
+    """Is the journal PERSISTENT? The harvest is only authoritative if it is.
+
+    /var/log/journal existing is what makes journald write to disk; /run/log/journal is
+    the volatile fallback and would mean every reboot erases the record.
+    """
+    return {
+        "var_log_journal": os.path.isdir("/var/log/journal"),
+        "run_log_journal": os.path.isdir("/run/log/journal"),
+        "persistent": os.path.isdir("/var/log/journal"),
+    }
+
+
+def smbios_installed_kb() -> tuple[int | None, str | None]:
+    """Installed RAM per SMBIOS type-17 records — the GROUND TRUTH for the verdict.
+
+    Deliberately not /proc/meminfo (that is the number under test) and not dmesg's
+    `Memory: x/yK` denominator (which is already post-e820 and post-reservation).
+    Hardware does not change between boots, so one read applies to harvested boots too.
+    """
+    rc, out, err = run(["dmidecode", "-t", "17"], timeout=60)
+    if rc != 0:
+        return None, f"dmidecode -t 17 failed (rc={rc}): {err.strip()[:200]}"
+    total = 0
+    seen = 0
+    for line in out.splitlines():
+        match = RE_DMI_SIZE.match(line)
+        if not match:
+            continue  # "No Module Installed" and every non-Size line
+        total += int(match.group("n")) * DMI_UNIT_KB[match.group("unit").lower()]
+        seen += 1
+    if seen == 0:
+        return None, "dmidecode -t 17 reported no populated memory devices"
+    return total, None
+
+
+def meminfo_total_kb() -> tuple[int | None, str | None]:
+    text = read_text("/proc/meminfo")
+    if text is None:
+        return None, "/proc/meminfo unreadable"
+    for line in text.splitlines():
+        if line.startswith("MemTotal:"):
+            return int(line.split()[1]), None
+    return None, "/proc/meminfo has no MemTotal"
+
+
+def classify_kernel_read(memattr_addr, diagnostics: list[str]) -> tuple[str, int | None]:
+    """What did the kernel make of the table? THREE outcomes, not two (§0b).
+
+    Returns (classification, table_version_if_reported).
+
+      no-table      firmware published no MEMATTR pointer at all (the dev nodes)
+      valid         a MEMATTR pointer AND no diagnostic => the kernel parsed a table it
+                    considered valid. This is SILENCE AS EVIDENCE, and it is a healthy
+                    state, not a missing measurement.
+      bail-version  garbage version > 2 => kernel warned and bailed. Memory intact.
+                    The LUCKY path, and it looks alarming in the log.
+      failed-map    the catastrophic path. The message comes from
+                    efi_memattr_apply_permissions() and is therefore PROOF the bogus
+                    memblock_reserve() already happened upstream of it.
+      bogus-size    a kernel >= 6.13 rejecting the table. Should never appear on 6.12;
+                    if it does, the kernel under test changed and the arm is void.
+    """
+    version = None
+    for line in diagnostics:
+        match = RE_TABLE_VERSION.search(line)
+        if match:
+            version = int(match.group("version"))
+    joined = " | ".join(diagnostics)
+    if "Failed to map" in joined:
+        return "failed-map", version
+    if "Corrupted EFI Memory Attributes Table" in joined:
+        return "bogus-size", version
+    if "Unexpected EFI Memory Attributes table version" in joined:
+        return "bail-version", version
+    if diagnostics:
+        return "diagnostic-unrecognised", version
+    if memattr_addr is None:
+        return "no-table", version
+    return "valid", version
+
+
+def derive_verdict(usable_kb, installed_kb) -> tuple[str, float | None]:
+    if usable_kb is None or not installed_kb:
+        return "unknown", None
+    fraction = usable_kb / installed_kb
+    if fraction >= VERDICT_GOOD_MIN_FRACTION:
+        return "good", fraction
+    if fraction <= VERDICT_BAD_MAX_FRACTION:
+        return "bad", fraction
+    return "indeterminate", fraction
+
+
+def build_record(
+    *,
+    boot_selector: str,
+    boot_meta: dict | None,
+    installed_kb,
+    installed_note,
+    round_number,
+    live: bool,
+) -> dict:
+    notes: list[str] = []
+    if installed_note:
+        notes.append(installed_note)
+
+    lines, log_note = kernel_log(boot_selector)
+    if log_note:
+        notes.append(log_note)
+
+    config_table_line = None
+    memattr_addr = None
+    diagnostics: list[str] = []
+    memory_line = None
+    node_data_line = None
+    cmdline = None
+
+    for line in lines:
+        stripped = line.strip()
+        if memattr_addr is None:
+            match = RE_MEMATTR_ADDR.search(stripped)
+            if match and stripped.startswith("efi:"):
+                memattr_addr = match.group("addr").lower()
+                config_table_line = stripped
+        # Every memattr-related diagnostic, verbatim. Matched on the pr_fmt prefix the
+        # kernel actually uses (`efi: memattr: `) AND on the message text, so a prefix
+        # change upstream cannot silently drop the line.
+        if stripped.startswith("efi: memattr:") or "EFI Memory Attributes" in stripped:
+            if stripped != config_table_line:
+                diagnostics.append(stripped)
+        if memory_line is None and stripped.startswith("Memory:"):
+            if RE_MEMORY_LINE.match(stripped):
+                memory_line = stripped
+        if node_data_line is None:
+            match = RE_NODE_DATA.match(stripped)
+            if match:
+                node_data_line = stripped
+        if cmdline is None:
+            match = RE_CMDLINE.match(stripped)
+            if match:
+                cmdline = match.group("cmdline").strip()
+
+    available_kb = present_kb = reserved_kb = None
+    if memory_line:
+        match = RE_MEMORY_LINE.match(memory_line)
+        available_kb = int(match.group("available"))
+        present_kb = int(match.group("present"))
+        reserved = match.group("reserved")
+        reserved_kb = int(reserved) if reserved else None
+    else:
+        notes.append("no `Memory: .../... available` line found in this boot's kernel log")
+
+    kernel_read, table_version = classify_kernel_read(memattr_addr, diagnostics)
+
+    # /proc/meminfo describes the CURRENT boot only. For a harvested past boot the
+    # equivalent measurement is the kernel's own `Memory: available` figure, which
+    # tracks MemTotal to within ~7% on both paths. Which one was used is recorded, so
+    # nobody has to guess later.
+    mem_total_kb, meminfo_note = (None, None)
+    if live:
+        mem_total_kb, meminfo_note = meminfo_total_kb()
+        if meminfo_note:
+            notes.append(meminfo_note)
+    usable_kb = mem_total_kb if mem_total_kb is not None else available_kb
+    basis = "MemTotal" if mem_total_kb is not None else "dmesg Memory: available"
+    verdict, fraction = derive_verdict(usable_kb, installed_kb)
+
+    record = {
+        "schema": SCHEMA,
+        "round": round_number,
+        "host": os.uname().nodename,
+        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+        "recorded_at_epoch": int(time.time()),
+        "live_boot": live,
+        "boot_index": (boot_meta or {}).get("index"),
+        "boot_id": (boot_meta or {}).get("boot_id"),
+        "boot_first_entry_usec": (boot_meta or {}).get("first_entry_usec"),
+        "kernel_release": os.uname().release,
+        "kernel_cmdline": cmdline,
+        # --- the independent variable -------------------------------------------------
+        "efi_config_table_line": config_table_line,
+        "memattr_address": memattr_addr,
+        "memattr_diagnostics": diagnostics,
+        "memattr_diagnostic_count": len(diagnostics),
+        # An explicit marker, never an empty field: "the kernel said nothing" is a
+        # measurement (it read a VALID table), not a gap in the data.
+        "memattr_diagnostic_present": bool(diagnostics),
+        "memattr_table_version": table_version,
+        "kernel_read": kernel_read,
+        # --- the dependent variable ---------------------------------------------------
+        "memory_line": memory_line,
+        "memory_available_kb": available_kb,
+        "memory_present_kb": present_kb,
+        "memory_reserved_kb": reserved_kb,
+        "mem_total_kb": mem_total_kb,
+        "smbios_installed_kb": installed_kb,
+        "usable_kb": usable_kb,
+        "usable_fraction": round(fraction, 5) if fraction is not None else None,
+        "verdict": verdict,
+        "verdict_basis": f"{basis} vs SMBIOS installed",
+        # --- corroborating evidence ---------------------------------------------------
+        "node_data_line": node_data_line,
+        "notes": notes,
+    }
+    return record
+
+
+def append_record(path: str, record: dict) -> str | None:
+    """One O_APPEND write of one line, so a reader never sees a torn record."""
+    try:
+        parent = os.path.dirname(path) or "."
+        os.makedirs(parent, mode=0o755, exist_ok=True)
+        payload = (json.dumps(record, sort_keys=True) + "\n").encode()
+        fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
+        try:
+            os.write(fd, payload)
+            os.fsync(fd)  # the point of the ledger is surviving the next reboot
+        finally:
+            os.close(fd)
+    except OSError as exc:
+        return f"could not append to {path}: {exc}"
+    return None
+
+
+def ledger_status(path: str) -> dict:
+    """Resume support: how far did a previous run get?
+
+    Round numbers come from the ledger itself rather than a line count, so the
+    baseline round 0 record and any hand-added line cannot shift the numbering.
+    """
+    rounds: list[int] = []
+    records = 0
+    malformed = 0
+    if os.path.exists(path):
+        with open(path, "r", encoding="utf-8", errors="replace") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                records += 1
+                try:
+                    parsed = json.loads(line)
+                except json.JSONDecodeError:
+                    malformed += 1
+                    continue
+                value = parsed.get("round")
+                if isinstance(value, int):
+                    rounds.append(value)
+    return {
+        "schema": SCHEMA,
+        "ledger": path,
+        "records": records,
+        "malformed_records": malformed,
+        "completed_rounds": max(rounds) if rounds else 0,
+        "rounds_recorded": sorted(rounds),
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--round", type=int, default=None, help="survey round number to stamp")
+    parser.add_argument("--append", default=None, help="on-node ledger to append to")
+    parser.add_argument("--all-boots", action="store_true", help="harvest every retained boot")
+    parser.add_argument("--status", default=None, metavar="LEDGER", help="report resume state")
+    args = parser.parse_args()
+
+    if args.status:
+        print(json.dumps(ledger_status(args.status), indent=2, sort_keys=True))
+        return 0
+
+    installed_kb, installed_note = smbios_installed_kb()
+    boots, boots_note = list_boots()
+    storage = journal_storage()
+
+    if args.all_boots:
+        records = []
+        for boot in boots:
+            # Address boots by BOOT ID, not by relative index: indices shift under the
+            # harvester's feet if journald rotates mid-harvest, and the ID is what the
+            # per-round ledger recorded, so the two records join cleanly.
+            selector = boot["boot_id"] or str(boot["index"])
+            records.append(
+                build_record(
+                    boot_selector=selector,
+                    boot_meta=boot,
+                    installed_kb=installed_kb,
+                    installed_note=installed_note,
+                    round_number=None,
+                    live=(boot["index"] == 0),
+                )
+            )
+        payload = {
+            "schema": SCHEMA,
+            "host": os.uname().nodename,
+            "harvested_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "journal_storage": storage,
+            "smbios_installed_kb": installed_kb,
+            "boots_listed": len(boots),
+            "notes": [n for n in [boots_note, installed_note] if n],
+            "boots": records,
+        }
+        print(json.dumps(payload, indent=2, sort_keys=True))
+        return 0
+
+    current = next((b for b in boots if b["index"] == 0), None)
+    record = build_record(
+        boot_selector="0",
+        boot_meta=current,
+        installed_kb=installed_kb,
+        installed_note=installed_note,
+        round_number=args.round,
+        live=True,
+    )
+    record["journal_storage"] = storage
+    if boots_note:
+        record["notes"].append(boots_note)
+    if args.append:
+        note = append_record(args.append, record)
+        if note:
+            record["notes"].append(note)
+    print(json.dumps(record, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

infra/ansible/roles/memattr_survey/tasks/analyse.yml

@@ -0,0 +1,42 @@
+---
+# Controller-side analysis. Runs on localhost over the artifact directory, so it is
+# independent of whether any surveyed node is still alive — and so it can be re-run on
+# its own (`--tags analyse`) without touching hardware.
+#
+# The statistics live in files/memattr_analyse.py, not in Jinja: Clopper-Pearson and
+# Fisher belong somewhere testable, and a confidence interval assembled out of template
+# filters is a confidence interval nobody will ever check.
+
+- name: Locate the survey artifacts
+  ansible.builtin.find:
+    paths: "{{ memattr_survey_artifact_dir }}"
+    patterns: "*.boots.json"
+  register: memattr_survey_artifacts
+
+- name: Report that there is nothing to analyse
+  ansible.builtin.debug:
+    msg: >-
+      No *.boots.json under {{ memattr_survey_artifact_dir }} — no survey has produced a
+      harvest yet. Run the playbook with -e memattr_survey_reboots_authorised=true.
+  when: memattr_survey_artifacts.matched | int == 0
+
+- name: Analyse the survey
+  ansible.builtin.command:
+    argv:
+      - /usr/bin/env
+      - python3
+      - "{{ role_path }}/files/memattr_analyse.py"
+      - "--artifact-dir"
+      - "{{ memattr_survey_artifact_dir }}"
+      - "--rounds"
+      - "{{ memattr_survey_rounds }}"
+      - "--out"
+      - "{{ memattr_survey_artifact_dir }}/summary.md"
+  register: memattr_survey_analysis
+  changed_when: false
+  when: memattr_survey_artifacts.matched | int > 0
+
+- name: Show the analysis
+  ansible.builtin.debug:
+    msg: "{{ memattr_survey_analysis.stdout.split('\n') }}"
+  when: memattr_survey_analysis is not skipped

infra/ansible/roles/memattr_survey/tasks/harvest.yml

@@ -0,0 +1,66 @@
+---
+# The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from
+# scratch in one pass by the same code that produced the per-round ledger.
+#
+# WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and
+# the append, that round exists on the machine but not in the ledger. The journal does not
+# care whether the driver was alive — the boot happened, the kernel logged it, journald
+# persisted it. So the harvest can only ever be a superset of the ledger, and the analysis
+# joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and
+# any boot the ledger missed still shows up.
+#
+# This runs even for a node that aborted mid-survey, which is exactly when it matters most.
+
+- name: Harvest every retained boot from the persistent journal
+  ansible.builtin.script:
+    cmd: memattr_probe.py --all-boots
+    executable: /usr/bin/python3
+  register: memattr_survey_harvest_raw
+  changed_when: false   # strictly a read of the journal
+
+- name: Parse the harvest
+  ansible.builtin.set_fact:
+    memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}"
+
+- name: Ensure the artifact directory exists (on the controller)
+  ansible.builtin.file:
+    path: "{{ memattr_survey_artifact_dir }}"
+    state: directory
+    mode: "0755"
+  delegate_to: localhost
+  become: false
+  # NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and
+  # run_once is unreliable there. `state: directory` is idempotent, so three hosts each
+  # asserting the directory exists is cheaper than a conditional that might not fire.
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Write the harvested boot record
+  ansible.builtin.copy:
+    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json"
+    mode: "0644"
+    content: "{{ memattr_survey_harvest | to_nice_json }}"
+  delegate_to: localhost
+  become: false
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Mirror the final ledger to the controller
+  ansible.builtin.fetch:
+    src: "{{ memattr_survey_state_file }}"
+    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
+    flat: true
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Harvest summary
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots
+      from the journal
+      ({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad,
+      {{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good,
+      {{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other)
+      {%- if memattr_survey_aborted | default(false) | bool %}
+      — NOTE: this node ABORTED ({{ memattr_survey_abort_reason }})
+      {%- endif %}

infra/ansible/roles/memattr_survey/tasks/main.yml

@@ -0,0 +1,23 @@
+---
+# memattr_survey — see defaults/main.yml for the question, README.md for how to run and
+# read it, and infra/PROD-PROVISIONING.md §0b for the settled mechanism this measures.
+#
+# ORDER OF OPERATIONS
+#   1. preflight.yml — refuse to touch a node that is not prod, not idle, or has no
+#      persistent journal; read the resume point; record the boot we found it on.
+#   2. round.yml x N — reboot, measure, append, mirror. Per-node fail-safe.
+#   3. harvest.yml   — re-derive every retained boot from the journal (authoritative).
+#
+# The playbook splits these across plays so that preflight is any_errors_fatal (one unsafe
+# node stops everything) while the rounds are not (one dead node stops only itself), and
+# so that the controller-side analysis runs even if every node dies. This file is the
+# whole-role entry point kept for completeness; the playbook calls the stages directly.
+
+- name: Preflight and baseline
+  ansible.builtin.include_tasks: preflight.yml
+
+- name: Run the survey rounds
+  ansible.builtin.include_tasks: rounds.yml
+
+- name: Harvest the authoritative record
+  ansible.builtin.include_tasks: harvest.yml

infra/ansible/roles/memattr_survey/tasks/preflight.yml

@@ -0,0 +1,149 @@
+---
+# Safety gate. Runs with any_errors_fatal at the play level (playbooks/memattr-survey.yml),
+# so a violation on ANY node stops the entire playbook before a single reboot — the
+# correct posture when the failure mode is "rebooted a machine that was doing something".
+#
+# VERIFY, DO NOT ASSUME. The premise that makes these reboots safe is that prod carries no
+# k3s and runs nothing. That premise is checked here on every run rather than trusted from
+# a document, because the document and the machine drift independently.
+
+- name: Assert the target host is in the prod group
+  # The dev cluster is LIVE. This is the one check that a mistyped --limit or a copied
+  # command line cannot slip past: membership is read from the inventory, not the pattern.
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_required_group in (group_names | default([]))
+    fail_msg: >-
+      REFUSING TO SURVEY {{ inventory_hostname }}: it is not a member of
+      '{{ memattr_survey_required_group }}' (groups: {{ group_names | default([]) | join(', ') }}).
+      This role reboots its targets. Point it at prod, or override
+      memattr_survey_required_group deliberately.
+    quiet: true
+
+- name: Check whether any cluster or container runtime unit is active
+  # `is-active` returns non-zero for inactive/unknown units, which is the expected answer
+  # here, so the rc is data and not a failure.
+  ansible.builtin.command:
+    argv: "{{ ['systemctl', 'is-active'] + memattr_survey_forbidden_units }}"
+  register: memattr_survey_units
+  changed_when: false
+  failed_when: false
+
+- name: Check whether k3s has been installed on this node
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  loop: "{{ memattr_survey_forbidden_paths }}"
+  register: memattr_survey_paths
+  changed_when: false
+
+- name: Assert nothing is running on this node
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_units.stdout_lines | reject('equalto', 'inactive') | reject('equalto', 'unknown') | list | length == 0
+      - memattr_survey_paths.results | selectattr('stat.exists') | list | length == 0
+    fail_msg: >-
+      REFUSING TO REBOOT {{ inventory_hostname }} — it is not idle.
+      units {{ memattr_survey_forbidden_units | zip(memattr_survey_units.stdout_lines) | map('join', '=') | join(' ') }};
+      present paths: {{ memattr_survey_paths.results | selectattr('stat.exists') | map(attribute='item') | list | join(', ') | default('none', true) }}.
+      The survey is only sanctioned because these boxes carry no k3s and run nothing.
+    quiet: true
+
+- name: Clear any abort marker from a previous run
+  # A marker left by a run that has since been resumed successfully would make this run's
+  # report describe the LAST failure rather than this one. The marker means "this run
+  # aborted", so it is cleared at the start of every run and re-written only if it does.
+  ansible.builtin.file:
+    path: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
+    state: absent
+  delegate_to: localhost
+  become: false
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Assert forks allows every node to be surveyed concurrently
+  # The survey play runs `strategy: free` so the nodes reboot in parallel, but forks is a
+  # hard cap on that: with fewer forks than hosts the survey quietly serialises and takes
+  # a multiple of the wall clock it should. Cheap to satisfy (`-f <n>`), so assert rather
+  # than discover it three hours later.
+  ansible.builtin.assert:
+    that:
+      - (lookup('ansible.builtin.config', 'DEFAULT_FORKS') | int) >= (ansible_play_hosts_all | length)
+    fail_msg: >-
+      forks={{ lookup('ansible.builtin.config', 'DEFAULT_FORKS') }} but this survey targets
+      {{ ansible_play_hosts_all | length }} hosts. The rounds would serialise. Re-run with
+      -f {{ ansible_play_hosts_all | length }} (or higher).
+    quiet: true
+  run_once: true
+
+- name: Read the node's resume state and baseline boot
+  # One call does double duty: it reports how far a previous run got (so this one resumes)
+  # and, with --round 0, records the boot we found the node on. Round 0 is a free data
+  # point — it is a boot nobody chose, so it is not selected for by the survey.
+  ansible.builtin.script:
+    cmd: >-
+      memattr_probe.py --status {{ memattr_survey_state_file | quote }}
+    executable: /usr/bin/python3
+  register: memattr_survey_status_raw
+  changed_when: false
+
+- name: Record the resume point
+  ansible.builtin.set_fact:
+    memattr_survey_status: "{{ memattr_survey_status_raw.stdout | from_json }}"
+    memattr_survey_aborted: false
+    memattr_survey_abort_reason: ""
+
+- name: Assert the probe schema is the one this role understands
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_status.schema == 'memattr-survey/1'
+    fail_msg: >-
+      Probe returned schema {{ memattr_survey_status.schema | default('<none>') }};
+      this role understands memattr-survey/1. Role and probe are out of step.
+    quiet: true
+
+- name: Probe the boot the node is currently on
+  # On a FIRST run this is stamped round 0 and appended: the boot the node was already
+  # sitting on is a free, unselected data point. On a RESUME it is read but NOT appended,
+  # because that boot is already round N of the interrupted run and double-counting it
+  # would bias the sample toward whatever outcome happened to survive the interruption.
+  ansible.builtin.script:
+    cmd: >-
+      memattr_probe.py
+      {% if memattr_survey_status.completed_rounds | int == 0 %}
+      --round 0 --append {{ memattr_survey_state_file | quote }}
+      {% endif %}
+    executable: /usr/bin/python3
+  register: memattr_survey_baseline_raw
+  # The append is the experiment's own ledger, not a change to the node under test: this
+  # role reports `changed` only for things that could affect a boot, and a file written
+  # under /var/lib long after the kernel parsed the table cannot.
+  changed_when: false
+
+- name: Parse the baseline record
+  ansible.builtin.set_fact:
+    memattr_survey_baseline: "{{ memattr_survey_baseline_raw.stdout | from_json }}"
+
+- name: Assert the journal is PERSISTENT
+  # The end-of-run harvest is the authoritative record precisely because it re-derives
+  # every boot from the journal. On a volatile journal (/run/log/journal only) each
+  # reboot erases the evidence and the harvest would silently return one boot — a survey
+  # that looks complete and holds nothing. Verified, not assumed.
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_baseline.journal_storage.persistent | bool
+    fail_msg: >-
+      {{ inventory_hostname }} has NO persistent journal (/var/log/journal missing).
+      Every reboot would erase the record this survey depends on. Refusing to run.
+    quiet: true
+
+- name: Report the starting state
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: idle, prod, persistent journal.
+      kernel {{ memattr_survey_baseline.kernel_release | default('?') }},
+      installed {{ (memattr_survey_baseline.smbios_installed_kb | default(0) | int / 1048576) | round(1) }} GiB.
+      baseline boot MEMATTR={{ memattr_survey_baseline.memattr_address | default('none') }}
+      kernel_read={{ memattr_survey_baseline.kernel_read | default('?') }}
+      verdict={{ memattr_survey_baseline.verdict | default('?') | upper }}.
+      resuming from round {{ memattr_survey_status.completed_rounds }}
+      of {{ memattr_survey_rounds }}.

infra/ansible/roles/memattr_survey/tasks/round.yml

@@ -0,0 +1,98 @@
+---
+# ONE round on ONE node: reboot, wait for it back, measure the new boot, record it twice.
+# Included in a loop from tasks/main.yml with loop_var `memattr_survey_round`.
+#
+# FAIL-SAFE, PER NODE. The block/rescue below is what keeps one dead node from ending the
+# survey: on any failure this host sets memattr_survey_aborted and every later round is
+# skipped FOR THIS HOST ONLY, while the other nodes keep going (Ansible runs the loop for
+# all hosts in lockstep, so the reboots are genuinely concurrent). The abort is reported
+# loudly here AND again in the final analysis, which counts rounds from the artifacts
+# rather than from Ansible's host state — so a node that vanishes so hard that it is
+# marked unreachable (which rescue cannot catch) still shows up as a short arm.
+
+- name: "Round {{ memattr_survey_round }} — survey step"
+  block:
+    - name: "Round {{ memattr_survey_round }} — reboot and wait for the node to return"
+      ansible.builtin.reboot:
+        reboot_timeout: "{{ memattr_survey_reboot_timeout | int }}"
+        connect_timeout: "{{ memattr_survey_connect_timeout | int }}"
+        post_reboot_delay: "{{ memattr_survey_post_reboot_delay | int }}"
+        msg: "memattr survey round {{ memattr_survey_round }} of {{ memattr_survey_rounds }}"
+
+    - name: "Round {{ memattr_survey_round }} — measure the new boot and append to the ledger"
+      ansible.builtin.script:
+        cmd: >-
+          memattr_probe.py --round {{ memattr_survey_round }}
+          --append {{ memattr_survey_state_file | quote }}
+        executable: /usr/bin/python3
+      register: memattr_survey_round_raw
+      changed_when: false   # see the note in preflight.yml — the ledger is not the node
+
+    - name: "Round {{ memattr_survey_round }} — parse the record"
+      ansible.builtin.set_fact:
+        memattr_survey_record: "{{ memattr_survey_round_raw.stdout | from_json }}"
+
+    - name: "Round {{ memattr_survey_round }} — result"
+      ansible.builtin.debug:
+        msg: >-
+          [{{ inventory_hostname }} r{{ memattr_survey_round }}/{{ memattr_survey_rounds }}]
+          MEMATTR={{ memattr_survey_record.memattr_address | default('none') }}
+          read={{ memattr_survey_record.kernel_read }}
+          {%- if memattr_survey_record.memattr_table_version is not none %}
+          (version {{ memattr_survey_record.memattr_table_version }})
+          {%- endif %}
+          MemTotal={{ (memattr_survey_record.mem_total_kb | default(0) | int / 1048576) | round(2) }}G
+          of {{ (memattr_survey_record.smbios_installed_kb | default(0) | int / 1048576) | round(0) | int }}G
+          => {{ memattr_survey_record.verdict | upper }}
+          boot={{ memattr_survey_record.boot_id | default('?') | truncate(9, true, '') }}
+      when: memattr_survey_show_progress | bool
+
+    # BRACES. The on-node ledger already survives a reboot; re-fetching it every round
+    # means the controller also holds every completed round if the DRIVER dies, without
+    # waiting for the end-of-run harvest. One tiny file per host, per-host destination
+    # paths, so three concurrent fetches cannot collide.
+    - name: "Round {{ memattr_survey_round }} — mirror the ledger to the controller"
+      ansible.builtin.fetch:
+        src: "{{ memattr_survey_state_file }}"
+        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
+        flat: true
+      changed_when: false
+      when: memattr_survey_write_artifacts | bool
+
+  rescue:
+    - name: "Round {{ memattr_survey_round }} — ABORT this node, continue the others"
+      ansible.builtin.set_fact:
+        memattr_survey_aborted: true
+        memattr_survey_abort_reason: >-
+          round {{ memattr_survey_round }} failed:
+          {{ ansible_failed_result.msg | default('unknown error') | trim }}
+
+    # The abort must survive the run, not live only in terminal scrollback. Without
+    # this the report can see that an arm is SHORT but not WHY, and "0 rounds" reads
+    # like a targeting mistake instead of a node that failed to come back.
+    - name: "Round {{ memattr_survey_round }} — record the abort as an artifact"
+      ansible.builtin.copy:
+        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
+        mode: "0644"
+        content: "{{ memattr_survey_abort_record | to_nice_json }}"
+      vars:
+        memattr_survey_abort_record:
+          schema: "memattr-survey/1"
+          host: "{{ inventory_hostname }}"
+          aborted_at_round: "{{ memattr_survey_round }}"
+          target_rounds: "{{ memattr_survey_rounds }}"
+          reason: "{{ memattr_survey_abort_reason | trim }}"
+          failed_task: "{{ ansible_failed_task.name | default('unknown') }}"
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      when: memattr_survey_write_artifacts | bool
+
+    - name: "Round {{ memattr_survey_round }} — announce the abort"
+      ansible.builtin.debug:
+        msg: >-
+          *** SURVEY ABORTED ON {{ inventory_hostname }} at round
+          {{ memattr_survey_round }}/{{ memattr_survey_rounds }}:
+          {{ memattr_survey_abort_reason }}
+          *** The other nodes continue. This node's arm is SHORT — the analysis will
+          say so, and its p(bad) interval will be correspondingly wider.

infra/ansible/roles/memattr_survey/tasks/rounds.yml

@@ -0,0 +1,40 @@
+---
+# The loop itself, kept separate from round.yml so the loop bounds (which encode the
+# RESUME logic) are readable in one place.
+#
+# RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a
+# re-run after an interruption continues where it stopped and can never lose or duplicate
+# an earlier round. Already at or past the target count => nothing to do, and the run
+# collapses to a harvest.
+#
+# NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which
+# makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest,
+# analysis) still runs, so a dry run exercises the entire pipeline without touching a
+# single machine.
+
+- name: Announce the plan
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}:
+      {%- if not (memattr_survey_reboots_authorised | bool) %}
+      DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm).
+      {%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %}
+      already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do.
+      {%- else %}
+      rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }}
+      to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots).
+      {%- endif %}
+
+- name: Survey round
+  ansible.builtin.include_tasks: round.yml
+  loop: >-
+    {{ range(memattr_survey_status.completed_rounds | int + 1,
+             memattr_survey_rounds | int + 1) | list }}
+  loop_control:
+    loop_var: memattr_survey_round
+    label: "{{ inventory_hostname }} round {{ memattr_survey_round }}"
+  # Evaluated per iteration, so the abort flag set by a rescue in one round skips every
+  # round after it on THIS host and no others.
+  when:
+    - memattr_survey_reboots_authorised | bool
+    - not (memattr_survey_aborted | bool)

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -18,22 +18,193 @@ node_baseline_packages:
   # node from first baseline — package installation is the role's job, never a manual step
   # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
   - smartmontools
+# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
+# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
+# baseline — not a one-off `apt install` someone has to remember on the next node.
+#
+# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
+# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
+# from its contents, so an unlucky boot reserves everything above the table and the node
+# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
+# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
+# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
+# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
+# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
+# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+#
+# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
+# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
+# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
+# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
+# encode the environment into the ROLE, would silently miss a fourth prod node named
+# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
+# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
+# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
+#
+# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
+# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
+# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
+# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
+#
+# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
+# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
+# delete this block. Nothing else in the repo depends on it.
+node_baseline_backports_kernel: false
+# The suite, mirror and components the source line is built from. Split out so a mirror
+# change is a variable, not an edit to a heredoc in tasks/main.yml.
+node_baseline_backports_suite: trixie-backports
+node_baseline_backports_mirror: "http://deb.debian.org/debian"
+# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
+# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
+# surface beyond what the node already trusts is a separate decision nobody has taken.
+node_baseline_backports_components:
+  - main
+  - non-free-firmware
+# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
+# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
+# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
+# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
+# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
+# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
+# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
+node_baseline_backports_kernel_packages:
+  - linux-image-amd64
+# Provenance/rollback note written on the node. An operator standing in front of a prod box
+# with no repo checkout must be able to answer "why is this machine not on the distro
+# kernel, and how do I undo it?" from the machine itself.
+node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
+
 node_baseline_kernel_modules:
   - overlay
   - br_netfilter
   - iscsi_tcp
   - dm_crypt
   - nfs
+
+# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
+# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
+# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
+#
+#   node_baseline_longhorn_source:
+#     device           dedicated block device at node_baseline_longhorn_device (DEV: the
+#                      tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
+#     claim-mountpoint re-purpose the partition the node ALREADY has mounted at
+#                      node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
+#                      path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
+#                      and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
+#                      is correct and a hardcoded one would eventually mkfs the wrong
+#                      partition on a future node. Destructive; heavily guarded — read
+#                      tasks/disks.yml before changing anything here.
+#
+#   node_baseline_etcd_source:
+#     device                dedicated block device at node_baseline_etcd_device (DEV:
+#                           /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
+#                           isolated from Longhorn rebuild I/O after dev was burned by HDD
+#                           contention).
+#     root-filesystem       etcd lives on the OS filesystem, with NO dedicated device and
+#                           NO claimed data partition under it. Kept as a valid, supported
+#                           shape; NOT what prod uses any more (see below).
+#     shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
+#                           CLAIMED data partition (PROD). No dedicated device, but also NOT
+#                           the OS filesystem: the whole point of the 2026-09-04 layout
+#                           revision is that /var (11.8 GiB) can hold neither the container
+#                           image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
+#
+# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
+# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
+# var, which still fails the `disks` assert.
+node_baseline_longhorn_source: device
 node_baseline_longhorn_device: /dev/vdb
-node_baseline_longhorn_mount: /var/lib/longhorn
-node_baseline_longhorn_fs_label: longhorn
+# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
+# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
+# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
+# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
+# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
+#
+# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
+# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
+# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
+# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
+# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
+node_baseline_data_mount: /var/lib/longhorn
+node_baseline_data_fs_label: longhorn
+# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
+# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
+# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
+# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
+# same fact, and the two are checked against each other by nothing but this comment.
+node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
+# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
+# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
+# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
+# install flags, this role reads the same value to create the directory and to place etcd,
+# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
+# value (higher precedence than any role default) is what actually resolves here, even
+# though roles/k3s_server is not in the node-baseline play; the default() keeps
+# node-baseline.yml runnable standalone.
+node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
+# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
+# default so a dev host cannot enter the claim path even by accident.
+node_baseline_longhorn_claim_mountpoint: ""
+# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
+# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
+# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
+# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
+# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
+# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
+# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
+node_baseline_data_retired_fs_label: ""
+node_baseline_data_retired_mount: ""
+# Sanity floor for the claim path: a data partition smaller than this is not "a small
+# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
+node_baseline_longhorn_min_size_gb: 100
 # Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
 # so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
 # See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
-# device renames (matches the Longhorn LABEL= approach).
+# device renames (matches the data partition's LABEL= approach).
+node_baseline_etcd_source: device
 node_baseline_etcd_device: /dev/vdc
-node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
+# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
+# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
+# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
+# to /data/k3s/server/db, inside the claimed partition.
+node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
 node_baseline_etcd_fs_label: k3s-etcd
+# Mounts whose backing devices the claim path must NEVER touch. Used to build the
+# protected-device set from the node's own gathered facts (plus live swap), so a node
+# partitioned differently from the fleet is judged on its own layout rather than against
+# a hardcoded list of partition names.
+node_baseline_protected_mounts:
+  - /
+  - /boot
+  - /boot/efi
+  - /usr
+  - /var
+  - /home
+  - /tmp
+
+# --- WIRELESS (tasks/main.yml, tag `wireless`) ---------------------------------------
+# A CLUSTER NODE GETS EXACTLY ONE PATH ONTO THE NETWORK. ON by default — unlike the
+# backports kernel above, which is a prod-only deviation, this is a cluster-wide policy and
+# it is a provable NO-OP where it does not apply: the tasks act only on interfaces the node
+# reports as real 802.11 devices (a `phy80211` link in sysfs), and dev's Proxmox VMs have
+# none, so on dev nothing is written, nothing is unloaded and nothing changes. Defaulting it
+# true is what makes a FOURTH node with a wifi card inherit the policy from being a cluster
+# node, rather than from someone remembering to set a flag.
+#
+# WHY IT EXISTS: k3s-prod-3 came up on both eno1 (192.168.2.12, the inventory address) and
+# wlp1s0 (192.168.2.195, DHCP on the same /24). The wifi lease's dhcpcd run rewrote
+# /etc/resolv.conf three minutes after this role wrote it, and a second address on the LAN
+# where kube-vip floats 192.168.2.9 and Cilium L2-announces .20-.30 is a problem in its own
+# right. See the `wireless` block in tasks/main.yml for the full reasoning, including why the
+# DRIVER MODULE is the layer that gets blocked rather than the link, a service, or rfkill.
+#
+# SET IT FALSE for a host that must genuinely keep wifi (there is none today). That does NOT
+# undo an already-applied block — delete the file below on the node as well.
+node_baseline_disable_wireless: true
+# Where the persistent block lives. 99- so it is read last and overrides any earlier
+# permissive rule; the name says what it does, on a node with no repo checkout to hand.
+node_baseline_wireless_modprobe_file: /etc/modprobe.d/99-k3s-no-wireless.conf
 
 # --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
 # These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml

infra/ansible/roles/node_baseline/tasks/disks.yml

@@ -0,0 +1,642 @@
+---
+# =====================================================================================
+# node_baseline — DATA DISKS.  Every task here carries the `disks` tag (applied once by
+# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
+# Split into its own file rather than living in main.yml because it is the one part of
+# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
+#
+# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
+#
+#   dev  (group_vars/all.yml, role defaults)
+#     node_baseline_longhorn_source: device          -> mkfs /dev/vdb, mount LABEL=longhorn
+#                                                       at /var/lib/longhorn (== dataPath)
+#     node_baseline_etcd_source:     device          -> mkfs /dev/vdc, mount LABEL=k3s-etcd
+#     k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
+#     THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
+#     exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
+#
+#   prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
+#     node_baseline_longhorn_source: claim-mountpoint  (claim mountpoint /srv)
+#     node_baseline_data_mount:      /data            (LABEL=k3s-data)
+#     node_baseline_etcd_source:     shared-data-partition
+#     k3s_data_dir:                  /data/k3s
+#     ONE physical disk per node, already fully partitioned by the Debian installer. The
+#     ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
+#     between TWO tenants:  /data/longhorn (Longhorn replicas)  +  /data/k3s (the whole k3s
+#     data dir: etcd AND the containerd image store).
+#
+# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
+# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
+# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
+# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
+# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
+# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
+# partition is the fix; the partition therefore serves two tenants, and neither its
+# mountpoint nor its filesystem label may name only one of them.
+#
+# WHY /data AND LABEL k3s-data.
+#   * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
+#     Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
+#     at, and the idempotency contract is that /srv is gone from /proc/mounts and
+#     /etc/fstab once the claim has happened. Re-using it as the destination would make
+#     "claimed" and "not yet claimed" indistinguishable.
+#   * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
+#     accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
+#     `data` as an example of somebody else's named volume. Labelling ours `data` would make
+#     a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
+#     cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
+#     convention in this repo.
+#
+# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
+# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
+# empty string, and the second one must never silently succeed. `root-filesystem` and
+# `shared-data-partition` are explicit, greppable statements of intent that the assert
+# below accepts; an unset or placeholder device is still a hard failure. The two differ in
+# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
+# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
+# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
+# than printing a claim it never checked (see the etcd section).
+#
+# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
+# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
+# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
+# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
+# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
+# point at the wrong partition on some future node and mkfs it. So the role asks the NODE
+# which block device is mounted at the claim mountpoint, and then proves that answer is
+# safe (see the assert wall below) before touching it. The inventory declares INTENT
+# ("the partition currently mounted at /srv"); the node supplies the FACT.
+# =====================================================================================
+
+# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
+# its data-disk layout. Two failure shapes are caught:
+#   * an undeclared/typo'd source enum, and
+#   * source: device with a device path that is a PLACEHOLDER rather than a real device.
+# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
+# vars now that prod uses neither: if anyone ever flips a source back to `device` without
+# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
+# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
+# stays in the token list because that was the pre-decision placeholder and may still be
+# carried by other environments/branches.
+- name: Assert this environment's data-disk layout has actually been declared
+  ansible.builtin.assert:
+    that:
+      - node_baseline_longhorn_source in ['device', 'claim-mountpoint']
+      - node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
+      # `shared-data-partition` is only coherent when there IS a claimed data partition to
+      # share, and only when the k3s data dir actually sits INSIDE it. Catch the
+      # contradiction here, in inventory terms, rather than three sections later when etcd
+      # has silently landed back on the OS filesystem.
+      - >-
+        node_baseline_etcd_source != 'shared-data-partition'
+        or node_baseline_longhorn_source == 'claim-mountpoint'
+      - >-
+        node_baseline_etcd_source != 'shared-data-partition'
+        or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
+      # The transition knobs are a matched PAIR: a retired label with no retired mountpoint
+      # (or vice versa) cannot describe a state to migrate from.
+      - >-
+        (node_baseline_data_retired_fs_label | length > 0)
+        == (node_baseline_data_retired_mount | length > 0)
+      - >-
+        node_baseline_longhorn_source != 'device'
+        or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
+      - >-
+        node_baseline_etcd_source != 'device'
+        or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
+      - >-
+        node_baseline_longhorn_source != 'claim-mountpoint'
+        or (node_baseline_longhorn_claim_mountpoint | length) > 0
+    fail_msg: >-
+      This host's data-disk layout is not declared.
+      longhorn_source={{ node_baseline_longhorn_source }}
+      (device={{ node_baseline_longhorn_device }},
+      claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
+      etcd_source={{ node_baseline_etcd_source }}
+      (device={{ node_baseline_etcd_device }}).
+      Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
+      `claim-mountpoint` (re-purpose the partition already mounted at
+      node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
+      `device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
+      it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
+      data partition — prod's shape since 2026-09-04, which additionally requires
+      longhorn_source `claim-mountpoint` and a k3s data dir under
+      node_baseline_data_mount={{ node_baseline_data_mount }};
+      k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
+      operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
+      Declare it in the environment's group_vars and record the reasoning in
+      §2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
+      `--tags packages`, which skips every disk task including this assert.
+    quiet: true
+
+# =====================================================================================
+# THE DATA PARTITION  (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
+# =====================================================================================
+
+# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
+# rather than a reformat. Once the claim has happened the partition carries the ext4
+# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
+# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
+# mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
+# "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
+# path means converged, rc != 0 means the claim has not been made. Everything destructive
+# below is gated on rc != 0.
+- name: "Data partition: look for a filesystem already carrying the data-partition label"
+  ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
+  register: node_baseline_lh_labelled
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+# --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
+# WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
+# label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
+# takes over — which then fails closed at "the claim mountpoint must actually be a
+# mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
+# behaviour but it is not convergence, and the prod nodes are in exactly that state today:
+# claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
+# So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
+#
+# WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
+# partition with the new label. It never touches the DATA and never reformats: e2label is a
+# metadata write, exactly as in the claim path below. The common mount task then mounts
+# LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
+# `blkid -L <new label>` succeeds and none of this is reachable.
+#
+# THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
+# volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
+# unmounting it would orphan every one of them:
+#   1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
+#      group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
+#   2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
+#      the same structural gate that keeps dev out of the claim path keeps it out of this one.
+#   3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
+#      replica directories on it fails closed and nothing is unmounted.
+# Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
+- name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
+  ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
+  register: node_baseline_lh_retired_labelled
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when:
+    - node_baseline_longhorn_source == 'claim-mountpoint'
+    - node_baseline_data_retired_fs_label | length > 0
+    - node_baseline_lh_labelled.rc != 0
+
+- name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
+  ansible.builtin.set_fact:
+    # A node claimed under the RETIRED label. Mutually exclusive with the claim below.
+    node_baseline_lh_transition_needed: >-
+      {{ node_baseline_longhorn_source == 'claim-mountpoint'
+         and node_baseline_data_retired_fs_label | length > 0
+         and node_baseline_lh_labelled.rc != 0
+         and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
+    # True only for a claim-mountpoint environment that has NOT been claimed under EITHER
+    # label. A `device` environment (dev) can never enter the claim path at all — that is
+    # the structural guarantee that no generalisation of prod's behaviour can reach the
+    # LIVE dev nodes and their /dev/vdb Longhorn replicas.
+    node_baseline_lh_claim_needed: >-
+      {{ node_baseline_longhorn_source == 'claim-mountpoint'
+         and node_baseline_lh_labelled.rc != 0
+         and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
+
+- name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_data_retired_mount }}
+  register: node_baseline_lh_retired_findmnt
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_transition_needed | bool
+
+# Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
+# not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
+# an unmount of a populated Longhorn store would orphan every replica on the node.
+- name: "Data partition (transition): read the contents of the retired mountpoint"
+  ansible.builtin.find:
+    paths: "{{ node_baseline_data_retired_mount }}"
+    file_type: any
+    hidden: true
+    recurse: false
+    excludes: ['lost+found']
+  register: node_baseline_lh_retired_contents
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): prove this node is safe to re-label"
+  ansible.builtin.assert:
+    that:
+      # 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
+      #    retired label names. The only state this role knows how to transition is the one
+      #    the previous version of it produced; anything else is an operator decision.
+      - (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
+      - >-
+        (node_baseline_lh_retired_findmnt.stdout | default('') | trim)
+        == (node_baseline_lh_retired_labelled.stdout | default('') | trim)
+      # 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
+      #    not the node we think it is, or a tenant has already written data — in both cases
+      #    unmounting and re-labelling under it is destructive and the role must stop.
+      - (node_baseline_lh_retired_contents.files | length) == 0
+    fail_msg: >-
+      REFUSING to transition this node's data partition from
+      LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
+      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
+      device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
+      device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
+      (findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
+      non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
+      {{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
+      If the partition holds DATA, this transition would orphan it — Longhorn replicas are
+      not recoverable from an unmounted, re-labelled filesystem. Migrate the data
+      deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
+      Do NOT relax this assert.
+    quiet: true
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): report exactly what is about to change"
+  ansible.builtin.debug:
+    msg: >-
+      Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
+      from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
+      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
+      unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
+      No filesystem is created and no data is moved.
+  when: node_baseline_lh_transition_needed | bool
+
+# e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
+# breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
+- name: "Data partition (transition): unmount the retired mountpoint"
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_retired_mount }}"
+    state: unmounted
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_retired_mount }}"
+    state: absent_from_fstab
+    backup: true
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): re-stamp the partition with the new filesystem label"
+  ansible.builtin.command: >-
+    e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
+    {{ node_baseline_data_fs_label }}
+  changed_when: true
+  when: node_baseline_lh_transition_needed | bool
+
+# --- The claim (prod, first run only) -------------------------------------------------
+# Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
+# mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
+# returning the device behind `/` — which is precisely the mistake that would hand the
+# root partition to e2label.
+- name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
+  register: node_baseline_lh_claim_findmnt
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
+  ansible.builtin.assert:
+    that:
+      - node_baseline_lh_claim_findmnt.rc == 0
+      - (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
+    fail_msg: >-
+      node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
+      yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
+      this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
+      stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
+      The role will NOT go looking for "a big partition" to claim instead. Either the
+      node is partitioned differently from the fleet — in which case fix the node or give
+      this host its own declaration — or the claim already happened and the label was
+      removed by hand.
+    quiet: true
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): inspect the candidate partition"
+  ansible.builtin.command: >-
+    lsblk --json --nodeps --bytes
+    --output PATH,TYPE,FSTYPE,LABEL,SIZE
+    {{ node_baseline_lh_claim_findmnt.stdout | trim }}
+  register: node_baseline_lh_claim_lsblk
+  changed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+# The devices this role must never, under any circumstances, hand to e2label. Built from
+# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
+# NODE's account of what is load-bearing, not a hardcoded list of partition names.
+- name: "Longhorn (claim): list the devices currently backing active swap"
+  ansible.builtin.command: swapon --noheadings --show=NAME
+  register: node_baseline_lh_swapdevs
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+# Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
+# ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
+- name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
+  ansible.builtin.find:
+    paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    file_type: any
+    hidden: true
+    recurse: false
+    excludes: ['lost+found']
+  register: node_baseline_lh_claim_contents
+  when: node_baseline_lh_claim_needed | bool
+
+# THE GUARD WALL. Five independent conditions, every one of which must hold before the
+# role modifies a filesystem it did not create. Any single failure stops the play with
+# the evidence in the message — nothing is unmounted, relabelled or mounted first.
+- name: "Longhorn (claim): prove the candidate partition is safe to claim"
+  ansible.builtin.assert:
+    that:
+      # 1. It is a real block device — a partition or a whole disk. Catches a claim
+      #    mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
+      #    which would have produced a non-/dev SOURCE or an lsblk row of another type.
+      - node_baseline_lh_candidate.type in ['part', 'disk']
+      # 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
+      #    so a differently-partitioned node is caught by its own facts.
+      - node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
+      # 3. It already holds ext4. The claim path RELABELS rather than reformats (see
+      #    below), which is only meaningful for the filesystem type we are about to
+      #    declare in fstab. Anything else is an operator decision, not a role's.
+      - node_baseline_lh_candidate.fstype == 'ext4'
+      # 4. It carries no filesystem label, or already carries ours. A partition labelled
+      #    `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
+      #    it would make it unfindable by its own fstab entry.
+      - (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
+      # 5. It is big enough to be the partition we meant. A data partition of a few GB is
+      #    not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
+      - (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
+      # 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
+      #    ext4 5% root reserve; this is where that premise is verified per node instead
+      #    of trusted. If any node has real data on it, the play stops here — the role
+      #    never wipes data it cannot account for.
+      - (node_baseline_lh_claim_contents.files | length) == 0
+    fail_msg: >-
+      REFUSING to claim {{ node_baseline_lh_candidate.path }}
+      (mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
+      type={{ node_baseline_lh_candidate.type }},
+      fstype={{ node_baseline_lh_candidate.fstype }},
+      label='{{ node_baseline_lh_candidate.label | default('', true) }}',
+      size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
+      (minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
+      non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
+      {{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
+      devices this node needs and that must never be claimed:
+      {{ node_baseline_lh_protected_devices }}.
+      This is the destructive step of the role and it fails CLOSED. If the partition
+      holds data, back it up or pick a different one; do NOT relax this assert.
+    quiet: true
+  vars:
+    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
+    # The node's OWN account of what it cannot afford to lose: the devices behind the
+    # mounts in node_baseline_protected_mounts (from gathered facts, so a node
+    # partitioned differently is judged on its own layout) plus every active swap device.
+    # Built here rather than hardcoded so no partition-name list can go stale.
+    node_baseline_lh_protected_devices: >-
+      {{ (ansible_facts.mounts | default([])
+          | selectattr('mount', 'in', node_baseline_protected_mounts)
+          | map(attribute='device') | list)
+         + (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): report exactly what is about to be claimed"
+  ansible.builtin.debug:
+    msg: >-
+      Claiming {{ node_baseline_lh_candidate.path }}
+      ({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
+      currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
+      as the data partition: unmount -> drop its fstab entry ->
+      relabel LABEL={{ node_baseline_data_fs_label }} ->
+      mount at {{ node_baseline_data_mount }}.
+  vars:
+    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
+  when: node_baseline_lh_claim_needed | bool
+
+# --- Why the role must unmount and relabel, not just point fstab at the device ---------
+# The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
+# /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
+# path does not. The installer-created /srv filesystem has NO label (fstab pins it by
+# UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
+# the unmount first — and the fstab entry must go in the same breath, or the next boot
+# re-mounts the same partition at /srv underneath Longhorn.
+#
+# RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
+# mkfs would buy nothing and would destroy anything the emptiness check got wrong.
+# e2label is a metadata write: if some assert above was too permissive, the worst case is
+# a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
+# cannot do this job: it declines to touch a device that already has a filesystem, so it
+# would silently no-op and then the LABEL= mount would fail with no useful message.)
+- name: "Longhorn (claim): unmount the claim mountpoint"
+  ansible.posix.mount:
+    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    state: unmounted
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
+  ansible.posix.mount:
+    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    state: absent_from_fstab
+    backup: true
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
+  ansible.builtin.command: >-
+    e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
+    {{ node_baseline_data_fs_label }}
+  changed_when: true
+  when: node_baseline_lh_claim_needed | bool
+
+# --- The dedicated-device path (dev) --------------------------------------------------
+# Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
+# never reformats a device that already carries a filesystem.
+- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_longhorn_device }}"
+    opts: "-L {{ node_baseline_data_fs_label }}"
+  when: node_baseline_longhorn_source == 'device'
+
+# --- Common to ALL paths (device, claim, transition) ----------------------------------
+# Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
+# mount task. Idempotent: on a converged node this reports ok, which is what makes a
+# re-run a no-op rather than a reformat. It is also what completes the transition above:
+# the retired mountpoint is gone from fstab and the partition now answers to the new label,
+# so this task mounts it at the new path.
+- name: Mount the data partition
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_mount }}"
+    src: "LABEL={{ node_baseline_data_fs_label }}"
+    fstype: ext4
+    opts: "defaults,nofail,x-systemd.device-timeout=5s"
+    state: mounted
+
+# --- The per-tenant subdirectories ---------------------------------------------------
+# WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
+# tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
+# subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
+# /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
+# either there would at best be a no-op and at worst re-mode the running dev cluster's data
+# dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
+# mounted — i.e. only the directories it is actually responsible for. That test is derived
+# from the paths themselves, so no environment has to remember to switch it off.
+#
+# MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
+# the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
+# etcd database, the node token and the containerd image store, and k3s would tighten it to
+# 0700 itself on first start — doing it here means it is never briefly world-readable.
+- name: Create the per-tenant data directories on the data partition
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
+    - { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
+  loop_control:
+    label: "{{ item.path }} ({{ item.tenant }})"
+  when: item.path.startswith(node_baseline_data_mount ~ '/')
+
+# =====================================================================================
+# ETCD DATA DIR
+# =====================================================================================
+
+# --- The dedicated-device path (dev) --------------------------------------------------
+# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
+# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
+# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
+# a disk that already carries a filesystem, so re-running against an already-migrated
+# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
+# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
+# role would mount over a populated directory.
+- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_etcd_device }}"
+    opts: "-L {{ node_baseline_etcd_fs_label }}"
+  when: node_baseline_etcd_source == 'device'
+
+- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
+  ansible.posix.mount:
+    path: "{{ node_baseline_etcd_mount }}"
+    src: "LABEL={{ node_baseline_etcd_fs_label }}"
+    fstype: ext4
+    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
+    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
+    opts: "defaults,noatime"
+    dump: "0"
+    passno: "2"
+    state: mounted
+  when: node_baseline_etcd_source == 'device'
+
+# --- The no-dedicated-device paths (prod) ---------------------------------------------
+# Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
+# populated — so etcd gets no device of its own. That is an operator decision
+# (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
+# UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
+# on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
+# latency rather than collapsing it. Accepted to be MEASURED, not assumed.
+#
+# WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
+# filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
+# 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
+# 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
+# could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
+# partition and etcd moved with it: `shared-data-partition`. Both enum values are still
+# supported, and the role now PROVES which one is true instead of printing a claim.
+#
+# There is nothing to CREATE here for either shape — the data-partition section above
+# already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
+# role's job is to verify the assumption rather than act on it.
+- name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_etcd_mount }}
+  register: node_baseline_etcd_existing_mount
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+- name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
+  ansible.builtin.assert:
+    that:
+      - node_baseline_etcd_existing_mount.rc != 0
+    fail_msg: >-
+      node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
+      NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
+      ALREADY a separate mount, backed by
+      {{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
+      contradiction the role will not resolve on its own: either this node was previously
+      built with a dedicated etcd device (in which case unmount it deliberately, after
+      accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
+      the node in a state where the declared layout and the real one disagree.
+    quiet: true
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+# Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
+# WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
+# existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
+# that does exist — the same filesystem the data dir will be created on. TARGET is captured
+# separately from the human-readable line because the assert below judges on it.
+- name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
+  ansible.builtin.shell: |
+    set -eu
+    p={{ node_baseline_etcd_mount | quote }}
+    while [ ! -d "$p" ]; do p=$(dirname "$p"); done
+    findmnt --noheadings --first-only --output TARGET --target "$p"
+    findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
+  register: node_baseline_etcd_backing_fs
+  changed_when: false
+  check_mode: false
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+# THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
+# `shared-data-partition` is a distinct enum value rather than a comment. Declaring
+# k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
+# above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
+# of container images would land back on the OS filesystem — the exact failure this change
+# exists to prevent, and completely invisible until the disk filled. So the role checks that
+# the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
+# run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
+# which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
+# whole record for that shape.
+- name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
+  ansible.builtin.assert:
+    that:
+      - >-
+        (node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
+        == node_baseline_data_mount
+    fail_msg: >-
+      node_baseline_etcd_source is `shared-data-partition`, which asserts that
+      {{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
+      {{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
+      '{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
+      (full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
+      The data partition is therefore NOT mounted where this layout requires, and k3s would
+      put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
+      filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
+      image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
+      this node until the mount is correct.
+    quiet: true
+  when: node_baseline_etcd_source == 'shared-data-partition'
+
+- name: "etcd (no dedicated device): show which filesystem etcd will live on"
+  ansible.builtin.debug:
+    msg: >-
+      etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
+      (source={{ node_baseline_etcd_source }}); it lands on
+      {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
+      (SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
+      measured risk — §2.
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1,6 +1,6 @@
 ---
-# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
-# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
+# TAGS — every task here carries one of `packages`, `kernel`, `system`, `wireless`, `disks` or
+# `dns`. An untagged run is unchanged (all six run, in this order). The split exists because the PACKAGE floor
 # and the DATA-DISK work have different readiness: packages are safe on any node from
 # first contact, while the disk tasks need block devices that a new environment may not
 # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -12,6 +12,16 @@
 # mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
 # at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
 #   ansible-playbook playbooks/node-baseline.yml --tags dns
+# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
+# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
+# it must be runnable without touching a disk decision that is still open:
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
+# `wireless` (a cluster node gets exactly ONE path onto the network) is separated for a third
+# reason: it must be runnable BEFORE `dns` and without touching packages, the kernel or a disk
+# decision, because a node with a second DHCP-managed interface reverts the `dns` write. It is
+# placed before the disk work for the same reason, and the pair is the whole remedy for the
+# k3s-prod-3 incident:
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags wireless,dns
 
 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
   ansible.builtin.apt:
@@ -21,6 +31,237 @@
     cache_valid_time: 3600
   tags: [packages]
 
+# =====================================================================================
+# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
+# =====================================================================================
+# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
+# FUTURE prod node gets it from provisioning instead of from someone remembering.
+#
+# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
+# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
+# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
+# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
+# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
+# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
+# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
+# (first released in v6.13) validates the size and rejects the table instead. It is absent
+# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
+# stable feed will ever pick it up. There is NO kernel cmdline workaround —
+# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
+# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
+# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
+#
+# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
+# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
+# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
+# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
+# change for no benefit. See defaults/main.yml for why this is a variable rather than a
+# `k3s-prod-*` hostname test.
+#
+# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
+# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
+# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
+# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
+# before touching the next, never node 2 first — and a role cannot enforce that. The last
+# task here reports the pending reboot loudly instead of letting a green play imply the node
+# is already fixed.
+- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
+  when: node_baseline_backports_kernel | bool
+  tags: [kernel]
+  block:
+
+    # The suite name and the running release must agree, or apt resolves no candidate and
+    # `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
+    # a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
+    # dist-upgraded to forky while group_vars still says trixie-backports).
+    - name: Assert the configured backports suite matches this node's Debian release
+      ansible.builtin.assert:
+        that:
+          - ansible_facts['distribution'] == 'Debian'
+          - node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
+        fail_msg: >-
+          node_baseline_backports_kernel is enabled but node_baseline_backports_suite
+          ({{ node_baseline_backports_suite }}) is not the backports suite for this node's
+          release ({{ ansible_facts['distribution'] }}
+          {{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
+          candidate and this play would report success while leaving the node on the
+          vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
+        quiet: true
+
+    # deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
+    # prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
+    # rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
+    # idempotent in a way `apt_repository` — which rewrites and reformats — is not.
+    #
+    # THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
+    # backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
+    # apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
+    # 500 for a package that was explicitly installed from there so it keeps getting updates.
+    # Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
+    # HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
+    # be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
+    - name: Enable the Debian backports apt source (deb822)
+      ansible.builtin.copy:
+        dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
+          # A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
+          # machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
+          # Table its firmware publishes (upstream 8fbe4c49c0cc). See
+          # infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
+          # on this node.
+          #
+          # Backports is NotAutomatic, so nothing is taken from here unless asked for by name
+          # with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
+          # ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
+          # Pin-Priority would switch that off.
+          Types: deb
+          URIs: {{ node_baseline_backports_mirror }}
+          Suites: {{ node_baseline_backports_suite }}
+          Components: {{ node_baseline_backports_components | join(' ') }}
+          Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
+      register: node_baseline_backports_source
+
+    - name: Refresh the apt cache after enabling backports
+      ansible.builtin.apt:
+        update_cache: true
+      when: node_baseline_backports_source is changed
+
+    # `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
+    # nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
+    # nothing at all — the single most likely way to write this task and have it silently fix
+    # nothing forever. `latest` + `default_release` resolves the candidate inside backports
+    # and upgrades the metapackage to it. Once the node is current a re-run is a genuine
+    # no-op: no change, no churn, no spurious reboot flag. Scope is exactly
+    # node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
+    # is dragged out of the main archive.
+    - name: Install the fixed kernel from backports (>= 6.13)
+      ansible.builtin.apt:
+        name: "{{ node_baseline_backports_kernel_packages }}"
+        state: latest
+        default_release: "{{ node_baseline_backports_suite }}"
+        update_cache: true
+        cache_valid_time: 3600
+      register: node_baseline_backports_kernel_apt
+
+    # WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
+    # metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
+    # answer anybody can roll back from six months later.
+    - name: Read back the installed backports kernel metapackage version
+      ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
+      register: node_baseline_backports_kernel_version
+      changed_when: false
+      check_mode: false
+
+    # `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
+    # neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
+    # list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
+    # GRUB's "Advanced options" without uninstalling anything.
+    - name: List every kernel image installed on this node, newest first
+      ansible.builtin.shell:
+        cmd: set -o pipefail; linux-version list | linux-version sort --reverse
+        executable: /bin/bash
+      register: node_baseline_installed_kernels
+      changed_when: false
+      check_mode: false
+
+    # The node must be able to explain itself to an operator standing in front of it with no
+    # repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
+    # role is a real no-op rather than a permanent daily "changed".
+    - name: Record the deviation, the exact version, and the rollback path on the node
+      ansible.builtin.copy:
+        dest: "{{ node_baseline_backports_kernel_stamp }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
+          #
+          # THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
+          #
+          # WHY
+          #   This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
+          #   Linux < 6.13 reserves memory from it without validating its size, and an unlucky
+          #   boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
+          #   k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
+          #   is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
+          #   v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
+          #   Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
+          #
+          # PROOF THE FIX IS ACTIVE (not merely a lucky boot)
+          #   BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
+          #   >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
+          #     zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
+          #   (roles/node_health does the version check for you; see the health report's
+          #   firmware.efi_memattr finding.)
+          #
+          #   FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
+          #   and each prints a DIFFERENT line — all three are rejections:
+          #     tbl->version > 2         'Unexpected EFI Memory Attributes table version %d'
+          #                              (NOTE: printed with %d, so garbage may read NEGATIVE)
+          #     bad desc_size            'Unexpected EFI Memory Attributes descriptor size ...'
+          #     num_entries > 64K        '[Firmware Bug]: Corrupted EFI Memory Attributes
+          #                               Table detected! (version == ..., num_entries == ...)'
+          #   Which one appears depends on WHICH field this boot's garbage corrupted, so do
+          #   NOT treat the absence of the FW_BUG line as the fix being absent — all three
+          #   prod nodes printed the *version* line on 2026-09-04.
+          #   The one line that means the fix is NOT active:
+          #     'Failed to map EFI Memory Attributes table @ 0x...'  — the kernel trusted the
+          #     corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
+          #     escalate, do not shrug.
+          #
+          # INSTALLED
+          #   source   /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
+          #   package  {{ node_baseline_backports_kernel_packages | join(' ') }}
+          #   version  {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
+          #   kernels installed on this node, newest first:
+          {% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
+          #     {{ kv }}
+          {% endfor %}
+          #
+          # SECURITY COST
+          #   Packages from *-backports are provided AS IS. They are NOT covered by the Debian
+          #   Security Team's guarantees for the main archive: no security-tracker commitment,
+          #   no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
+          #
+          # ROLLBACK — no repo checkout needed
+          #   1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
+          #      and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
+          #      GNU/Linux". That answers "is the new kernel the problem?" without
+          #      uninstalling anything. Note it reinstates the memattr bug for that boot.
+          #   2. PERMANENT. Set node_baseline_backports_kernel: false in
+          #      inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
+          #      puts the backports kernel straight back — then, on this node:
+          #        apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
+          #        apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
+          #        rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
+          #        apt-get update && update-grub && reboot
+          #
+          # EXIT CONDITION
+          #   If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
+          #   rebase), this deviation can be removed entirely: flip the variable to false,
+          #   roll back as above, and delete the `kernel` block from roles/node_baseline.
+
+    # Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
+    # it means the fixed kernel is on disk. See the block header for why the role will not
+    # reboot on its own.
+    - name: Report whether this node is running the kernel it now has installed
+      ansible.builtin.debug:
+        msg: >-
+          {{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
+              ~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
+              prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
+              node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
+              with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
+             if newest != ansible_facts['kernel']
+             else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
+                   ~ '); no reboot pending for the memattr fix.') }}
+      vars:
+        newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
+
 - name: Persist kernel modules for k3s/Cilium/Longhorn
   ansible.builtin.copy:
     dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
@@ -79,70 +320,330 @@
     state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
   tags: [system]
 
-# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
-# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
-# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
-# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
-# which do not exist on bare metal either — a confusing failure that looks like a wiring
-# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
-# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
-- name: Assert this environment's data-disk devices have actually been decided
+# =====================================================================================
+# WIRELESS — tag `wireless`: a cluster node gets exactly ONE path onto the network
+# =====================================================================================
+# WHAT WENT WRONG (k3s-prod-3, reported by the operator 2026-09-04). The node was up on BOTH
+# eno1 (192.168.2.12 — the inventory address) and wlp1s0 (192.168.2.195, DHCP, the SAME /24).
+# The wifi lease carried a dhcpcd run with it, and dhcpcd's `20-resolv.conf` hook REWROTE
+# /etc/resolv.conf, undoing the resolvers this role had just written: the file read
+# "# Generated by dhcpcd from wlp1s0.dhcp, wlp1s0.dhcp6, wlp1s0.ra" with nameserver
+# 192.168.2.1 plus two Google IPv6 servers, timestamped THREE MINUTES AFTER the role's own
+# write (01:32 -> 01:35). The role had reported `changed` and green. That is exactly the
+# "successful run that fixed nothing" the `contended` branch of the DNS block below exists to
+# prevent, and the detection blind spot that let it through is closed there.
+#
+# WHY THE INTERFACE IS REMOVED RATHER THAN JUST REPORTED. Closing the detection hole alone
+# would have turned a silent revert into a loud failure — an improvement, but it leaves the
+# node with a second, DHCP-managed path onto the LAN, and on a k3s server that is three
+# distinct problems beyond DNS:
+#   * a SECOND DEFAULT ROUTE, whose metric decides which source address the node uses;
+#   * a SECOND ADDRESS ON THE SAME L2 — the segment where kube-vip floats 192.168.2.9 and
+#     Cilium L2-announces 192.168.2.20-.30 (§4). One host answering ARP on one subnet from
+#     two NICs is the exact shape neither mechanism can reason about;
+#   * an UNMANAGED address: nothing in this repo knows about .195, and the DHCP server is
+#     free to hand out a different one after any lease expiry.
+# None of that is a resolver bug, and none of it is fixed by writing resolv.conf harder.
+#
+# WHY THE DRIVER LAYER — the four alternatives, and why each is weaker. What has to hold is:
+# survives a REBOOT, survives a LEASE RENEWAL, and cannot be undone by a routine `ifup`.
+# Measured on the nodes themselves — all three are HP EliteDesk 800 G5 minis, eno1 is
+# `e1000e`, the wifi NIC is an Intel AX200 on `iwlwifi`, and the OS is Debian 13 trixie with
+# ifupdown + `networking.service`, NO NetworkManager, NO systemd-networkd, NO systemd-resolved:
+#   `ip link set <wifi> down` — NOT persistent. Any `ifup`, any hotplug event, any reboot
+#       undoes it. It is what had already been done by hand on node 3, and the stale dhcpcd
+#       /etc/resolv.conf outlived it — the link was down and the wrong resolvers were still
+#       in the file, across a reboot.
+#   mask the service that manages it — THERE IS NO SUCH SERVICE HERE. dhcpcd on these nodes
+#       is `dhcpcd-base`: no dhcpcd.service exists at all (`systemctl is-enabled dhcpcd` =
+#       not-found), it is spawned per-interface by ifupdown. The only unit that can bring
+#       wireless up is `networking.service` — which ALSO owns eno1's static address, so
+#       masking it would cut the one SSH path to a machine whose operator is not on site.
+#   `rfkill` soft-block — weaker on every axis. The netdev still EXISTS, so ifupdown and
+#       dhcpcd can still be pointed at it; persistence depends on systemd-rfkill restoring
+#       saved state (the kernel's own `persistent` flag reads 0 on these devices, and the
+#       `rfkill` CLI is not even installed); and one `rfkill unblock wlan` undoes it.
+#   the BIOS `M.2 WLAN/BT` toggle — genuinely the deepest layer, and it reads `Enable` on all
+#       three nodes (hp-bioscfg survey, §1b). NOT taken here: it needs a REBOOT to take
+#       effect, and a prod reboot is a serialized, supervised operator step (§0b "SECOND
+#       FAULT" — k3s-prod-2 hung on a warm reboot and needed the power button). It also
+#       disables Bluetooth on the same M.2 card. Left as an operator recommendation in §4.
+#   THE DRIVER MODULE — chosen. `blacklist` plus `install <module> /bin/false` in
+#       /etc/modprobe.d means kmod refuses to load the wifi driver on this boot and every
+#       future one, so THE NETDEV NEVER EXISTS. No interface means no DHCP lease, no
+#       resolv.conf hook, no second route, no second address — and every re-activation path
+#       above (udev hotplug, a hand-run `ifup`, wpa_supplicant, a hand-run dhcpcd) needs a
+#       device that is not there. `blacklist` alone would still allow a dependency- or
+#       hand-triggered `modprobe`; the `install ... /bin/false` line is what makes it refuse
+#       outright. It is ONE declarative file, readable on the node, and undone by deleting
+#       it. Paired with the runtime unload below, it needs NO reboot to take effect.
+#
+# SAFE AND IDEMPOTENT ON ALL THREE PROD NODES *AND* ON THE LIVE DEV CLUSTER. Nothing here is
+# hostname- or hardware-specific: the block acts only on interfaces the node itself reports as
+# real 802.11 devices (a `phy80211` link in sysfs) and on the driver modules behind them.
+#   * dev (Proxmox VMs, virtio NICs) has no such interface, so the discovery comes back empty,
+#     every task below is skipped, and NOT EVEN THE modprobe.d FILE IS WRITTEN. The block
+#     cannot touch dev's networking, which is why it is safe to have on by default.
+#   * k3s-prod-1 (wlp1s0) and k3s-prod-2 (wlp2s0) already have the link DOWN, so the link-down
+#     task is skipped by its own `when` and only the modprobe.d file and the module unload
+#     remain — and both are no-ops on a re-run.
+#   * k3s-prod-3 (wlp1s0) is the reported node.
+# A re-run changes nothing on any of them: the file is content-identical, `ip link set down`
+# is gated on the link not already being down, and the unload is gated on the module still
+# being loaded.
+#
+# AND IT REFUSES TO CUT ITS OWN PATH IN. The assert below stops the play if the address
+# Ansible is connected to is bound to a wireless interface, or if the default route leaves
+# over one, or if the wireless driver module is also the driver of the default-route
+# interface. That is the one case where doing this would lock everybody out of a machine
+# nobody can walk up to, so it is checked per node, on every run, from that node's own facts.
+- name: Survey this node's 802.11 devices, their drivers and any live DHCP client
+  ansible.builtin.shell:
+    executable: /bin/sh
+    cmd: |
+      set -u
+      # Flat KEY|value lines so the Ansible side needs one single-group regex per fact and
+      # no parsing cleverness. A wireless interface is one with a phy80211 link in sysfs —
+      # that is the kernel's own answer to "is this 802.11", not a name pattern like wlp*.
+      wifi=''
+      for d in /sys/class/net/*; do
+        [ -e "$d/phy80211" ] || continue
+        i=$(basename "$d")
+        wifi="$wifi $i"
+        m=$(basename "$(readlink -f "$d/device/driver" 2>/dev/null || echo unknown)")
+        state=$(cat "$d/operstate" 2>/dev/null || echo unknown)
+        printf 'IFACE|%s\n' "$i"
+        printf 'DETAIL|%s driver=%s operstate=%s\n' "$i" "$m" "$state"
+        # LIVE means "not already down" — the link-down task keys off this, so a node whose
+        # radio is already down reports no change.
+        [ "$state" = down ] || printf 'IFACE_LIVE|%s\n' "$i"
+        ip -o addr show dev "$i" 2>/dev/null | awk '{print "IFACE_ADDR|" $4}'
+        [ "$m" = unknown ] || printf 'MODULE|%s\n' "$m"
+      done
+      for m in $(printf '%s\n' $wifi | while read -r i; do
+            [ -n "$i" ] && basename "$(readlink -f "/sys/class/net/$i/device/driver" 2>/dev/null || echo unknown)"
+          done | sort -u); do
+        [ "$m" = unknown ] && continue
+        [ -d "/sys/module/$m" ] || continue
+        printf 'MODULE_LOADED|%s\n' "$m"
+        # Holders must be unloaded FIRST: `modprobe -r iwlwifi` fails while iwlmvm still
+        # holds it. Read from sysfs rather than assumed, so a different wifi driver stack
+        # (ath*, rtw*, mt76*) works without editing this role.
+        for h in $(ls "/sys/module/$m/holders" 2>/dev/null); do printf 'MODULE_HOLDER|%s\n' "$h"; done
+      done
+      dr=$(ip -o route show default 2>/dev/null | awk '{for (i = 1; i < NF; i++) if ($i == "dev") print $(i + 1)}' | head -1)
+      printf 'DEFROUTE_IFACE|%s\n' "${dr:-none}"
+      if [ -n "${dr:-}" ] && [ -e "/sys/class/net/$dr" ]; then
+        printf 'DEFROUTE_MODULE|%s\n' "$(basename "$(readlink -f "/sys/class/net/$dr/device/driver" 2>/dev/null || echo unknown)")"
+      fi
+      # Any DHCP client process, with the interfaces it was started for. Used only to stop a
+      # client that holds a WIRELESS interface and nothing else.
+      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
+        printf 'DHCPCLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
+      done
+      exit 0
+  register: node_baseline_wifi_scan
+  changed_when: false
+  check_mode: false
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Interpret the wireless survey
+  ansible.builtin.set_fact:
+    node_baseline_wifi_ifaces: "{{ scan | regex_findall('(?m)^IFACE\\|(.+)$') }}"
+    node_baseline_wifi_live: "{{ scan | regex_findall('(?m)^IFACE_LIVE\\|(.+)$') }}"
+    node_baseline_wifi_addrs: "{{ scan | regex_findall('(?m)^IFACE_ADDR\\|(\\S+)') | map('regex_replace', '/.*$', '') | list }}"
+    node_baseline_wifi_modules: "{{ scan | regex_findall('(?m)^MODULE\\|(.+)$') | unique }}"
+    node_baseline_wifi_modules_loaded: "{{ scan | regex_findall('(?m)^MODULE_LOADED\\|(.+)$') | unique }}"
+    node_baseline_wifi_module_holders: "{{ scan | regex_findall('(?m)^MODULE_HOLDER\\|(.+)$') | unique }}"
+    node_baseline_wifi_detail: "{{ scan | regex_findall('(?m)^DETAIL\\|(.+)$') }}"
+    node_baseline_wifi_dhcp_clients: "{{ scan | regex_findall('(?m)^DHCPCLIENT\\|(.+)$') }}"
+    node_baseline_defroute_iface: "{{ (scan | regex_findall('(?m)^DEFROUTE_IFACE\\|(.+)$') | first) | default('none', true) }}"
+    node_baseline_defroute_module: "{{ (scan | regex_findall('(?m)^DEFROUTE_MODULE\\|(.+)$') | first) | default('unknown', true) }}"
+  vars:
+    scan: "{{ node_baseline_wifi_scan.stdout | default('') }}"
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Report this node's wireless hardware
+  ansible.builtin.debug:
+    msg: >-
+      wireless devices = {{ node_baseline_wifi_detail | default([]) | join('; ') | default('(none)', true) }}
+      | driver modules = {{ node_baseline_wifi_modules | default([]) | join(', ') | default('(none)', true) }}
+      (loaded: {{ node_baseline_wifi_modules_loaded | default([]) | join(', ') | default('(none)', true) }},
+      held by: {{ node_baseline_wifi_module_holders | default([]) | join(', ') | default('(none)', true) }})
+      | default route via {{ node_baseline_defroute_iface | default('?') }}
+      (driver {{ node_baseline_defroute_module | default('?') }})
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+# The one way this block could do real harm is by removing the interface it is being run
+# over. Checked from the node's own facts, per node, every run — not assumed from the fact
+# that prod "should" be wired.
+- name: Refuse to disable wireless if the path to this node runs OVER the wireless interface
   ansible.builtin.assert:
     that:
-      - "'BLOCKED' not in node_baseline_longhorn_device"
-      - "'BLOCKED' not in node_baseline_etcd_device"
+      - node_baseline_defroute_iface not in node_baseline_wifi_ifaces
+      - node_baseline_defroute_module not in node_baseline_wifi_modules
+      - (ansible_host | default(inventory_hostname)) not in node_baseline_wifi_addrs
     fail_msg: >-
-      node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
-      placeholders for this host's environment
-      (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
-      The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
-      and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
-      the SMART read that decision waits on can be taken) re-run this play with
-      `--tags packages`, which skips every disk task including this assert.
+      REFUSING to disable wireless on this node: the connection to it appears to BE the
+      wireless path. ansible_host={{ ansible_host | default(inventory_hostname) }}, wireless
+      interfaces={{ node_baseline_wifi_ifaces | join(', ') }} carrying
+      {{ node_baseline_wifi_addrs | join(', ') | default('no addresses', true) }}, default route
+      via {{ node_baseline_defroute_iface }} (driver {{ node_baseline_defroute_module }}).
+      Disabling the driver here would drop this node off the network with no remote way back.
+      Give the node a wired address, point the inventory at it, and re-run.
     quiet: true
-  tags: [disks]
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_ifaces | length > 0
+  tags: [wireless]
 
-- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
-  community.general.filesystem:
-    fstype: ext4
-    dev: "{{ node_baseline_longhorn_device }}"
-    opts: "-L {{ node_baseline_longhorn_fs_label }}"
-  tags: [disks]
+# THE PERSISTENT HALF. This file is the whole reboot story: kmod consults /etc/modprobe.d on
+# every load attempt, so from here on the driver cannot be loaded by udev at boot, as a
+# dependency, or by hand. Written ONLY when the node actually has a wireless driver, so a
+# node without wifi hardware (every dev VM) gets no file and no change at all.
+# `update-initramfs` is deliberately NOT run: no wifi module is loaded from the initramfs on
+# these nodes (root is a local SATA/NVMe device), the driver is autoloaded by udev after
+# switch-root, and re-generating an initramfs on a prod node is a bigger risk than the one it
+# would cover.
+- name: Block the wireless driver module(s) from ever loading again
+  ansible.builtin.copy:
+    dest: "{{ node_baseline_wireless_modprobe_file }}"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # Managed by the Ansible node_baseline role (tags: wireless) — do not edit by hand.
+      #
+      # WHY THIS NODE HAS NO WIRELESS. A k3s server gets exactly ONE path onto the network.
+      # A second, DHCP-managed interface on the same LAN gives the node a second default
+      # route, a second address nothing in the cluster's config knows about, and a DHCP
+      # client whose resolv.conf hook silently overwrites the resolvers Ansible manages
+      # (that is what happened on k3s-prod-3, 2026-09-04). The link is also brought down and
+      # the module unloaded at the same time, so no reboot is needed for this to take effect.
+      #
+      # `blacklist` stops udev's alias-driven autoload; the `install ... /bin/false` line is
+      # what also refuses a dependency-triggered or hand-run `modprobe`. Both are needed.
+      #
+      # TO RE-ENABLE WIRELESS ON THIS NODE: delete this file and `modprobe` the module(s)
+      # below (or reboot). To stop Ansible putting it back, set
+      # node_baseline_disable_wireless: false for this host first — otherwise the next
+      # node-baseline run recreates it. Full rationale: roles/node_baseline/tasks/main.yml,
+      # the `wireless` block, and infra/PROD-PROVISIONING.md §4.
+      {% for module in node_baseline_wifi_modules %}
+      blacklist {{ module }}
+      install {{ module }} /bin/false
+      {% endfor %}
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_modules | length > 0
+  tags: [wireless]
 
-- name: Mount the Longhorn data disk
-  ansible.posix.mount:
-    path: "{{ node_baseline_longhorn_mount }}"
-    src: "LABEL={{ node_baseline_longhorn_fs_label }}"
-    fstype: ext4
-    opts: "defaults,nofail,x-systemd.device-timeout=5s"
-    state: mounted
-  tags: [disks]
+# THE RUNTIME HALF — everything below makes the change true NOW, so the node does not have to
+# be rebooted to stop carrying a second address (this role never reboots: §0b "SECOND FAULT").
+#
+# Stop a DHCP client that is holding a wireless interface first, so it releases its lease
+# instead of being left holding a device that is about to disappear. HEAVILY GUARDED: it runs
+# only for clients whose command line names a wireless interface and names NO non-wireless
+# interface, because `dhcpcd -x` against a shared master process would also drop a wired
+# lease. On these prod nodes eno1 is a static ifupdown address with no DHCP client at all, so
+# this task has nothing to do; it exists so the ordering is correct on a node where wifi is
+# still actively leasing.
+- name: Stop the dhcpcd instance that holds a wireless interface (and nothing else)
+  ansible.builtin.command:
+    argv: [dhcpcd, --exit, "{{ item }}"]
+  loop: "{{ node_baseline_wifi_ifaces }}"
+  register: node_baseline_wifi_dhcpcd_exit
+  changed_when: rc == 0
+  failed_when: false
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_dhcp_wifi_only | length > 0
+    - node_baseline_wifi_dhcp_wifi_only | select('search', 'dhcpcd') | list | length > 0
+  vars:
+    # A client is "wireless-only" when its argv mentions a wireless interface and mentions no
+    # other interface this node has. Empty interface sets are turned into a pattern that
+    # cannot match, so an odd node fails towards doing nothing rather than towards guessing.
+    node_baseline_non_wifi_ifaces: "{{ ansible_facts['interfaces'] | default([]) | difference(node_baseline_wifi_ifaces) | difference(['lo']) }}"
+    node_baseline_wifi_dhcp_wifi_only: >-
+      {{ node_baseline_wifi_dhcp_clients
+         | select('search', '\b(' ~ ((node_baseline_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
+         | reject('search', '\b(' ~ ((node_baseline_non_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
+         | list }}
+  tags: [wireless]
 
-# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
-# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
-# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
-# a disk that already carries a filesystem, so re-running against an already-migrated
-# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
-# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
-# role would mount over a populated directory.
-- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
-  community.general.filesystem:
-    fstype: ext4
-    dev: "{{ node_baseline_etcd_device }}"
-    opts: "-L {{ node_baseline_etcd_fs_label }}"
-  tags: [disks]
+- name: Take each wireless link down now
+  ansible.builtin.command:
+    argv: [ip, link, set, dev, "{{ item }}", down]
+  loop: "{{ node_baseline_wifi_live }}"
+  changed_when: true
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
 
-- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
-  ansible.posix.mount:
-    path: "{{ node_baseline_etcd_mount }}"
-    src: "LABEL={{ node_baseline_etcd_fs_label }}"
-    fstype: ext4
-    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
-    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
-    opts: "defaults,noatime"
-    dump: "0"
-    passno: "2"
-    state: mounted
+# Unload the driver so the netdev is gone immediately — holders (iwlmvm on these nodes) first,
+# then the driver itself, because `modprobe -r` refuses a module that is still held.
+# NOT fatal on failure: the persistent block above is already in place, so a module that
+# cannot be unloaded right now is gone at the next boot regardless. The report below says so
+# out loud rather than letting a green play imply the radio is already off.
+- name: Unload the wireless driver module(s) now (no reboot needed)
+  community.general.modprobe:
+    name: "{{ item }}"
+    state: absent
+  loop: "{{ node_baseline_wifi_module_holders + node_baseline_wifi_modules_loaded }}"
+  register: node_baseline_wifi_unload
+  failed_when: false
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Report the wireless state this node is left in
+  ansible.builtin.debug:
+    msg: >-
+      {{ 'No 802.11 device on this node — nothing to disable (no modprobe.d file written).'
+         if (node_baseline_wifi_ifaces | length) == 0
+         else ((('WIRELESS DISABLED: ' ~ (node_baseline_wifi_modules | join(', ')) ~ ' blocked in '
+                 ~ node_baseline_wireless_modprobe_file ~ ' (survives reboot) and unloaded now; '
+                 ~ (node_baseline_wifi_ifaces | join(', ')) ~ ' is down and the netdev is gone.')
+                if (failed_unloads | length) == 0
+                else ('WIRELESS BLOCKED FOR THE NEXT BOOT BUT STILL LOADED NOW: could not unload '
+                      ~ (failed_unloads | join(', ')) ~ '. ' ~ node_baseline_wireless_modprobe_file
+                      ~ ' is in place, so the driver will not load again after a reboot, but the '
+                      ~ 'radio is live until then — check `ip link` and whether something still '
+                      ~ 'holds the device.'))) }}
+  vars:
+    failed_unloads: >-
+      {{ (node_baseline_wifi_unload.results | default([]))
+         | selectattr('failed', 'defined') | selectattr('failed') | map(attribute='item') | list }}
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+# =====================================================================================
+# DATA DISKS — tag `disks`, in tasks/disks.yml
+# =====================================================================================
+# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
+# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
+# should be readable end-to-end on its own; and both environments' storage shapes — dev's
+# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
+# the OS filesystem — are declared there as an explicit enum rather than inferred from
+# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
+#
+# The `disks` tag is applied ONCE here, in BOTH places it is needed:
+#   * `tags:` on the include itself, so `--tags packages` never even reaches the include
+#     and a placeholder device path is not so much as templated; and
+#   * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
+#     optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
+#     its own tags to its children the way `import_tasks` does, so without `apply` the
+#     included tasks are untagged and `--tags disks` selects the include and then runs
+#     NOTHING inside it. That failure is silent and reports success, which is the worst
+#     shape a guard can have.
+# A FULL run against a node whose layout is undeclared stops on the assert at the top of
+# that file instead of failing obscurely at mkfs time.
+- name: Provision the data disks (Longhorn store, etcd data dir)
+  ansible.builtin.include_tasks:
+    file: disks.yml
+    apply:
+      tags: [disks]
   tags: [disks]
 
 - name: Disable swap now
@@ -180,6 +681,11 @@
 # node and the matching seam is written. Adding a third mechanism means adding a branch
 # here, not rewriting the role.
 #
+# AND DETECTION IS ONLY AS GOOD AS THE SET OF OWNERS IT KNOWS ABOUT. It originally looked for
+# resolvconf/openresolv and NetworkManager, missed dhcpcd, and therefore reported a green,
+# `changed` run on k3s-prod-3 whose resolvers a wifi lease reverted three minutes later. dhcpcd
+# is now part of that set — see the detection task below and the `contended` failure.
+#
 # All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
 # documented at the top of this file) still installs nothing but packages.
 
@@ -199,6 +705,55 @@
   check_mode: false
   tags: [dns]
 
+# THE dhcpcd BLIND SPOT, CLOSED (k3s-prod-3, 2026-09-04). dhcpcd's `20-resolv.conf` hook
+# rewrites /etc/resolv.conf on every lease, renewal and carrier change, and it is a THIRD
+# way to own that file — neither resolvconf nor NetworkManager. On node 3 it silently reverted
+# this role's write three minutes later, and because nothing below looked for it the role had
+# already reported `changed` and green: the precise failure the `contended` branch exists to
+# make impossible.
+#
+# WHY THIS LOOKS FOR A LIVE CLIENT AND NOT FOR AN INSTALLED PACKAGE. `dhcpcd-base` is
+# installed on all three prod nodes and always will be — it is the DHCP client ifupdown
+# reaches for — and there is NO dhcpcd.service on them at all (`systemctl is-active dhcpcd` =
+# not-found; it is spawned per-interface by ifupdown). Keying off the package, or off a unit
+# that does not exist, would put every prod node permanently in `contended` and leave the DNS
+# block unable to configure a node it is perfectly safe to configure. What actually matters is
+# whether a DHCP client CAN still write the file, so all three real signals are checked:
+#   CLIENT| a dhcpcd/dhclient process running right now (with the argv that started it);
+#   UNIT|   a dhcpcd/dhclient unit that IS active or IS enabled, on a distro that has one;
+#   STANZA| an UNCOMMENTED `iface <x> inet[6] dhcp` in /etc/network/interfaces{,.d} — the
+#           configuration that makes ifupdown spawn a client at the next boot or `ifup`, even
+#           when none is running at this instant.
+# Any one of them means "another manager is in play" and the play stops below. None of them
+# means no client can run, and the file is genuinely ours to own.
+- name: Determine whether a DHCP client (dhcpcd/dhclient) can rewrite /etc/resolv.conf
+  ansible.builtin.shell:
+    executable: /bin/sh
+    cmd: |
+      set -u
+      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
+        printf 'CLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
+      done
+      for u in dhcpcd dhcpcd5 dhclient isc-dhcp-client; do
+        a=$(systemctl is-active "$u" 2>/dev/null || true)
+        e=$(systemctl is-enabled "$u" 2>/dev/null || true)
+        # Compared exactly, never with a substring match: `is-active` answers "inactive",
+        # which contains the word "active".
+        if [ "$a" = active ] || [ "$e" = enabled ] || [ "$e" = enabled-runtime ]; then
+          printf 'UNIT|%s active=%s enabled=%s\n' "$u" "$a" "$e"
+        fi
+      done
+      # `^[[:space:]]*iface` cannot match a commented-out stanza, which is how the wireless
+      # stanza is left on these nodes.
+      grep -rhE '^[[:space:]]*iface[[:space:]]+[^[:space:]]+[[:space:]]+inet6?[[:space:]]+dhcp' \
+        /etc/network/interfaces /etc/network/interfaces.d/ 2>/dev/null \
+        | sed -e 's/^[[:space:]]*/STANZA|/'
+      exit 0
+  register: node_baseline_dhcp_client_scan
+  changed_when: false
+  check_mode: false
+  tags: [dns]
+
 # follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
 # points, which is the strongest single signal of who owns the file.
 - name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
@@ -229,9 +784,10 @@
 #       resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
 #       then the ONLY thing that changes real behaviour; writing resolv.conf would be
 #       overwritten by resolved (or, worse, land in /run and vanish at reboot).
-#   contended — nobody's stub, but resolvconf or NetworkManager is present/running and
-#       would regenerate the file underneath us. We do NOT write. The role fails loudly
-#       below rather than leaving an operator believing a change stuck when it did not.
+#   contended — nobody's stub, but resolvconf, NetworkManager or a DHCP client (dhcpcd /
+#       dhclient) is present/running/configured and would regenerate the file underneath us.
+#       We do NOT write. The role fails loudly below rather than leaving an operator
+#       believing a change stuck when it did not.
 #   resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
 #       Safe to own outright.
 # resolved running but NOT owning resolv.conf correctly falls through to the file branch:
@@ -241,13 +797,21 @@
     node_baseline_dns_mechanism: >-
       {{
         'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
-        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
+        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present
+                              or node_baseline_dhcp_client_live)
               else 'resolv-conf-file')
       }}
+    # Kept as facts (not just task vars) so the failure message, the report and the
+    # reclaim note below all quote the SAME evidence this decision was made on.
+    node_baseline_dhcp_client_evidence: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) | regex_findall('(?m)^(?:CLIENT|UNIT|STANZA)\\|(.+)$') }}"
+    # A file dhcpcd WROTE, versus a dhcpcd that can still write it — two different facts, and
+    # only the second one is contention. See the reclaim note below.
+    node_baseline_resolv_dhcpcd_written: "{{ 'Generated by dhcpcd' in (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
   vars:
     node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
     node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
     node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
+    node_baseline_dhcp_client_live: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) is search('(?m)^(?:CLIENT|UNIT|STANZA)\\|') }}"
     node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
     node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
     node_baseline_resolved_owns_resolv: >-
@@ -264,7 +828,9 @@
       NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
       /etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
       target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
-      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
+      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }},
+      DHCP client evidence={{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }},
+      resolv.conf written by dhcpcd={{ node_baseline_resolv_dhcpcd_written | default(false) }})
   tags: [dns]
 
 # Same shape, and the same reason, as the data-disk assert above: this environment's
@@ -297,16 +863,45 @@
 - name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
   ansible.builtin.fail:
     msg: >-
-      /etc/resolv.conf on this node is not the systemd-resolved stub, but
-      resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
-      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
-      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
-      would regenerate the file underneath this role. Writing it here would look like it
-      worked and silently revert. Add the matching branch to roles/node_baseline (the
-      `dns`-tagged block) for whichever manager this node really uses — do not force it.
+      /etc/resolv.conf on this node is not the systemd-resolved stub, but something else would
+      regenerate the file underneath this role: resolvconf/openresolv
+      (/etc/resolvconf/resolv.conf.d present:
+      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}), NetworkManager
+      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}), or a DHCP client
+      that can still write it — dhcpcd/dhclient evidence:
+      {{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }}.
+      Writing it here would look like it worked and silently revert — which is exactly what
+      happened on k3s-prod-3 on 2026-09-04, where a wifi lease's dhcpcd hook reverted this
+      role's resolvers three minutes after a green run. If the evidence above is a DHCP client
+      on a WIRELESS interface, the fix is the `wireless` block earlier in this role (a cluster
+      node gets one network path) — run it, then re-run this. If it is a DHCP client on the
+      node's real interface, or another manager entirely, add the matching branch to
+      roles/node_baseline for the manager this node really uses — do not force it.
   when: node_baseline_dns_mechanism == 'contended'
   tags: [dns]
 
+# THE CORPSE CASE, and why it is NOT a hole in the fail-closed rule above. After the
+# `wireless` block disables the radio, k3s-prod-3 is left with a /etc/resolv.conf that dhcpcd
+# WROTE (its "# Generated by dhcpcd from wlp1s0.dhcp..." header, the gateway as sole IPv4
+# resolver, two Google IPv6 servers) and NO dhcpcd that can ever run again. `contended` means
+# "another manager is live and will undo this write"; a file left behind by a client that no
+# longer exists is not a manager, it is the damage — and refusing to write here would leave the
+# node permanently mis-resolved, which is the very outcome the operator reported. So the file
+# branch below reclaims it, and says so.
+- name: Note that a dhcpcd-written /etc/resolv.conf is being reclaimed
+  ansible.builtin.debug:
+    msg: >-
+      /etc/resolv.conf on this node was written by dhcpcd, but no DHCP client can write it any
+      more (no process, no active/enabled unit, no `inet dhcp` stanza). Taking ownership and
+      replacing it with the declared upstreams
+      ({{ node_baseline_dns_servers | join(', ') }}). If a DHCP client ever comes back on this
+      node, the `contended` check above will stop the play instead of letting it revert
+      silently.
+  when:
+    - node_baseline_dns_mechanism == 'resolv-conf-file'
+    - node_baseline_resolv_dhcpcd_written | default(false)
+  tags: [dns]
+
 # --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
 # A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
 # never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses

infra/ansible/roles/node_health/README.md

@@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI,
   EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified
   against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM).
-- **Never `become`s unless it must.** Only SMART needs root.
+- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring
+  buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM
+  inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and
+  the privileged second pass re-runs the *same* collector with `--privileged` so the
+  parsing has one home. Without escalation those checks SKIP with the exact reason.
 
 ## What it checks
 
@@ -44,15 +48,99 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
 | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
+| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
 | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
+| `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
 
 Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
 in task logic.
 
+
+## The EFI memattr check
+
+`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
+
+HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
+does not validate it before reserving memory from its contents. **Two different kernel
+functions matter, and the check used to conflate them:**
+
+- `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both
+  fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so
+  **catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through
+  `phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything
+  above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and
+  costs nothing measurable.
+- `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints
+  "Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or
+  merely large.
+
+So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not*
+prove memory was lost: that depends on the garbage **value** at that address on that boot,
+not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03)
+booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on
+measured memory, never on the log line alone:**
+
+| kernel log | measured shortfall | verdict |
+|---|---|---|
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic |
+| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working |
+| no memattr line, log reaches back to boot | (not consulted) | PASS |
+| no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
+| `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
+
+The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt
+table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt
+table is not passing a health check even when it kept its RAM.
+
+### The FW_BUG line is proof, but its ABSENCE proves nothing
+
+`efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG
+line:
+
+| gate | message | reachable when |
+|---|---|---|
+| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
+| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not |
+| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not |
+
+**All three are rejections.** Which one a boot prints is decided by *which field* that boot's
+garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very
+often prints the *first* message, and looking only for the FW_BUG line will make a working
+fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed
+the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version
+alongside the log line, and why the check must never demand the FW_BUG string as its
+evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in
+the kernel image itself, not a line in `dmesg`.
+
+**The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged
+*negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that
+until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected
+a corrupt table was reported as one that "accepted it without complaint" — a false PASS on
+the exact bug this check exists to find.
+
+**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
+`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
+workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
+rebooting/CMOS/BIOS/USB changes only move where the table lands.
+
+### The two memory checks cannot contradict each other
+
+`memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine
+lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus
+the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in
+`tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads
+it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of
+its own, so there is exactly one verdict on lost memory per host and both checks quote it.
+A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a
+future memory loss with a different cause is still caught.
+
 ## Output
 
 1. A human summary printed in the play output.

infra/ansible/roles/node_health/defaults/main.yml

@@ -45,6 +45,33 @@ node_health_thresholds:
   # Memory (percentage of MemTotal still available)
   mem_available_pct_warn: 15
   mem_available_pct_fail: 5
+  # --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ----------------------------
+  # Two independent measurements of the same question, both fed by memory.physical_total:
+  #
+  #   unmanaged  /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own
+  #              memory map says exists, that never reached the allocator. Needs no root
+  #              and no tools, works identically on metal and in a VM.
+  #   shortfall  MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is
+  #              installed. Needs root, and only bare metal reports it. Independent of the
+  #              kernel's memory map entirely, so it also catches RAM the firmware never
+  #              handed the kernel in the first place.
+  #
+  # MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and
+  # not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2%
+  # k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state.
+  # The healthy figure is the kernel image, memmap and firmware reservations; it is a
+  # near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10%
+  # keeps clearance for that. There is no plausible tuning of a threshold in this band
+  # that changes any of those five verdicts.
+  #
+  # The DIMM shortfall runs a little higher than the unmanaged figure on the same node
+  # (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and
+  # the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the
+  # kernel's to manage. Same bands: the signal being caught is 83%, not 3%.
+  mem_unmanaged_pct_warn: 10
+  mem_unmanaged_pct_fail: 25
+  mem_dimm_shortfall_pct_warn: 10
+  mem_dimm_shortfall_pct_fail: 25
   # Temperatures (deg C, any sensor)
   temp_warn_c: 70
   temp_fail_c: 85

infra/ansible/roles/node_health/files/node_health_collect.py

@@ -97,10 +97,22 @@ def read_kv(path, sep=":"):
     return out
 
 
+def which(tool):
+    """Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits.
+
+    `dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the
+    unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed
+    but unreachable would be a false reason on a SKIP, which is the one thing this
+    collector may never do."""
+    return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+
+
 def run(argv, timeout=20):
     """Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent."""
-    if shutil.which(argv[0]) is None:
+    exe = which(argv[0])
+    if exe is None:
         return None, "", "%s not installed" % argv[0]
+    argv = [exe] + list(argv[1:])
     try:
         proc = subprocess.run(
             argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout
@@ -284,9 +296,300 @@ def collect_memory():
         edac["ce_total"] = ce_total
         edac["ue_total"] = ue_total
     mem["edac"] = edac
+    # Physically present vs actually usable — the cause-independent memory-loss signal.
+    mem["zones"] = collect_zoneinfo(mem["mem_total_kb"])
+    # Firmware's own inventory of what is physically installed. Root-only; attempted
+    # here anyway so the unprivileged pass records the exact reason it could not read.
+    mem["devices"] = collect_memory_devices()
     return mem
 
 
+# --- Memory the kernel has but never hands out --------------------------------------
+# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
+# no DMI. Per zone it reports:
+#   spanned  pages in the zone's address range, holes included
+#   present  spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says
+#            physically exists here
+#   managed  present MINUS everything memblock reserved before the buddy allocator was
+#            handed the rest (kernel image, memmap, initrd, crashkernel, firmware
+#            reservations). sum(managed) IS MemTotal, exactly, on every node measured.
+# So `present - managed` is, by construction, "RAM this machine has that the OS will
+# never allocate", whatever the reason. On a healthy node it is the couple of hundred
+# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and
+# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% —
+# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude
+# separation, which is why this needs no clever tuning to be trustworthy.
+#
+# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if
+# a future firmware, kernel or hypervisor lost the memory for an entirely different
+# reason, and it needs neither root nor a parseable kernel log.
+def collect_zoneinfo(mem_total_kb):
+    raw = read_text("/proc/zoneinfo", optional=False)
+    page_kb = os.sysconf("SC_PAGE_SIZE") // 1024
+    out = {
+        "available": raw is not None,
+        "reason": None if raw is not None else "/proc/zoneinfo could not be read",
+        "page_size_kb": page_kb,
+        "zones": [],
+        "present_kb": None,
+        "managed_kb": None,
+        "unmanaged_kb": None,
+        "unmanaged_pct": None,
+    }
+    if raw is None:
+        return out
+    node = zone = None
+    cur = {}
+    present_pages = managed_pages = 0
+    for line in raw.splitlines():
+        fields = line.split()
+        if line.startswith("Node ") and "zone" in line:
+            # "Node 0, zone   Normal"
+            if cur:
+                out["zones"].append(cur)
+            node, zone = fields[1].rstrip(","), fields[-1]
+            cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None}
+        elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"):
+            try:
+                cur[fields[0]] = int(fields[1])
+            except ValueError:
+                cur[fields[0]] = None
+    if cur:
+        out["zones"].append(cur)
+    # An empty parse is a PARSE failure, not "this node has no memory" — say so and stay
+    # null rather than emitting a confident 0.
+    if not out["zones"]:
+        out["available"] = False
+        out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it"
+        return out
+    for z in out["zones"]:
+        present_pages += z["present"] or 0
+        managed_pages += z["managed"] or 0
+    out["present_kb"] = present_pages * page_kb
+    out["managed_kb"] = managed_pages * page_kb
+    out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"]
+    if out["present_kb"] > 0:
+        out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1)
+    # Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the
+    # two readings disagree and the discrepancy itself is worth showing.
+    out["matches_mem_total"] = (
+        None if (mem_total_kb is None or out["managed_kb"] is None)
+        else abs(out["managed_kb"] - mem_total_kb) <= page_kb
+    )
+    return out
+
+
+# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) -----------------------------------
+# The independent answer to "how much RAM is physically in this box", from the firmware's
+# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads
+# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that
+# failure is recorded as the SKIP reason — never as zero DIMMs.
+#
+# On a VM there is usually either no type-17 structure at all or a single synthetic entry
+# that merely restates the configured guest RAM; both are handled by reporting what was
+# found and letting the check SKIP when nothing populated was reported.
+DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def collect_memory_devices():
+    out = {
+        "available": False,
+        "reason": None,
+        "devices": [],
+        "slots": 0,
+        "populated_slots": 0,
+        "installed_kb": None,
+    }
+    if which("dmidecode") is None:
+        out["reason"] = ("dmidecode is not installed on this node; it ships with "
+                         "roles/node_baseline (node_baseline_packages). "
+                         "This role never installs packages.")
+        return out
+    rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30)
+    if rc != 0:
+        out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % (
+            rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output")
+        return out
+    # Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which
+    # may or may not hold a module ("Size: No Module Installed").
+    total_kb = 0
+    saw_size = False
+    for block in stdout.split("\n\n"):
+        if "Memory Device" not in block:
+            continue
+        fields = {}
+        for line in block.splitlines():
+            key, sep, value = line.partition(":")
+            if sep:
+                fields[key.strip()] = value.strip()
+        if "Size" not in fields:
+            continue
+        out["slots"] += 1
+        size_kb = None
+        parts = fields["Size"].split()
+        if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS:
+            try:
+                size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()]
+            except ValueError:
+                size_kb = None
+        if size_kb:
+            saw_size = True
+            total_kb += size_kb
+            out["populated_slots"] += 1
+        out["devices"].append(
+            {
+                "locator": fields.get("Locator"),
+                "bank_locator": fields.get("Bank Locator"),
+                # Verbatim, so an unparsed spelling is visible rather than silently null.
+                "size_raw": fields["Size"],
+                "size_kb": size_kb,
+                "type": fields.get("Type"),
+                "speed": fields.get("Speed"),
+                "configured_speed": fields.get("Configured Memory Speed"),
+                "manufacturer": fields.get("Manufacturer"),
+                "part_number": fields.get("Part Number"),
+                "rank": fields.get("Rank"),
+                # Serial deliberately NOT collected: the artifact already carries drive
+                # serials and a DIMM serial adds identifying data no check reads.
+            }
+        )
+    out["available"] = saw_size
+    if not saw_size:
+        out["reason"] = ("dmidecode ran but reported no populated memory device "
+                         "(%d slot structure(s) found) — typical of a VM with no SMBIOS "
+                         "type 17 inventory" % out["slots"])
+    else:
+        out["installed_kb"] = total_kb
+    return out
+
+
+# --- Kernel ring buffer: the EFI Memory Attributes Table ----------------------------
+# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative
+# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
+# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
+# the product). The kernel prints one of two lines:
+#
+#   "Failed to map EFI Memory Attributes table @ 0x..."   the garbage version read <= 2, so
+#       the kernel TRUSTED the table and reserved memory from it. This is emitted later, by
+#       efi_memattr_apply_permissions()'s memremap(), and it fails for ANY sufficiently
+#       wrong tbl_size. So it proves the table was corrupt and trusted — it does NOT prove
+#       memory was lost: only a NEGATIVE tbl_size (bit 31 set, sign-extended and clamped to
+#       PHYS_ADDR_MAX) reserves everything above the table, while a garbage-but-positive
+#       tbl_size costs nothing measurable. Whether memory actually went missing is decided
+#       by MEASUREMENT (collect_zoneinfo below), not by this line: k3s-prod-2 lost 13.3 GiB
+#       twice at 0xbeeb0018 and then booted with all 15.1 GiB at the same address, same line.
+#   "Unexpected EFI Memory Attributes table version <N>"  the garbage version read > 2, so
+#       the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
+#       firmware is still publishing a corrupt table and the outcome is decided afresh by
+#       what the garbage reads on every boot. That is luck, not immunity.
+#
+# This collector reports the LINES and the addresses; it never rules on memory. Classifying
+# is roles/node_health/tasks/assess.yml, which rules on the measured shortfall.
+#
+# It is boot-to-boot NON-DETERMINISTIC, which is precisely why it is detected on every
+# health run instead of diagnosed once.
+#
+# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
+# every node here). The unprivileged pass therefore records WHY it could not read, and
+# roles/node_health re-runs this collector with `--privileged` when escalation works.
+BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:")
+EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
+MEMATTR_FAILED_RE = re.compile(
+    r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
+# NOTE the `-?`. The kernel prints this version with %d — SIGNED — so a garbage field with
+# bit 31 set is logged NEGATIVE (k3s-prod-2, 2026-09-04: "version -1569980416"). Without the
+# minus sign this regex missed those lines entirely and the state fell through to "clean",
+# i.e. a node whose kernel had just REJECTED a corrupt table was reported as one whose
+# kernel "accepted it without complaint" — a false PASS on the exact bug this check exists
+# to find. Anchor future edits on that: the field is signed, and every diagnostic that
+# quotes it must survive a negative value.
+MEMATTR_VERSION_RE = re.compile(
+    r"Unexpected EFI Memory Attributes table version\s+(-?\d+)", re.I)
+MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
+# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
+# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
+# captured separately, above, and quoting it as evidence would bury the one line that
+# matters in 200 characters of unrelated table addresses.
+MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I)
+# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line
+# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict.
+MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available")
+MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved")
+
+
+def collect_klog():
+    out = {
+        "available": False,
+        "reason": None,
+        "boot_covered": None,
+        "memattr_state": None,
+        "memattr_published": None,
+        "memattr_table_address": None,
+        "memattr_version": None,
+        "memattr_lines": [],
+        "memory_line": None,
+    }
+    rc, stdout, stderr = run(["dmesg"], timeout=30)
+    if rc is None:
+        out["reason"] = "dmesg is not installed on this node"
+        return out
+    if rc != 0:
+        detail = (stderr or stdout).strip().splitlines()
+        out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % (
+            rc, os.geteuid(), detail[-1] if detail else "no output",
+            " — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG"
+            if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "")
+        return out
+
+    out["available"] = True
+    # Does the buffer still reach back to early boot? The ring buffer WRAPS, and every
+    # memattr line is printed in the first fraction of a second. If the boot header is
+    # gone then so may be a memattr line that was printed — absence would prove nothing,
+    # and the check must SKIP rather than PASS. This is the difference between
+    # "unaffected" and "could not tell", and the role is not allowed to blur it.
+    out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout))
+
+    published = EFI_CONFIG_RE.search(stdout)
+    if published:
+        out["memattr_published"] = True
+        out["memattr_table_address"] = published.group(1)
+    elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M):
+        # UEFI booted and the config-table line is present, but it lists no MEMATTR=.
+        out["memattr_published"] = False
+    # else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None.
+
+    for line in stdout.splitlines():
+        if MEMATTR_LINE_RE.search(line):
+            out["memattr_lines"].append(line.strip())
+    joined = "\n".join(out["memattr_lines"])
+    if MEMATTR_CORRUPT_RE.search(joined):
+        # Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected.
+        out["memattr_state"] = "corrupt_rejected"
+    elif MEMATTR_FAILED_RE.search(joined):
+        out["memattr_state"] = "failed_to_map"
+        addr = MEMATTR_FAILED_RE.search(joined).group(1)
+        if addr:
+            out["memattr_table_address"] = addr
+    elif MEMATTR_VERSION_RE.search(joined):
+        out["memattr_state"] = "bad_version"
+        out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1))
+    else:
+        out["memattr_state"] = "clean"
+
+    mem_line = MEMORY_LINE_RE.search(stdout)
+    if mem_line:
+        full = [l for l in stdout.splitlines() if mem_line.group(0) in l]
+        reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None
+        out["memory_line"] = {
+            "available_kb": int(mem_line.group(1)),
+            "total_kb": int(mem_line.group(2)),
+            "reserved_kb": int(reserved.group(1)) if reserved else None,
+            "raw": full[0].strip() if full else None,
+        }
+    return out
+
+
 # --- Thermals -----------------------------------------------------------------------
 def collect_thermal():
     zones = []
@@ -708,15 +1011,34 @@ def collect_os():
 def collect_tools():
     tools = {}
     for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
-        tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+        tools[tool] = which(tool)
     return tools
 
 
 def main():
+    # --privileged: re-collect ONLY the fields that genuinely need root, for the second
+    # pass roles/node_health makes when (and only when) escalation is proven to work
+    # (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place —
+    # re-implementing the kernel-log grep in Jinja would be a second source of truth for
+    # the same fact, and the two would drift.
+    if "--privileged" in sys.argv[1:]:
+        doc = {
+            "schema": SCHEMA,
+            "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "collector_uid": os.geteuid(),
+            "klog": collect_klog(),
+            "memory": {"devices": collect_memory_devices()},
+        }
+        doc["unreadable"] = UNREADABLE
+        json.dump(doc, sys.stdout, indent=2, sort_keys=True)
+        sys.stdout.write("\n")
+        return
+
     doc = {
         "schema": SCHEMA,
         "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
         "collector_uid": os.geteuid(),
+        "klog": collect_klog(),
         "os": collect_os(),
         "dmi": collect_dmi(),
         "cpu": collect_cpu(),

infra/ansible/roles/node_health/tasks/assess.yml

@@ -387,6 +387,118 @@
                ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
                ~ avail_pct | string ~ '%)') }}
 
+# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
+# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
+# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
+# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
+# from the presence of a kernel-log line — and they duly contradicted each other on the same
+# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
+# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
+# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
+# measurement was right; the inference from the log line was wrong — the memattr block below
+# has the refined mechanism.
+#
+# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
+# place and both checks read this fact: neither derives a shortfall of its own, and the
+# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
+# disagree about whether memory was lost — there is only one verdict on that question, and
+# it is this one. Thresholds and their measured baselines: defaults/main.yml.
+- name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
+  ansible.builtin.set_fact:
+    node_health_mem_shortfall: "{{ {
+      'zones_pct': z_pct, 'zones_status': z_status,
+      'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
+      'status': worst_status } }}"
+  vars:
+    ansible_become: false
+    th: "{{ node_health_thresholds }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+
+    # 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
+    #    actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
+    #    tools, identical on metal and in a VM — so this half is available almost always.
+    z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
+    z_status: >-
+      {{ 'SKIP' if z_pct is not number
+         else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
+               else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
+
+    # 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
+    #    VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
+    #    never as a zero DIMM count. Independent of the kernel's memory map entirely.
+    inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
+    d_pct: >-
+      {{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
+         if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
+         else none }}
+    d_status: >-
+      {{ 'SKIP' if d_pct is not number
+         else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
+               else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
+
+    # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
+    # SKIPs must never add up to a PASS.
+    worst_status: >-
+      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
+         | default('SKIP', true) }}
+
+# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
+# firmware.efi_memattr names the one cause we have actually hit here; this one asks the
+# blunter question — "is this machine using the RAM it has?" — and answers it from two
+# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
+# losing memory for some other reason still shows up.
+- name: Assess — usable memory vs the memory this machine physically has
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'memory.physical_total',
+      'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
+      'status': phys_status,
+      'detail': phys_detail }] }}"
+  vars:
+    ansible_become: false
+    ms: "{{ node_health_mem_shortfall }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+    # Straight from the shared measurement — this check re-derives nothing.
+    z_pct: "{{ ms.zones_pct }}"
+    inst_kb: "{{ ms.installed_kb }}"
+    d_pct: "{{ ms.dimm_pct }}"
+    phys_status: "{{ ms.status }}"
+    # What the kernel log says, IF it was read — quoted here only as attribution for the
+    # measured shortfall. The verdict above is the shared measurement and never depends on
+    # the log; firmware.efi_memattr reads that same verdict, so the two always agree.
+    phys_cause: >-
+      {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
+           firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
+           if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
+         else ('. The kernel log was NOT read here, so this shortfall is unattributed —
+                see check firmware.efi_memattr')
+           if not (node_health_facts.klog.available | bool)
+         else '' }}
+    phys_detail: >-
+      {{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
+           ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
+           ~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
+             was never handed to the allocator (/proc/zoneinfo)')
+          if z_pct is number
+          else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
+         ~ '. '
+         ~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
+             ~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
+             ~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
+             ~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
+             ~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
+            if d_pct is number
+            else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
+         ~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
+              reservations and (on this platform) 64 MiB of graphics stolen memory.
+              A shortfall of this size is memory the machine HAS and cannot use')
+            if phys_status in ['WARN', 'FAIL'] else '')
+         ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
+
 - name: Assess — swap (kubelet requires it off)
   ansible.builtin.set_fact:
     node_health_checks: "{{ node_health_checks + [{
@@ -674,3 +786,245 @@
                ~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
                ~ (' — serial numbers not read (root-only DMI fields)'
                   if d.product_serial is none else '')) }}
+
+# --- EFI Memory Attributes Table ------------------------------------------------------
+# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
+# what follows is only what this check needs to be read correctly.
+#
+# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
+# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
+# are involved, and conflating them is what made this check wrong until 2026-09-03:
+#
+#   efi_memattr_init()              tbl_size = sizeof(*tbl) + num_entries * desc_size, then
+#                                   memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
+#                                   is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
+#                                   NEGATIVE: a garbage product with bit 31 set sign-extends
+#                                   through phys_addr_t and memblock_cap_size() clamps it to
+#                                   PHYS_ADDR_MAX — everything above the table is reserved.
+#                                   A garbage-but-POSITIVE tbl_size reserves a modest, wrong
+#                                   amount and costs nothing measurable.
+#   efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
+#                                   THIS is what prints "Failed to map". It fails for ANY
+#                                   sufficiently wrong tbl_size — negative OR merely large.
+#
+# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
+# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
+# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
+# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
+# is the counter-example that falsified the old inference.
+#
+# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
+#
+#   "Failed to map ... @ 0x..."   the bug FIRED: corrupt table, trusted by the kernel.
+#       FAIL when the shared measurement (node_health_mem_shortfall) says memory is
+#       catastrophically missing — memory is lost NOW.
+#       WARN otherwise — this boot drew a survivable garbage value; the next may not.
+#   "Unexpected ... version <N>"  version read > 2, so the kernel bailed out before
+#       reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
+#       safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
+#       gates and no draw can reach the fatal arithmetic any more. Same line, opposite
+#       meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
+#   FW_BUG "Corrupted EFI Memory Attributes Table detected!"  the >= 6.13 THIRD gate
+#       (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
+#       must not be treated as such: it is only reachable on a boot whose garbage leaves the
+#       version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
+#       line instead. All three prod nodes did exactly that on 2026-09-04.
+#
+# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
+# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
+# measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
+# firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
+# the draw still happens but every outcome is a rejection, so the same lines mean the
+# validation is working.
+#
+# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
+# so it cannot be diagnosed once. It has to be measured on every health run.
+#
+# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
+# node_health_mem_shortfall, computed once further up this file; this check does not measure
+# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
+# one verdict on any host and both checks quote it.
+#
+# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
+# ONLY if the kernel log was actually read AND still reaches back to early boot — every
+# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
+# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
+# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
+# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
+# log at all, so a node whose log is unreadable is still not unmeasured.
+- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'firmware.efi_memattr',
+      'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
+      'status': memattr_status,
+      'detail': memattr_detail }] }}"
+  vars:
+    ansible_become: false
+    k: "{{ node_health_facts.klog }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    # THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
+    # near memory.physical_total. This check contributes the kernel-log facts only.
+    ms: "{{ node_health_mem_shortfall }}"
+    ma_state: "{{ k.memattr_state | default(none) }}"
+    # DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
+    # carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
+    # THREE checks before it computes tbl_size at all:
+    #     tbl->version > 2                       -> "Unexpected ... table version %d"
+    #     desc_size implausible                  -> "Unexpected ... descriptor size %u ..."
+    #     num_entries > SZ_64K                   -> FW_BUG "Corrupted EFI Memory Attributes
+    #                                               Table detected! ..."
+    # ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
+    # given boot produces is decided by WHICH field that boot's garbage lands in — it is not
+    # a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
+    # version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
+    # memblock_reserve(). That is the whole difference, and it is why the verdict for
+    # `bad_version` below is opposite on the two kernels: same log line, completely different
+    # meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
+    # something the `version` test can parse.
+    memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
+    memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
+    # `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
+    # COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
+    # this check and memory.physical_total can no longer disagree about lost memory.
+    # `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
+    # boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
+    # WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
+    # forever would make the check unable to ever show the fix landing, which is exactly what
+    # it is for.
+    memattr_status: >-
+      {{ 'SKIP' if not (k.available | bool)
+         else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
+               else ('WARN' if ma_state == 'failed_to_map'
+                     else ('PASS' if ma_state == 'corrupt_rejected'
+                           else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
+                                 if ma_state == 'bad_version'
+                                 else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
+    # One remediation, quoted by both findings. NOTE what it must NOT say: there is no
+    # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
+    # could take effect. Suggesting one would be a fix that quietly does nothing.
+    memattr_fix: >-
+      FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
+      upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
+      bogus", first released in v6.13 and absent from v6.12 and from the current
+      linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
+      There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
+      bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
+      unplugging USB only change where the table lands and may APPEAR to fix it — they are
+      a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+    # The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
+    # to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
+    # kernel log has no part in; the verdict word comes from ms.status, so it always agrees
+    # with memory.physical_total. When the measurement is missing it says so, and says why.
+    memattr_figures: >-
+      {{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+          ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
+          ~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
+            allocator (/proc/zoneinfo)')
+         if ms.zones_pct is number
+         else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
+                reports as installed (/proc/zoneinfo NOT read — '
+                ~ (z.reason | default('no reason recorded', true)) ~ ')')
+               if ms.dimm_pct is number else '') }}
+    memattr_corroboration: >-
+      {{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
+          ~ ms.status ~ '): ' ~ memattr_figures
+          ~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
+             else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
+                   else ' — the normal few percent (kernel image, memmap, firmware
+                        reservations, graphics stolen memory). NO measurable memory was lost
+                        on this boot.')))
+         if (memattr_figures | trim | length) > 0
+         else (' Whether memory was lost on this boot could NOT be measured here: neither
+                present-vs-managed pages nor installed DIMM capacity was read — see check
+                memory.physical_total for the reasons.') }}
+    # The kernel log lines themselves, verbatim. A verdict the operator cannot check
+    # against the evidence is not much better than a wrong one.
+    memattr_evidence: >-
+      {{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
+          if (k.memattr_lines | default([]) | length) > 0 else '')
+         ~ ((' At boot the kernel reported '
+             ~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+             ~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
+             ~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
+             ~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
+            if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
+            else '') }}
+    memattr_detail: >-
+      {%- if not (k.available | bool) -%}
+      the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
+      Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
+      lives only in dmesg.{{ memattr_corroboration }}
+      {%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
+      THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
+      line. The kernel failed to map the EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }}, which proves the
+      table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
+      from its contents; the measurement below shows that reservation was the catastrophic
+      kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
+      through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
+      table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'failed_to_map' -%}
+      THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
+      loss either. The kernel failed to map the EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }}: the table was
+      corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
+      WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
+      the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
+      with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
+      tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
+      measurable and still fails the later memremap() that printed this line. This boot drew a survivable
+      value; the draw is repeated on EVERY boot, at the same address — this platform has
+      been seen to lose 13.3 GiB and to keep its full memory at one identical table address
+      (infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
+      {{ memattr_fix }}
+      {%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
+      the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
+      kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
+      VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
+      boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
+      outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
+      efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
+      WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
+      Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
+      (num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
+      version field <= 2. Both messages are rejections, and which one a boot prints is the
+      lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
+      path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
+      PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
+      {{ memattr_corroboration }}
+      {%- elif ma_state == 'bad_version' -%}
+      EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
+      CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
+      version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
+      kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
+      EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
+      tbl_size the garbage produces — nothing measurable if that value is positive,
+      everything above the table if it is negative.{{ memattr_evidence }}
+      {{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'corrupt_rejected' -%}
+      the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
+      validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
+      cannot cost this node memory on this or any future boot; no
+      action.{{ memattr_evidence }}{{ memattr_corroboration }}
+      {%- elif not (k.boot_covered | bool) -%}
+      NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
+      back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
+      the ring buffer, which wraps). Every memattr diagnostic is printed within the first
+      fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
+      pass. Read /var/log/kern.log for this boot, or re-run after a
+      reboot.{{ memattr_corroboration }}
+      {%- elif k.memattr_published is sameas true -%}
+      the firmware publishes an EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }} and the kernel
+      accepted it without complaint, in a log that still reaches back to early boot. Not
+      affected on this boot.
+      {%- elif k.memattr_published is sameas false -%}
+      this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
+      lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
+      guest.
+      {%- else -%}
+      no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
+      so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
+      boot, so that absence is measured, not assumed.
+      {%- endif -%}

infra/ansible/roles/node_health/tasks/collect_privileged.yml

@@ -0,0 +1,62 @@
+---
+# The SECOND pass of the collector, run with escalation, for the two facts that are
+# genuinely root-gated:
+#
+#   * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an
+#     unprivileged read gets EPERM. This is where the EFI Memory Attributes Table
+#     diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference
+#     between a node that has lost 13.3 GiB and one that has not.
+#   * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed
+#     DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem.
+#
+# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a
+# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line-
+# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND
+# source of truth for the same fact, and the two would drift. `--privileged` makes the
+# collector emit exactly those subtrees and nothing else, so the parsing stays in one
+# place and is identical whichever pass produced it.
+#
+# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass
+# actually failed to read one of them (main.yml checks both). On a node where
+# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule
+# is that everything readable WITHOUT root is read without root.
+#
+# If escalation is unavailable the checks that consume these SKIP with the exact reason
+# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line"
+# and "could not read the kernel log" are different answers and the role says which.
+
+- name: Re-collect the root-only facts with escalation
+  ansible.builtin.script:
+    cmd: node_health_collect.py --privileged
+    executable: /usr/bin/python3
+  become: true
+  vars:
+    ansible_become: true
+  register: node_health_privileged_raw
+  changed_when: false   # strictly read-only; it can never change the node
+
+- name: Assert the privileged payload is the schema this role understands
+  ansible.builtin.assert:
+    that:
+      - (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1'
+    fail_msg: >-
+      Privileged collector pass returned an unexpected schema; role and collector are
+      out of step.
+    quiet: true
+  vars:
+    ansible_become: false
+
+# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive
+# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine
+# would REPLACE it and throw away the reasons the first pass recorded.
+- name: Merge the privileged facts over the unprivileged ones
+  ansible.builtin.set_fact:
+    node_health_facts: >-
+      {{ node_health_facts | combine({
+           'klog': priv.klog,
+           'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}),
+           'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])),
+         }) }}
+  vars:
+    ansible_become: false
+    priv: "{{ node_health_privileged_raw.stdout | from_json }}"

infra/ansible/roles/node_health/tasks/main.yml

@@ -4,9 +4,11 @@
 # ORDER OF OPERATIONS
 #   1. Collect everything readable WITHOUT root (one script, one round trip).
 #   2. Probe whether privilege escalation actually works on this host.
-#   3. Only if it does AND smartmontools is installed: read SMART.
-#   4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
-#   5. Render the human summary + write the JSON artifact.
+#   3. Only if it does, and only for what the first pass could NOT read: re-run the same
+#      collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
+#   4. Only if it does AND smartmontools is installed: read SMART.
+#   5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#   6. Render the human summary + write the JSON artifact.
 #
 # BECOME HANDLING — the subtle bit, read before editing.
 #   `ansible_become` set as an inventory/group var (which is exactly what
@@ -43,6 +45,16 @@
   ansible.builtin.include_tasks: probe_privilege.yml
   when: node_health_try_privileged | bool
 
+# Only when the unprivileged pass actually failed to read one of them. On a node with
+# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
+# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
+- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
+  ansible.builtin.include_tasks: collect_privileged.yml
+  when:
+    - node_health_root_available | bool
+    - (not (node_health_facts.klog.available | bool))
+      or (not (node_health_facts.memory.devices.available | bool))
+
 - name: Read SMART data (privileged)
   ansible.builtin.include_tasks: collect_smart.yml
   when:

infra/ansible/roles/node_health/templates/summary.txt.j2

@@ -12,6 +12,17 @@
 {% macro gib(n) -%}
 {{ (n | int / 1073741824) | round(1) }} GiB
 {%- endmacro %}
+{# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not
+   report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of
+   its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised
+   "object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it
+   as `n/a` (the same token this template already uses for unreported readings) rather than
+   defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The
+   collector already guards this — tasks/collect_smart.yml uses selectattr('thresh',
+   'defined') and `a.thresh | default(0)` — so only the template was exposed. #}
+{% macro thr(a) -%}
+{{ a.thresh | default('n/a') }}
+{%- endmacro %}
 ================================================================================
  NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
  collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
@@ -45,6 +56,15 @@
 {% endif %}
  Memory       {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
  ECC/EDAC     {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
+{% set mz = node_health_facts.memory.zones %}
+{% set md = node_health_facts.memory.devices %}
+ Present RAM  {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }}
+ DIMMs        {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }}
+{% for dimm in md.devices | default([]) if dimm.size_kb %}
+              {{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }}
+{% endfor %}
+{% set klog = node_health_facts.klog %}
+ EFI memattr  {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
  Load         {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
 
 --------------------------------------------------------------------------------
@@ -64,19 +84,19 @@
               lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }}  temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
               pending(197)={{ sm.pending_sectors | default('?') }}  offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }}  reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }}  crc(199)={{ sm.crc_errors | default('?') }}
 {% if sm.failing_now or sm.failed_past %}
-              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %}
+              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %}
 
 {% elif sm.thresholds_present %}
-              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %}
+              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %}
 
 {% endif %}
               vendor context (NOT used for the verdict — raw units are vendor-defined):
               reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }}  grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }}  erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }}  program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }}  unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
 {% if sm.realloc_attr %}
-              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged)
+              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged)
 {% endif %}
 {% for a in sm.attributes | default([]) %}
-              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
+              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
 
 {% endfor %}
 {% else %}

platform/coredns/coredns-custom.yaml

@@ -29,9 +29,9 @@ metadata:
 data:
   # Filename `home.server` -> imported as a standalone server block for the whole zone.
   home.server: |
-    home.keithjackson.dev:53 {
+    ${BASE_DOMAIN}:53 {
         errors
-        # Answer EVERY name in the zone (apex + all `<svc>.home.keithjackson.dev`) with the Gateway
+        # Answer EVERY name in the zone (apex + all `<svc>.${BASE_DOMAIN}`) with the Gateway
         # VIP. The template plugin's default match is `.*`, so this covers current and FUTURE app
         # hostnames without editing this file per service. `{{ .Name }}` echoes the queried FQDN.
         template IN A {

platform/crafty/deployment.yaml

@@ -125,7 +125,7 @@ spec:
             - --http-address=0.0.0.0:4180
             - --reverse-proxy=true
             - --provider=oidc
-            - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+            - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
             # Gate the co-located Crafty UI. Crafty serves HTTPS with a SELF-SIGNED cert, so skip verify
             # on this in-pod (127.0.0.1) hop; the browser-facing TLS is the trusted wildcard at the Gateway.
             - --upstream=https://127.0.0.1:8443
@@ -138,9 +138,9 @@ spec:
             - --allowed-group=platform-admins
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
-            - --redirect-url=https://crafty.home.keithjackson.dev/oauth2/callback
-            - --cookie-domain=.home.keithjackson.dev
-            - --whitelist-domain=.home.keithjackson.dev
+            - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
+            - --cookie-domain=.${BASE_DOMAIN}
+            - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true
             - --set-xauthrequest=true

platform/crafty/httproute.yaml

@@ -25,7 +25,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - crafty.home.keithjackson.dev
+    - crafty.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/crafty/service-l4.yaml

@@ -42,7 +42,7 @@ metadata:
     lbipam.cilium.io/sharing-key: "crafty"
     lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
     # external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
-    external-dns.alpha.kubernetes.io/hostname: "crafty-mc.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

platform/externaldns/helmrelease.yaml

@@ -29,15 +29,46 @@ spec:
     sources:
       - gateway-httproute
       - service
-    # domain-filter MUST be the actual Cloudflare ZONE (keithjackson.dev), not the platform subdomain
-    # (home.keithjackson.dev) — ExternalDNS filters the account's zones by this value, and a subdomain
-    # filter drops the parent zone → "no hosted zone matching record DNS Name" and nothing is created.
-    # Scope stays effectively home.keithjackson.dev because that's the only suffix services request;
-    # policy=sync + registry=txt only ever touch ExternalDNS-OWNED records, never the pre-existing
-    # wildcard or other keithjackson.dev records. (docs/09; resolves the WS-NET TODO below.)
+    # ---------------------------------------------------------------------------------------------
+    # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
+    #
+    # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
+    # `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because
+    # NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore
+    # read dev's TXT registry entries as its own, found no local resource claiming those names, and
+    # garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
+    # Full write-up: infra/PROD-PROVISIONING.md §10.
+    #
+    # THE TWO GUARDS THAT ACTUALLY PREVENT A RECURRENCE — note that neither is domainFilters:
+    #   1. txtOwnerId  — per-environment identity; a cluster only ever touches records it owns.
+    #   2. DISJOINT HOSTNAME SETS — the two environments serve different suffixes, so they never
+    #      describe the same record in the first place. dev = *.dev.home.keithjackson.dev,
+    #      prod = *.home.keithjackson.dev (prod IS the apex; operator decision 2026-09-04).
+    # ---------------------------------------------------------------------------------------------
+    # Which names this instance may manage at all. BOTH ENVIRONMENTS RENDER THE CLOUDFLARE **ZONE**
+    # `keithjackson.dev` HERE, and this is NOT an oversight to be tightened.
+    #
+    # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for ZONE
+    # SELECTION as well as record matching, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+    # matches NO zone — external-dns generates the record and then skips it with "no hosted zone
+    # matching record DNS Name", publishing NOTHING. This has been "improved" and reverted more than
+    # once (an agent narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in
+    # review). It is not defence in depth; it is a total outage of DNS publication. Isolation comes
+    # from the two guards above, which a domain filter cannot provide anyway — both environments live
+    # in the ONE zone regardless of what this says. See clusters/base/cluster-vars.yaml.
     domainFilters:
-      - "${DNS_ZONE}"
-    policy: sync            # create + update + delete records as services come and go
+      - "${EXTERNALDNS_DOMAIN_FILTER}"
+    # The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
+    # caused the incident. dev renders `dev`, prod renders `prod`, and the BASE default is a
+    # non-colliding sentinel so a new environment that forgets the knob owns nothing and can delete
+    # nothing. NOTHING renders `default` any more (the external-dns chart default, and the value both
+    # clusters silently shared): dev held it only because its live apex records were registered under
+    # it, and dev's move to *.dev.home.keithjackson.dev retired that constraint along with those
+    # records. Prod must NOT be given `default` even though it inherits the apex NAMES — that would
+    # hand prod ownership of dev's old apex records and let `policy: sync` below delete them.
+    txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
+    policy: sync            # create + update + delete records as services come and go — but ONLY
+                            # records this owner id owns AND this domainFilter admits (see above).
     # Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007).
     env:
       - name: CF_API_TOKEN

platform/gateway/gateway.yaml

@@ -13,7 +13,7 @@ spec:
     - name: https
       protocol: HTTPS
       port: 443
-      hostname: "*.home.keithjackson.dev"
+      hostname: "*.${BASE_DOMAIN}"
       tls:
         mode: Terminate
         certificateRefs:
@@ -30,7 +30,7 @@ spec:
     - name: http-redirect
       protocol: HTTP
       port: 80
-      hostname: "*.home.keithjackson.dev"
+      hostname: "*.${BASE_DOMAIN}"
       allowedRoutes:
         namespaces:
           from: Selector
@@ -46,7 +46,7 @@ spec:
     - name: https-apex
       protocol: HTTPS
       port: 443
-      hostname: "home.keithjackson.dev"
+      hostname: "${BASE_DOMAIN}"
       tls:
         mode: Terminate
         certificateRefs:
@@ -61,7 +61,7 @@ spec:
     - name: http-apex-redirect
       protocol: HTTP
       port: 80
-      hostname: "home.keithjackson.dev"
+      hostname: "${BASE_DOMAIN}"
       allowedRoutes:
         namespaces:
           from: Selector

platform/gateway/wildcard-certificate.yaml

@@ -10,8 +10,8 @@ metadata:
 spec:
   secretName: wildcard-home-tls        # consumed by the Gateway HTTPS listener below
   dnsNames:
-    - "*.home.keithjackson.dev"
-    - "home.keithjackson.dev"          # apex for the C2 UI (docs/09)
+    - "*.${BASE_DOMAIN}"
+    - "${BASE_DOMAIN}"          # apex for the C2 UI (docs/09)
   issuerRef:
     kind: ClusterIssuer
     # ${ACME_CLUSTER_ISSUER}: dev -> letsencrypt-staging, prod -> letsencrypt-prod (cluster-vars).

platform/headlamp/httproute.yaml

@@ -24,7 +24,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - headlamp.home.keithjackson.dev
+    - headlamp.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/homepage/configmap.yaml

@@ -36,63 +36,63 @@ data:
     ---
     - Platform:
         - Headlamp:
-            href: https://headlamp.home.keithjackson.dev
+            href: https://headlamp.${BASE_DOMAIN}
             description: Kubernetes dashboard (per-user OIDC RBAC)
             icon: mdi-kubernetes
         - Grafana:
-            href: https://grafana.home.keithjackson.dev
+            href: https://grafana.${BASE_DOMAIN}
             description: Metrics, logs & dashboards (fronts Prometheus/Loki/Alertmanager)
             icon: mdi-chart-areaspline
         - Semaphore:
-            href: https://semaphore.home.keithjackson.dev
+            href: https://semaphore.${BASE_DOMAIN}
             description: Ansible automation & CI
             icon: mdi-cog-play
         - Gitea:
-            href: https://gitea.home.keithjackson.dev
+            href: https://gitea.${BASE_DOMAIN}
             description: Git hosting & Actions (GitOps source of truth)
             icon: mdi-git
         - Keycloak:
-            href: https://keycloak.home.keithjackson.dev
+            href: https://keycloak.${BASE_DOMAIN}
             description: Identity provider (SSO / OIDC)
             icon: mdi-account-key
 
     - Media:
         - Plex:
-            href: https://plex.home.keithjackson.dev
+            href: https://plex.${BASE_DOMAIN}
             description: Media streaming
             icon: mdi-plex
         - Seerr:
-            href: https://seerr.home.keithjackson.dev
+            href: https://seerr.${BASE_DOMAIN}
             description: Media requests
             icon: mdi-play-box-multiple
         - Sonarr:
-            href: https://sonarr.home.keithjackson.dev
+            href: https://sonarr.${BASE_DOMAIN}
             description: TV library manager
             icon: mdi-television-classic
         - Radarr:
-            href: https://radarr.home.keithjackson.dev
+            href: https://radarr.${BASE_DOMAIN}
             description: Movie library manager
             icon: mdi-movie-open
         - Jackett:
-            href: https://jackett.home.keithjackson.dev
+            href: https://jackett.${BASE_DOMAIN}
             description: Indexer proxy
             icon: mdi-magnify
         - qBittorrent:
-            href: https://qbittorrent.home.keithjackson.dev
+            href: https://qbittorrent.${BASE_DOMAIN}
             description: Download client (VPN egress)
             icon: mdi-download-network
 
     - Network & Infra:
         - Pi-hole:
-            href: https://pihole.home.keithjackson.dev
+            href: https://pihole.${BASE_DOMAIN}
             description: Network-wide DNS & ad-blocking
             icon: mdi-shield-check
         - UniFi:
-            href: https://unifi.home.keithjackson.dev:11443
+            href: https://unifi.${BASE_DOMAIN}:11443
             description: Network controller (UOS)
             icon: mdi-router-network
         - Nextcloud:
-            href: https://nextcloud.home.keithjackson.dev
+            href: https://nextcloud.${BASE_DOMAIN}
             description: Files, calendar & contacts
             icon: mdi-cloud
 
@@ -104,7 +104,7 @@ data:
               href: https://gitea.keithjackson.dev
         - Auth portal:
             - abbr: SSO
-              href: https://auth.home.keithjackson.dev
+              href: https://auth.${BASE_DOMAIN}
     - Reference:
         - Homepage docs:
             - abbr: HP

platform/homepage/deployment.yaml

@@ -54,7 +54,7 @@ spec:
             # REQUIRED by Homepage >= 1.0 — requests whose Host header is not in this list get 403.
             # The public apex host (served via the Gateway) + the pod IP (for kubelet probes).
             - name: HOMEPAGE_ALLOWED_HOSTS
-              value: "home.keithjackson.dev,$(MY_POD_IP):3000"
+              value: "${BASE_DOMAIN},$(MY_POD_IP):3000"
           resources:
             requests:
               cpu: 25m

platform/homepage/httproute.yaml

@@ -28,7 +28,7 @@ spec:
       namespace: gateway
       sectionName: https-apex     # the apex listener (wildcard `https` won't match the bare apex)
   hostnames:
-    - home.keithjackson.dev
+    - ${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/keycloak/httproute.yaml

@@ -16,7 +16,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - keycloak.home.keithjackson.dev
+    - keycloak.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/keycloak/keycloak-instance.yaml

@@ -30,7 +30,7 @@ spec:
     poolMaxSize: 20
   # --- Exposure: TLS at the Gateway; Keycloak serves HTTP and trusts the proxy headers. ---
   hostname:
-    hostname: https://keycloak.home.keithjackson.dev
+    hostname: https://keycloak.${BASE_DOMAIN}
   http:
     httpEnabled: true                      # TLS is terminated at the Gateway (edge), not in Keycloak
   proxy:

platform/keycloak/keycloak-realm-import.yaml

@@ -98,9 +98,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:GRAFANA_CLIENT_SECRET)"
         redirectUris:
-          - https://grafana.home.keithjackson.dev/login/generic_oauth
+          - https://grafana.${BASE_DOMAIN}/login/generic_oauth
         webOrigins:
-          - https://grafana.home.keithjackson.dev
+          - https://grafana.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -125,10 +125,10 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:OAUTH2_PROXY_CLIENT_SECRET)"
         redirectUris:
-          - https://*.home.keithjackson.dev/oauth2/callback
-          - https://auth.home.keithjackson.dev/oauth2/callback
+          - https://*.${BASE_DOMAIN}/oauth2/callback
+          - https://auth.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://*.home.keithjackson.dev
+          - https://*.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -209,9 +209,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:GITEA_CLIENT_SECRET)"
         redirectUris:
-          - https://gitea.home.keithjackson.dev/user/oauth2/keycloak/callback
+          - https://gitea.${BASE_DOMAIN}/user/oauth2/keycloak/callback
         webOrigins:
-          - https://gitea.home.keithjackson.dev
+          - https://gitea.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -240,9 +240,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:NEXTCLOUD_CLIENT_SECRET)"
         redirectUris:
-          - https://nextcloud.home.keithjackson.dev/apps/user_oidc/code
+          - https://nextcloud.${BASE_DOMAIN}/apps/user_oidc/code
         webOrigins:
-          - https://nextcloud.home.keithjackson.dev
+          - https://nextcloud.${BASE_DOMAIN}
         # Post-logout redirect allow-list (#logout "redirect failed"). On sign-out user_oidc redirects the
         # browser to Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud base '/'> and
         # client_id=nextcloud but NO id_token_hint (provider sendIdTokenHint=0), so Keycloak validates the
@@ -250,7 +250,7 @@ spec:
         # "redirect failed". The wildcard covers the base URL and any logout landing path. Keycloak stores
         # this as the client attribute `post.logout.redirect.uris` (## separates multiple values).
         attributes:
-          post.logout.redirect.uris: https://nextcloud.home.keithjackson.dev/*
+          post.logout.redirect.uris: https://nextcloud.${BASE_DOMAIN}/*
         defaultClientScopes:
           - email
           - profile
@@ -279,9 +279,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:SEMAPHORE_CLIENT_SECRET)"
         redirectUris:
-          - https://semaphore.home.keithjackson.dev/api/auth/oidc/keycloak/redirect
+          - https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect
         webOrigins:
-          - https://semaphore.home.keithjackson.dev
+          - https://semaphore.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -318,9 +318,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:HEADLAMP_CLIENT_SECRET)"
         redirectUris:
-          - https://headlamp.home.keithjackson.dev/oidc-callback
+          - https://headlamp.${BASE_DOMAIN}/oidc-callback
         webOrigins:
-          - https://headlamp.home.keithjackson.dev
+          - https://headlamp.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -363,9 +363,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:CRAFTY_CLIENT_SECRET)"
         redirectUris:
-          - https://crafty.home.keithjackson.dev/oauth2/callback
+          - https://crafty.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://crafty.home.keithjackson.dev
+          - https://crafty.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -398,9 +398,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:PIHOLE_CLIENT_SECRET)"
         redirectUris:
-          - https://pihole.home.keithjackson.dev/oauth2/callback
+          - https://pihole.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://pihole.home.keithjackson.dev
+          - https://pihole.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -433,9 +433,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:JACKETT_CLIENT_SECRET)"
         redirectUris:
-          - https://jackett.home.keithjackson.dev/oauth2/callback
+          - https://jackett.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://jackett.home.keithjackson.dev
+          - https://jackett.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -459,9 +459,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:QBITTORRENT_CLIENT_SECRET)"
         redirectUris:
-          - https://qbittorrent.home.keithjackson.dev/oauth2/callback
+          - https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://qbittorrent.home.keithjackson.dev
+          - https://qbittorrent.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -485,9 +485,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:RADARR_CLIENT_SECRET)"
         redirectUris:
-          - https://radarr.home.keithjackson.dev/oauth2/callback
+          - https://radarr.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://radarr.home.keithjackson.dev
+          - https://radarr.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -511,9 +511,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:SONARR_CLIENT_SECRET)"
         redirectUris:
-          - https://sonarr.home.keithjackson.dev/oauth2/callback
+          - https://sonarr.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://sonarr.home.keithjackson.dev
+          - https://sonarr.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile

platform/kubevirt/alerts.yaml

@@ -30,7 +30,7 @@ spec:
           annotations:
             summary: "KubeVirt VM(s) failing to launch in {{ $labels.namespace }}"
             description: "One or more VirtualMachineInstances are in the Failed phase in namespace {{ $labels.namespace }} for 5m — a failed VM launch (image import, scheduling, or qemu error). Check `kubectl get vmi -n {{ $labels.namespace }}` and the virt-launcher pod events."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-launch-failed"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-launch-failed"
         # Crash-loop: the VM's virt-launcher pod is restarting repeatedly (guest kernel panic, OOM, or a
         # crashing qemu). >3 restarts in 15m is a crash-loop, not a one-off reboot.
         - alert: KubeVirtVMCrashLooping
@@ -42,7 +42,7 @@ spec:
           annotations:
             summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
             description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-crashloop"
         # Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle
         # operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod
         # 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole
@@ -59,7 +59,7 @@ spec:
           annotations:
             summary: "KubeVirt control plane degraded"
             description: "The KubeVirt control plane (virt-controller/virt-api) is not ready or its metrics are absent for 10m — VM lifecycle operations (create/start/stop) will fail (docs/06)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-control-plane"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-control-plane"
         # CDI import stuck: a DataVolume import/clone still in flight after 30m blocks the VM it backs.
         # kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a positive value
         # sustained means an import is not progressing (metric name confirmed against CDI at bring-up).
@@ -71,4 +71,4 @@ spec:
           annotations:
             summary: "CDI DataVolume import stuck: {{ $labels.namespace }}/{{ $labels.name }}"
             description: "A CDI DataVolume import/clone in {{ $labels.namespace }} has not completed after 30m — the VM disk it backs cannot start. Check the importer pod (often an egress/default-deny block on the image source, or a slow mirror)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cdi-import-stuck"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cdi-import-stuck"

platform/longhorn/README.md

@@ -25,7 +25,7 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
 | Requirement | Summary | How this component serves it |
 |---|---|---|
 | **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
-| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
+| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the node's data disk at `${LONGHORN_DATA_PATH}` (a `cluster-vars` knob: dev `/var/lib/longhorn`, a dedicated disk; prod `/data/longhorn`, a subdirectory of the partition it shares with k3s's data dir). `replicaAutoBalance: best-effort` re-spreads on node return. |
 | **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
 
 ## Dev-scoped choices (change for prod)
@@ -48,9 +48,19 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
 
 ## Node prerequisites (already handled, informational)
 
-Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
-`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
-`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
+Each node presents ext4 storage at `${LONGHORN_DATA_PATH}` and installs `open-iscsi` +
+`nfs-common` + kernel modules — all via `roles/node_baseline` (`infra/cloud-init/` is a README
+now; provisioning became API-only). This component assumes that floor. The shape differs per
+environment, which is why the path is a knob:
+
+| | |
+|---|---|
+| dev | a **dedicated** disk (`/dev/vdb`, `LABEL=longhorn`) mounted at `/var/lib/longhorn` — the mount root *is* the dataPath |
+| prod | `/data/longhorn`, a **subdirectory** of the ~424 GB partition (`LABEL=k3s-data`) mounted at `/data`, whose sibling `/data/k3s` holds k3s's entire data dir (etcd + the containerd image store, moved off the 11.8 GiB `/var` — `infra/PROD-PROVISIONING.md` §2) |
+
+`[parity]`: the value for an environment MUST equal `node_baseline_longhorn_data_path` in that
+environment's Ansible group_vars. Ansible creates and mounts the directory; this component tells
+Longhorn to use it.
 
 ## Contract (consumed by everyone with state)
 

platform/longhorn/helmrelease.yaml

@@ -42,10 +42,22 @@ spec:
       defaultClass: false
 
     defaultSettings:
-      # --- data path (R-STATE-02) ---
-      # Dedicated Longhorn data disk (/dev/vdb → ext4) is mounted here by cloud-init on every node
-      # (infra/cloud-init/user-data.yaml.tftpl). Longhorn stores replicas on this disk only.
-      defaultDataPath: "/var/lib/longhorn"
+      # --- data path (R-STATE-02, R-ENV-02) ---
+      # The on-node directory every replica is written into. A cluster-vars KNOB, not a literal,
+      # because the two environments mount their data disk differently and a hardcoded path would
+      # put one of them on the OS filesystem:
+      #   dev  /var/lib/longhorn  — a dedicated ext4 disk (/dev/vdb) mounted there by
+      #                             roles/node_baseline. The mount root IS the dataPath: one disk,
+      #                             one tenant. 18 live volumes; this value must never change.
+      #   prod /data/longhorn     — a SUBDIRECTORY of the claimed ~424 GB partition mounted at
+      #                             /data, whose sibling /data/k3s holds k3s's entire data dir
+      #                             (etcd + the containerd image store, moved off the 11.8 GiB
+      #                             /var). Two tenants, so Longhorn gets a subdirectory rather
+      #                             than the mount root. infra/PROD-PROVISIONING.md §2.
+      # [parity] MUST equal roles/node_baseline's node_baseline_longhorn_data_path for the same
+      # environment — Ansible creates and mounts the directory, Helm tells Longhorn to use it.
+      # Set in BOTH overlays (clusters/dev, clusters/prod); base default is the dev value.
+      defaultDataPath: "${LONGHORN_DATA_PATH}"
 
       # --- replication (R-STATE-02) ---
       # Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly

platform/longhorn/values.yaml

@@ -8,7 +8,9 @@
 #   defaultClass: false                              # we own StorageClasses (storageclasses.yaml)
 #
 # defaultSettings:
-#   defaultDataPath: /var/lib/longhorn               # dedicated data disk (infra/cloud-init)
+#   defaultDataPath: ${LONGHORN_DATA_PATH}           # cluster-vars knob: dev /var/lib/longhorn
+#                                                    # (dedicated /dev/vdb), prod /data/longhorn
+#                                                    # (a subdirectory of the shared data partition)
 #   defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT}   # overlay knob (=3 in dev)
 #   replicaSoftAntiAffinity: false                   # HARD anti-affinity: one replica per node
 #   replicaZoneSoftAntiAffinity: true                # single zone (3 mini-PCs, no failure zones)

platform/monitoring/alerts.yaml

@@ -38,7 +38,7 @@ spec:
           annotations:
             summary: "Node exporter down on {{ $labels.instance }}"
             description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
         - alert: NodeMemoryPressureHigh
           expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
           for: 10m
@@ -47,7 +47,7 @@ spec:
           annotations:
             summary: "Memory pressure high on {{ $labels.instance }}"
             description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-memory"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
         - alert: NodeDiskFillingUp
           expr: |
             (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
@@ -58,7 +58,7 @@ spec:
           annotations:
             summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
             description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-disk"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
         - alert: NodeCPUPressureHigh
           # PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
           # run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
@@ -71,7 +71,7 @@ spec:
           annotations:
             summary: "CPU pressure high on {{ $labels.instance }}"
             description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
         - alert: NodeCPULoadHigh
           # 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
           # runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
@@ -86,7 +86,7 @@ spec:
           annotations:
             summary: "Load per core high on {{ $labels.instance }}"
             description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
 
 # =====================================================================================
 # Storage — Longhorn
@@ -113,7 +113,7 @@ spec:
           annotations:
             summary: "Longhorn volume degraded: {{ $labels.volume }}"
             description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-degraded"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
         - alert: LonghornVolumeFaulted
           expr: longhorn_volume_robustness == 3
           for: 1m
@@ -122,7 +122,7 @@ spec:
           annotations:
             summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
             description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-faulted"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
         - alert: LonghornVolumeCapacityHigh
           expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
           for: 30m
@@ -131,7 +131,7 @@ spec:
           annotations:
             summary: "Longhorn volume near capacity: {{ $labels.volume }}"
             description: "Volume {{ $labels.volume }} is over 85% used for 30m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
         - alert: LonghornNodeStorageHigh
           expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
           for: 30m
@@ -140,7 +140,7 @@ spec:
           annotations:
             summary: "Longhorn node storage near capacity: {{ $labels.node }}"
             description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
 
 # =====================================================================================
 # GitOps — Flux
@@ -169,7 +169,7 @@ spec:
           annotations:
             summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/flux-reconcile"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
         - alert: FluxResourceSuspended
           expr: |
             max by (namespace, name, kind) (gotk_suspend_status) == 1
@@ -179,7 +179,7 @@ spec:
           annotations:
             summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
-            runbook_url: "https://home.keithjackson.dev/runbooks/flux-suspended"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
 
 # =====================================================================================
 # Certificates — cert-manager
@@ -205,7 +205,7 @@ spec:
           annotations:
             summary: "Certificate expiring soon: {{ $labels.name }}"
             description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-expiry"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
         - alert: CertificateExpiryImminent
           expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
           for: 1h
@@ -214,7 +214,7 @@ spec:
           annotations:
             summary: "Certificate expiry imminent: {{ $labels.name }}"
             description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-expiry"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
         - alert: CertificateNotReady
           expr: certmanager_certificate_ready_status{condition="False"} == 1
           for: 1h
@@ -223,7 +223,7 @@ spec:
           annotations:
             summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
             description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-issuance"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"
 
 # =====================================================================================
 # Networking — VPN / egress-gateway / denied flows (Hubble)
@@ -255,7 +255,7 @@ spec:
           annotations:
             summary: "Egress VPN tunnel down"
             description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/vpn-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
         # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
         # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
         # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
@@ -270,7 +270,7 @@ spec:
           annotations:
             summary: "Spike in policy-denied network flows"
             description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/denied-flows"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"
 
 # =====================================================================================
 # Backups — Velero
@@ -296,7 +296,7 @@ spec:
           annotations:
             summary: "Velero backup failed: schedule {{ $labels.schedule }}"
             description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-failure"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
         - alert: VeleroBackupPartialFailure
           expr: increase(velero_backup_partial_failure_total[1h]) > 0
           for: 5m
@@ -305,7 +305,7 @@ spec:
           annotations:
             summary: "Velero backup partially failed"
             description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-partial"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
         - alert: VeleroNoRecentBackup
           expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
           for: 1h
@@ -314,7 +314,7 @@ spec:
           annotations:
             summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
             description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-missed"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
 
 # =====================================================================================
 # Identity — Keycloak (SSO is a critical dependency — docs/15)
@@ -345,7 +345,7 @@ spec:
           annotations:
             summary: "Keycloak IdP down"
             description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
         - alert: KeycloakDatabaseDown
           # Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
           # DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
@@ -358,7 +358,7 @@ spec:
           annotations:
             summary: "Keycloak database unavailable"
             description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-db"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
 
 # =====================================================================================
 # The monitor's own capacity — don't let the monitor blind itself (docs/10)
@@ -386,7 +386,7 @@ spec:
           annotations:
             summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
             description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
-            runbook_url: "https://home.keithjackson.dev/runbooks/monitoring-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
         - alert: PrometheusTargetsMissing
           expr: |
             (count(up == 0) / count(up)) > 0.10
@@ -396,7 +396,7 @@ spec:
           annotations:
             summary: "More than 10% of Prometheus targets are down"
             description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
-            runbook_url: "https://home.keithjackson.dev/runbooks/targets-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"
 
 # =====================================================================================
 # Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
@@ -430,7 +430,7 @@ spec:
           annotations:
             summary: "Flux image-update-automation failing: {{ $labels.name }}"
             description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
-            runbook_url: "https://home.keithjackson.dev/runbooks/image-automation"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
         # An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
         # auth / bad range) → that service stops receiving updates even though automation is healthy.
         - alert: ImagePolicyStalled
@@ -444,7 +444,7 @@ spec:
           annotations:
             summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/image-policy-stalled"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
         # A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
         # holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
         # (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
@@ -461,4 +461,4 @@ spec:
           annotations:
             summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
             description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/update-rollback"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"

platform/monitoring/grafana-httproute.yaml

@@ -14,7 +14,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - grafana.home.keithjackson.dev
+    - grafana.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/monitoring/kube-prometheus-stack.yaml

@@ -194,15 +194,15 @@ spec:
             optional: true
       grafana.ini:
         server:
-          root_url: https://grafana.home.keithjackson.dev
+          root_url: https://grafana.${BASE_DOMAIN}
         auth.generic_oauth:
           enabled: true
           name: Keycloak
           client_id: grafana
           scopes: "openid email profile"
-          auth_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/auth
-          token_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/token
-          api_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/userinfo
+          auth_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/auth
+          token_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/token
+          api_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/userinfo
           login_attribute_path: preferred_username
           email_attribute_path: email
           # Map Keycloak groups (the `groups` claim) to Grafana roles (R-IDP-06): platform-admins ->

platform/oauth2-proxy/helmrelease.yaml

@@ -32,7 +32,7 @@ spec:
       existingSecret: oauth2-proxy-credentials
     extraArgs:
       provider: oidc
-      oidc-issuer-url: https://keycloak.home.keithjackson.dev/realms/homelab
+      oidc-issuer-url: https://keycloak.${BASE_DOMAIN}/realms/homelab
       # Forward-auth (ext-auth) mode: no upstream app; return 202 for an authenticated session so the
       # Gateway ext-authz can allow/deny. Auth headers are surfaced to the backend.
       reverse-proxy: "true"
@@ -41,9 +41,9 @@ spec:
       pass-access-token: "true"
       email-domain: "*"
       # Single sign-in/callback host for all forward-auth-protected services on the platform zone.
-      redirect-url: https://auth.home.keithjackson.dev/oauth2/callback
-      whitelist-domain: ".home.keithjackson.dev"
-      cookie-domain: ".home.keithjackson.dev"
+      redirect-url: https://auth.${BASE_DOMAIN}/oauth2/callback
+      whitelist-domain: ".${BASE_DOMAIN}"
+      cookie-domain: ".${BASE_DOMAIN}"
       cookie-secure: "true"
       # Expose group membership so per-service group authorization (R-IDP-06) can be enforced. The
       # Keycloak `oauth2-proxy` client emits a `groups` claim (full.path=false => bare names like

platform/oauth2-proxy/httproute.yaml

@@ -14,7 +14,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - auth.home.keithjackson.dev
+    - auth.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

schemas/servicedefinition.schema.json

@@ -9,7 +9,7 @@
   "properties": {
     "name": {
       "type": "string",
-      "description": "DNS label -> <name>.home.keithjackson.dev (docs/13, docs/09). Must be a valid RFC 1123 DNS label.",
+      "description": "DNS label -> <name>.${BASE_DOMAIN}, i.e. <name>.home.keithjackson.dev on dev and <name>.prod.home.keithjackson.dev on prod (docs/13, docs/09). Must be a valid RFC 1123 DNS label.",
       "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$",
       "minLength": 1,
       "maxLength": 63
@@ -223,8 +223,8 @@
         "properties": {
           "host": {
             "type": "string",
-            "description": "FQDN under the platform zone. Must be a <label>.home.keithjackson.dev name (docs/09).",
-            "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.home\\.keithjackson\\.dev$"
+            "description": "Hostname under the platform zone, as a single DNS label plus the ${BASE_DOMAIN} token — e.g. 'gitea.${BASE_DOMAIN}' (docs/09). ${BASE_DOMAIN} is the per-environment hostname suffix from clusters/base/cluster-vars.yaml, resolved by Flux postBuild substitution at reconcile time (dev: home.keithjackson.dev, prod: prod.home.keithjackson.dev). AUTHOR THE TOKEN, NOT A LITERAL: a literal suffix is what let prod be authored to serve dev's public hostnames out of the shared Cloudflare zone, and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10). The literal 'home.keithjackson.dev' form stays accepted so the L1 render fixtures — which are renderer INPUTS, not reconciled manifests — keep validating unchanged.",
+            "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.(\\$\\{BASE_DOMAIN\\}|home\\.keithjackson\\.dev)$"
           },
           "port": {
             "type": "integer",