feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex

*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***

Operator simplified the architecture, inverting c8c1d29:
  dev  -> *.dev.home.keithjackson.dev        (was the apex)
  prod -> *.home.keithjackson.dev            (the apex; no prod. subdomain, no CNAME layer)

WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.

Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.

VALUES
                BASE_DOMAIN                  TXT_OWNER_ID   DOMAIN_FILTER
  base          dev.home.keithjackson.dev    sentinel       keithjackson.dev
  dev           dev.home.keithjackson.dev    dev            keithjackson.dev
  prod          home.keithjackson.dev        prod           keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.

dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.

DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.

VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.

THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.

Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.

Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.

RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.

ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.

The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.

Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/vms/unifi/30-virtualmachine.yaml

@@ -102,7 +102,7 @@ spec:
             userData: |-
               #cloud-config
               hostname: unifi
-              fqdn: unifi.home.keithjackson.dev
+              fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
               packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]

apps/vms/unifi/50-service-l4.yaml

@@ -46,7 +46,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
     # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
     # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
-    external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

apps/wave-0/gitea.yaml

@@ -90,9 +90,9 @@ spec:
               name: gitea-bootstrap
               key: secret-key
         - name: GITEA__server__DOMAIN
-          value: gitea.home.keithjackson.dev
+          value: gitea.${BASE_DOMAIN}
         - name: GITEA__server__ROOT_URL
-          value: https://gitea.home.keithjackson.dev/
+          value: https://gitea.${BASE_DOMAIN}/
         - name: GITEA__actions__ENABLED
           value: 'true'
         - name: GITEA__actions__DEFAULT_ACTIONS_URL
@@ -127,7 +127,7 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
                 \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
@@ -207,7 +207,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - gitea.home.keithjackson.dev
+  - gitea.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/jackett.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:9117
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - jackett.home.keithjackson.dev
+  - jackett.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/pihole.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:80
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -229,7 +229,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - pihole.home.keithjackson.dev
+  - pihole.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/plex.yaml

@@ -183,7 +183,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - plex.home.keithjackson.dev
+  - plex.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/qbittorrent.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - qbittorrent.home.keithjackson.dev
+  - qbittorrent.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/gitea-actions.yaml

@@ -165,7 +165,7 @@ spec:
               exec /sbin/tini -- /opt/act/run.sh
           env:
             - name: GITEA_INSTANCE_URL
-              value: https://gitea.home.keithjackson.dev
+              value: https://gitea.${BASE_DOMAIN}
             - name: GITEA_RUNNER_REGISTRATION_TOKEN
               valueFrom:
                 secretKeyRef:

apps/wave-1/nextcloud.yaml

@@ -104,13 +104,13 @@ spec:
               name: nextcloud-bootstrap
               key: admin-password
         - name: NEXTCLOUD_TRUSTED_DOMAINS
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEHOST
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEPROTOCOL
           value: https
         - name: OVERWRITECLIURL
-          value: https://nextcloud.home.keithjackson.dev
+          value: https://nextcloud.${BASE_DOMAIN}
         - name: NEXTCLOUD_OIDC_CLIENT_SECRET
           valueFrom:
             secretKeyRef:
@@ -125,9 +125,9 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc]\
-                [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
+                \ client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
-                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
+                ); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -228,7 +228,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - nextcloud.home.keithjackson.dev
+  - nextcloud.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/radarr.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:7878
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - radarr.home.keithjackson.dev
+  - radarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/semaphore.yaml

@@ -110,7 +110,7 @@ spec:
         - name: SEMAPHORE_ADMIN_NAME
           value: Semaphore Admin
         - name: SEMAPHORE_ADMIN_EMAIL
-          value: admin@home.keithjackson.dev
+          value: admin@${BASE_DOMAIN}
         - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
           valueFrom:
             secretKeyRef:
@@ -231,7 +231,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - semaphore.home.keithjackson.dev
+  - semaphore.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/sonarr.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - sonarr.home.keithjackson.dev
+  - sonarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-2/seerr.yaml

@@ -223,7 +223,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - seerr.home.keithjackson.dev
+  - seerr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

bootstrap/flux/bootstrap.sh

@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #
-# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
+# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
+# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
+# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
+# same script bootstraps dev and prod with no fork (R-ENV-02).
 #
 # Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
 # =============================================================================
@@ -33,7 +36,7 @@
 #   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
+#   3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
 set -euo pipefail
 
 # --- config (env-overridable) ------------------------------------------------
-FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"     # dev is the ONLY Phase-1 build target
+# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
+# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
+# every existing caller — override it, never edit it.
+FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
 FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
 SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
 # The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,7 +68,7 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
 
-echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
+echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
 
 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
 for bin in flux kubectl; do
@@ -134,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
   exit 1
 fi
 
-# --- 3. apply the authored dev entrypoint ------------------------------------
+# --- 3. apply the authored environment entrypoint ----------------------------
 echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
 

bootstrap/gitea/bootstrap-gitea.sh

@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 # bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
-# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
+# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
-# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
+# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
-# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
+# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
-# CONTRIBUTING.md).
+# using it now.
 #
-# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
+# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
+# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
+#
+#   PROD (bare metal, current use):  HOST_NAT_IP=192.168.2.132
+#       The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
+#       NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
+#       three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
+#       apply to prod, so do not "fix" prod by opening zones.
+#
+#   DEV (libvirt VMs, HISTORICAL):   HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
+#       *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
+#       bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
+#       recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
+#
+# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
+# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
+# author-only work and must never be fired off casually or "to see if it works": it starts a
+# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
+# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
+# it will then reconcile. Know which HEAD you are publishing before you run it.
 set -euo pipefail
 ENGINE="${ENGINE:-podman}"
 GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}"   # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
 # subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
 DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
 DATA_DIR="podman volume ${DATA_VOLUME}"   # display label only (data lives in the named volume)
-# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
+# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
 # clone/redirect URLs are correct from inside the cluster.
-# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
+#
+# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
+# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
+# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
+# Pass it explicitly, e.g. for prod:   HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
+#
+# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
+# VM->host:3000 by default and had to be opened once with
 #   sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
-HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
+# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
+if [ -z "${HOST_NAT_IP:-}" ]; then
+  echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
+  echo "       prod: HOST_NAT_IP=192.168.2.132   (same LAN as the nodes, no NAT, no firewall change)" >&2
+  echo "       There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
+  exit 1
+fi
 ORG="${GITEA_ORG:-Home-Network}"
 REPO="${GITEA_REPO:-home-network-k3s-impl}"
 
@@ -75,8 +107,17 @@ cat <<EOF
 
 Gitea bootstrap source ready (${DATA_DIR}).
   UI (host):                     ${BASE}/${ORG}/${REPO}
-  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git   (clusters/dev)
+  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
+                                 (patch this url into the overlay being bootstrapped:
+                                  clusters/prod today; clusters/dev has already cut over)
   Admin creds (host, 0600):      ${ADMIN_CRED_FILE}
-Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
+Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
+firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
+must allow VM->host:${HTTP_PORT} —
   ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
+REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
+at this source will reconcile THAT revision.
+TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
+any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
+procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
 EOF

catalog/gitea.yaml

@@ -24,9 +24,9 @@ env:
         name: gitea-bootstrap
         key: secret-key
   - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.home.keithjackson.dev
+    value: gitea.${BASE_DOMAIN}
   - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
   - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
     value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
   - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,7 +104,7 @@ lifecycle:
           # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
           # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
           # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           gitea_oidc_fail() {
             echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
@@ -168,7 +168,7 @@ lifecycle:
           # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
           # the moment its OIDC guard was fixed and the block actually ran for the first time.
           exit 0
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: data
@@ -178,7 +178,7 @@ storage:
                                 # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
     port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)

catalog/jackett.yaml

@@ -17,7 +17,7 @@ name: jackett
 kind: container
 source:
   image: lscr.io/linuxserver/jackett:latest
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
@@ -25,7 +25,7 @@ storage:
     class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
     port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)

catalog/keycloak.yaml

@@ -18,7 +18,7 @@ dependencies:
     port: 5432
     protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
     port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
   mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.

catalog/nextcloud.yaml

@@ -50,13 +50,13 @@ env:
   #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
   #     though it speaks plain HTTP behind the proxy. ---
   - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEPROTOCOL
     value: https
   - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
   # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
   #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
   #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
           # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
           # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
           OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           nextcloud_oidc_fail() {
             echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -228,7 +228,7 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
     port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/pihole.yaml

@@ -14,7 +14,7 @@ storage:
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)

catalog/plex.yaml

@@ -72,7 +72,7 @@ storage:
                               # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
     port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
   mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/qbittorrent.yaml

@@ -32,7 +32,7 @@ name: qbittorrent
 kind: container
 source:
   image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
@@ -40,7 +40,7 @@ storage:
     class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
     port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)

catalog/radarr.yaml

@@ -9,7 +9,7 @@ name: radarr
 kind: container
 source:
   image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)

catalog/render.py

@@ -44,7 +44,16 @@ except ImportError:  # pragma: no cover - explicit, actionable failure
 
 # --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
 LBL = "homelab.keithjackson.dev"
-ZONE = "home.keithjackson.dev"
+# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
+# Every rendered manifest lands under a Flux path whose Kustomization declares
+# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
+# reconcile time: dev renders dev.home.keithjackson.dev, prod renders home.keithjackson.dev (the
+# APEX — operator decision 2026-09-04; prod IS the apex, so there is no `prod.` label and no CNAMEs).
+# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
+# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
+# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
+# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
+ZONE = "${BASE_DOMAIN}"
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
@@ -598,7 +607,7 @@ def r_internet_egress(s: dict) -> dict:
 def r_httproutes(s: dict) -> list:
     """One HTTPRoute per ingressRoute, attaching host -> service port on the shared Gateway.
 
-    TLS is the Gateway's wildcard *.home.keithjackson.dev cert (no per-service cert step);
+    TLS is the Gateway's wildcard *.${BASE_DOMAIN} cert (no per-service cert step);
     ExternalDNS manages the record from the HTTPRoute (R-TLS/R-DNS, renderer-contract).
     """
     out = []

catalog/seerr.yaml

@@ -57,7 +57,7 @@ dependencies:
     port: 32400
     protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
     port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/semaphore.yaml

@@ -58,7 +58,7 @@ env:
   - name: SEMAPHORE_ADMIN_NAME
     value: Semaphore Admin
   - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
   # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
   #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
   - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -110,7 +110,7 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
     port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/sonarr.yaml

@@ -9,7 +9,7 @@ name: sonarr
 kind: container
 source:
   image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)

clusters/base/cluster-vars.yaml

@@ -17,15 +17,103 @@ data:
   CLUSTER_ENV: "base"                       # dev | prod  (overlays set this)
 
   # --- DNS / TLS ---
-  BASE_DOMAIN: "home.keithjackson.dev"      # platform hostname suffix (same both envs; split-horizon)
+  # THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
-  DNS_ZONE: "keithjackson.dev"              # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
+  # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
-                                            # ExternalDNS domain-filter must be the zone, not the subdomain
+  # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
+  #
+  # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
+  # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
+  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
+  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
+  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
+  # suffix, so the two clusters can no longer describe the same name.
+  #
+  #   dev   -> dev.home.keithjackson.dev        (a subdomain; = this base default)
+  #   prod  -> home.keithjackson.dev            (THE APEX — prod IS the apex, no `prod.` label)
+  #
+  # OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
+  # had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
+  # whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
+  # DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
+  # default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
+  # string dev holds.
+  #
+  # DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
+  # now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
+  # URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
+  # push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
+  # Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
+  # cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
+  # infra/PROD-PROVISIONING.md §10.
+  #
+  # DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
+  # `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
+  BASE_DOMAIN: "dev.home.keithjackson.dev"
+
+  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
+  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
+  # so it cannot be created, updated or deleted regardless of TXT ownership.
+  #
+  # BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
+  # environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
+  #
+  # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
+  # SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+  # matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
+  # record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
+  # narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
+  # it — the change is not defence in depth, it is a total outage of DNS publication.
+  #
+  # WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
+  #   (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
+  #   (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
+  # That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
+  # both environments live in the one zone either way.
+  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
+
+  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
+  # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
+  # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
+  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
+  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
+  #
+  #   dev -> `dev`      prod -> `prod`      base default -> a non-colliding sentinel (below)
+  #
+  # DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
+  # this migration dev HAD to render `default`, because its LIVE apex records were already registered
+  # in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
+  # THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
+  # *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
+  # is now free to be chosen on its merits. Two reasons to take `dev`:
+  #   - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
+  #     clusters silently shared it, which IS the incident). An environment whose identity is the
+  #     forget-value can never be told apart from a misconfigured newcomer. No environment should
+  #     hold it.
+  #   - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
+  #     owned by `default`, which no environment now claims, so nothing deletes them implicitly on
+  #     some future reconcile — they are retired deliberately, and prod's publish is gated on that
+  #     retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
+  #     race between two reconcile loops.
+  #
+  # PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
+  # dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
+  # the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
+  # would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
+  # roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
+  # environment's id. A new environment that forgets this knob must not silently inherit another's
+  # identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
+  # can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
+  # instead of destroying a live environment's DNS.
+  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
+
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
   GATEWAY_VIP: "192.168.1.53"                # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
                                             # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
                                             # takes the pool's first free IP, and coredns-custom answers
-                                            # *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
+                                            # *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
                                             # hairpins through the Gateway (correct SNI -> wildcard cert).
                                             # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
                                             # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
@@ -74,6 +162,17 @@ data:
 
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
+  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
+  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
+  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
+  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
+  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
+  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
+  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
+  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
+  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
+  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
+  LONGHORN_DATA_PATH: "/var/lib/longhorn"
 
   # --- observability sizing (docs/10) ---
   MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long

clusters/dev/kustomization.yaml

@@ -5,6 +5,20 @@
 # reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
 # It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
 # on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
+#
+# ##############################################################################################
+# ###  *** DO NOT MERGE THIS FILE TO `main` UNATTENDED (as of 2026-09-04) ***                 ###
+# ###                                                                                        ###
+# ###  This overlay now carries dev's DNS MIGRATION target: BASE_DOMAIN moves from the apex   ###
+# ###  `home.keithjackson.dev` to `dev.home.keithjackson.dev`, and the GitRepository url       ###
+# ###  moves with it. sync.yaml reconciles ./clusters/dev from `main` EVERY 10 MINUTES, so a   ###
+# ###  merge to main IS the cutover — it executes on the live dev cluster with nobody present. ###
+# ###  Dev's Gitea is both its GitOps source and the push target: if the new host is not up    ###
+# ###  with a Ready cert first, dev stops reconciling and pushing cannot repair it.            ###
+# ###                                                                                        ###
+# ###  Run the ordered runbook: infra/PROD-PROVISIONING.md §10 (steps D1-D8), operator        ###
+# ###  present. Merging is step D3, not step one.                                             ###
+# ##############################################################################################
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
@@ -131,7 +145,24 @@ patches:
         # via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
         # secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
         # revert this url (docs/05).
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        #
+        # *** THE STRAND. THIS URL IS THE MOST DANGEROUS LINE IN THE REPOSITORY. ***
+        # It is BOTH dev's GitOps source AND the operator's push target, and it is NOT tokenized —
+        # it cannot be: the `flux-system` root Kustomization that renders ./clusters/dev declares NO
+        # postBuild.substituteFrom, so `gitea.${BASE_DOMAIN}` here would ship the LITERAL token and
+        # break the source outright (the L0 flux-postbuild-substitution check asserts exactly this).
+        # So it is a hand-maintained literal that must be moved in lockstep with BASE_DOMAIN below.
+        #
+        # It has been moved to dev's NEW suffix as part of the 2026-09-04 migration authoring. That
+        # makes this file the TARGET state, NOT a safe unattended merge: dev's root Kustomization
+        # (sync.yaml) reconciles ./clusters/dev from `main` every 10m, so MERGING THIS TO main WHILE
+        # DEV IS LIVE MIGRATES DEV UNATTENDED. If gitea.dev.home.keithjackson.dev does not already
+        # resolve and serve a valid cert at that instant, dev stops reconciling and CANNOT BE FIXED
+        # BY PUSHING — the broken thing is the path Flux uses to read pushes.
+        # ORDERING THAT AVOIDS IT (infra/PROD-PROVISIONING.md §10, steps D1-D3): suspend dev's root
+        # Kustomization or park this url on the bootstrap Gitea (http://192.168.2.132:3000/...) for
+        # the migration window, and only restore the in-cluster url after the new host is proven.
+        url: https://gitea.dev.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   # --- dev-scoped cluster-vars (R-ENV-02) ---
@@ -148,6 +179,60 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # ==========================================================================================
+      # DEV DNS IDENTITY (R-DNS-01/02/03) — DEV MOVES OFF THE APEX. **NOT YET APPLIED.**
+      # ==========================================================================================
+      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day:
+      #
+      #   dev   -> dev.home.keithjackson.dev      (this knob; = the base default, restated)
+      #   prod  -> home.keithjackson.dev          (THE APEX — prod IS the apex, no `prod.` label,
+      #                                            and therefore no apex CNAME layer at all)
+      #
+      # *** THIS VALUE IS AHEAD OF THE LIVE CLUSTER. *** The dev cluster is serving the APEX right
+      # now. Merging this file does not perform the migration — it DEFINES the target. The cutover is
+      # an ORDERED, OPERATOR-PRESENT runbook: infra/PROD-PROVISIONING.md §10.
+      #
+      # WHY IT CANNOT BE A PLAIN VARIABLE FLIP — THE STRAND. `spec.url` of the GitRepository ~20
+      # lines above is https://gitea.home.keithjackson.dev/... . Dev's Gitea is BOTH dev's GitOps
+      # source AND the operator's push target. The moment BASE_DOMAIN moves, dev's Gitea HTTPRoute,
+      # DNS record and wildcard cert all move to gitea.dev.home.keithjackson.dev. If that host does
+      # not ALREADY resolve with a valid, Ready cert, dev stops reconciling — and it CANNOT BE FIXED
+      # BY PUSHING, because the thing that is broken is the path Flux uses to read pushes. Recovery
+      # is then hands-on (point the source at the bootstrap Gitea on 192.168.2.132:3000, or repair
+      # in-cluster by hand). §10 states the ordering that avoids it: publish and verify the new host
+      # FIRST, or park the source on the bootstrap Gitea for the window.
+      #
+      # ALSO MOVED BY THIS ONE LINE, all on a live cluster: the `*.` wildcard Certificate (a fresh
+      # ACME order), Keycloak's issuer URL and ~12 OIDC clients' redirect URIs, kubectl SSO (a k3s
+      # server flag => a rolling restart + every user re-issuing their kubeconfig), coredns-custom's
+      # split-horizon zone, Homepage's links, and 6 L2 chainsaw tests that hardcode the apex.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "dev.home.keithjackson.dev"
+      # ExternalDNS scope. MUST stay the Cloudflare ZONE — NOT dev.home.keithjackson.dev. FINDING
+      # F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this for ZONE SELECTION, so a
+      # strict-subdomain filter matches no zone and publishes NOTHING ("no hosted zone matching
+      # record DNS Name"). Dev moving to a subdomain does NOT make narrowing safe — it makes it
+      # newly tempting, which is why this comment is here. Same value the retired DNS_ZONE knob held.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        value: "keithjackson.dev"
+      # OWNER ID: `dev` (was `default` until 2026-09-04). The old value was FORCED — dev's live apex
+      # records were already in Cloudflare's TXT registry under owner `default`, so renaming would
+      # have orphaned every one of them. THE MIGRATION DISSOLVES THAT CONSTRAINT: on the new suffix
+      # dev publishes a brand-new record set with no prior registry entries to recognise, so the id
+      # is chosen on its merits, and `default` has none — it is the value external-dns uses when you
+      # FORGET the flag, which is exactly how both clusters came to share it. Consequences, both
+      # deliberate:
+      #   - dev's OLD apex records remain owned by `default`, which no environment now claims, so
+      #     nothing deletes them implicitly. Retiring them is an explicit §10 step, and prod's first
+      #     publish is gated on that retirement being CONFIRMED — not on two reconcile loops racing.
+      #   - prod stays `prod` and must NEVER be given `default`, or prod would own the live dev
+      #     cluster's apex records and sync-delete them. Prod inherits the apex NAMES, not its OWNER.
+      # dev != prod is the load-bearing invariant; the base default stays a non-colliding sentinel.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "dev"
       # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
       # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
       # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -156,7 +241,7 @@ patches:
       #
       # Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
       # (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
-      # *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
+      # *.dev.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
       - op: replace
         path: /data/GATEWAY_VIP
         value: "192.168.1.53"
@@ -190,6 +275,16 @@ patches:
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
+      # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
+      # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
+      # else would orphan every one of them (they are not recoverable from the old path by
+      # rescheduling). Restated explicitly even though it equals the base default, exactly like the
+      # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
+      # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/var/lib/longhorn"
       # smaller retention / volumes to keep the dev workstation footprint light.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

clusters/prod/kustomization.yaml

@@ -115,24 +115,78 @@ patches:
             name: sops-age
   # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
   #
-  # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
+  # ############################################################################################
+  # ###                                                                                      ###
+  # ###   *** T E M P O R A R Y   B O O T S T R A P   S O U R C E  —  NOT PERMANENT ***       ###
+  # ###                                                                                      ###
+  # ###   The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside   ###
+  # ###   any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST   ###
+  # ###   be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this  ###
+  # ###   and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is  ###
+  # ###   a live single point of failure, not a settled design. See "CUTOVER" below.          ###
+  # ###                                                                                      ###
+  # ############################################################################################
   #
-  # Two independent reasons, both consequences of the split-horizon design:
+  # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
-  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
+  # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
-  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
+  # REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
-  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
+  # which changes WHY the blocker holds but not THAT it holds:
-  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
+  #   1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
-  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
+  #      pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
-  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
+  #      inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
-  #      gitea.home.keithjackson.dev to itself and never reach a repository.
+  #      BEFORE dev is retired, or prod loses its source of truth.
+  #      NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
+  #      gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
+  #      the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
+  #      cutover below a one-line change once prod actually runs Gitea.
+  #   2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
+  #      easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
+  #      ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
+  #      prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
+  #      listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
+  #      to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
+  #      remains is purely reason 1 — the Gitea is not there YET.
   #
-  # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
+  # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
-  # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
+  #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
-  # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
+  #   operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
+  #   reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
+  #   change was needed. This is the same "local bootstrap source" pattern dev itself used before
+  #   its Phase-6 cutover (docs/05 "Source-of-truth").
+  #
+  # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
+  # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
+  # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
+  # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
+  #
+  # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
+  # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
+  #   curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
+  #   git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git  -> refs/heads/main
+  # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
+  # THAT FIRST: a private repo here fails exactly that way.
+  #
+  # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
+  #   WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
+  #         through prod's own Gateway VIP with a publicly-trusted cert.
+  #   DO:   1. push/mirror this repo into the prod in-cluster Gitea;
+  #         2. change the url below to
+  #            https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+  #            and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
+  #            clusters/dev/kustomization.yaml, is what "done" looks like);
+  #         3. confirm `flux get sources git -n flux-system` reconciles the new revision;
+  #         4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
+  #            exactly as dev retained its bootstrap Gitea.
+  #   UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
+  #   powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
+  #   it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
   #
   # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
   # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
   # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
+  # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
+  # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
+  # is the revision you intend prod to run (`git ls-remote` above, against local `main`).
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository
@@ -140,7 +194,9 @@ patches:
         name: flux-system
         namespace: flux-system
       spec:
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
+        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
+        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   - target:
@@ -154,9 +210,96 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # ==========================================================================================
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
+      # ==========================================================================================
+      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
+      # (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
+      #
+      #   dev   -> dev.home.keithjackson.dev     (dev carries the label; clusters/dev)
+      #   prod  -> home.keithjackson.dev         (THE APEX — this knob)
+      #
+      # There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
+      # name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
+      # Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
+      #
+      # BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
+      # HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
+      # EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
+      # consequences, one of which already happened:
+      #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
+      #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
+      #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
+      #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
+      #      have raced each other's `_acme-challenge` TXT records on that one zone.
+      #
+      # *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
+      # Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
+      # therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
+      # tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
+      #   (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
+      #   (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
+      #       (verified against Cloudflare directly — a pfSense override is NOT evidence about public
+      #       DNS; that mistake was made on 2026-09-04), AND
+      #   (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
+      #       that same name set while dev's still exists counts against LE's duplicate-certificate
+      #       limit (5 per week per identical name set).
+      # Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
+      # the 2026-09-04 incident with the roles reversed.
+      #
+      # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
+      # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
+      # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
+      # trusts a different Keycloak than the one prod actually runs.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "home.keithjackson.dev"
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        # MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
+        # like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
+        # F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
+        # zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
+        # record DNS Name", so prod publishes NOTHING. An agent narrowed this to
+        # prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
+        # Isolation from dev does NOT come from this filter — it comes from
+        # (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
+        # (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
+        # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
+        value: "keithjackson.dev"
+      # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
+      # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
+      # its own and, under policy: sync, deleted the records behind them.
+      #
+      # PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
+      # NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
+      # the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
+      # prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
+      # never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
+      # invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
+      # simply fails to publish until those records are retired, which is the correct behaviour.
+      # dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
+      # non-colliding sentinel. All three differ.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "prod"
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
+      # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
+      # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
+      # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
+      # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
+      # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
+      #
+      # [parity] MUST equal node_baseline_longhorn_data_path in
+      # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
+      # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
+      # path on the OS filesystem instead of the data partition and /var fills up.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/data/longhorn"
       # longer retention / larger volumes on prod NVMe.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -68,7 +68,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
@@ -76,8 +76,8 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -91,7 +91,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -91,7 +91,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

infra/ansible/README.md

@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOP
 > ```
 > (Both take the same target-group override as the rest — see **Targeting prod** below.)
 
+> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
+> ```
+> ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
+> ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey
+> ```
+> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
+> until `-e memattr_survey_reboots_authorised=true` is passed. See
+> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
+
 ## Re-provisioning a node (tofu -replace / fresh OS)
 A node rebuild wipes **everything on the host filesystem** — including host-level config
 that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
@@ -109,6 +118,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
 `playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
 keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
 bring-up chain.
+`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
+for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
+role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
+can be mistyped; inventory membership cannot.
 
 ```bash
 # dev — completely unchanged, no extra vars

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -87,24 +87,155 @@ kube_vip_address: "192.168.2.9"
 # — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
 cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
 
-# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
+# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
-# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
+# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
-# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
+# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
-# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
+# backports kernel as a side effect of sharing the role), and this line switches it on for
-# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
+# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
-# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
+# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
-# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
+# rather than from anyone remembering.
 #
-# The overrides below are deliberately present-but-broken: leaving them UNSET would
+# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
-# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
+# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
-# hardware — a confusing failure that looks like a wiring bug. This value fails just
+# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
-# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
+# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
-# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
+# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
-# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
+# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
-# updates these two lines. All three nodes must end up with an identical layout
+# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
-# (Longhorn replica placement assumes symmetry) before any of this is used for real.
+# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
-node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
-node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# in §3.
+#
+# COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
+# guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
+# the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
+# back to false, roll back per the note the role writes to
+# /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
+# roles/node_baseline.
+#
+# The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
+# FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
+node_baseline_backports_kernel: true
+
+# --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
+# Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
+#
+#   DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
+#                     disk, claimed and mounted at the NEUTRAL mountpoint `/data`
+#                     (LABEL=k3s-data), then split between TWO tenants:
+#                       /data/longhorn   Longhorn's dataPath
+#                       /data/k3s        k3s --data-dir (etcd AND the containerd image store)
+#   ETCD           -> NO dedicated device. It lives inside /data/k3s on that partition.
+#
+# ================================ THE 2026-09-04 REVISION ================================
+# The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
+# /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
+# data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
+# deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
+# would therefore have filled /var during its FIRST image pulls — and with etcd on the same
+# partition, a full /var means etcd cannot write its WAL. That is cluster death, not
+# degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
+# mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
+# tenants. Nothing was installed on prod when this was decided, which is exactly why it was
+# decided then — it is cheap now and very expensive after bring-up.
+#
+# THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
+# were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
+# contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
+# MEASURED, not assumed — exit condition in §2.
+# =========================================================================================
+#
+# Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
+# not as a device path and a blank. That distinction is the point of this block:
+#
+# 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
+#    k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
+#    (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
+#    Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
+#    freezes a fact about today's hardware into inventory, where it goes stale silently the
+#    first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
+#    nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
+#    So inventory declares the INTENT ("the partition currently mounted at /srv") and the
+#    role discovers the device on the node, then proves it is safe to claim (a partition or
+#    disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
+#    unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
+#
+# 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
+#    describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
+#    layout etcd is inside /data/k3s on the claimed data partition, which is neither a
+#    dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
+#    Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
+#    and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
+#    on every run, that the filesystem backing the etcd data dir really is /data. If the
+#    mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
+#    WAL would land back on the OS filesystem, invisibly, until the disk filled. The
+#    `root-filesystem` value makes no such claim and gets no such assert — which is exactly
+#    why the two are distinct enum values rather than one with a comment.
+#
+# 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
+#    the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
+#    the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
+#    this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
+#    k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
+#    Longhorn volumes are untouched.
+node_baseline_longhorn_source: claim-mountpoint
+node_baseline_longhorn_claim_mountpoint: /srv
+node_baseline_etcd_source: shared-data-partition
+
+# --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
+# `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
+# CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
+# /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
+# as the destination would make "claimed" and "not claimed" indistinguishable.
+# `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
+# partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
+# else's named volume — so labelling ours `data` would make a stranger's `data` volume look
+# already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
+# `k3s-etcd` label convention already in this repo.
+node_baseline_data_mount: /data
+node_baseline_data_fs_label: k3s-data
+# Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
+# equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
+# side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
+# default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
+node_baseline_longhorn_data_path: /data/longhorn
+
+# --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
+# The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
+# `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
+# label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
+# would walk into the greenfield claim path — which fails closed at "the claim mountpoint
+# must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
+# is right but it is not convergence, so the transition is DECLARED here: the role unmounts
+# the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
+# the partition is empty (nothing orphaned) and is the device the retired label names.
+# Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
+# SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
+# then skip on the retired-label lookup and nothing else depends on them.
+node_baseline_data_retired_fs_label: longhorn
+node_baseline_data_retired_mount: /var/lib/longhorn
+
+# --- k3s --data-dir (§2) -----------------------------------------------------------
+# THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
+# claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
+# `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
+# which defers to this variable) creates the directory 0700 on the data partition and derives
+# etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
+# Ansible side cannot disagree with itself.
+#
+# Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
+# it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
+# dir would orphan that mount under a running cluster.
+k3s_data_dir: /data/k3s
+
+# Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
+# source uses a device path any more, but leaving these UNSET would let prod silently
+# inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
+# metal) the moment anyone flipped a source back to `device`. These values can never parse
+# as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
+# before mkfs is ever reached.
+node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
+node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
 
 # --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
 # roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
@@ -158,15 +289,70 @@ node_baseline_dns_servers:
   #   LAN clients       -> PiHole. Not a cluster concern.
   #   nodes             -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
   #
-  # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
+  # THE k3s API SERVER is a node process and carries `oidc-issuer-url=...` (built from
-  # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
+  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
-  # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
+  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
-  # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
-  # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
-  # with the dev Gateway VIP.
   #
-  # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
+  # ORDERING HAZARD, NOT YET CLEARED. Prod's issuer is keycloak.home.keithjackson.dev — an APEX
-  # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
+  # name, because prod IS the apex under the 2026-09-04 inversion (k3s_base_domain below).
-  # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
+  # That is the name the LIVE DEV CLUSTER answers today, and its public A record currently
-  # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
+  # points at DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. A prod
+  # node resolving its own issuer through these public upstreams therefore reaches dev until:
+  #   (1) dev has migrated to *.dev.home.keithjackson.dev and RETIRED its apex records, and
+  #   (2) PROD's own external-dns has published keycloak.home.keithjackson.dev at PROD's
+  #       Gateway VIP (192.168.2.20).
+  # Only then do the public answer and the in-cluster split-horizon answer AGREE, and no
+  # /etc/hosts pin is needed. Until then kubectl OIDC SSO on prod cannot work; the cert-based
+  # admin kubeconfig is the break-glass path and always works. Ordering, not a design gap —
+  # the ordered steps are infra/PROD-PROVISIONING.md §10.
 node_baseline_dns_search: []
+
+# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
+# PROD **IS** THE APEX. [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml —
+# the same one-fact-in-two-places contract as k3s_data_dir and k3s_api_host. The Flux side
+# cannot supply this: the kube-apiserver's oidc-issuer-url is a k3s server FLAG written to
+# /etc/rancher/k3s/config.yaml, not a manifest, so postBuild.substituteFrom never sees it.
+# Full rationale in roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url derives.
+#
+# WHY THE SUFFIXES SPLIT (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve
+# the SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
+# deleted every one of dev's public records.
+#
+# WHICH WAY ROUND — INVERTED 2026-09-04 by operator decision, superseding the interim split
+# committed earlier the same day (prod on `prod.home.` behind apex CNAMEs). The simpler target:
+#   prod -> home.keithjackson.dev       (THE APEX; this variable — no `prod.` label, no CNAMEs)
+#   dev  -> dev.home.keithjackson.dev   (roles/k3s_server/defaults/main.yml)
+# So prod's Keycloak issuer is https://keycloak.home.keithjackson.dev/realms/homelab, and the
+# `kubernetes` OIDC client in PROD's OWN Keycloak realm is what prod's kubectl SSO authenticates
+# against.
+#
+# ORDERING HAZARD SPECIFIC TO THIS VARIABLE. keycloak.home.keithjackson.dev is the name the LIVE
+# dev cluster answers today, and prod's nodes resolve the issuer through PUBLIC upstreams (see
+# node_baseline_dns_search above). So until dev has vacated the apex and PROD's external-dns has
+# published that name at PROD's Gateway VIP (192.168.2.20), a prod node resolving its own issuer
+# gets DEV's VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. kubectl OIDC SSO on prod
+# therefore does not work until §10's dev-vacate steps are complete; the cert-based admin
+# kubeconfig is the break-glass path and always works. Ordering, not a design gap.
+k3s_base_domain: "home.keithjackson.dev"
+
+# --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
+# roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
+# and group_vars/all.yml carries no override, so dev keeps resolving exactly that).
+# Prod bootstraps its OWN overlay. Before this seam existed the role could bootstrap
+# clusters/dev ONLY — bootstrap/flux/bootstrap.sh's FLUX_ENV_PATH default was never
+# exported by the role — so `flux bootstrap ... --path=clusters/prod` (§8 step 4 of
+# infra/PROD-PROVISIONING.md) was simply not reachable without editing the role.
+#
+# Same seam as kube_vip_address / k3s_api_host / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel above: set once for the [k3s_prod] umbrella group, so a
+# FOURTH prod node inherits it from group membership with no per-host or per-run flag.
+#
+# WHAT clusters/prod STILL NEEDS before this is safe to run — do NOT invoke the
+# flux_bootstrap role until these are true (PROD-PROVISIONING.md §7):
+#   - the remaining TBD-PROD-* placeholders in clusters/prod/kustomization.yaml are real
+#     (S3_BACKUP_ENDPOINT); they are INVALID on purpose so an early apply fails loudly;
+#   - the GitRepository url there is the TEMPORARY workstation Gitea
+#     (http://192.168.2.132:3000/...) and that container is actually running and reachable
+#     from the prod nodes — it is outside every cluster and can simply be powered off;
+#   - the `sops-age` Secret is installed out of band on prod (bootstrap/sops).
+flux_bootstrap_env_path: "clusters/prod"

infra/ansible/inventory/hosts.ini

@@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 
 # =============================================================================
 # PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
-# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
+# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
-# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
+# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
-# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
+# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
-# nothing has been run against node 1, and the operator has decided the CLUSTER stays
+# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
-# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
+# check and the node-baseline PACKAGE floor has been run, and the operator has decided
-# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
+# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
-# later member-add dance.
+# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
+# single-server-etcd window, no later member-add dance.
 #
 # KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
 # hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
@@ -92,9 +93,18 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
 # node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
 # partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
 # nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
-# node 3 remains commented until it is racked.
+# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
+# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
+# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
+# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
+# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
+# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
+# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
+# three-way, not a node-1 outlier.
+# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
+# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
 k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
-# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
+k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
 
 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
 # pattern; named to match the group_vars directory so Ansible auto-loads it).

infra/ansible/playbooks/memattr-survey.yml

@@ -0,0 +1,137 @@
+---
+# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
+# R-ENV-03, R-TEST-02).
+#
+# WHAT QUESTION THIS ANSWERS
+#   infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
+#   EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
+#   Linux 6.12 reserves everything from the table address to the top of RAM without
+#   validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
+#   playbook measures them:
+#
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
+#        failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
+#        fleet-wide were at other addresses. If bad outcomes map onto a narrow address
+#        range, the causal chain closes with nothing touched.
+#     2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+#   §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
+#   is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
+#   a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
+#   boots per arm; detecting a mere odds shift needs ~50.
+#
+# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
+# package installs, no hardware moves, no node_baseline run. Its job is to establish real
+# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
+# a Fast Boot flip, a 6.13 kernel) compares against it.
+#
+# HOW TO RUN IT
+#   Dry run first — every stage except the reboots, so the pipeline is proven before 45
+#   minutes of wall clock are committed:
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml'
+#
+#   The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml \
+#           -e memattr_survey_reboots_authorised=true'
+#
+#   Fewer rounds, or one node:
+#     -e memattr_survey_rounds=5        --limit k3s-prod-2
+#
+#   RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
+#   rounds are never lost or double-counted. Analysis alone, over existing artifacts:
+#     ansible-playbook playbooks/memattr-survey.yml --tags analyse
+#
+# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
+# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
+# not a member of memattr_survey_required_group — the dev cluster is live, and a survey
+# that reboots it 20 times because of a mistyped --limit is exactly the accident worth
+# engineering against.
+#
+# HOW TO READ THE OUTPUT
+#   infra/ansible/_artifacts/memattr-survey/  (gitignored, regenerated every run)
+#     <host>.rounds.jsonl  one JSON line per round, mirrored off the node after EVERY round
+#     <host>.boots.json    every boot the persistent journal holds, re-derived in one pass
+#                          at the end — the AUTHORITATIVE record
+#     summary.md           the analysis, also printed at the end of the run
+#   See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
+#   with NO memattr log line is a healthy third state and not a missing measurement.
+
+# =====================================================================================
+# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
+# before a single reboot. That is the right asymmetry: the cost of a false stop is a
+# re-run, the cost of a false start is rebooting a machine that was doing something.
+# =====================================================================================
+- name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  any_errors_fatal: true
+  tags: [survey]
+  tasks:
+    - name: Preflight and baseline probe
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: preflight
+
+# =====================================================================================
+# PLAY 2 — THE SURVEY.
+#
+# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
+# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
+# keep collecting.
+#
+# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
+# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
+# that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
+# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
+# blocks belongs to exactly one host, so the other two sit idle through it and the survey
+# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
+# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
+# appeared in the output. `free` lets each host walk its own task list at its own pace,
+# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
+#
+# Consequences worth knowing before editing: output INTERLEAVES between hosts (every
+# progress line is host-tagged for exactly this reason), and `run_once` is unreliable
+# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
+# directory create on every host instead. Concurrency is also capped by `forks`
+# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
+# than letting a bigger fleet silently serialise again.
+# =====================================================================================
+- name: memattr survey — reboot rounds and journal harvest
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  strategy: free
+  tags: [survey]
+  tasks:
+    - name: Run the reboot rounds
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: rounds
+
+    - name: Harvest every retained boot from the persistent journal
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: harvest
+
+# =====================================================================================
+# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
+# even if every surveyed node died, because the artifacts are already on disk and the
+# analysis counts rounds from THOSE rather than from Ansible's host state. A node that
+# vanished shows up as a short arm with a wider interval, not as a missing report.
+# =====================================================================================
+- name: memattr survey — analyse
+  hosts: localhost
+  gather_facts: false
+  become: false
+  tags: [analyse]
+  tasks:
+    - name: Produce the analysis
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: analyse

infra/ansible/roles/cilium/defaults/main.yml

@@ -15,9 +15,20 @@ cilium_namespace: "kube-system"
 # Cilium 1.20.1 targets Gateway API v1.6.1.
 gateway_api_version: "v1.6.1"
 gateway_api_crd_urls:
+  # ONE URL, deliberately. As of v1.6.0 the STANDARD channel bundle already contains
+  # TLSRoute (plus TCPRoute, UDPRoute, ListenerSet and BackendTLSPolicy), so the separate
+  # experimental TLSRoute manifest this list used to carry is redundant — and worse than
+  # redundant: v1.6 also ships a ValidatingAdmissionPolicy,
+  # `safe-upgrades.gateway.networking.k8s.io`, which REJECTS an experimental-channel CRD
+  # applied over a standard-channel one. Applying the old second URL therefore fails hard:
+  #   ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' ... denied request:
+  #   Installing experimental CRDs on top of standard channel CRDs is prohibited by default.
+  # Verified 2026-09-04 on the prod bootstrap: standard-install.yaml v1.6.1 alone produces the
+  # exact 10 gateway.networking.k8s.io CRDs the LIVE dev cluster is running, tlsroutes included
+  # and annotated channel=standard, bundle-version=v1.6.1 — so this is parity with dev, not a
+  # reduction. Dev needs no reapply; it already has them. If gateway_api_version is ever moved
+  # BACK below v1.6.0, the experimental TLSRoute URL has to come back with it.
   - "https://github.com/kubernetes-sigs/gateway-api/releases/download/{{ gateway_api_version }}/standard-install.yaml"
-  # TLSRoute (experimental) is required by Cilium's Gateway API support.
-  - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
 
 # Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
 # comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev

infra/ansible/roles/flux_bootstrap/defaults/main.yml

@@ -19,5 +19,22 @@ flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
 flux_bootstrap_sops_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/sops/install-age-key.sh"
 
 # Assumed interface of bootstrap/flux (GitOps Engineer, T-1.4):
-#   runs `flux bootstrap ... --path=clusters/dev` against this cluster.
+#   runs `flux install` + `kubectl apply -k <env path>` against this cluster.
 flux_bootstrap_flux_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/flux/bootstrap.sh"
+
+# --- WHICH ENVIRONMENT OVERLAY THIS BOOTSTRAPS (R-ENV-02) -------------------------
+# The one thing that makes this role environment-agnostic. bootstrap/flux/bootstrap.sh
+# reads FLUX_ENV_PATH (default clusters/dev) and ends with `kubectl apply -k <that>`;
+# tasks/main.yml exports this value into that variable, so the role NEVER names an
+# environment and the task graph is identical for dev and prod.
+#
+# THE DEFAULT IS DEV ON PURPOSE — dev is LIVE, and group_vars/all.yml carries no
+# override, so a dev run resolves exactly the value it always had (clusters/dev) and
+# behaves byte-for-byte as before.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same group_vars
+# seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel. No role fork, no per-run extra-var, no second
+# playbook: group membership decides. Before this existed the role could bootstrap
+# ONLY clusters/dev, so prod bring-up was impossible without editing the role.
+flux_bootstrap_env_path: "clusters/dev"

infra/ansible/roles/flux_bootstrap/tasks/main.yml

@@ -4,11 +4,17 @@
 # — it calls the entrypoints the Security + GitOps Engineers author there.
 #
 # Order matters: install the age decryption key FIRST (so Flux can decrypt on its
-# very first reconcile), THEN run `flux bootstrap` at clusters/dev.
+# very first reconcile), THEN bootstrap Flux at the environment's overlay.
 #
 # The two commands are variables (see defaults/main.yml). If the bootstrap workers
 # expose different entrypoints, override flux_bootstrap_sops_cmd / _flux_cmd — the
 # task graph does not change.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): this role names NO environment. Which overlay gets
+# bootstrapped comes from `flux_bootstrap_env_path` (defaults/main.yml, default
+# clusters/dev), exported below as FLUX_ENV_PATH — the variable bootstrap/flux/
+# bootstrap.sh already reads. Prod overrides it in group_vars/k3s_prod/vars.yml, the
+# same seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name.
 
 - name: Assert the bootstrap/ subtree exists (authored by the other worker)
   ansible.builtin.stat:
@@ -29,11 +35,14 @@
   run_once: true
   changed_when: true
 
-- name: Bootstrap Flux against clusters/dev (bootstrap/flux)
+- name: Bootstrap Flux against this environment's overlay (bootstrap/flux)
   ansible.builtin.command:
     cmd: "{{ flux_bootstrap_flux_cmd }}"
   environment:
     KUBECONFIG: "{{ flux_bootstrap_kubeconfig }}"
+    # The ONLY per-environment input. clusters/dev by default (dev unchanged);
+    # clusters/prod for [k3s_prod] hosts via group_vars.
+    FLUX_ENV_PATH: "{{ flux_bootstrap_env_path }}"
   delegate_to: localhost
   become: false
   run_once: true

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
 # the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
 k3s_disable: "traefik,servicelb,local-storage"
 
+# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
+# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
+# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
+# image store (agent/containerd) — which is by far the largest tenant.
+#
+# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
+# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
+# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
+# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
+# running cluster's entire state directory out from under it and orphan that etcd mount.
+#
+# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
+# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
+# node inherits it from group membership rather than from anyone remembering. It is set
+# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
+# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
+# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
+# this same variable). One line moves both.
+#
+# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
+# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
+# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
+# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
+# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
+#
+# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
+# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
+# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
+# Restart handler for a value that is not changing. The install flags are the correct and
+# sufficient place, and prod has no k3s installed yet.
+k3s_data_dir: "/var/lib/rancher/k3s"
+
 # Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
 k3s_server_mode: "secondary"
 
@@ -34,3 +66,44 @@ k3s_tls_sans: >-
   {{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
       + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
      | unique }}
+
+# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
+# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
+#
+# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
+# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
+# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
+# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
+#
+#   Flux side     clusters/base/cluster-vars.yaml   BASE_DOMAIN      (per-overlay)
+#   Ansible side  THIS variable                     k3s_base_domain  (per-inventory-group)
+#
+# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
+# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
+# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
+# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
+# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
+#
+# THE DEFAULT IS THE DEV VALUE — the same convention as k3s_data_dir above and cluster-vars'
+# GATEWAY_VIP / LONGHORN_DATA_PATH: the role default is dev's value and prod overrides it in
+# inventory/group_vars/k3s_prod/vars.yml (the same inventory seam k3s_data_dir / k3s_api_host /
+# kube_vip_address use, so a FOURTH prod server inherits it from group membership rather than
+# from anyone remembering).
+#
+# *** WHICH DEV VALUE — UPDATED 2026-09-04, AND IT IS AHEAD OF THE LIVE DEV CLUSTER. ***
+# The operator inverted the suffix split: PROD is now the apex `home.keithjackson.dev` and DEV
+# carries the label `dev.home.keithjackson.dev`. This default therefore moved from the apex to
+# dev's subdomain, mirroring BASE_DOMAIN in clusters/dev/kustomization.yaml.
+#
+# CONSEQUENCE — this default no longer matches what the live dev nodes are running, so unlike
+# every previous edit to this file A DEV RE-RUN IS NOT A NO-OP. Rendering config.yaml with the
+# new issuer CHANGES the file, FIRES the Restart handler, and rolls k3s on all three dev
+# servers, which also invalidates every user's OIDC kubeconfig (SSO must be re-issued). That is
+# a deliberate step of the dev DNS migration, NOT a side effect to discover during unrelated
+# work: DO NOT run the k3s_server role against dev until step D6 of
+# infra/PROD-PROVISIONING.md §10, and never while the operator is away. The break-glass
+# cert-based admin kubeconfig keeps working throughout, which is what makes the step safe.
+k3s_base_domain: "dev.home.keithjackson.dev"
+
+# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
+k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -22,6 +22,7 @@
       --write-kubeconfig-mode=644
       {% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
       --node-ip={{ k3s_node_ip }}
+      --data-dir={{ k3s_data_dir }}
 
 # On an ALREADY-installed cluster the install task above is idempotent (creates:
 # k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
@@ -74,13 +75,19 @@
       # OIDC (Keycloak SSO for kubectl). The API server trusts ID tokens from the homelab realm;
       # the `kubernetes` PUBLIC client (PKCE) + kubelogin mint them, and RBAC binds the `groups`
       # claim (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins->cluster-admin,
-      # oidc:viewers->view). Nodes resolve keycloak.home.keithjackson.dev to the Cilium gateway VIP
+      # oidc:viewers->view). Nodes resolve keycloak.<k3s_base_domain> to the Cilium gateway VIP
-      # (192.168.1.53) and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
+      # and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
       # prefixes namespace federated identities away from cert / service-account subjects.
       # ADDITIVE + break-glass-safe: the cert-based admin kubeconfig keeps full access with or
       # without Keycloak, so a bad realm/OIDC state can never lock you out of the cluster.
+      #
+      # NOT a Flux ${BASE_DOMAIN} token: this is a k3s SERVER FLAG, not a manifest, so
+      # postBuild.substituteFrom never touches it. k3s_oidc_issuer_url derives from
+      # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
+      # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
+      # config.yaml is unchanged and the Restart handler does not fire.
       kube-apiserver-arg:
-        - "oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab"
+        - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
         - "oidc-client-id=kubernetes"
         - "oidc-username-claim=email"
         - "oidc-username-prefix=oidc:"
@@ -97,15 +104,20 @@
     INSTALL_K3S_VERSION: "{{ k3s_version }}"
   when: k3s_server_mode == 'primary'
 
+# PATH FOLLOWS --data-dir. k3s writes the node-token to <data-dir>/server/token, NOT to
+# the literal /var/lib/rancher/k3s. On dev k3s_data_dir IS /var/lib/rancher/k3s (the role
+# default, restated explicitly above) so this is byte-for-byte the old path; on prod
+# --data-dir is /data/k3s (group_vars/k3s_prod) and the hardcoded path did not exist at all
+# — this task would have timed out after 180s and the two secondaries could never join.
 - name: Wait for the k3s node-token to exist (primary)
   ansible.builtin.wait_for:
-    path: /var/lib/rancher/k3s/server/token
+    path: "{{ k3s_data_dir }}/server/token"
     timeout: 180
   when: k3s_server_mode == 'primary'
 
 - name: Read the k3s node-token (primary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   register: k3s_token_raw
   when: k3s_server_mode == 'primary'
 
@@ -125,9 +137,11 @@
 # "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
 # dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
 # delegates to the PROD primary using this same role, unchanged (R-ENV-02).
+# k3s_data_dir is a CLUSTER-WIDE value (group_vars for prod, role default for dev), so the
+# secondary's own copy of it is also the primary's data dir — correct for the delegated read.
 - name: Fetch the k3s node-token directly from the primary (secondary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   delegate_to: "{{ groups[k3s_primary_group][0] }}"
   register: k3s_primary_token_raw
   when:

infra/ansible/roles/kube_vip/defaults/main.yml

@@ -31,4 +31,14 @@ kube_vip_retry_period: 2
 
 # k3s auto-deploy manifests dir: k3s applies anything here on start, so kube-vip
 # is reproducible and independent of Flux/Helm.
-kube_vip_manifest_path: "/var/lib/rancher/k3s/server/manifests/kube-vip.yaml"
+#
+# THE PATH FOLLOWS --data-dir. k3s reads auto-deploy manifests from
+# <data-dir>/server/manifests, so this must be derived from the cluster-wide
+# k3s_data_dir (roles/k3s_server turns the same variable into --data-dir), not from the
+# literal /var/lib/rancher/k3s. Dev leaves k3s_data_dir unset outside roles/k3s_server's
+# own defaults, so the fallback below reproduces dev's existing path exactly; prod sets
+# k3s_data_dir: /data/k3s in group_vars/k3s_prod, where the old hardcoded path was simply
+# not the directory k3s watches — the manifest would have been written and silently never
+# applied, and the control-plane VIP would never have come up.
+kube_vip_manifest_path: >-
+  {{ k3s_data_dir | default('/var/lib/rancher/k3s') }}/server/manifests/kube-vip.yaml

infra/ansible/roles/kube_vip/tasks/main.yml

@@ -1,13 +1,22 @@
 ---
 # Deploy kube-vip (control-plane HA VIP) via the k3s auto-deploy manifests dir.
-# k3s applies /var/lib/rancher/k3s/server/manifests/*.yaml on start, so this is
+# k3s applies <data-dir>/server/manifests/*.yaml on start, so this is
-# reproducible and independent of Flux. The same manifest is written on every
+# reproducible and independent of Flux (see kube_vip_manifest_path in defaults/main.yml —
+# the dir follows k3s's --data-dir, which prod moves to /data/k3s). The same manifest is written on every
 # server so the DaemonSet survives the loss of any single node.
 
-- name: Detect the LAN interface holding the node's 192.168.1.5x address
+# WHICH INTERFACE kube-vip ARPs THE VIP ON. Detected from this node's OWN k3s_node_ip
+# (inventory, per host) rather than a hardcoded subnet: the previous awk matched the
+# literal 192.168.1.5[0-2], i.e. DEV's node addresses, so on prod (192.168.2.10/.11/.12)
+# it returned nothing and the assert below aborted the play. k3s_node_ip is the same
+# address k3s advertises with --node-ip and is defined for dev and prod alike, so this
+# resolves eno1 on prod and the identical dev interface it always resolved before.
+# `ip addr show to <ip>/32` does the matching in iproute2 instead of a regex over dotted
+# quads (where `.` would match any character).
+- name: Detect the LAN interface holding this node's k3s_node_ip
   ansible.builtin.shell: >-
     set -o pipefail;
-    ip -o -4 addr show | awk '/192\.168\.1\.5[0-2]\//{print $2; exit}'
+    ip -o -4 addr show to {{ k3s_node_ip }}/32 | awk '{print $2; exit}'
   args:
     executable: /bin/bash
   register: kube_vip_iface_detect
@@ -25,7 +34,7 @@
       - kube_vip_vip_interface is defined
       - kube_vip_vip_interface | length > 0
     fail_msg: >-
-      Could not detect the 192.168.1.5x LAN interface for kube-vip on
+      Could not detect the interface holding {{ k3s_node_ip }} for kube-vip on
       {{ inventory_hostname }} — set kube_vip_interface explicitly.
 
 - name: Ensure the k3s auto-deploy manifests directory exists

infra/ansible/roles/memattr_survey/README.md

@@ -0,0 +1,201 @@
+# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery
+
+**Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02
+**Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism
+is settled there and this role does **not** re-derive it.
+
+## The question
+
+§0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes
+Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it,
+and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from
+the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.
+
+Two things §0b leaves unmeasured, and this role measures:
+
+1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed
+   failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes
+   fleet-wide were at other addresses. If bad outcomes map onto a narrow address range,
+   the causal chain closes without touching hardware or firmware.
+2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each.
+
+## Why 20 rounds
+
+Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson
+interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is
+worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a
+mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds`
+is a variable; lowering it below 20 produces an arm that cannot answer the question it was
+built for, and the analysis will say so.
+
+## This is the BASELINE arm
+
+**Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware
+moves, no `node_baseline` run. The role installs nothing and configures nothing; its only
+write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's
+own allocation footprint — initrd size, module set, boot device — is *part of the causal
+chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that
+the kernel release and cmdline were constant on every node, for exactly this reason.
+
+## Running it
+
+```bash
+# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml'
+
+# The real thing. Reboots are OPT-IN; the interlock is deliberate.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml \
+      -e memattr_survey_reboots_authorised=true'
+
+# Re-analyse existing artifacts, no hardware touched.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml --tags analyse'
+```
+
+Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`,
+`-e memattr_survey_hosts=<group>`.
+
+20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are
+independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That
+parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count;
+preflight asserts the latter. See the playbook comment for why the default linear
+strategy silently serialises a looped `include_tasks` (it was measured, not assumed).
+
+## Safety — verified, not assumed
+
+The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That
+premise is re-checked on **every** run, because a document and a machine drift apart:
+
+| Check | Failure mode it prevents |
+|---|---|
+| Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. |
+| `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. |
+| `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. |
+| `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. |
+| `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. |
+
+Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the
+whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a
+false start is a rebooted production machine.
+
+## Fail-safe and resume
+
+* **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node
+  that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the
+  observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that
+  host only** and the others keep collecting. The abort is announced immediately and again
+  in the report — which counts rounds from the **artifacts**, not from Ansible's host
+  state, so even a node that goes fully unreachable (which `rescue` cannot catch) still
+  appears as a short arm with a wider interval, never as a silently missing report.
+* **Resumable.** Round numbers come from the on-node ledger, so re-running after an
+  interruption continues where it stopped and can neither lose nor duplicate a round. The
+  boot found on a resume is *not* re-appended: it is already round N of the interrupted
+  run, and double-counting it would bias the sample toward whichever outcome survived the
+  interruption.
+* **The abort is durable, not scrollback.** `<host>.aborted.json` records the round, the
+  failing task and the reason, so the report can say *why* an arm is short instead of
+  leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every
+  run and re-written only if that run aborts. Its *absence* is not proof of success — a
+  node can be lost abruptly enough that Ansible calls it unreachable before the rescue
+  runs — so completeness is still judged on round count, and the marker only supplies the
+  reason when there is one.
+* **Analysis survives everything.** It is a separate play on `localhost`, so it runs even
+  if every surveyed node died. With a node down, `--tags analyse` is the way in: a full
+  run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host.
+
+## Belt and braces on the data
+
+Two independent records, joined on `boot_id`:
+
+| | written by | authoritative for |
+|---|---|---|
+| `<host>.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** |
+| `<host>.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** |
+| `<host>.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** |
+
+The journal does not care whether the driver was alive: the boot happened, the kernel
+logged it, journald persisted it. So the harvest can only ever be a *superset* of the
+ledger, and a round the driver failed to append still shows up.
+
+## Reading the output
+
+Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as
+`roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never
+committed. `summary.md` is also printed at the end of the run.
+
+### The verdict is measured, never inferred from a log line
+
+This is the subtlest part of the design. There are **three** kernel readings, not two:
+
+| `kernel_read` | What the kernel logged | Memory |
+|---|---|---|
+| `valid` | **nothing at all** — it parsed a table it considered valid | intact |
+| `bail-version` | `Unexpected EFI Memory Attributes table version <garbage>` | intact — the *lucky* path, and it looks alarming |
+| `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** |
+
+Silence is therefore *evidence of health*, not a missing measurement — which is why the
+record carries an explicit `memattr_diagnostic_present: false` rather than an empty field.
+And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`,
+*later* than the `memblock_reserve()` that did the damage: it is a consequence, not the
+cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and
+`kernel_read` is recorded alongside it as an independent observation. §5a of the report
+cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.
+
+Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed
+clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything
+landing between them is reported as `indeterminate` and excluded from `n` — a new third
+thing deserves human eyes, not a coin flip into a bucket.
+
+### What is and is not pooled
+
+Boots the survey did **not cause** are shown in the per-round table for context and
+excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot
+each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so
+pooling it would double-count prior evidence into a run whose point is fresh trials).
+
+### The statistics
+
+`files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is.
+
+* **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the
+  Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly
+  instead of needing a special case. A bare fraction is what this exercise exists to stop
+  reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
+* **The address↔outcome question is tested three ways**, because the tempting version is
+  circular:
+  * **4b determinism** — was any single address seen with *both* outcomes? One such
+    address falsifies "the address determines the outcome" outright.
+  * **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed
+    up as a p-value: the bad set is defined after seeing the data.
+  * **4d the pre-registered prediction** — §0b, written before this run, names
+    `0xbeeb0018`. Fisher exact against this run's boots only. This is the one
+    non-circular test available.
+  * **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a
+    run produces no bad boot, `verdict` has no variance to explain, but `kernel_read`
+    does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in
+    §0b's mechanism. An address that reproduces the same *parse* boot after boot is the
+    address doing causal work, even when none of those parses was catastrophic.
+
+## Files
+
+| Path | What it is |
+|---|---|
+| `tasks/preflight.yml` | Safety gate, resume point, baseline probe |
+| `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock |
+| `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue |
+| `tasks/harvest.yml` | The authoritative journal pass |
+| `tasks/analyse.yml` | Controller-side analysis |
+| `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable |
+| `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report |
+
+## What this cannot tell you
+
+It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute
+the difference between nodes, because node 2 is simultaneously the only single-DIMM node
+and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled
+rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS
+settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this
+role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all
+three nodes, and that is not in dispute.

infra/ansible/roles/memattr_survey/defaults/main.yml

@@ -0,0 +1,92 @@
+---
+# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
+# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
+# settled and is NOT re-derived here.
+#
+# THE QUESTION THIS ROLE ANSWERS
+#   §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
+#   Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
+#        observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
+#        successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
+#        address range, the causal chain is closed with no hardware or firmware touched.
+#     2. Is the fault node-2-specific, or does any node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+# WHY 20 ROUNDS AND NOT FIVE
+#   Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
+#   interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
+#   boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
+#   needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
+#   not a round number.
+#
+# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
+#   No BIOS writes, no kernel change, no package installs, no hardware moves. The role
+#   installs nothing and configures nothing; its only write to a node is its own append-
+#   only ledger under /var/lib. Anything that alters the boot chain's allocation
+#   footprint — and per §0b that includes the initrd, the module set and the boot device —
+#   changes the variable under test and voids the arm.
+
+# --- Sample size --------------------------------------------------------------------
+# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
+# machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
+memattr_survey_rounds: 20
+
+# --- Safety interlock ----------------------------------------------------------------
+# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
+# preflight, a baseline probe of the current boot, the journal harvest and the analysis.
+# That is a genuinely useful dry run and it is also how you prove the pipeline works
+# before committing 45 minutes of wall clock to it.
+memattr_survey_reboots_authorised: false
+
+# The role refuses to run against a host that is not in this group. The prod boxes carry
+# no k3s and nothing is running on them, which is the entire reason rebooting them is
+# sanctioned; the dev cluster is live and must never be caught by a stray --limit.
+memattr_survey_required_group: k3s_prod
+
+# Unit names whose presence or activity means something IS running on this node. Any hit
+# aborts the whole playbook before the first reboot (see tasks/preflight.yml).
+memattr_survey_forbidden_units:
+  - k3s
+  - k3s-agent
+  - kubelet
+  - containerd
+  - docker
+# Paths whose existence means k3s has been installed here, even if not currently running.
+memattr_survey_forbidden_paths:
+  - /etc/rancher/k3s
+  - /var/lib/rancher/k3s
+  - /usr/local/bin/k3s
+
+# --- Reboot behaviour -----------------------------------------------------------------
+# A node that does not answer within this is declared lost: the role stops rebooting THAT
+# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
+# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
+# slow boot.
+memattr_survey_reboot_timeout: 600
+memattr_survey_connect_timeout: 20
+# Let journald settle before reading the new boot's kernel log. The kernel log itself is
+# complete long before SSH is up; this is purely margin.
+memattr_survey_post_reboot_delay: 15
+
+# --- On-node ledger (belt) -------------------------------------------------------------
+# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
+# the controller dies. This is what makes the survey resumable: re-running skips rounds
+# already recorded here rather than starting over.
+memattr_survey_state_dir: /var/lib/memattr-survey
+memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
+
+# --- Controller artifacts (braces) ------------------------------------------------------
+# Same convention as roles/node_health and conformance/_artifacts/: generated output under
+# a `_artifacts/` directory that .gitignore excludes, never committed.
+#   <host>.rounds.jsonl  the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
+#                        mid-survey still leaves every completed round on the controller
+#   <host>.boots.json    the journal harvest — every retained boot re-derived in one
+#                        pass. THIS is the authoritative record; the ledger is the backup.
+#   summary.md           the analysis (also printed)
+memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
+memattr_survey_write_artifacts: true
+
+# Print each round's one-line result as it happens. A 45-minute unattended run with no
+# output is indistinguishable from a hung one.
+memattr_survey_show_progress: true

infra/ansible/roles/memattr_survey/files/memattr_analyse.py

@@ -0,0 +1,521 @@
+#!/usr/bin/env python3
+"""memattr_analyse.py — turn a memattr survey's artifacts into the answer.
+
+Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads
+  <host>.boots.json    the journal harvest — authoritative, one entry per retained boot
+  <host>.rounds.jsonl  the per-round ledger — authoritative for ROUND NUMBERS
+and joins them on boot_id.
+
+WHAT IT ANSWERS, in order:
+
+  1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated
+     loudly and up front, because every number below it is weaker than it looks.
+
+  2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing
+     this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and
+     completely different evidence. Clopper-Pearson is the exact (conservative) binomial
+     interval — no normal approximation, correct at 0/n and n/n where the approximation is
+     degenerate and where a clean arm is most likely to land.
+
+  3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three
+     ways, deliberately, because the tempting version of this test is circular:
+       (a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such
+           address falsifies "the address determines the outcome" outright, and no amount
+           of correlation elsewhere repairs it.
+       (b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported
+           as sorted min/max per outcome plus the count of good addresses falling inside
+           the bad range. This is descriptive; it is NOT dressed up as a p-value, because
+           the bad set was chosen after seeing the data.
+       (c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts
+           that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots
+           this run observed, which is the one non-circular test available.
+
+  4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory
+     ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell.
+
+Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import math
+import os
+import sys
+from collections import Counter, defaultdict
+
+# §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad
+# both times, while all five successes fleet-wide were at other addresses.
+PREREGISTERED_BAD_ADDRESS = "0xbeeb0018"
+
+
+# --- Exact binomial interval ----------------------------------------------------------
+def binom_tail_ge(n: int, k: int, p: float) -> float:
+    """P(X >= k) for X ~ Bin(n, p)."""
+    if k <= 0:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1))
+
+
+def binom_tail_le(n: int, k: int, p: float) -> float:
+    """P(X <= k) for X ~ Bin(n, p)."""
+    if k >= n:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1))
+
+
+def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
+    """Exact 95% (by default) binomial confidence interval for k successes in n trials.
+
+    Solved by bisection on the exact tail probabilities rather than via the Beta/F
+    quantile identity, so it needs nothing outside the standard library and the
+    boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing:
+      lower bound solves P(X >= k | p) = alpha/2,  and is 0 when k = 0
+      upper bound solves P(X <= k | p) = alpha/2,  and is 1 when k = n
+    """
+    if n == 0:
+        return (0.0, 1.0)
+
+    def bisect(fn, target, lo, hi):
+        for _ in range(200):
+            mid = (lo + hi) / 2
+            if fn(mid) < target:
+                lo = mid
+            else:
+                hi = mid
+        return (lo + hi) / 2
+
+    lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0)
+    upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0)
+    return (lower, upper)
+
+
+def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float:
+    """Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only.
+
+    Used ONLY for the pre-registered prediction (address == the §0b address vs outcome),
+    never for a bad set defined after the fact — that would be a garden of forking paths
+    with a p-value stapled on.
+    """
+    n = a + b + c + d
+    if n == 0:
+        return 1.0
+    row1, col1 = a + b, a + c
+
+    def prob(x):
+        return (
+            math.comb(row1, x)
+            * math.comb(n - row1, col1 - x)
+            / math.comb(n, col1)
+        )
+
+    observed = prob(a)
+    lo = max(0, col1 - (n - row1))
+    hi = min(row1, col1)
+    # 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating
+    # float noise in the equality case.
+    return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9)))
+
+
+# --- Loading ---------------------------------------------------------------------------
+def load_host(art_dir: str, host: str) -> dict:
+    """Harvest (authoritative outcomes) joined to ledger (authoritative round numbers)."""
+    with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle:
+        harvest = json.load(handle)
+
+    rounds_by_boot: dict[str, int] = {}
+    ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl")
+    ledger: list[dict] = []
+    if os.path.exists(ledger_path):
+        with open(ledger_path, encoding="utf-8") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                try:
+                    record = json.loads(line)
+                except json.JSONDecodeError:
+                    continue
+                ledger.append(record)
+                if record.get("boot_id") and isinstance(record.get("round"), int):
+                    rounds_by_boot[record["boot_id"]] = record["round"]
+
+    boots = []
+    for boot in harvest.get("boots", []):
+        entry = dict(boot)
+        entry["round"] = rounds_by_boot.get(boot.get("boot_id"))
+        boots.append(entry)
+    boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0)))
+
+    # Written by the per-round rescue when a node stops coming back. Its absence is not
+    # proof of success — a node can be lost so abruptly that Ansible calls it unreachable
+    # before the rescue runs — so completeness is still judged on round COUNT, and this
+    # only supplies the reason when one exists.
+    abort = None
+    abort_path = os.path.join(art_dir, f"{host}.aborted.json")
+    if os.path.exists(abort_path):
+        try:
+            with open(abort_path, encoding="utf-8") as handle:
+                abort = json.load(handle)
+        except (OSError, json.JSONDecodeError):
+            abort = {"reason": "abort marker present but unreadable"}
+
+    return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort}
+
+
+def discover_hosts(art_dir: str) -> list[str]:
+    return sorted(
+        os.path.basename(p)[: -len(".boots.json")]
+        for p in glob.glob(os.path.join(art_dir, "*.boots.json"))
+    )
+
+
+# --- Rendering ---------------------------------------------------------------------------
+def gib(kb) -> str:
+    return "?" if kb is None else f"{kb / 1048576:.2f}"
+
+
+def pct(x) -> str:
+    return "?" if x is None else f"{100 * x:.0f}%"
+
+
+def render(hosts: list[dict], expected_rounds: int) -> str:
+    out: list[str] = []
+    w = out.append
+
+    all_boots = [b for h in hosts for b in h["boots"]]
+    # Only boots this survey CAUSED count toward any statistic. Excluded, deliberately:
+    #   * pre-existing journal boots (round None) — observed under an unknown selection
+    #     process, and already counted once in §0b's table;
+    #   * round 0, the boot each node was found sitting on — it is a boot nobody chose,
+    #     but for node 2 it is literally §0b's "boot 0" row, so pooling it would double-
+    #     count prior evidence into a run whose entire point is fresh, independent trials.
+    # Both are still SHOWN in the per-round table, because context is not contamination.
+    surveyed = [b for b in all_boots if b.get("round") not in (None, 0)]
+
+    w("# EFI Memory Attributes Table — baseline reboot survey")
+    w("")
+    w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no")
+    w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this")
+    w("measures p(bad) per node and tests whether the outcome is predicted by the address")
+    w("the firmware published the table at.")
+    w("")
+
+    # --- 1. Completeness -----------------------------------------------------------------
+    w("## 1. Completeness — read this before any number below it")
+    w("")
+    w("| node | rounds recorded | target | journal boots harvested | short? |")
+    w("|---|---:|---:|---:|---|")
+    short_arms = []
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        is_short = n < expected_rounds
+        if is_short:
+            short_arms.append((h["host"], n))
+        w(
+            f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | "
+            f"{'**YES — arm incomplete**' if is_short else 'no'} |"
+        )
+    w("")
+    if short_arms:
+        w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms))
+        w("> Every interval below is correspondingly wider than planned.")
+        w("")
+        for h in hosts:
+            if h.get("abort"):
+                a = h["abort"]
+                w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of "
+                  f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}")
+                w(f"> (failing task: {a.get('failed_task', 'unknown')})")
+                w("")
+
+    # --- 2. Per-round table ---------------------------------------------------------------
+    w("## 2. Per-round results")
+    w("")
+    for h in hosts:
+        installed = h["harvest"].get("smbios_installed_kb")
+        w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)")
+        w("")
+        w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |")
+        w("|---:|---|---|---|---:|---:|---:|---|")
+        for b in h["boots"]:
+            rnd = b.get("round")
+            label = "pre" if rnd is None else str(rnd)
+            usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb")
+            w(
+                f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | "
+                f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | "
+                f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | "
+                f"{gib(usable)} | {pct(b.get('usable_fraction'))} | "
+                f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |"
+            )
+        w("")
+        w("`round` = survey round. `pre` = a boot already in the journal before the survey;")
+        w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED")
+        w("from every statistic below. `kernel read`: `valid` = table")
+        w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage")
+        w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =")
+        w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from")
+        w("the presence of a log line.")
+        w("")
+
+    # --- 3. p(bad) -------------------------------------------------------------------------
+    w("## 3. p(bad) per node — 95% Clopper-Pearson")
+    w("")
+    w("| node | bad | n | p(bad) | 95% CI | reading |")
+    w("|---|---:|---:|---:|---|---|")
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        bad = sum(1 for b in rounds if b.get("verdict") == "bad")
+        lo, hi = clopper_pearson(bad, n)
+        if n == 0:
+            reading = "no data"
+        elif bad == 0:
+            reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}"
+        elif bad == n:
+            reading = f"bad every time in {n}; p(bad) at least {lo:.2f}"
+        else:
+            reading = f"p(bad) between {lo:.2f} and {hi:.2f}"
+        w(
+            f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | "
+            f"{lo:.2f} – {hi:.2f} | {reading} |"
+        )
+    w("")
+    indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")]
+    if indet:
+        w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are")
+        w("> reported as `indeterminate` rather than forced into a bucket. They are")
+        w("> excluded from n above. Inspect them by hand.")
+        w("")
+
+    # --- 4. Address vs outcome -----------------------------------------------------------
+    w("## 4. Is the outcome predicted by where the table landed?")
+    w("")
+    by_addr: dict[str, Counter] = defaultdict(Counter)
+    addr_hosts: dict[str, set] = defaultdict(set)
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        by_addr[addr][b.get("verdict")] += 1
+        addr_hosts[addr].add(b.get("host") or "?")
+
+    w("### 4a. Every address observed in the survey")
+    w("")
+    w("| MEMATTR | good | bad | other | nodes | verdict for this address |")
+    w("|---|---:|---:|---:|---|---|")
+    ambiguous = []
+    for addr in sorted(by_addr, key=lambda a: (a == "none", a)):
+        counts = by_addr[addr]
+        good, bad = counts.get("good", 0), counts.get("bad", 0)
+        other = sum(v for k, v in counts.items() if k not in ("good", "bad"))
+        if good and bad:
+            note = "**BOTH — address does not determine outcome**"
+            ambiguous.append(addr)
+        elif bad:
+            note = "always bad"
+        elif good:
+            note = "always good"
+        else:
+            note = "—"
+        w(
+            f"| `{addr}` | {good} | {bad} | {other} | "
+            f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |"
+        )
+    w("")
+
+    w("### 4b. Determinism")
+    w("")
+    if not by_addr:
+        w("No surveyed boots — nothing to test.")
+    elif ambiguous:
+        w(f"**FALSIFIED for {len(ambiguous)} address(es):** "
+          + ", ".join(f"`{a}`" for a in ambiguous)
+          + " was seen with BOTH outcomes. The address alone cannot determine the outcome.")
+    else:
+        w("**Not falsified.** No address in this survey was ever seen with both outcomes —")
+        w("consistent with the address determining the outcome, though with this many")
+        w("distinct addresses most were seen only once, which makes the test weak.")
+        singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1)
+        w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)")
+    w("")
+
+    w("### 4c. Separation of the address ranges")
+    w("")
+    bad_addrs = sorted({b["memattr_address"] for b in surveyed
+                        if b.get("verdict") == "bad" and b.get("memattr_address")})
+    good_addrs = sorted({b["memattr_address"] for b in surveyed
+                         if b.get("verdict") == "good" and b.get("memattr_address")})
+    if bad_addrs and good_addrs:
+        blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16)
+        inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi]
+        w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`")
+        w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`")
+        w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**"
+          + (" — the ranges interleave, so 'a narrow bad range' does not hold"
+             if inside else " — the ranges are disjoint"))
+        w("")
+        w("Descriptive only. The bad set was defined after seeing the data, so this is a")
+        w("picture, not a test.")
+    elif not bad_addrs:
+        w("No bad boot was observed in this survey, so there is no bad range to separate.")
+    else:
+        w("No good boot was observed in this survey.")
+    w("")
+
+    w("### 4d. The pre-registered prediction")
+    w("")
+    w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the")
+    w("bad address (both prior failures were there; all five prior successes elsewhere).")
+    w("Tested against this survey's boots only:")
+    w("")
+    a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    w(f"| | bad | good |")
+    w(f"|---|---:|---:|")
+    w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |")
+    w(f"| elsewhere | {c} | {d} |")
+    w("")
+    if a + b_ == 0:
+        w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the")
+        w("prediction had no opportunity to be right or wrong. That is itself informative:")
+        w("the address moves every boot, so a specific address is not a reproducible handle.")
+    else:
+        p = fisher_exact_2x2(a, b_, c, d)
+        w(f"Fisher exact (two-sided) p = **{p:.4g}**"
+          + ("  — the prediction holds." if p < 0.05 and a and not b_
+             else "  — not significant at 0.05." if p >= 0.05
+             else "  — significant, but read the cell counts before believing it."))
+    w("")
+
+    w("### 4e. Does the address predict how the KERNEL READ the table?")
+    w("")
+    w("The strongest test this data can support when no bad boot occurred. `verdict` has")
+    w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both")
+    w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:")
+    w("the kernel reads whatever bytes happen to be at the address the firmware published.")
+    w("If the address determines the reading, the address is doing causal work, one step")
+    w("short of the catastrophic path.")
+    w("")
+    reads_by_addr: dict[str, set] = defaultdict(set)
+    n_by_addr: Counter = Counter()
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        reads_by_addr[addr].add(b.get("kernel_read"))
+        n_by_addr[addr] += 1
+    repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1}
+    inconsistent = {a: r for a, r in repeated.items() if len(r) > 1}
+    distinct_reads = {r for rs in reads_by_addr.values() for r in rs}
+    w(f"- distinct addresses: **{len(reads_by_addr)}**, of which "
+      f"**{len(repeated)}** were observed more than once")
+    w(f"- distinct kernel readings across the survey: "
+      + ", ".join(f"`{r}`" for r in sorted(distinct_reads)))
+    w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**")
+    w("")
+    if len(distinct_reads) < 2:
+        w("Only one reading occurred in the whole survey, so there is nothing to predict.")
+    elif inconsistent:
+        w("**The address does NOT determine the reading:** "
+          + "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items()))
+    elif not repeated:
+        w("No address recurred, so the test could not be run.")
+    else:
+        w(f"**The address determines the reading, on every address observed more than once**")
+        w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both")
+        w("readings present in the sample). This is direct evidence for the causal chain")
+        w("§0b describes: the same address reproduces the same parse, boot after boot.")
+        w("It does NOT establish which addresses are catastrophic — no bad boot occurred —")
+        w("only that WHERE the table lands is what decides what the kernel finds there.")
+    w("")
+    w("| MEMATTR | boots | kernel reading(s) |")
+    w("|---|---:|---|")
+    for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)):
+        marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else ""
+        w(f"| `{addr}` | {n_by_addr[addr]} | "
+          f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |")
+    w("")
+
+    # --- 5. Corroboration -------------------------------------------------------------------
+    w("## 5. Corroborating evidence")
+    w("")
+    w("### 5a. Kernel's own reading vs the measured outcome")
+    w("")
+    cross: dict[tuple, int] = Counter()
+    for b in surveyed:
+        cross[(b.get("kernel_read"), b.get("verdict"))] += 1
+    w("| kernel read | verdict | boots |")
+    w("|---|---|---:|")
+    for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))):
+        w(f"| {read} | {verdict} | {count} |")
+    w("")
+    disagreements = [
+        (read, verdict, count)
+        for (read, verdict), count in cross.items()
+        if (read == "failed-map") != (verdict == "bad")
+    ]
+    if disagreements:
+        w("> **The log line and the measurement DISAGREE** in "
+          f"{sum(c for _, _, c in disagreements)} boot(s): "
+          + "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements)
+          + ". That is a finding in its own right — §0b's model says `failed-map` and only")
+        w("> `failed-map` costs memory.")
+    else:
+        w("`failed-map` and only `failed-map` cost memory, in every boot observed —")
+        w("exactly as §0b's mechanism predicts, and independent of the address question.")
+    w("")
+
+    w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)")
+    w("")
+    w("| verdict | NODE_DATA end address | boots |")
+    w("|---|---|---:|")
+    nd: Counter = Counter()
+    for b in surveyed:
+        line = b.get("node_data_line") or ""
+        end = line.split("-")[-1].rstrip("]") if "-" in line else "?"
+        nd[(b.get("verdict"), end)] += 1
+    for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
+        w(f"| {verdict} | `{end}` | {count} |")
+    w("")
+
+    # --- 6. Invariants ------------------------------------------------------------------
+    w("## 6. Nothing under test changed")
+    w("")
+    w("| node | kernels seen | distinct cmdlines | SMBIOS installed |")
+    w("|---|---|---:|---|")
+    for h in hosts:
+        kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")})
+        cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")}
+        w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | "
+          f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |")
+    w("")
+    w("More than one kernel or more than one cmdline on a node would void that node's arm.")
+    w("")
+    return "\n".join(out) + "\n"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--artifact-dir", required=True)
+    parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted")
+    parser.add_argument("--out", default=None, help="also write the report here")
+    args = parser.parse_args()
+
+    names = discover_hosts(args.artifact_dir)
+    if not names:
+        print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr)
+        return 1
+    hosts = [load_host(args.artifact_dir, name) for name in names]
+    report = render(hosts, args.rounds)
+    if args.out:
+        with open(args.out, "w", encoding="utf-8") as handle:
+            handle.write(report)
+    sys.stdout.write(report)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

infra/ansible/roles/memattr_survey/files/memattr_probe.py

@@ -0,0 +1,489 @@
+#!/usr/bin/env python3
+"""memattr_probe.py — read one boot's EFI Memory Attributes Table outcome off a node.
+
+Part of roles/memattr_survey (infra/ansible). See that role's README.md for the
+question this answers; see infra/PROD-PROVISIONING.md §0b for the settled mechanism.
+
+WHY A SCRIPT AND NOT A HANDFUL OF `command:` TASKS
+  The record for one boot needs six sources cross-referenced (kernel log config-table
+  line, kernel log memattr diagnostic, kernel log `Memory:` line, kernel log
+  `NODE_DATA` line, /proc/meminfo, SMBIOS) and one derived verdict. Doing that in
+  Jinja across six registered results is unreadable and would put the classification
+  logic in a template instead of in one testable place. It also has to be exactly the
+  same logic for a live boot and for a boot harvested out of the journal, which a
+  single --all-boots mode gives for free.
+
+MODES
+  (default)      one JSON record for the CURRENT boot, on stdout
+  --all-boots    one JSON object with a `boots` list: EVERY boot the persistent
+                 journal still holds, re-derived from scratch. This is the
+                 AUTHORITATIVE record — it survives the driver script dying mid-run.
+  --status       {"completed_rounds": N, "records": M} from the on-node ledger, so the
+                 playbook can resume without re-deriving anything.
+
+  --round N      stamp the record with the survey round number
+  --append PATH  ALSO append the record to PATH as one JSON line (the on-node ledger),
+                 created 0644 with parents. Appended with O_APPEND in one write() so a
+                 concurrent reader never sees a half line.
+
+VERDICT IS MEASURED, NOT INFERRED FROM A LOG LINE
+  A node whose kernel reads a VALID table logs no memattr diagnostic at all and is
+  perfectly healthy; a node whose kernel bails out on a garbage version > 2 logs a
+  scary warning and is ALSO perfectly healthy. Only the third state — "Failed to map"
+  — costs memory, and even that message is emitted by a LATER function than the one
+  that did the damage. So the verdict comes from usable RAM versus SMBIOS installed
+  RAM, and `kernel_read` is recorded ALONGSIDE it as an independent observation. If
+  the two ever disagree, that disagreement is itself the finding.
+
+EXIT STATUS is 0 whenever a record could be produced, even a mostly-null one: a field
+this script could not read comes back as null with a reason in `notes`, never as an
+empty string and never as a hard failure. A survey that aborts on one unreadable field
+loses the other 19 rounds.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+
+SCHEMA = "memattr-survey/1"
+
+# --- Verdict thresholds -------------------------------------------------------------
+# Observed on this fleet: a good boot yields usable/installed of 0.95-0.98 (kernel code,
+# reserved firmware regions and struct page overhead account for the rest); the bad path
+# yields 0.15-0.17 (everything above the table address is reserved). Two orders of
+# separation, so the thresholds are nowhere near either cluster. Anything landing in the
+# gap is deliberately NOT called: it is a new third thing and wants human eyes.
+VERDICT_GOOD_MIN_FRACTION = 0.85
+VERDICT_BAD_MAX_FRACTION = 0.60
+
+# `Memory: 2568544K/16546088K available (... 13958980K reserved ...)`
+RE_MEMORY_LINE = re.compile(
+    r"^Memory:\s+(?P<available>\d+)K/(?P<present>\d+)K available\b.*?"
+    r"(?:(?P<reserved>\d+)K reserved)?"
+)
+# `efi: ACPI=0x... SMBIOS=0x... MEMATTR=0xbeeb0018 MOKvar=0x... INITRD=0x...`
+RE_MEMATTR_ADDR = re.compile(r"\bMEMATTR=(?P<addr>0x[0-9a-fA-F]+)")
+# `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749`
+RE_TABLE_VERSION = re.compile(r"table version\s+(?P<version>-?\d+)")
+# `NODE_DATA(0) allocated [mem 0xbee85680-0xbeeaffff]` — per §0b the cleanest A/B tell:
+# on the bad path pgdat is forced immediately BELOW the table address instead of landing
+# at the top of RAM. Captured because it is free and it is independent evidence.
+RE_NODE_DATA = re.compile(r"^NODE_DATA\(\d+\) allocated \[mem (?P<range>[0-9a-fx]+-[0-9a-fx]+)\]")
+RE_CMDLINE = re.compile(r"^Command line:\s*(?P<cmdline>.*)$")
+# dmidecode type 17 `Size: 16 GB` / `Size: 8192 MB` / `Size: No Module Installed`
+RE_DMI_SIZE = re.compile(r"^\s*Size:\s+(?P<n>\d+)\s+(?P<unit>[kKmMgGtT]B)\s*$")
+
+DMI_UNIT_KB = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def run(argv: list[str], timeout: int = 60) -> tuple[int, str, str]:
+    """Run a command, never raise. Returns (rc, stdout, stderr); rc 127 if absent."""
+    try:
+        proc = subprocess.run(
+            argv, capture_output=True, text=True, timeout=timeout, check=False
+        )
+    except FileNotFoundError:
+        return 127, "", f"{argv[0]}: not found"
+    except subprocess.TimeoutExpired:
+        return 124, "", f"{' '.join(argv)}: timed out after {timeout}s"
+    except OSError as exc:  # pragma: no cover - defensive
+        return 126, "", f"{' '.join(argv)}: {exc}"
+    return proc.returncode, proc.stdout, proc.stderr
+
+
+def read_text(path: str) -> str | None:
+    try:
+        with open(path, "r", encoding="utf-8", errors="replace") as handle:
+            return handle.read()
+    except OSError:
+        return None
+
+
+def kernel_log(boot: str) -> tuple[list[str], str | None]:
+    """Kernel ring buffer for `boot` ('0' = current, '-1' = previous, or a boot ID).
+
+    `-o cat` is deliberate: it yields the message VERBATIM with no host/facility
+    decoration, which is what the survey wants to record, and it is stable across
+    systemd versions. `-k` restricts to _TRANSPORT=kernel.
+    """
+    rc, out, err = run(["journalctl", "-k", "-b", boot, "-o", "cat", "--no-pager"], timeout=180)
+    if rc != 0:
+        return [], f"journalctl -k -b {boot} failed (rc={rc}): {err.strip()[:200]}"
+    return out.splitlines(), None
+
+
+def list_boots() -> tuple[list[dict], str | None]:
+    """Every boot the journal still holds, oldest first.
+
+    Prefers `--list-boots -o json` (systemd >= 250, so always on Debian 13) and falls
+    back to parsing the table, because a survey that cannot enumerate boots has lost
+    its authoritative record and must say so rather than silently reporting one boot.
+    """
+    rc, out, err = run(["journalctl", "--list-boots", "-o", "json", "--no-pager"], timeout=120)
+    if rc == 0:
+        try:
+            parsed = json.loads(out)
+        except json.JSONDecodeError as exc:
+            return [], f"--list-boots -o json unparseable: {exc}"
+        boots = [
+            {
+                "index": int(entry.get("index", 0)),
+                "boot_id": entry.get("boot_id"),
+                "first_entry_usec": entry.get("first_entry"),
+                "last_entry_usec": entry.get("last_entry"),
+            }
+            for entry in parsed
+        ]
+        return sorted(boots, key=lambda b: b["index"]), None
+
+    rc, out, err = run(["journalctl", "--list-boots", "--no-pager"], timeout=120)
+    if rc != 0:
+        return [], f"journalctl --list-boots failed (rc={rc}): {err.strip()[:200]}"
+    boots = []
+    for line in out.splitlines():
+        fields = line.split()
+        if len(fields) < 2 or not re.fullmatch(r"-?\d+", fields[0]):
+            continue  # header row
+        boots.append(
+            {
+                "index": int(fields[0]),
+                "boot_id": fields[1],
+                "first_entry_usec": None,
+                "last_entry_usec": None,
+            }
+        )
+    return sorted(boots, key=lambda b: b["index"]), None
+
+
+def journal_storage() -> dict:
+    """Is the journal PERSISTENT? The harvest is only authoritative if it is.
+
+    /var/log/journal existing is what makes journald write to disk; /run/log/journal is
+    the volatile fallback and would mean every reboot erases the record.
+    """
+    return {
+        "var_log_journal": os.path.isdir("/var/log/journal"),
+        "run_log_journal": os.path.isdir("/run/log/journal"),
+        "persistent": os.path.isdir("/var/log/journal"),
+    }
+
+
+def smbios_installed_kb() -> tuple[int | None, str | None]:
+    """Installed RAM per SMBIOS type-17 records — the GROUND TRUTH for the verdict.
+
+    Deliberately not /proc/meminfo (that is the number under test) and not dmesg's
+    `Memory: x/yK` denominator (which is already post-e820 and post-reservation).
+    Hardware does not change between boots, so one read applies to harvested boots too.
+    """
+    rc, out, err = run(["dmidecode", "-t", "17"], timeout=60)
+    if rc != 0:
+        return None, f"dmidecode -t 17 failed (rc={rc}): {err.strip()[:200]}"
+    total = 0
+    seen = 0
+    for line in out.splitlines():
+        match = RE_DMI_SIZE.match(line)
+        if not match:
+            continue  # "No Module Installed" and every non-Size line
+        total += int(match.group("n")) * DMI_UNIT_KB[match.group("unit").lower()]
+        seen += 1
+    if seen == 0:
+        return None, "dmidecode -t 17 reported no populated memory devices"
+    return total, None
+
+
+def meminfo_total_kb() -> tuple[int | None, str | None]:
+    text = read_text("/proc/meminfo")
+    if text is None:
+        return None, "/proc/meminfo unreadable"
+    for line in text.splitlines():
+        if line.startswith("MemTotal:"):
+            return int(line.split()[1]), None
+    return None, "/proc/meminfo has no MemTotal"
+
+
+def classify_kernel_read(memattr_addr, diagnostics: list[str]) -> tuple[str, int | None]:
+    """What did the kernel make of the table? THREE outcomes, not two (§0b).
+
+    Returns (classification, table_version_if_reported).
+
+      no-table      firmware published no MEMATTR pointer at all (the dev nodes)
+      valid         a MEMATTR pointer AND no diagnostic => the kernel parsed a table it
+                    considered valid. This is SILENCE AS EVIDENCE, and it is a healthy
+                    state, not a missing measurement.
+      bail-version  garbage version > 2 => kernel warned and bailed. Memory intact.
+                    The LUCKY path, and it looks alarming in the log.
+      failed-map    the catastrophic path. The message comes from
+                    efi_memattr_apply_permissions() and is therefore PROOF the bogus
+                    memblock_reserve() already happened upstream of it.
+      bogus-size    a kernel >= 6.13 rejecting the table. Should never appear on 6.12;
+                    if it does, the kernel under test changed and the arm is void.
+    """
+    version = None
+    for line in diagnostics:
+        match = RE_TABLE_VERSION.search(line)
+        if match:
+            version = int(match.group("version"))
+    joined = " | ".join(diagnostics)
+    if "Failed to map" in joined:
+        return "failed-map", version
+    if "Corrupted EFI Memory Attributes Table" in joined:
+        return "bogus-size", version
+    if "Unexpected EFI Memory Attributes table version" in joined:
+        return "bail-version", version
+    if diagnostics:
+        return "diagnostic-unrecognised", version
+    if memattr_addr is None:
+        return "no-table", version
+    return "valid", version
+
+
+def derive_verdict(usable_kb, installed_kb) -> tuple[str, float | None]:
+    if usable_kb is None or not installed_kb:
+        return "unknown", None
+    fraction = usable_kb / installed_kb
+    if fraction >= VERDICT_GOOD_MIN_FRACTION:
+        return "good", fraction
+    if fraction <= VERDICT_BAD_MAX_FRACTION:
+        return "bad", fraction
+    return "indeterminate", fraction
+
+
+def build_record(
+    *,
+    boot_selector: str,
+    boot_meta: dict | None,
+    installed_kb,
+    installed_note,
+    round_number,
+    live: bool,
+) -> dict:
+    notes: list[str] = []
+    if installed_note:
+        notes.append(installed_note)
+
+    lines, log_note = kernel_log(boot_selector)
+    if log_note:
+        notes.append(log_note)
+
+    config_table_line = None
+    memattr_addr = None
+    diagnostics: list[str] = []
+    memory_line = None
+    node_data_line = None
+    cmdline = None
+
+    for line in lines:
+        stripped = line.strip()
+        if memattr_addr is None:
+            match = RE_MEMATTR_ADDR.search(stripped)
+            if match and stripped.startswith("efi:"):
+                memattr_addr = match.group("addr").lower()
+                config_table_line = stripped
+        # Every memattr-related diagnostic, verbatim. Matched on the pr_fmt prefix the
+        # kernel actually uses (`efi: memattr: `) AND on the message text, so a prefix
+        # change upstream cannot silently drop the line.
+        if stripped.startswith("efi: memattr:") or "EFI Memory Attributes" in stripped:
+            if stripped != config_table_line:
+                diagnostics.append(stripped)
+        if memory_line is None and stripped.startswith("Memory:"):
+            if RE_MEMORY_LINE.match(stripped):
+                memory_line = stripped
+        if node_data_line is None:
+            match = RE_NODE_DATA.match(stripped)
+            if match:
+                node_data_line = stripped
+        if cmdline is None:
+            match = RE_CMDLINE.match(stripped)
+            if match:
+                cmdline = match.group("cmdline").strip()
+
+    available_kb = present_kb = reserved_kb = None
+    if memory_line:
+        match = RE_MEMORY_LINE.match(memory_line)
+        available_kb = int(match.group("available"))
+        present_kb = int(match.group("present"))
+        reserved = match.group("reserved")
+        reserved_kb = int(reserved) if reserved else None
+    else:
+        notes.append("no `Memory: .../... available` line found in this boot's kernel log")
+
+    kernel_read, table_version = classify_kernel_read(memattr_addr, diagnostics)
+
+    # /proc/meminfo describes the CURRENT boot only. For a harvested past boot the
+    # equivalent measurement is the kernel's own `Memory: available` figure, which
+    # tracks MemTotal to within ~7% on both paths. Which one was used is recorded, so
+    # nobody has to guess later.
+    mem_total_kb, meminfo_note = (None, None)
+    if live:
+        mem_total_kb, meminfo_note = meminfo_total_kb()
+        if meminfo_note:
+            notes.append(meminfo_note)
+    usable_kb = mem_total_kb if mem_total_kb is not None else available_kb
+    basis = "MemTotal" if mem_total_kb is not None else "dmesg Memory: available"
+    verdict, fraction = derive_verdict(usable_kb, installed_kb)
+
+    record = {
+        "schema": SCHEMA,
+        "round": round_number,
+        "host": os.uname().nodename,
+        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+        "recorded_at_epoch": int(time.time()),
+        "live_boot": live,
+        "boot_index": (boot_meta or {}).get("index"),
+        "boot_id": (boot_meta or {}).get("boot_id"),
+        "boot_first_entry_usec": (boot_meta or {}).get("first_entry_usec"),
+        "kernel_release": os.uname().release,
+        "kernel_cmdline": cmdline,
+        # --- the independent variable -------------------------------------------------
+        "efi_config_table_line": config_table_line,
+        "memattr_address": memattr_addr,
+        "memattr_diagnostics": diagnostics,
+        "memattr_diagnostic_count": len(diagnostics),
+        # An explicit marker, never an empty field: "the kernel said nothing" is a
+        # measurement (it read a VALID table), not a gap in the data.
+        "memattr_diagnostic_present": bool(diagnostics),
+        "memattr_table_version": table_version,
+        "kernel_read": kernel_read,
+        # --- the dependent variable ---------------------------------------------------
+        "memory_line": memory_line,
+        "memory_available_kb": available_kb,
+        "memory_present_kb": present_kb,
+        "memory_reserved_kb": reserved_kb,
+        "mem_total_kb": mem_total_kb,
+        "smbios_installed_kb": installed_kb,
+        "usable_kb": usable_kb,
+        "usable_fraction": round(fraction, 5) if fraction is not None else None,
+        "verdict": verdict,
+        "verdict_basis": f"{basis} vs SMBIOS installed",
+        # --- corroborating evidence ---------------------------------------------------
+        "node_data_line": node_data_line,
+        "notes": notes,
+    }
+    return record
+
+
+def append_record(path: str, record: dict) -> str | None:
+    """One O_APPEND write of one line, so a reader never sees a torn record."""
+    try:
+        parent = os.path.dirname(path) or "."
+        os.makedirs(parent, mode=0o755, exist_ok=True)
+        payload = (json.dumps(record, sort_keys=True) + "\n").encode()
+        fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
+        try:
+            os.write(fd, payload)
+            os.fsync(fd)  # the point of the ledger is surviving the next reboot
+        finally:
+            os.close(fd)
+    except OSError as exc:
+        return f"could not append to {path}: {exc}"
+    return None
+
+
+def ledger_status(path: str) -> dict:
+    """Resume support: how far did a previous run get?
+
+    Round numbers come from the ledger itself rather than a line count, so the
+    baseline round 0 record and any hand-added line cannot shift the numbering.
+    """
+    rounds: list[int] = []
+    records = 0
+    malformed = 0
+    if os.path.exists(path):
+        with open(path, "r", encoding="utf-8", errors="replace") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                records += 1
+                try:
+                    parsed = json.loads(line)
+                except json.JSONDecodeError:
+                    malformed += 1
+                    continue
+                value = parsed.get("round")
+                if isinstance(value, int):
+                    rounds.append(value)
+    return {
+        "schema": SCHEMA,
+        "ledger": path,
+        "records": records,
+        "malformed_records": malformed,
+        "completed_rounds": max(rounds) if rounds else 0,
+        "rounds_recorded": sorted(rounds),
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--round", type=int, default=None, help="survey round number to stamp")
+    parser.add_argument("--append", default=None, help="on-node ledger to append to")
+    parser.add_argument("--all-boots", action="store_true", help="harvest every retained boot")
+    parser.add_argument("--status", default=None, metavar="LEDGER", help="report resume state")
+    args = parser.parse_args()
+
+    if args.status:
+        print(json.dumps(ledger_status(args.status), indent=2, sort_keys=True))
+        return 0
+
+    installed_kb, installed_note = smbios_installed_kb()
+    boots, boots_note = list_boots()
+    storage = journal_storage()
+
+    if args.all_boots:
+        records = []
+        for boot in boots:
+            # Address boots by BOOT ID, not by relative index: indices shift under the
+            # harvester's feet if journald rotates mid-harvest, and the ID is what the
+            # per-round ledger recorded, so the two records join cleanly.
+            selector = boot["boot_id"] or str(boot["index"])
+            records.append(
+                build_record(
+                    boot_selector=selector,
+                    boot_meta=boot,
+                    installed_kb=installed_kb,
+                    installed_note=installed_note,
+                    round_number=None,
+                    live=(boot["index"] == 0),
+                )
+            )
+        payload = {
+            "schema": SCHEMA,
+            "host": os.uname().nodename,
+            "harvested_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "journal_storage": storage,
+            "smbios_installed_kb": installed_kb,
+            "boots_listed": len(boots),
+            "notes": [n for n in [boots_note, installed_note] if n],
+            "boots": records,
+        }
+        print(json.dumps(payload, indent=2, sort_keys=True))
+        return 0
+
+    current = next((b for b in boots if b["index"] == 0), None)
+    record = build_record(
+        boot_selector="0",
+        boot_meta=current,
+        installed_kb=installed_kb,
+        installed_note=installed_note,
+        round_number=args.round,
+        live=True,
+    )
+    record["journal_storage"] = storage
+    if boots_note:
+        record["notes"].append(boots_note)
+    if args.append:
+        note = append_record(args.append, record)
+        if note:
+            record["notes"].append(note)
+    print(json.dumps(record, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

infra/ansible/roles/memattr_survey/tasks/analyse.yml

@@ -0,0 +1,42 @@
+---
+# Controller-side analysis. Runs on localhost over the artifact directory, so it is
+# independent of whether any surveyed node is still alive — and so it can be re-run on
+# its own (`--tags analyse`) without touching hardware.
+#
+# The statistics live in files/memattr_analyse.py, not in Jinja: Clopper-Pearson and
+# Fisher belong somewhere testable, and a confidence interval assembled out of template
+# filters is a confidence interval nobody will ever check.
+
+- name: Locate the survey artifacts
+  ansible.builtin.find:
+    paths: "{{ memattr_survey_artifact_dir }}"
+    patterns: "*.boots.json"
+  register: memattr_survey_artifacts
+
+- name: Report that there is nothing to analyse
+  ansible.builtin.debug:
+    msg: >-
+      No *.boots.json under {{ memattr_survey_artifact_dir }} — no survey has produced a
+      harvest yet. Run the playbook with -e memattr_survey_reboots_authorised=true.
+  when: memattr_survey_artifacts.matched | int == 0
+
+- name: Analyse the survey
+  ansible.builtin.command:
+    argv:
+      - /usr/bin/env
+      - python3
+      - "{{ role_path }}/files/memattr_analyse.py"
+      - "--artifact-dir"
+      - "{{ memattr_survey_artifact_dir }}"
+      - "--rounds"
+      - "{{ memattr_survey_rounds }}"
+      - "--out"
+      - "{{ memattr_survey_artifact_dir }}/summary.md"
+  register: memattr_survey_analysis
+  changed_when: false
+  when: memattr_survey_artifacts.matched | int > 0
+
+- name: Show the analysis
+  ansible.builtin.debug:
+    msg: "{{ memattr_survey_analysis.stdout.split('\n') }}"
+  when: memattr_survey_analysis is not skipped

infra/ansible/roles/memattr_survey/tasks/harvest.yml

@@ -0,0 +1,66 @@
+---
+# The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from
+# scratch in one pass by the same code that produced the per-round ledger.
+#
+# WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and
+# the append, that round exists on the machine but not in the ledger. The journal does not
+# care whether the driver was alive — the boot happened, the kernel logged it, journald
+# persisted it. So the harvest can only ever be a superset of the ledger, and the analysis
+# joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and
+# any boot the ledger missed still shows up.
+#
+# This runs even for a node that aborted mid-survey, which is exactly when it matters most.
+
+- name: Harvest every retained boot from the persistent journal
+  ansible.builtin.script:
+    cmd: memattr_probe.py --all-boots
+    executable: /usr/bin/python3
+  register: memattr_survey_harvest_raw
+  changed_when: false   # strictly a read of the journal
+
+- name: Parse the harvest
+  ansible.builtin.set_fact:
+    memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}"
+
+- name: Ensure the artifact directory exists (on the controller)
+  ansible.builtin.file:
+    path: "{{ memattr_survey_artifact_dir }}"
+    state: directory
+    mode: "0755"
+  delegate_to: localhost
+  become: false
+  # NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and
+  # run_once is unreliable there. `state: directory` is idempotent, so three hosts each
+  # asserting the directory exists is cheaper than a conditional that might not fire.
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Write the harvested boot record
+  ansible.builtin.copy:
+    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json"
+    mode: "0644"
+    content: "{{ memattr_survey_harvest | to_nice_json }}"
+  delegate_to: localhost
+  become: false
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Mirror the final ledger to the controller
+  ansible.builtin.fetch:
+    src: "{{ memattr_survey_state_file }}"
+    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
+    flat: true
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Harvest summary
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots
+      from the journal
+      ({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad,
+      {{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good,
+      {{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other)
+      {%- if memattr_survey_aborted | default(false) | bool %}
+      — NOTE: this node ABORTED ({{ memattr_survey_abort_reason }})
+      {%- endif %}

infra/ansible/roles/memattr_survey/tasks/main.yml

@@ -0,0 +1,23 @@
+---
+# memattr_survey — see defaults/main.yml for the question, README.md for how to run and
+# read it, and infra/PROD-PROVISIONING.md §0b for the settled mechanism this measures.
+#
+# ORDER OF OPERATIONS
+#   1. preflight.yml — refuse to touch a node that is not prod, not idle, or has no
+#      persistent journal; read the resume point; record the boot we found it on.
+#   2. round.yml x N — reboot, measure, append, mirror. Per-node fail-safe.
+#   3. harvest.yml   — re-derive every retained boot from the journal (authoritative).
+#
+# The playbook splits these across plays so that preflight is any_errors_fatal (one unsafe
+# node stops everything) while the rounds are not (one dead node stops only itself), and
+# so that the controller-side analysis runs even if every node dies. This file is the
+# whole-role entry point kept for completeness; the playbook calls the stages directly.
+
+- name: Preflight and baseline
+  ansible.builtin.include_tasks: preflight.yml
+
+- name: Run the survey rounds
+  ansible.builtin.include_tasks: rounds.yml
+
+- name: Harvest the authoritative record
+  ansible.builtin.include_tasks: harvest.yml

infra/ansible/roles/memattr_survey/tasks/preflight.yml

@@ -0,0 +1,149 @@
+---
+# Safety gate. Runs with any_errors_fatal at the play level (playbooks/memattr-survey.yml),
+# so a violation on ANY node stops the entire playbook before a single reboot — the
+# correct posture when the failure mode is "rebooted a machine that was doing something".
+#
+# VERIFY, DO NOT ASSUME. The premise that makes these reboots safe is that prod carries no
+# k3s and runs nothing. That premise is checked here on every run rather than trusted from
+# a document, because the document and the machine drift independently.
+
+- name: Assert the target host is in the prod group
+  # The dev cluster is LIVE. This is the one check that a mistyped --limit or a copied
+  # command line cannot slip past: membership is read from the inventory, not the pattern.
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_required_group in (group_names | default([]))
+    fail_msg: >-
+      REFUSING TO SURVEY {{ inventory_hostname }}: it is not a member of
+      '{{ memattr_survey_required_group }}' (groups: {{ group_names | default([]) | join(', ') }}).
+      This role reboots its targets. Point it at prod, or override
+      memattr_survey_required_group deliberately.
+    quiet: true
+
+- name: Check whether any cluster or container runtime unit is active
+  # `is-active` returns non-zero for inactive/unknown units, which is the expected answer
+  # here, so the rc is data and not a failure.
+  ansible.builtin.command:
+    argv: "{{ ['systemctl', 'is-active'] + memattr_survey_forbidden_units }}"
+  register: memattr_survey_units
+  changed_when: false
+  failed_when: false
+
+- name: Check whether k3s has been installed on this node
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  loop: "{{ memattr_survey_forbidden_paths }}"
+  register: memattr_survey_paths
+  changed_when: false
+
+- name: Assert nothing is running on this node
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_units.stdout_lines | reject('equalto', 'inactive') | reject('equalto', 'unknown') | list | length == 0
+      - memattr_survey_paths.results | selectattr('stat.exists') | list | length == 0
+    fail_msg: >-
+      REFUSING TO REBOOT {{ inventory_hostname }} — it is not idle.
+      units {{ memattr_survey_forbidden_units | zip(memattr_survey_units.stdout_lines) | map('join', '=') | join(' ') }};
+      present paths: {{ memattr_survey_paths.results | selectattr('stat.exists') | map(attribute='item') | list | join(', ') | default('none', true) }}.
+      The survey is only sanctioned because these boxes carry no k3s and run nothing.
+    quiet: true
+
+- name: Clear any abort marker from a previous run
+  # A marker left by a run that has since been resumed successfully would make this run's
+  # report describe the LAST failure rather than this one. The marker means "this run
+  # aborted", so it is cleared at the start of every run and re-written only if it does.
+  ansible.builtin.file:
+    path: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
+    state: absent
+  delegate_to: localhost
+  become: false
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Assert forks allows every node to be surveyed concurrently
+  # The survey play runs `strategy: free` so the nodes reboot in parallel, but forks is a
+  # hard cap on that: with fewer forks than hosts the survey quietly serialises and takes
+  # a multiple of the wall clock it should. Cheap to satisfy (`-f <n>`), so assert rather
+  # than discover it three hours later.
+  ansible.builtin.assert:
+    that:
+      - (lookup('ansible.builtin.config', 'DEFAULT_FORKS') | int) >= (ansible_play_hosts_all | length)
+    fail_msg: >-
+      forks={{ lookup('ansible.builtin.config', 'DEFAULT_FORKS') }} but this survey targets
+      {{ ansible_play_hosts_all | length }} hosts. The rounds would serialise. Re-run with
+      -f {{ ansible_play_hosts_all | length }} (or higher).
+    quiet: true
+  run_once: true
+
+- name: Read the node's resume state and baseline boot
+  # One call does double duty: it reports how far a previous run got (so this one resumes)
+  # and, with --round 0, records the boot we found the node on. Round 0 is a free data
+  # point — it is a boot nobody chose, so it is not selected for by the survey.
+  ansible.builtin.script:
+    cmd: >-
+      memattr_probe.py --status {{ memattr_survey_state_file | quote }}
+    executable: /usr/bin/python3
+  register: memattr_survey_status_raw
+  changed_when: false
+
+- name: Record the resume point
+  ansible.builtin.set_fact:
+    memattr_survey_status: "{{ memattr_survey_status_raw.stdout | from_json }}"
+    memattr_survey_aborted: false
+    memattr_survey_abort_reason: ""
+
+- name: Assert the probe schema is the one this role understands
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_status.schema == 'memattr-survey/1'
+    fail_msg: >-
+      Probe returned schema {{ memattr_survey_status.schema | default('<none>') }};
+      this role understands memattr-survey/1. Role and probe are out of step.
+    quiet: true
+
+- name: Probe the boot the node is currently on
+  # On a FIRST run this is stamped round 0 and appended: the boot the node was already
+  # sitting on is a free, unselected data point. On a RESUME it is read but NOT appended,
+  # because that boot is already round N of the interrupted run and double-counting it
+  # would bias the sample toward whatever outcome happened to survive the interruption.
+  ansible.builtin.script:
+    cmd: >-
+      memattr_probe.py
+      {% if memattr_survey_status.completed_rounds | int == 0 %}
+      --round 0 --append {{ memattr_survey_state_file | quote }}
+      {% endif %}
+    executable: /usr/bin/python3
+  register: memattr_survey_baseline_raw
+  # The append is the experiment's own ledger, not a change to the node under test: this
+  # role reports `changed` only for things that could affect a boot, and a file written
+  # under /var/lib long after the kernel parsed the table cannot.
+  changed_when: false
+
+- name: Parse the baseline record
+  ansible.builtin.set_fact:
+    memattr_survey_baseline: "{{ memattr_survey_baseline_raw.stdout | from_json }}"
+
+- name: Assert the journal is PERSISTENT
+  # The end-of-run harvest is the authoritative record precisely because it re-derives
+  # every boot from the journal. On a volatile journal (/run/log/journal only) each
+  # reboot erases the evidence and the harvest would silently return one boot — a survey
+  # that looks complete and holds nothing. Verified, not assumed.
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_baseline.journal_storage.persistent | bool
+    fail_msg: >-
+      {{ inventory_hostname }} has NO persistent journal (/var/log/journal missing).
+      Every reboot would erase the record this survey depends on. Refusing to run.
+    quiet: true
+
+- name: Report the starting state
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: idle, prod, persistent journal.
+      kernel {{ memattr_survey_baseline.kernel_release | default('?') }},
+      installed {{ (memattr_survey_baseline.smbios_installed_kb | default(0) | int / 1048576) | round(1) }} GiB.
+      baseline boot MEMATTR={{ memattr_survey_baseline.memattr_address | default('none') }}
+      kernel_read={{ memattr_survey_baseline.kernel_read | default('?') }}
+      verdict={{ memattr_survey_baseline.verdict | default('?') | upper }}.
+      resuming from round {{ memattr_survey_status.completed_rounds }}
+      of {{ memattr_survey_rounds }}.

infra/ansible/roles/memattr_survey/tasks/round.yml

@@ -0,0 +1,98 @@
+---
+# ONE round on ONE node: reboot, wait for it back, measure the new boot, record it twice.
+# Included in a loop from tasks/main.yml with loop_var `memattr_survey_round`.
+#
+# FAIL-SAFE, PER NODE. The block/rescue below is what keeps one dead node from ending the
+# survey: on any failure this host sets memattr_survey_aborted and every later round is
+# skipped FOR THIS HOST ONLY, while the other nodes keep going (Ansible runs the loop for
+# all hosts in lockstep, so the reboots are genuinely concurrent). The abort is reported
+# loudly here AND again in the final analysis, which counts rounds from the artifacts
+# rather than from Ansible's host state — so a node that vanishes so hard that it is
+# marked unreachable (which rescue cannot catch) still shows up as a short arm.
+
+- name: "Round {{ memattr_survey_round }} — survey step"
+  block:
+    - name: "Round {{ memattr_survey_round }} — reboot and wait for the node to return"
+      ansible.builtin.reboot:
+        reboot_timeout: "{{ memattr_survey_reboot_timeout | int }}"
+        connect_timeout: "{{ memattr_survey_connect_timeout | int }}"
+        post_reboot_delay: "{{ memattr_survey_post_reboot_delay | int }}"
+        msg: "memattr survey round {{ memattr_survey_round }} of {{ memattr_survey_rounds }}"
+
+    - name: "Round {{ memattr_survey_round }} — measure the new boot and append to the ledger"
+      ansible.builtin.script:
+        cmd: >-
+          memattr_probe.py --round {{ memattr_survey_round }}
+          --append {{ memattr_survey_state_file | quote }}
+        executable: /usr/bin/python3
+      register: memattr_survey_round_raw
+      changed_when: false   # see the note in preflight.yml — the ledger is not the node
+
+    - name: "Round {{ memattr_survey_round }} — parse the record"
+      ansible.builtin.set_fact:
+        memattr_survey_record: "{{ memattr_survey_round_raw.stdout | from_json }}"
+
+    - name: "Round {{ memattr_survey_round }} — result"
+      ansible.builtin.debug:
+        msg: >-
+          [{{ inventory_hostname }} r{{ memattr_survey_round }}/{{ memattr_survey_rounds }}]
+          MEMATTR={{ memattr_survey_record.memattr_address | default('none') }}
+          read={{ memattr_survey_record.kernel_read }}
+          {%- if memattr_survey_record.memattr_table_version is not none %}
+          (version {{ memattr_survey_record.memattr_table_version }})
+          {%- endif %}
+          MemTotal={{ (memattr_survey_record.mem_total_kb | default(0) | int / 1048576) | round(2) }}G
+          of {{ (memattr_survey_record.smbios_installed_kb | default(0) | int / 1048576) | round(0) | int }}G
+          => {{ memattr_survey_record.verdict | upper }}
+          boot={{ memattr_survey_record.boot_id | default('?') | truncate(9, true, '') }}
+      when: memattr_survey_show_progress | bool
+
+    # BRACES. The on-node ledger already survives a reboot; re-fetching it every round
+    # means the controller also holds every completed round if the DRIVER dies, without
+    # waiting for the end-of-run harvest. One tiny file per host, per-host destination
+    # paths, so three concurrent fetches cannot collide.
+    - name: "Round {{ memattr_survey_round }} — mirror the ledger to the controller"
+      ansible.builtin.fetch:
+        src: "{{ memattr_survey_state_file }}"
+        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
+        flat: true
+      changed_when: false
+      when: memattr_survey_write_artifacts | bool
+
+  rescue:
+    - name: "Round {{ memattr_survey_round }} — ABORT this node, continue the others"
+      ansible.builtin.set_fact:
+        memattr_survey_aborted: true
+        memattr_survey_abort_reason: >-
+          round {{ memattr_survey_round }} failed:
+          {{ ansible_failed_result.msg | default('unknown error') | trim }}
+
+    # The abort must survive the run, not live only in terminal scrollback. Without
+    # this the report can see that an arm is SHORT but not WHY, and "0 rounds" reads
+    # like a targeting mistake instead of a node that failed to come back.
+    - name: "Round {{ memattr_survey_round }} — record the abort as an artifact"
+      ansible.builtin.copy:
+        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
+        mode: "0644"
+        content: "{{ memattr_survey_abort_record | to_nice_json }}"
+      vars:
+        memattr_survey_abort_record:
+          schema: "memattr-survey/1"
+          host: "{{ inventory_hostname }}"
+          aborted_at_round: "{{ memattr_survey_round }}"
+          target_rounds: "{{ memattr_survey_rounds }}"
+          reason: "{{ memattr_survey_abort_reason | trim }}"
+          failed_task: "{{ ansible_failed_task.name | default('unknown') }}"
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      when: memattr_survey_write_artifacts | bool
+
+    - name: "Round {{ memattr_survey_round }} — announce the abort"
+      ansible.builtin.debug:
+        msg: >-
+          *** SURVEY ABORTED ON {{ inventory_hostname }} at round
+          {{ memattr_survey_round }}/{{ memattr_survey_rounds }}:
+          {{ memattr_survey_abort_reason }}
+          *** The other nodes continue. This node's arm is SHORT — the analysis will
+          say so, and its p(bad) interval will be correspondingly wider.

infra/ansible/roles/memattr_survey/tasks/rounds.yml

@@ -0,0 +1,40 @@
+---
+# The loop itself, kept separate from round.yml so the loop bounds (which encode the
+# RESUME logic) are readable in one place.
+#
+# RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a
+# re-run after an interruption continues where it stopped and can never lose or duplicate
+# an earlier round. Already at or past the target count => nothing to do, and the run
+# collapses to a harvest.
+#
+# NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which
+# makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest,
+# analysis) still runs, so a dry run exercises the entire pipeline without touching a
+# single machine.
+
+- name: Announce the plan
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}:
+      {%- if not (memattr_survey_reboots_authorised | bool) %}
+      DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm).
+      {%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %}
+      already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do.
+      {%- else %}
+      rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }}
+      to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots).
+      {%- endif %}
+
+- name: Survey round
+  ansible.builtin.include_tasks: round.yml
+  loop: >-
+    {{ range(memattr_survey_status.completed_rounds | int + 1,
+             memattr_survey_rounds | int + 1) | list }}
+  loop_control:
+    loop_var: memattr_survey_round
+    label: "{{ inventory_hostname }} round {{ memattr_survey_round }}"
+  # Evaluated per iteration, so the abort flag set by a rescue in one round skips every
+  # round after it on THIS host and no others.
+  when:
+    - memattr_survey_reboots_authorised | bool
+    - not (memattr_survey_aborted | bool)

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -18,22 +18,193 @@ node_baseline_packages:
   # node from first baseline — package installation is the role's job, never a manual step
   # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
   - smartmontools
+# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
+# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
+# baseline — not a one-off `apt install` someone has to remember on the next node.
+#
+# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
+# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
+# from its contents, so an unlucky boot reserves everything above the table and the node
+# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
+# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
+# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
+# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
+# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
+# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+#
+# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
+# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
+# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
+# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
+# encode the environment into the ROLE, would silently miss a fourth prod node named
+# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
+# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
+# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
+#
+# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
+# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
+# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
+# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
+#
+# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
+# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
+# delete this block. Nothing else in the repo depends on it.
+node_baseline_backports_kernel: false
+# The suite, mirror and components the source line is built from. Split out so a mirror
+# change is a variable, not an edit to a heredoc in tasks/main.yml.
+node_baseline_backports_suite: trixie-backports
+node_baseline_backports_mirror: "http://deb.debian.org/debian"
+# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
+# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
+# surface beyond what the node already trusts is a separate decision nobody has taken.
+node_baseline_backports_components:
+  - main
+  - non-free-firmware
+# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
+# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
+# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
+# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
+# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
+# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
+# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
+node_baseline_backports_kernel_packages:
+  - linux-image-amd64
+# Provenance/rollback note written on the node. An operator standing in front of a prod box
+# with no repo checkout must be able to answer "why is this machine not on the distro
+# kernel, and how do I undo it?" from the machine itself.
+node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
+
 node_baseline_kernel_modules:
   - overlay
   - br_netfilter
   - iscsi_tcp
   - dm_crypt
   - nfs
+
+# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
+# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
+# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
+#
+#   node_baseline_longhorn_source:
+#     device           dedicated block device at node_baseline_longhorn_device (DEV: the
+#                      tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
+#     claim-mountpoint re-purpose the partition the node ALREADY has mounted at
+#                      node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
+#                      path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
+#                      and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
+#                      is correct and a hardcoded one would eventually mkfs the wrong
+#                      partition on a future node. Destructive; heavily guarded — read
+#                      tasks/disks.yml before changing anything here.
+#
+#   node_baseline_etcd_source:
+#     device                dedicated block device at node_baseline_etcd_device (DEV:
+#                           /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
+#                           isolated from Longhorn rebuild I/O after dev was burned by HDD
+#                           contention).
+#     root-filesystem       etcd lives on the OS filesystem, with NO dedicated device and
+#                           NO claimed data partition under it. Kept as a valid, supported
+#                           shape; NOT what prod uses any more (see below).
+#     shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
+#                           CLAIMED data partition (PROD). No dedicated device, but also NOT
+#                           the OS filesystem: the whole point of the 2026-09-04 layout
+#                           revision is that /var (11.8 GiB) can hold neither the container
+#                           image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
+#
+# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
+# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
+# var, which still fails the `disks` assert.
+node_baseline_longhorn_source: device
 node_baseline_longhorn_device: /dev/vdb
-node_baseline_longhorn_mount: /var/lib/longhorn
+# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
-node_baseline_longhorn_fs_label: longhorn
+# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
+# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
+# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
+# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
+#
+# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
+# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
+# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
+# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
+# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
+node_baseline_data_mount: /var/lib/longhorn
+node_baseline_data_fs_label: longhorn
+# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
+# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
+# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
+# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
+# same fact, and the two are checked against each other by nothing but this comment.
+node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
+# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
+# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
+# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
+# install flags, this role reads the same value to create the directory and to place etcd,
+# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
+# value (higher precedence than any role default) is what actually resolves here, even
+# though roles/k3s_server is not in the node-baseline play; the default() keeps
+# node-baseline.yml runnable standalone.
+node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
+# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
+# default so a dev host cannot enter the claim path even by accident.
+node_baseline_longhorn_claim_mountpoint: ""
+# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
+# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
+# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
+# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
+# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
+# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
+# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
+node_baseline_data_retired_fs_label: ""
+node_baseline_data_retired_mount: ""
+# Sanity floor for the claim path: a data partition smaller than this is not "a small
+# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
+node_baseline_longhorn_min_size_gb: 100
 # Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
 # so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
 # See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
-# device renames (matches the Longhorn LABEL= approach).
+# device renames (matches the data partition's LABEL= approach).
+node_baseline_etcd_source: device
 node_baseline_etcd_device: /dev/vdc
-node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
+# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
+# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
+# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
+# to /data/k3s/server/db, inside the claimed partition.
+node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
 node_baseline_etcd_fs_label: k3s-etcd
+# Mounts whose backing devices the claim path must NEVER touch. Used to build the
+# protected-device set from the node's own gathered facts (plus live swap), so a node
+# partitioned differently from the fleet is judged on its own layout rather than against
+# a hardcoded list of partition names.
+node_baseline_protected_mounts:
+  - /
+  - /boot
+  - /boot/efi
+  - /usr
+  - /var
+  - /home
+  - /tmp
+
+# --- WIRELESS (tasks/main.yml, tag `wireless`) ---------------------------------------
+# A CLUSTER NODE GETS EXACTLY ONE PATH ONTO THE NETWORK. ON by default — unlike the
+# backports kernel above, which is a prod-only deviation, this is a cluster-wide policy and
+# it is a provable NO-OP where it does not apply: the tasks act only on interfaces the node
+# reports as real 802.11 devices (a `phy80211` link in sysfs), and dev's Proxmox VMs have
+# none, so on dev nothing is written, nothing is unloaded and nothing changes. Defaulting it
+# true is what makes a FOURTH node with a wifi card inherit the policy from being a cluster
+# node, rather than from someone remembering to set a flag.
+#
+# WHY IT EXISTS: k3s-prod-3 came up on both eno1 (192.168.2.12, the inventory address) and
+# wlp1s0 (192.168.2.195, DHCP on the same /24). The wifi lease's dhcpcd run rewrote
+# /etc/resolv.conf three minutes after this role wrote it, and a second address on the LAN
+# where kube-vip floats 192.168.2.9 and Cilium L2-announces .20-.30 is a problem in its own
+# right. See the `wireless` block in tasks/main.yml for the full reasoning, including why the
+# DRIVER MODULE is the layer that gets blocked rather than the link, a service, or rfkill.
+#
+# SET IT FALSE for a host that must genuinely keep wifi (there is none today). That does NOT
+# undo an already-applied block — delete the file below on the node as well.
+node_baseline_disable_wireless: true
+# Where the persistent block lives. 99- so it is read last and overrides any earlier
+# permissive rule; the name says what it does, on a node with no repo checkout to hand.
+node_baseline_wireless_modprobe_file: /etc/modprobe.d/99-k3s-no-wireless.conf
 
 # --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
 # These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml

infra/ansible/roles/node_baseline/tasks/disks.yml

@@ -0,0 +1,642 @@
+---
+# =====================================================================================
+# node_baseline — DATA DISKS.  Every task here carries the `disks` tag (applied once by
+# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
+# Split into its own file rather than living in main.yml because it is the one part of
+# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
+#
+# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
+#
+#   dev  (group_vars/all.yml, role defaults)
+#     node_baseline_longhorn_source: device          -> mkfs /dev/vdb, mount LABEL=longhorn
+#                                                       at /var/lib/longhorn (== dataPath)
+#     node_baseline_etcd_source:     device          -> mkfs /dev/vdc, mount LABEL=k3s-etcd
+#     k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
+#     THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
+#     exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
+#
+#   prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
+#     node_baseline_longhorn_source: claim-mountpoint  (claim mountpoint /srv)
+#     node_baseline_data_mount:      /data            (LABEL=k3s-data)
+#     node_baseline_etcd_source:     shared-data-partition
+#     k3s_data_dir:                  /data/k3s
+#     ONE physical disk per node, already fully partitioned by the Debian installer. The
+#     ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
+#     between TWO tenants:  /data/longhorn (Longhorn replicas)  +  /data/k3s (the whole k3s
+#     data dir: etcd AND the containerd image store).
+#
+# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
+# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
+# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
+# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
+# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
+# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
+# partition is the fix; the partition therefore serves two tenants, and neither its
+# mountpoint nor its filesystem label may name only one of them.
+#
+# WHY /data AND LABEL k3s-data.
+#   * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
+#     Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
+#     at, and the idempotency contract is that /srv is gone from /proc/mounts and
+#     /etc/fstab once the claim has happened. Re-using it as the destination would make
+#     "claimed" and "not yet claimed" indistinguishable.
+#   * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
+#     accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
+#     `data` as an example of somebody else's named volume. Labelling ours `data` would make
+#     a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
+#     cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
+#     convention in this repo.
+#
+# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
+# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
+# empty string, and the second one must never silently succeed. `root-filesystem` and
+# `shared-data-partition` are explicit, greppable statements of intent that the assert
+# below accepts; an unset or placeholder device is still a hard failure. The two differ in
+# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
+# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
+# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
+# than printing a claim it never checked (see the etcd section).
+#
+# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
+# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
+# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
+# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
+# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
+# point at the wrong partition on some future node and mkfs it. So the role asks the NODE
+# which block device is mounted at the claim mountpoint, and then proves that answer is
+# safe (see the assert wall below) before touching it. The inventory declares INTENT
+# ("the partition currently mounted at /srv"); the node supplies the FACT.
+# =====================================================================================
+
+# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
+# its data-disk layout. Two failure shapes are caught:
+#   * an undeclared/typo'd source enum, and
+#   * source: device with a device path that is a PLACEHOLDER rather than a real device.
+# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
+# vars now that prod uses neither: if anyone ever flips a source back to `device` without
+# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
+# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
+# stays in the token list because that was the pre-decision placeholder and may still be
+# carried by other environments/branches.
+- name: Assert this environment's data-disk layout has actually been declared
+  ansible.builtin.assert:
+    that:
+      - node_baseline_longhorn_source in ['device', 'claim-mountpoint']
+      - node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
+      # `shared-data-partition` is only coherent when there IS a claimed data partition to
+      # share, and only when the k3s data dir actually sits INSIDE it. Catch the
+      # contradiction here, in inventory terms, rather than three sections later when etcd
+      # has silently landed back on the OS filesystem.
+      - >-
+        node_baseline_etcd_source != 'shared-data-partition'
+        or node_baseline_longhorn_source == 'claim-mountpoint'
+      - >-
+        node_baseline_etcd_source != 'shared-data-partition'
+        or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
+      # The transition knobs are a matched PAIR: a retired label with no retired mountpoint
+      # (or vice versa) cannot describe a state to migrate from.
+      - >-
+        (node_baseline_data_retired_fs_label | length > 0)
+        == (node_baseline_data_retired_mount | length > 0)
+      - >-
+        node_baseline_longhorn_source != 'device'
+        or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
+      - >-
+        node_baseline_etcd_source != 'device'
+        or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
+      - >-
+        node_baseline_longhorn_source != 'claim-mountpoint'
+        or (node_baseline_longhorn_claim_mountpoint | length) > 0
+    fail_msg: >-
+      This host's data-disk layout is not declared.
+      longhorn_source={{ node_baseline_longhorn_source }}
+      (device={{ node_baseline_longhorn_device }},
+      claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
+      etcd_source={{ node_baseline_etcd_source }}
+      (device={{ node_baseline_etcd_device }}).
+      Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
+      `claim-mountpoint` (re-purpose the partition already mounted at
+      node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
+      `device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
+      it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
+      data partition — prod's shape since 2026-09-04, which additionally requires
+      longhorn_source `claim-mountpoint` and a k3s data dir under
+      node_baseline_data_mount={{ node_baseline_data_mount }};
+      k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
+      operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
+      Declare it in the environment's group_vars and record the reasoning in
+      §2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
+      `--tags packages`, which skips every disk task including this assert.
+    quiet: true
+
+# =====================================================================================
+# THE DATA PARTITION  (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
+# =====================================================================================
+
+# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
+# rather than a reformat. Once the claim has happened the partition carries the ext4
+# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
+# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
+# mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
+# "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
+# path means converged, rc != 0 means the claim has not been made. Everything destructive
+# below is gated on rc != 0.
+- name: "Data partition: look for a filesystem already carrying the data-partition label"
+  ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
+  register: node_baseline_lh_labelled
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+# --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
+# WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
+# label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
+# takes over — which then fails closed at "the claim mountpoint must actually be a
+# mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
+# behaviour but it is not convergence, and the prod nodes are in exactly that state today:
+# claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
+# So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
+#
+# WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
+# partition with the new label. It never touches the DATA and never reformats: e2label is a
+# metadata write, exactly as in the claim path below. The common mount task then mounts
+# LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
+# `blkid -L <new label>` succeeds and none of this is reachable.
+#
+# THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
+# volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
+# unmounting it would orphan every one of them:
+#   1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
+#      group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
+#   2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
+#      the same structural gate that keeps dev out of the claim path keeps it out of this one.
+#   3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
+#      replica directories on it fails closed and nothing is unmounted.
+# Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
+- name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
+  ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
+  register: node_baseline_lh_retired_labelled
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when:
+    - node_baseline_longhorn_source == 'claim-mountpoint'
+    - node_baseline_data_retired_fs_label | length > 0
+    - node_baseline_lh_labelled.rc != 0
+
+- name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
+  ansible.builtin.set_fact:
+    # A node claimed under the RETIRED label. Mutually exclusive with the claim below.
+    node_baseline_lh_transition_needed: >-
+      {{ node_baseline_longhorn_source == 'claim-mountpoint'
+         and node_baseline_data_retired_fs_label | length > 0
+         and node_baseline_lh_labelled.rc != 0
+         and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
+    # True only for a claim-mountpoint environment that has NOT been claimed under EITHER
+    # label. A `device` environment (dev) can never enter the claim path at all — that is
+    # the structural guarantee that no generalisation of prod's behaviour can reach the
+    # LIVE dev nodes and their /dev/vdb Longhorn replicas.
+    node_baseline_lh_claim_needed: >-
+      {{ node_baseline_longhorn_source == 'claim-mountpoint'
+         and node_baseline_lh_labelled.rc != 0
+         and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
+
+- name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_data_retired_mount }}
+  register: node_baseline_lh_retired_findmnt
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_transition_needed | bool
+
+# Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
+# not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
+# an unmount of a populated Longhorn store would orphan every replica on the node.
+- name: "Data partition (transition): read the contents of the retired mountpoint"
+  ansible.builtin.find:
+    paths: "{{ node_baseline_data_retired_mount }}"
+    file_type: any
+    hidden: true
+    recurse: false
+    excludes: ['lost+found']
+  register: node_baseline_lh_retired_contents
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): prove this node is safe to re-label"
+  ansible.builtin.assert:
+    that:
+      # 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
+      #    retired label names. The only state this role knows how to transition is the one
+      #    the previous version of it produced; anything else is an operator decision.
+      - (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
+      - >-
+        (node_baseline_lh_retired_findmnt.stdout | default('') | trim)
+        == (node_baseline_lh_retired_labelled.stdout | default('') | trim)
+      # 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
+      #    not the node we think it is, or a tenant has already written data — in both cases
+      #    unmounting and re-labelling under it is destructive and the role must stop.
+      - (node_baseline_lh_retired_contents.files | length) == 0
+    fail_msg: >-
+      REFUSING to transition this node's data partition from
+      LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
+      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
+      device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
+      device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
+      (findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
+      non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
+      {{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
+      If the partition holds DATA, this transition would orphan it — Longhorn replicas are
+      not recoverable from an unmounted, re-labelled filesystem. Migrate the data
+      deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
+      Do NOT relax this assert.
+    quiet: true
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): report exactly what is about to change"
+  ansible.builtin.debug:
+    msg: >-
+      Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
+      from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
+      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
+      unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
+      No filesystem is created and no data is moved.
+  when: node_baseline_lh_transition_needed | bool
+
+# e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
+# breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
+- name: "Data partition (transition): unmount the retired mountpoint"
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_retired_mount }}"
+    state: unmounted
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_retired_mount }}"
+    state: absent_from_fstab
+    backup: true
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): re-stamp the partition with the new filesystem label"
+  ansible.builtin.command: >-
+    e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
+    {{ node_baseline_data_fs_label }}
+  changed_when: true
+  when: node_baseline_lh_transition_needed | bool
+
+# --- The claim (prod, first run only) -------------------------------------------------
+# Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
+# mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
+# returning the device behind `/` — which is precisely the mistake that would hand the
+# root partition to e2label.
+- name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
+  register: node_baseline_lh_claim_findmnt
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
+  ansible.builtin.assert:
+    that:
+      - node_baseline_lh_claim_findmnt.rc == 0
+      - (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
+    fail_msg: >-
+      node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
+      yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
+      this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
+      stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
+      The role will NOT go looking for "a big partition" to claim instead. Either the
+      node is partitioned differently from the fleet — in which case fix the node or give
+      this host its own declaration — or the claim already happened and the label was
+      removed by hand.
+    quiet: true
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): inspect the candidate partition"
+  ansible.builtin.command: >-
+    lsblk --json --nodeps --bytes
+    --output PATH,TYPE,FSTYPE,LABEL,SIZE
+    {{ node_baseline_lh_claim_findmnt.stdout | trim }}
+  register: node_baseline_lh_claim_lsblk
+  changed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+# The devices this role must never, under any circumstances, hand to e2label. Built from
+# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
+# NODE's account of what is load-bearing, not a hardcoded list of partition names.
+- name: "Longhorn (claim): list the devices currently backing active swap"
+  ansible.builtin.command: swapon --noheadings --show=NAME
+  register: node_baseline_lh_swapdevs
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+# Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
+# ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
+- name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
+  ansible.builtin.find:
+    paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    file_type: any
+    hidden: true
+    recurse: false
+    excludes: ['lost+found']
+  register: node_baseline_lh_claim_contents
+  when: node_baseline_lh_claim_needed | bool
+
+# THE GUARD WALL. Five independent conditions, every one of which must hold before the
+# role modifies a filesystem it did not create. Any single failure stops the play with
+# the evidence in the message — nothing is unmounted, relabelled or mounted first.
+- name: "Longhorn (claim): prove the candidate partition is safe to claim"
+  ansible.builtin.assert:
+    that:
+      # 1. It is a real block device — a partition or a whole disk. Catches a claim
+      #    mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
+      #    which would have produced a non-/dev SOURCE or an lsblk row of another type.
+      - node_baseline_lh_candidate.type in ['part', 'disk']
+      # 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
+      #    so a differently-partitioned node is caught by its own facts.
+      - node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
+      # 3. It already holds ext4. The claim path RELABELS rather than reformats (see
+      #    below), which is only meaningful for the filesystem type we are about to
+      #    declare in fstab. Anything else is an operator decision, not a role's.
+      - node_baseline_lh_candidate.fstype == 'ext4'
+      # 4. It carries no filesystem label, or already carries ours. A partition labelled
+      #    `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
+      #    it would make it unfindable by its own fstab entry.
+      - (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
+      # 5. It is big enough to be the partition we meant. A data partition of a few GB is
+      #    not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
+      - (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
+      # 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
+      #    ext4 5% root reserve; this is where that premise is verified per node instead
+      #    of trusted. If any node has real data on it, the play stops here — the role
+      #    never wipes data it cannot account for.
+      - (node_baseline_lh_claim_contents.files | length) == 0
+    fail_msg: >-
+      REFUSING to claim {{ node_baseline_lh_candidate.path }}
+      (mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
+      type={{ node_baseline_lh_candidate.type }},
+      fstype={{ node_baseline_lh_candidate.fstype }},
+      label='{{ node_baseline_lh_candidate.label | default('', true) }}',
+      size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
+      (minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
+      non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
+      {{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
+      devices this node needs and that must never be claimed:
+      {{ node_baseline_lh_protected_devices }}.
+      This is the destructive step of the role and it fails CLOSED. If the partition
+      holds data, back it up or pick a different one; do NOT relax this assert.
+    quiet: true
+  vars:
+    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
+    # The node's OWN account of what it cannot afford to lose: the devices behind the
+    # mounts in node_baseline_protected_mounts (from gathered facts, so a node
+    # partitioned differently is judged on its own layout) plus every active swap device.
+    # Built here rather than hardcoded so no partition-name list can go stale.
+    node_baseline_lh_protected_devices: >-
+      {{ (ansible_facts.mounts | default([])
+          | selectattr('mount', 'in', node_baseline_protected_mounts)
+          | map(attribute='device') | list)
+         + (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): report exactly what is about to be claimed"
+  ansible.builtin.debug:
+    msg: >-
+      Claiming {{ node_baseline_lh_candidate.path }}
+      ({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
+      currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
+      as the data partition: unmount -> drop its fstab entry ->
+      relabel LABEL={{ node_baseline_data_fs_label }} ->
+      mount at {{ node_baseline_data_mount }}.
+  vars:
+    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
+  when: node_baseline_lh_claim_needed | bool
+
+# --- Why the role must unmount and relabel, not just point fstab at the device ---------
+# The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
+# /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
+# path does not. The installer-created /srv filesystem has NO label (fstab pins it by
+# UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
+# the unmount first — and the fstab entry must go in the same breath, or the next boot
+# re-mounts the same partition at /srv underneath Longhorn.
+#
+# RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
+# mkfs would buy nothing and would destroy anything the emptiness check got wrong.
+# e2label is a metadata write: if some assert above was too permissive, the worst case is
+# a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
+# cannot do this job: it declines to touch a device that already has a filesystem, so it
+# would silently no-op and then the LABEL= mount would fail with no useful message.)
+- name: "Longhorn (claim): unmount the claim mountpoint"
+  ansible.posix.mount:
+    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    state: unmounted
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
+  ansible.posix.mount:
+    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    state: absent_from_fstab
+    backup: true
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
+  ansible.builtin.command: >-
+    e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
+    {{ node_baseline_data_fs_label }}
+  changed_when: true
+  when: node_baseline_lh_claim_needed | bool
+
+# --- The dedicated-device path (dev) --------------------------------------------------
+# Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
+# never reformats a device that already carries a filesystem.
+- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_longhorn_device }}"
+    opts: "-L {{ node_baseline_data_fs_label }}"
+  when: node_baseline_longhorn_source == 'device'
+
+# --- Common to ALL paths (device, claim, transition) ----------------------------------
+# Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
+# mount task. Idempotent: on a converged node this reports ok, which is what makes a
+# re-run a no-op rather than a reformat. It is also what completes the transition above:
+# the retired mountpoint is gone from fstab and the partition now answers to the new label,
+# so this task mounts it at the new path.
+- name: Mount the data partition
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_mount }}"
+    src: "LABEL={{ node_baseline_data_fs_label }}"
+    fstype: ext4
+    opts: "defaults,nofail,x-systemd.device-timeout=5s"
+    state: mounted
+
+# --- The per-tenant subdirectories ---------------------------------------------------
+# WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
+# tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
+# subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
+# /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
+# either there would at best be a no-op and at worst re-mode the running dev cluster's data
+# dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
+# mounted — i.e. only the directories it is actually responsible for. That test is derived
+# from the paths themselves, so no environment has to remember to switch it off.
+#
+# MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
+# the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
+# etcd database, the node token and the containerd image store, and k3s would tighten it to
+# 0700 itself on first start — doing it here means it is never briefly world-readable.
+- name: Create the per-tenant data directories on the data partition
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
+    - { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
+  loop_control:
+    label: "{{ item.path }} ({{ item.tenant }})"
+  when: item.path.startswith(node_baseline_data_mount ~ '/')
+
+# =====================================================================================
+# ETCD DATA DIR
+# =====================================================================================
+
+# --- The dedicated-device path (dev) --------------------------------------------------
+# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
+# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
+# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
+# a disk that already carries a filesystem, so re-running against an already-migrated
+# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
+# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
+# role would mount over a populated directory.
+- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_etcd_device }}"
+    opts: "-L {{ node_baseline_etcd_fs_label }}"
+  when: node_baseline_etcd_source == 'device'
+
+- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
+  ansible.posix.mount:
+    path: "{{ node_baseline_etcd_mount }}"
+    src: "LABEL={{ node_baseline_etcd_fs_label }}"
+    fstype: ext4
+    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
+    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
+    opts: "defaults,noatime"
+    dump: "0"
+    passno: "2"
+    state: mounted
+  when: node_baseline_etcd_source == 'device'
+
+# --- The no-dedicated-device paths (prod) ---------------------------------------------
+# Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
+# populated — so etcd gets no device of its own. That is an operator decision
+# (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
+# UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
+# on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
+# latency rather than collapsing it. Accepted to be MEASURED, not assumed.
+#
+# WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
+# filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
+# 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
+# 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
+# could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
+# partition and etcd moved with it: `shared-data-partition`. Both enum values are still
+# supported, and the role now PROVES which one is true instead of printing a claim.
+#
+# There is nothing to CREATE here for either shape — the data-partition section above
+# already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
+# role's job is to verify the assumption rather than act on it.
+- name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_etcd_mount }}
+  register: node_baseline_etcd_existing_mount
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+- name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
+  ansible.builtin.assert:
+    that:
+      - node_baseline_etcd_existing_mount.rc != 0
+    fail_msg: >-
+      node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
+      NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
+      ALREADY a separate mount, backed by
+      {{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
+      contradiction the role will not resolve on its own: either this node was previously
+      built with a dedicated etcd device (in which case unmount it deliberately, after
+      accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
+      the node in a state where the declared layout and the real one disagree.
+    quiet: true
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+# Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
+# WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
+# existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
+# that does exist — the same filesystem the data dir will be created on. TARGET is captured
+# separately from the human-readable line because the assert below judges on it.
+- name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
+  ansible.builtin.shell: |
+    set -eu
+    p={{ node_baseline_etcd_mount | quote }}
+    while [ ! -d "$p" ]; do p=$(dirname "$p"); done
+    findmnt --noheadings --first-only --output TARGET --target "$p"
+    findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
+  register: node_baseline_etcd_backing_fs
+  changed_when: false
+  check_mode: false
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+# THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
+# `shared-data-partition` is a distinct enum value rather than a comment. Declaring
+# k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
+# above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
+# of container images would land back on the OS filesystem — the exact failure this change
+# exists to prevent, and completely invisible until the disk filled. So the role checks that
+# the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
+# run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
+# which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
+# whole record for that shape.
+- name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
+  ansible.builtin.assert:
+    that:
+      - >-
+        (node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
+        == node_baseline_data_mount
+    fail_msg: >-
+      node_baseline_etcd_source is `shared-data-partition`, which asserts that
+      {{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
+      {{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
+      '{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
+      (full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
+      The data partition is therefore NOT mounted where this layout requires, and k3s would
+      put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
+      filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
+      image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
+      this node until the mount is correct.
+    quiet: true
+  when: node_baseline_etcd_source == 'shared-data-partition'
+
+- name: "etcd (no dedicated device): show which filesystem etcd will live on"
+  ansible.builtin.debug:
+    msg: >-
+      etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
+      (source={{ node_baseline_etcd_source }}); it lands on
+      {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
+      (SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
+      measured risk — §2.
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1,6 +1,6 @@
 ---
-# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
+# TAGS — every task here carries one of `packages`, `kernel`, `system`, `wireless`, `disks` or
-# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
+# `dns`. An untagged run is unchanged (all six run, in this order). The split exists because the PACKAGE floor
 # and the DATA-DISK work have different readiness: packages are safe on any node from
 # first contact, while the disk tasks need block devices that a new environment may not
 # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -12,6 +12,16 @@
 # mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
 # at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
 #   ansible-playbook playbooks/node-baseline.yml --tags dns
+# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
+# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
+# it must be runnable without touching a disk decision that is still open:
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
+# `wireless` (a cluster node gets exactly ONE path onto the network) is separated for a third
+# reason: it must be runnable BEFORE `dns` and without touching packages, the kernel or a disk
+# decision, because a node with a second DHCP-managed interface reverts the `dns` write. It is
+# placed before the disk work for the same reason, and the pair is the whole remedy for the
+# k3s-prod-3 incident:
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags wireless,dns
 
 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
   ansible.builtin.apt:
@@ -21,6 +31,237 @@
     cache_valid_time: 3600
   tags: [packages]
 
+# =====================================================================================
+# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
+# =====================================================================================
+# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
+# FUTURE prod node gets it from provisioning instead of from someone remembering.
+#
+# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
+# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
+# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
+# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
+# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
+# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
+# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
+# (first released in v6.13) validates the size and rejects the table instead. It is absent
+# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
+# stable feed will ever pick it up. There is NO kernel cmdline workaround —
+# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
+# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
+# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
+#
+# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
+# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
+# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
+# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
+# change for no benefit. See defaults/main.yml for why this is a variable rather than a
+# `k3s-prod-*` hostname test.
+#
+# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
+# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
+# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
+# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
+# before touching the next, never node 2 first — and a role cannot enforce that. The last
+# task here reports the pending reboot loudly instead of letting a green play imply the node
+# is already fixed.
+- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
+  when: node_baseline_backports_kernel | bool
+  tags: [kernel]
+  block:
+
+    # The suite name and the running release must agree, or apt resolves no candidate and
+    # `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
+    # a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
+    # dist-upgraded to forky while group_vars still says trixie-backports).
+    - name: Assert the configured backports suite matches this node's Debian release
+      ansible.builtin.assert:
+        that:
+          - ansible_facts['distribution'] == 'Debian'
+          - node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
+        fail_msg: >-
+          node_baseline_backports_kernel is enabled but node_baseline_backports_suite
+          ({{ node_baseline_backports_suite }}) is not the backports suite for this node's
+          release ({{ ansible_facts['distribution'] }}
+          {{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
+          candidate and this play would report success while leaving the node on the
+          vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
+        quiet: true
+
+    # deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
+    # prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
+    # rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
+    # idempotent in a way `apt_repository` — which rewrites and reformats — is not.
+    #
+    # THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
+    # backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
+    # apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
+    # 500 for a package that was explicitly installed from there so it keeps getting updates.
+    # Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
+    # HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
+    # be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
+    - name: Enable the Debian backports apt source (deb822)
+      ansible.builtin.copy:
+        dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
+          # A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
+          # machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
+          # Table its firmware publishes (upstream 8fbe4c49c0cc). See
+          # infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
+          # on this node.
+          #
+          # Backports is NotAutomatic, so nothing is taken from here unless asked for by name
+          # with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
+          # ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
+          # Pin-Priority would switch that off.
+          Types: deb
+          URIs: {{ node_baseline_backports_mirror }}
+          Suites: {{ node_baseline_backports_suite }}
+          Components: {{ node_baseline_backports_components | join(' ') }}
+          Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
+      register: node_baseline_backports_source
+
+    - name: Refresh the apt cache after enabling backports
+      ansible.builtin.apt:
+        update_cache: true
+      when: node_baseline_backports_source is changed
+
+    # `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
+    # nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
+    # nothing at all — the single most likely way to write this task and have it silently fix
+    # nothing forever. `latest` + `default_release` resolves the candidate inside backports
+    # and upgrades the metapackage to it. Once the node is current a re-run is a genuine
+    # no-op: no change, no churn, no spurious reboot flag. Scope is exactly
+    # node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
+    # is dragged out of the main archive.
+    - name: Install the fixed kernel from backports (>= 6.13)
+      ansible.builtin.apt:
+        name: "{{ node_baseline_backports_kernel_packages }}"
+        state: latest
+        default_release: "{{ node_baseline_backports_suite }}"
+        update_cache: true
+        cache_valid_time: 3600
+      register: node_baseline_backports_kernel_apt
+
+    # WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
+    # metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
+    # answer anybody can roll back from six months later.
+    - name: Read back the installed backports kernel metapackage version
+      ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
+      register: node_baseline_backports_kernel_version
+      changed_when: false
+      check_mode: false
+
+    # `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
+    # neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
+    # list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
+    # GRUB's "Advanced options" without uninstalling anything.
+    - name: List every kernel image installed on this node, newest first
+      ansible.builtin.shell:
+        cmd: set -o pipefail; linux-version list | linux-version sort --reverse
+        executable: /bin/bash
+      register: node_baseline_installed_kernels
+      changed_when: false
+      check_mode: false
+
+    # The node must be able to explain itself to an operator standing in front of it with no
+    # repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
+    # role is a real no-op rather than a permanent daily "changed".
+    - name: Record the deviation, the exact version, and the rollback path on the node
+      ansible.builtin.copy:
+        dest: "{{ node_baseline_backports_kernel_stamp }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
+          #
+          # THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
+          #
+          # WHY
+          #   This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
+          #   Linux < 6.13 reserves memory from it without validating its size, and an unlucky
+          #   boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
+          #   k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
+          #   is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
+          #   v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
+          #   Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
+          #
+          # PROOF THE FIX IS ACTIVE (not merely a lucky boot)
+          #   BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
+          #   >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
+          #     zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
+          #   (roles/node_health does the version check for you; see the health report's
+          #   firmware.efi_memattr finding.)
+          #
+          #   FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
+          #   and each prints a DIFFERENT line — all three are rejections:
+          #     tbl->version > 2         'Unexpected EFI Memory Attributes table version %d'
+          #                              (NOTE: printed with %d, so garbage may read NEGATIVE)
+          #     bad desc_size            'Unexpected EFI Memory Attributes descriptor size ...'
+          #     num_entries > 64K        '[Firmware Bug]: Corrupted EFI Memory Attributes
+          #                               Table detected! (version == ..., num_entries == ...)'
+          #   Which one appears depends on WHICH field this boot's garbage corrupted, so do
+          #   NOT treat the absence of the FW_BUG line as the fix being absent — all three
+          #   prod nodes printed the *version* line on 2026-09-04.
+          #   The one line that means the fix is NOT active:
+          #     'Failed to map EFI Memory Attributes table @ 0x...'  — the kernel trusted the
+          #     corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
+          #     escalate, do not shrug.
+          #
+          # INSTALLED
+          #   source   /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
+          #   package  {{ node_baseline_backports_kernel_packages | join(' ') }}
+          #   version  {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
+          #   kernels installed on this node, newest first:
+          {% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
+          #     {{ kv }}
+          {% endfor %}
+          #
+          # SECURITY COST
+          #   Packages from *-backports are provided AS IS. They are NOT covered by the Debian
+          #   Security Team's guarantees for the main archive: no security-tracker commitment,
+          #   no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
+          #
+          # ROLLBACK — no repo checkout needed
+          #   1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
+          #      and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
+          #      GNU/Linux". That answers "is the new kernel the problem?" without
+          #      uninstalling anything. Note it reinstates the memattr bug for that boot.
+          #   2. PERMANENT. Set node_baseline_backports_kernel: false in
+          #      inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
+          #      puts the backports kernel straight back — then, on this node:
+          #        apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
+          #        apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
+          #        rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
+          #        apt-get update && update-grub && reboot
+          #
+          # EXIT CONDITION
+          #   If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
+          #   rebase), this deviation can be removed entirely: flip the variable to false,
+          #   roll back as above, and delete the `kernel` block from roles/node_baseline.
+
+    # Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
+    # it means the fixed kernel is on disk. See the block header for why the role will not
+    # reboot on its own.
+    - name: Report whether this node is running the kernel it now has installed
+      ansible.builtin.debug:
+        msg: >-
+          {{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
+              ~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
+              prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
+              node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
+              with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
+             if newest != ansible_facts['kernel']
+             else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
+                   ~ '); no reboot pending for the memattr fix.') }}
+      vars:
+        newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
+
 - name: Persist kernel modules for k3s/Cilium/Longhorn
   ansible.builtin.copy:
     dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
@@ -79,70 +320,330 @@
     state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
   tags: [system]
 
-# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
+# =====================================================================================
-# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
+# WIRELESS — tag `wireless`: a cluster node gets exactly ONE path onto the network
-# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
+# =====================================================================================
-# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
+# WHAT WENT WRONG (k3s-prod-3, reported by the operator 2026-09-04). The node was up on BOTH
-# which do not exist on bare metal either — a confusing failure that looks like a wiring
+# eno1 (192.168.2.12 — the inventory address) and wlp1s0 (192.168.2.195, DHCP, the SAME /24).
-# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
+# The wifi lease carried a dhcpcd run with it, and dhcpcd's `20-resolv.conf` hook REWROTE
-# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
+# /etc/resolv.conf, undoing the resolvers this role had just written: the file read
-- name: Assert this environment's data-disk devices have actually been decided
+# "# Generated by dhcpcd from wlp1s0.dhcp, wlp1s0.dhcp6, wlp1s0.ra" with nameserver
+# 192.168.2.1 plus two Google IPv6 servers, timestamped THREE MINUTES AFTER the role's own
+# write (01:32 -> 01:35). The role had reported `changed` and green. That is exactly the
+# "successful run that fixed nothing" the `contended` branch of the DNS block below exists to
+# prevent, and the detection blind spot that let it through is closed there.
+#
+# WHY THE INTERFACE IS REMOVED RATHER THAN JUST REPORTED. Closing the detection hole alone
+# would have turned a silent revert into a loud failure — an improvement, but it leaves the
+# node with a second, DHCP-managed path onto the LAN, and on a k3s server that is three
+# distinct problems beyond DNS:
+#   * a SECOND DEFAULT ROUTE, whose metric decides which source address the node uses;
+#   * a SECOND ADDRESS ON THE SAME L2 — the segment where kube-vip floats 192.168.2.9 and
+#     Cilium L2-announces 192.168.2.20-.30 (§4). One host answering ARP on one subnet from
+#     two NICs is the exact shape neither mechanism can reason about;
+#   * an UNMANAGED address: nothing in this repo knows about .195, and the DHCP server is
+#     free to hand out a different one after any lease expiry.
+# None of that is a resolver bug, and none of it is fixed by writing resolv.conf harder.
+#
+# WHY THE DRIVER LAYER — the four alternatives, and why each is weaker. What has to hold is:
+# survives a REBOOT, survives a LEASE RENEWAL, and cannot be undone by a routine `ifup`.
+# Measured on the nodes themselves — all three are HP EliteDesk 800 G5 minis, eno1 is
+# `e1000e`, the wifi NIC is an Intel AX200 on `iwlwifi`, and the OS is Debian 13 trixie with
+# ifupdown + `networking.service`, NO NetworkManager, NO systemd-networkd, NO systemd-resolved:
+#   `ip link set <wifi> down` — NOT persistent. Any `ifup`, any hotplug event, any reboot
+#       undoes it. It is what had already been done by hand on node 3, and the stale dhcpcd
+#       /etc/resolv.conf outlived it — the link was down and the wrong resolvers were still
+#       in the file, across a reboot.
+#   mask the service that manages it — THERE IS NO SUCH SERVICE HERE. dhcpcd on these nodes
+#       is `dhcpcd-base`: no dhcpcd.service exists at all (`systemctl is-enabled dhcpcd` =
+#       not-found), it is spawned per-interface by ifupdown. The only unit that can bring
+#       wireless up is `networking.service` — which ALSO owns eno1's static address, so
+#       masking it would cut the one SSH path to a machine whose operator is not on site.
+#   `rfkill` soft-block — weaker on every axis. The netdev still EXISTS, so ifupdown and
+#       dhcpcd can still be pointed at it; persistence depends on systemd-rfkill restoring
+#       saved state (the kernel's own `persistent` flag reads 0 on these devices, and the
+#       `rfkill` CLI is not even installed); and one `rfkill unblock wlan` undoes it.
+#   the BIOS `M.2 WLAN/BT` toggle — genuinely the deepest layer, and it reads `Enable` on all
+#       three nodes (hp-bioscfg survey, §1b). NOT taken here: it needs a REBOOT to take
+#       effect, and a prod reboot is a serialized, supervised operator step (§0b "SECOND
+#       FAULT" — k3s-prod-2 hung on a warm reboot and needed the power button). It also
+#       disables Bluetooth on the same M.2 card. Left as an operator recommendation in §4.
+#   THE DRIVER MODULE — chosen. `blacklist` plus `install <module> /bin/false` in
+#       /etc/modprobe.d means kmod refuses to load the wifi driver on this boot and every
+#       future one, so THE NETDEV NEVER EXISTS. No interface means no DHCP lease, no
+#       resolv.conf hook, no second route, no second address — and every re-activation path
+#       above (udev hotplug, a hand-run `ifup`, wpa_supplicant, a hand-run dhcpcd) needs a
+#       device that is not there. `blacklist` alone would still allow a dependency- or
+#       hand-triggered `modprobe`; the `install ... /bin/false` line is what makes it refuse
+#       outright. It is ONE declarative file, readable on the node, and undone by deleting
+#       it. Paired with the runtime unload below, it needs NO reboot to take effect.
+#
+# SAFE AND IDEMPOTENT ON ALL THREE PROD NODES *AND* ON THE LIVE DEV CLUSTER. Nothing here is
+# hostname- or hardware-specific: the block acts only on interfaces the node itself reports as
+# real 802.11 devices (a `phy80211` link in sysfs) and on the driver modules behind them.
+#   * dev (Proxmox VMs, virtio NICs) has no such interface, so the discovery comes back empty,
+#     every task below is skipped, and NOT EVEN THE modprobe.d FILE IS WRITTEN. The block
+#     cannot touch dev's networking, which is why it is safe to have on by default.
+#   * k3s-prod-1 (wlp1s0) and k3s-prod-2 (wlp2s0) already have the link DOWN, so the link-down
+#     task is skipped by its own `when` and only the modprobe.d file and the module unload
+#     remain — and both are no-ops on a re-run.
+#   * k3s-prod-3 (wlp1s0) is the reported node.
+# A re-run changes nothing on any of them: the file is content-identical, `ip link set down`
+# is gated on the link not already being down, and the unload is gated on the module still
+# being loaded.
+#
+# AND IT REFUSES TO CUT ITS OWN PATH IN. The assert below stops the play if the address
+# Ansible is connected to is bound to a wireless interface, or if the default route leaves
+# over one, or if the wireless driver module is also the driver of the default-route
+# interface. That is the one case where doing this would lock everybody out of a machine
+# nobody can walk up to, so it is checked per node, on every run, from that node's own facts.
+- name: Survey this node's 802.11 devices, their drivers and any live DHCP client
+  ansible.builtin.shell:
+    executable: /bin/sh
+    cmd: |
+      set -u
+      # Flat KEY|value lines so the Ansible side needs one single-group regex per fact and
+      # no parsing cleverness. A wireless interface is one with a phy80211 link in sysfs —
+      # that is the kernel's own answer to "is this 802.11", not a name pattern like wlp*.
+      wifi=''
+      for d in /sys/class/net/*; do
+        [ -e "$d/phy80211" ] || continue
+        i=$(basename "$d")
+        wifi="$wifi $i"
+        m=$(basename "$(readlink -f "$d/device/driver" 2>/dev/null || echo unknown)")
+        state=$(cat "$d/operstate" 2>/dev/null || echo unknown)
+        printf 'IFACE|%s\n' "$i"
+        printf 'DETAIL|%s driver=%s operstate=%s\n' "$i" "$m" "$state"
+        # LIVE means "not already down" — the link-down task keys off this, so a node whose
+        # radio is already down reports no change.
+        [ "$state" = down ] || printf 'IFACE_LIVE|%s\n' "$i"
+        ip -o addr show dev "$i" 2>/dev/null | awk '{print "IFACE_ADDR|" $4}'
+        [ "$m" = unknown ] || printf 'MODULE|%s\n' "$m"
+      done
+      for m in $(printf '%s\n' $wifi | while read -r i; do
+            [ -n "$i" ] && basename "$(readlink -f "/sys/class/net/$i/device/driver" 2>/dev/null || echo unknown)"
+          done | sort -u); do
+        [ "$m" = unknown ] && continue
+        [ -d "/sys/module/$m" ] || continue
+        printf 'MODULE_LOADED|%s\n' "$m"
+        # Holders must be unloaded FIRST: `modprobe -r iwlwifi` fails while iwlmvm still
+        # holds it. Read from sysfs rather than assumed, so a different wifi driver stack
+        # (ath*, rtw*, mt76*) works without editing this role.
+        for h in $(ls "/sys/module/$m/holders" 2>/dev/null); do printf 'MODULE_HOLDER|%s\n' "$h"; done
+      done
+      dr=$(ip -o route show default 2>/dev/null | awk '{for (i = 1; i < NF; i++) if ($i == "dev") print $(i + 1)}' | head -1)
+      printf 'DEFROUTE_IFACE|%s\n' "${dr:-none}"
+      if [ -n "${dr:-}" ] && [ -e "/sys/class/net/$dr" ]; then
+        printf 'DEFROUTE_MODULE|%s\n' "$(basename "$(readlink -f "/sys/class/net/$dr/device/driver" 2>/dev/null || echo unknown)")"
+      fi
+      # Any DHCP client process, with the interfaces it was started for. Used only to stop a
+      # client that holds a WIRELESS interface and nothing else.
+      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
+        printf 'DHCPCLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
+      done
+      exit 0
+  register: node_baseline_wifi_scan
+  changed_when: false
+  check_mode: false
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Interpret the wireless survey
+  ansible.builtin.set_fact:
+    node_baseline_wifi_ifaces: "{{ scan | regex_findall('(?m)^IFACE\\|(.+)$') }}"
+    node_baseline_wifi_live: "{{ scan | regex_findall('(?m)^IFACE_LIVE\\|(.+)$') }}"
+    node_baseline_wifi_addrs: "{{ scan | regex_findall('(?m)^IFACE_ADDR\\|(\\S+)') | map('regex_replace', '/.*$', '') | list }}"
+    node_baseline_wifi_modules: "{{ scan | regex_findall('(?m)^MODULE\\|(.+)$') | unique }}"
+    node_baseline_wifi_modules_loaded: "{{ scan | regex_findall('(?m)^MODULE_LOADED\\|(.+)$') | unique }}"
+    node_baseline_wifi_module_holders: "{{ scan | regex_findall('(?m)^MODULE_HOLDER\\|(.+)$') | unique }}"
+    node_baseline_wifi_detail: "{{ scan | regex_findall('(?m)^DETAIL\\|(.+)$') }}"
+    node_baseline_wifi_dhcp_clients: "{{ scan | regex_findall('(?m)^DHCPCLIENT\\|(.+)$') }}"
+    node_baseline_defroute_iface: "{{ (scan | regex_findall('(?m)^DEFROUTE_IFACE\\|(.+)$') | first) | default('none', true) }}"
+    node_baseline_defroute_module: "{{ (scan | regex_findall('(?m)^DEFROUTE_MODULE\\|(.+)$') | first) | default('unknown', true) }}"
+  vars:
+    scan: "{{ node_baseline_wifi_scan.stdout | default('') }}"
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Report this node's wireless hardware
+  ansible.builtin.debug:
+    msg: >-
+      wireless devices = {{ node_baseline_wifi_detail | default([]) | join('; ') | default('(none)', true) }}
+      | driver modules = {{ node_baseline_wifi_modules | default([]) | join(', ') | default('(none)', true) }}
+      (loaded: {{ node_baseline_wifi_modules_loaded | default([]) | join(', ') | default('(none)', true) }},
+      held by: {{ node_baseline_wifi_module_holders | default([]) | join(', ') | default('(none)', true) }})
+      | default route via {{ node_baseline_defroute_iface | default('?') }}
+      (driver {{ node_baseline_defroute_module | default('?') }})
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+# The one way this block could do real harm is by removing the interface it is being run
+# over. Checked from the node's own facts, per node, every run — not assumed from the fact
+# that prod "should" be wired.
+- name: Refuse to disable wireless if the path to this node runs OVER the wireless interface
   ansible.builtin.assert:
     that:
-      - "'BLOCKED' not in node_baseline_longhorn_device"
+      - node_baseline_defroute_iface not in node_baseline_wifi_ifaces
-      - "'BLOCKED' not in node_baseline_etcd_device"
+      - node_baseline_defroute_module not in node_baseline_wifi_modules
+      - (ansible_host | default(inventory_hostname)) not in node_baseline_wifi_addrs
     fail_msg: >-
-      node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
+      REFUSING to disable wireless on this node: the connection to it appears to BE the
-      placeholders for this host's environment
+      wireless path. ansible_host={{ ansible_host | default(inventory_hostname) }}, wireless
-      (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
+      interfaces={{ node_baseline_wifi_ifaces | join(', ') }} carrying
-      The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
+      {{ node_baseline_wifi_addrs | join(', ') | default('no addresses', true) }}, default route
-      and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
+      via {{ node_baseline_defroute_iface }} (driver {{ node_baseline_defroute_module }}).
-      the SMART read that decision waits on can be taken) re-run this play with
+      Disabling the driver here would drop this node off the network with no remote way back.
-      `--tags packages`, which skips every disk task including this assert.
+      Give the node a wired address, point the inventory at it, and re-run.
     quiet: true
-  tags: [disks]
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_ifaces | length > 0
+  tags: [wireless]
 
-- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
+# THE PERSISTENT HALF. This file is the whole reboot story: kmod consults /etc/modprobe.d on
-  community.general.filesystem:
+# every load attempt, so from here on the driver cannot be loaded by udev at boot, as a
-    fstype: ext4
+# dependency, or by hand. Written ONLY when the node actually has a wireless driver, so a
-    dev: "{{ node_baseline_longhorn_device }}"
+# node without wifi hardware (every dev VM) gets no file and no change at all.
-    opts: "-L {{ node_baseline_longhorn_fs_label }}"
+# `update-initramfs` is deliberately NOT run: no wifi module is loaded from the initramfs on
-  tags: [disks]
+# these nodes (root is a local SATA/NVMe device), the driver is autoloaded by udev after
+# switch-root, and re-generating an initramfs on a prod node is a bigger risk than the one it
+# would cover.
+- name: Block the wireless driver module(s) from ever loading again
+  ansible.builtin.copy:
+    dest: "{{ node_baseline_wireless_modprobe_file }}"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # Managed by the Ansible node_baseline role (tags: wireless) — do not edit by hand.
+      #
+      # WHY THIS NODE HAS NO WIRELESS. A k3s server gets exactly ONE path onto the network.
+      # A second, DHCP-managed interface on the same LAN gives the node a second default
+      # route, a second address nothing in the cluster's config knows about, and a DHCP
+      # client whose resolv.conf hook silently overwrites the resolvers Ansible manages
+      # (that is what happened on k3s-prod-3, 2026-09-04). The link is also brought down and
+      # the module unloaded at the same time, so no reboot is needed for this to take effect.
+      #
+      # `blacklist` stops udev's alias-driven autoload; the `install ... /bin/false` line is
+      # what also refuses a dependency-triggered or hand-run `modprobe`. Both are needed.
+      #
+      # TO RE-ENABLE WIRELESS ON THIS NODE: delete this file and `modprobe` the module(s)
+      # below (or reboot). To stop Ansible putting it back, set
+      # node_baseline_disable_wireless: false for this host first — otherwise the next
+      # node-baseline run recreates it. Full rationale: roles/node_baseline/tasks/main.yml,
+      # the `wireless` block, and infra/PROD-PROVISIONING.md §4.
+      {% for module in node_baseline_wifi_modules %}
+      blacklist {{ module }}
+      install {{ module }} /bin/false
+      {% endfor %}
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_modules | length > 0
+  tags: [wireless]
 
-- name: Mount the Longhorn data disk
+# THE RUNTIME HALF — everything below makes the change true NOW, so the node does not have to
-  ansible.posix.mount:
+# be rebooted to stop carrying a second address (this role never reboots: §0b "SECOND FAULT").
-    path: "{{ node_baseline_longhorn_mount }}"
+#
-    src: "LABEL={{ node_baseline_longhorn_fs_label }}"
+# Stop a DHCP client that is holding a wireless interface first, so it releases its lease
-    fstype: ext4
+# instead of being left holding a device that is about to disappear. HEAVILY GUARDED: it runs
-    opts: "defaults,nofail,x-systemd.device-timeout=5s"
+# only for clients whose command line names a wireless interface and names NO non-wireless
-    state: mounted
+# interface, because `dhcpcd -x` against a shared master process would also drop a wired
-  tags: [disks]
+# lease. On these prod nodes eno1 is a static ifupdown address with no DHCP client at all, so
+# this task has nothing to do; it exists so the ordering is correct on a node where wifi is
+# still actively leasing.
+- name: Stop the dhcpcd instance that holds a wireless interface (and nothing else)
+  ansible.builtin.command:
+    argv: [dhcpcd, --exit, "{{ item }}"]
+  loop: "{{ node_baseline_wifi_ifaces }}"
+  register: node_baseline_wifi_dhcpcd_exit
+  changed_when: rc == 0
+  failed_when: false
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_dhcp_wifi_only | length > 0
+    - node_baseline_wifi_dhcp_wifi_only | select('search', 'dhcpcd') | list | length > 0
+  vars:
+    # A client is "wireless-only" when its argv mentions a wireless interface and mentions no
+    # other interface this node has. Empty interface sets are turned into a pattern that
+    # cannot match, so an odd node fails towards doing nothing rather than towards guessing.
+    node_baseline_non_wifi_ifaces: "{{ ansible_facts['interfaces'] | default([]) | difference(node_baseline_wifi_ifaces) | difference(['lo']) }}"
+    node_baseline_wifi_dhcp_wifi_only: >-
+      {{ node_baseline_wifi_dhcp_clients
+         | select('search', '\b(' ~ ((node_baseline_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
+         | reject('search', '\b(' ~ ((node_baseline_non_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
+         | list }}
+  tags: [wireless]
 
-# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
+- name: Take each wireless link down now
-# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
+  ansible.builtin.command:
-# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
+    argv: [ip, link, set, dev, "{{ item }}", down]
-# a disk that already carries a filesystem, so re-running against an already-migrated
+  loop: "{{ node_baseline_wifi_live }}"
-# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
+  changed_when: true
-# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
+  when: node_baseline_disable_wireless | bool
-# role would mount over a populated directory.
+  tags: [wireless]
-- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
-  community.general.filesystem:
-    fstype: ext4
-    dev: "{{ node_baseline_etcd_device }}"
-    opts: "-L {{ node_baseline_etcd_fs_label }}"
-  tags: [disks]
 
-- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
+# Unload the driver so the netdev is gone immediately — holders (iwlmvm on these nodes) first,
-  ansible.posix.mount:
+# then the driver itself, because `modprobe -r` refuses a module that is still held.
-    path: "{{ node_baseline_etcd_mount }}"
+# NOT fatal on failure: the persistent block above is already in place, so a module that
-    src: "LABEL={{ node_baseline_etcd_fs_label }}"
+# cannot be unloaded right now is gone at the next boot regardless. The report below says so
-    fstype: ext4
+# out loud rather than letting a green play imply the radio is already off.
-    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
+- name: Unload the wireless driver module(s) now (no reboot needed)
-    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
+  community.general.modprobe:
-    opts: "defaults,noatime"
+    name: "{{ item }}"
-    dump: "0"
+    state: absent
-    passno: "2"
+  loop: "{{ node_baseline_wifi_module_holders + node_baseline_wifi_modules_loaded }}"
-    state: mounted
+  register: node_baseline_wifi_unload
+  failed_when: false
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Report the wireless state this node is left in
+  ansible.builtin.debug:
+    msg: >-
+      {{ 'No 802.11 device on this node — nothing to disable (no modprobe.d file written).'
+         if (node_baseline_wifi_ifaces | length) == 0
+         else ((('WIRELESS DISABLED: ' ~ (node_baseline_wifi_modules | join(', ')) ~ ' blocked in '
+                 ~ node_baseline_wireless_modprobe_file ~ ' (survives reboot) and unloaded now; '
+                 ~ (node_baseline_wifi_ifaces | join(', ')) ~ ' is down and the netdev is gone.')
+                if (failed_unloads | length) == 0
+                else ('WIRELESS BLOCKED FOR THE NEXT BOOT BUT STILL LOADED NOW: could not unload '
+                      ~ (failed_unloads | join(', ')) ~ '. ' ~ node_baseline_wireless_modprobe_file
+                      ~ ' is in place, so the driver will not load again after a reboot, but the '
+                      ~ 'radio is live until then — check `ip link` and whether something still '
+                      ~ 'holds the device.'))) }}
+  vars:
+    failed_unloads: >-
+      {{ (node_baseline_wifi_unload.results | default([]))
+         | selectattr('failed', 'defined') | selectattr('failed') | map(attribute='item') | list }}
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+# =====================================================================================
+# DATA DISKS — tag `disks`, in tasks/disks.yml
+# =====================================================================================
+# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
+# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
+# should be readable end-to-end on its own; and both environments' storage shapes — dev's
+# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
+# the OS filesystem — are declared there as an explicit enum rather than inferred from
+# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
+#
+# The `disks` tag is applied ONCE here, in BOTH places it is needed:
+#   * `tags:` on the include itself, so `--tags packages` never even reaches the include
+#     and a placeholder device path is not so much as templated; and
+#   * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
+#     optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
+#     its own tags to its children the way `import_tasks` does, so without `apply` the
+#     included tasks are untagged and `--tags disks` selects the include and then runs
+#     NOTHING inside it. That failure is silent and reports success, which is the worst
+#     shape a guard can have.
+# A FULL run against a node whose layout is undeclared stops on the assert at the top of
+# that file instead of failing obscurely at mkfs time.
+- name: Provision the data disks (Longhorn store, etcd data dir)
+  ansible.builtin.include_tasks:
+    file: disks.yml
+    apply:
+      tags: [disks]
   tags: [disks]
 
 - name: Disable swap now
@@ -180,6 +681,11 @@
 # node and the matching seam is written. Adding a third mechanism means adding a branch
 # here, not rewriting the role.
 #
+# AND DETECTION IS ONLY AS GOOD AS THE SET OF OWNERS IT KNOWS ABOUT. It originally looked for
+# resolvconf/openresolv and NetworkManager, missed dhcpcd, and therefore reported a green,
+# `changed` run on k3s-prod-3 whose resolvers a wifi lease reverted three minutes later. dhcpcd
+# is now part of that set — see the detection task below and the `contended` failure.
+#
 # All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
 # documented at the top of this file) still installs nothing but packages.
 
@@ -199,6 +705,55 @@
   check_mode: false
   tags: [dns]
 
+# THE dhcpcd BLIND SPOT, CLOSED (k3s-prod-3, 2026-09-04). dhcpcd's `20-resolv.conf` hook
+# rewrites /etc/resolv.conf on every lease, renewal and carrier change, and it is a THIRD
+# way to own that file — neither resolvconf nor NetworkManager. On node 3 it silently reverted
+# this role's write three minutes later, and because nothing below looked for it the role had
+# already reported `changed` and green: the precise failure the `contended` branch exists to
+# make impossible.
+#
+# WHY THIS LOOKS FOR A LIVE CLIENT AND NOT FOR AN INSTALLED PACKAGE. `dhcpcd-base` is
+# installed on all three prod nodes and always will be — it is the DHCP client ifupdown
+# reaches for — and there is NO dhcpcd.service on them at all (`systemctl is-active dhcpcd` =
+# not-found; it is spawned per-interface by ifupdown). Keying off the package, or off a unit
+# that does not exist, would put every prod node permanently in `contended` and leave the DNS
+# block unable to configure a node it is perfectly safe to configure. What actually matters is
+# whether a DHCP client CAN still write the file, so all three real signals are checked:
+#   CLIENT| a dhcpcd/dhclient process running right now (with the argv that started it);
+#   UNIT|   a dhcpcd/dhclient unit that IS active or IS enabled, on a distro that has one;
+#   STANZA| an UNCOMMENTED `iface <x> inet[6] dhcp` in /etc/network/interfaces{,.d} — the
+#           configuration that makes ifupdown spawn a client at the next boot or `ifup`, even
+#           when none is running at this instant.
+# Any one of them means "another manager is in play" and the play stops below. None of them
+# means no client can run, and the file is genuinely ours to own.
+- name: Determine whether a DHCP client (dhcpcd/dhclient) can rewrite /etc/resolv.conf
+  ansible.builtin.shell:
+    executable: /bin/sh
+    cmd: |
+      set -u
+      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
+        printf 'CLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
+      done
+      for u in dhcpcd dhcpcd5 dhclient isc-dhcp-client; do
+        a=$(systemctl is-active "$u" 2>/dev/null || true)
+        e=$(systemctl is-enabled "$u" 2>/dev/null || true)
+        # Compared exactly, never with a substring match: `is-active` answers "inactive",
+        # which contains the word "active".
+        if [ "$a" = active ] || [ "$e" = enabled ] || [ "$e" = enabled-runtime ]; then
+          printf 'UNIT|%s active=%s enabled=%s\n' "$u" "$a" "$e"
+        fi
+      done
+      # `^[[:space:]]*iface` cannot match a commented-out stanza, which is how the wireless
+      # stanza is left on these nodes.
+      grep -rhE '^[[:space:]]*iface[[:space:]]+[^[:space:]]+[[:space:]]+inet6?[[:space:]]+dhcp' \
+        /etc/network/interfaces /etc/network/interfaces.d/ 2>/dev/null \
+        | sed -e 's/^[[:space:]]*/STANZA|/'
+      exit 0
+  register: node_baseline_dhcp_client_scan
+  changed_when: false
+  check_mode: false
+  tags: [dns]
+
 # follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
 # points, which is the strongest single signal of who owns the file.
 - name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
@@ -229,9 +784,10 @@
 #       resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
 #       then the ONLY thing that changes real behaviour; writing resolv.conf would be
 #       overwritten by resolved (or, worse, land in /run and vanish at reboot).
-#   contended — nobody's stub, but resolvconf or NetworkManager is present/running and
+#   contended — nobody's stub, but resolvconf, NetworkManager or a DHCP client (dhcpcd /
-#       would regenerate the file underneath us. We do NOT write. The role fails loudly
+#       dhclient) is present/running/configured and would regenerate the file underneath us.
-#       below rather than leaving an operator believing a change stuck when it did not.
+#       We do NOT write. The role fails loudly below rather than leaving an operator
+#       believing a change stuck when it did not.
 #   resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
 #       Safe to own outright.
 # resolved running but NOT owning resolv.conf correctly falls through to the file branch:
@@ -241,13 +797,21 @@
     node_baseline_dns_mechanism: >-
       {{
         'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
-        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
+        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present
+                              or node_baseline_dhcp_client_live)
               else 'resolv-conf-file')
       }}
+    # Kept as facts (not just task vars) so the failure message, the report and the
+    # reclaim note below all quote the SAME evidence this decision was made on.
+    node_baseline_dhcp_client_evidence: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) | regex_findall('(?m)^(?:CLIENT|UNIT|STANZA)\\|(.+)$') }}"
+    # A file dhcpcd WROTE, versus a dhcpcd that can still write it — two different facts, and
+    # only the second one is contention. See the reclaim note below.
+    node_baseline_resolv_dhcpcd_written: "{{ 'Generated by dhcpcd' in (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
   vars:
     node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
     node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
     node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
+    node_baseline_dhcp_client_live: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) is search('(?m)^(?:CLIENT|UNIT|STANZA)\\|') }}"
     node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
     node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
     node_baseline_resolved_owns_resolv: >-
@@ -264,7 +828,9 @@
       NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
       /etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
       target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
-      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
+      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }},
+      DHCP client evidence={{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }},
+      resolv.conf written by dhcpcd={{ node_baseline_resolv_dhcpcd_written | default(false) }})
   tags: [dns]
 
 # Same shape, and the same reason, as the data-disk assert above: this environment's
@@ -297,16 +863,45 @@
 - name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
   ansible.builtin.fail:
     msg: >-
-      /etc/resolv.conf on this node is not the systemd-resolved stub, but
+      /etc/resolv.conf on this node is not the systemd-resolved stub, but something else would
-      resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
+      regenerate the file underneath this role: resolvconf/openresolv
-      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
+      (/etc/resolvconf/resolv.conf.d present:
-      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
+      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}), NetworkManager
-      would regenerate the file underneath this role. Writing it here would look like it
+      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}), or a DHCP client
-      worked and silently revert. Add the matching branch to roles/node_baseline (the
+      that can still write it — dhcpcd/dhclient evidence:
-      `dns`-tagged block) for whichever manager this node really uses — do not force it.
+      {{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }}.
+      Writing it here would look like it worked and silently revert — which is exactly what
+      happened on k3s-prod-3 on 2026-09-04, where a wifi lease's dhcpcd hook reverted this
+      role's resolvers three minutes after a green run. If the evidence above is a DHCP client
+      on a WIRELESS interface, the fix is the `wireless` block earlier in this role (a cluster
+      node gets one network path) — run it, then re-run this. If it is a DHCP client on the
+      node's real interface, or another manager entirely, add the matching branch to
+      roles/node_baseline for the manager this node really uses — do not force it.
   when: node_baseline_dns_mechanism == 'contended'
   tags: [dns]
 
+# THE CORPSE CASE, and why it is NOT a hole in the fail-closed rule above. After the
+# `wireless` block disables the radio, k3s-prod-3 is left with a /etc/resolv.conf that dhcpcd
+# WROTE (its "# Generated by dhcpcd from wlp1s0.dhcp..." header, the gateway as sole IPv4
+# resolver, two Google IPv6 servers) and NO dhcpcd that can ever run again. `contended` means
+# "another manager is live and will undo this write"; a file left behind by a client that no
+# longer exists is not a manager, it is the damage — and refusing to write here would leave the
+# node permanently mis-resolved, which is the very outcome the operator reported. So the file
+# branch below reclaims it, and says so.
+- name: Note that a dhcpcd-written /etc/resolv.conf is being reclaimed
+  ansible.builtin.debug:
+    msg: >-
+      /etc/resolv.conf on this node was written by dhcpcd, but no DHCP client can write it any
+      more (no process, no active/enabled unit, no `inet dhcp` stanza). Taking ownership and
+      replacing it with the declared upstreams
+      ({{ node_baseline_dns_servers | join(', ') }}). If a DHCP client ever comes back on this
+      node, the `contended` check above will stop the play instead of letting it revert
+      silently.
+  when:
+    - node_baseline_dns_mechanism == 'resolv-conf-file'
+    - node_baseline_resolv_dhcpcd_written | default(false)
+  tags: [dns]
+
 # --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
 # A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
 # never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses

infra/ansible/roles/node_health/README.md

@@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI,
   EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified
   against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM).
-- **Never `become`s unless it must.** Only SMART needs root.
+- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring
+  buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM
+  inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and
+  the privileged second pass re-runs the *same* collector with `--privileged` so the
+  parsing has one home. Without escalation those checks SKIP with the exact reason.
 
 ## What it checks
 
@@ -44,15 +48,99 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
 | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
+| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
 | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
+| `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
 
 Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
 in task logic.
 
+
+## The EFI memattr check
+
+`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
+
+HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
+does not validate it before reserving memory from its contents. **Two different kernel
+functions matter, and the check used to conflate them:**
+
+- `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both
+  fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so
+  **catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through
+  `phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything
+  above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and
+  costs nothing measurable.
+- `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints
+  "Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or
+  merely large.
+
+So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not*
+prove memory was lost: that depends on the garbage **value** at that address on that boot,
+not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03)
+booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on
+measured memory, never on the log line alone:**
+
+| kernel log | measured shortfall | verdict |
+|---|---|---|
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic |
+| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working |
+| no memattr line, log reaches back to boot | (not consulted) | PASS |
+| no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
+| `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
+
+The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt
+table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt
+table is not passing a health check even when it kept its RAM.
+
+### The FW_BUG line is proof, but its ABSENCE proves nothing
+
+`efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG
+line:
+
+| gate | message | reachable when |
+|---|---|---|
+| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
+| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not |
+| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not |
+
+**All three are rejections.** Which one a boot prints is decided by *which field* that boot's
+garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very
+often prints the *first* message, and looking only for the FW_BUG line will make a working
+fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed
+the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version
+alongside the log line, and why the check must never demand the FW_BUG string as its
+evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in
+the kernel image itself, not a line in `dmesg`.
+
+**The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged
+*negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that
+until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected
+a corrupt table was reported as one that "accepted it without complaint" — a false PASS on
+the exact bug this check exists to find.
+
+**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
+`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
+workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
+rebooting/CMOS/BIOS/USB changes only move where the table lands.
+
+### The two memory checks cannot contradict each other
+
+`memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine
+lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus
+the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in
+`tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads
+it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of
+its own, so there is exactly one verdict on lost memory per host and both checks quote it.
+A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a
+future memory loss with a different cause is still caught.
+
 ## Output
 
 1. A human summary printed in the play output.

infra/ansible/roles/node_health/defaults/main.yml

@@ -45,6 +45,33 @@ node_health_thresholds:
   # Memory (percentage of MemTotal still available)
   mem_available_pct_warn: 15
   mem_available_pct_fail: 5
+  # --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ----------------------------
+  # Two independent measurements of the same question, both fed by memory.physical_total:
+  #
+  #   unmanaged  /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own
+  #              memory map says exists, that never reached the allocator. Needs no root
+  #              and no tools, works identically on metal and in a VM.
+  #   shortfall  MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is
+  #              installed. Needs root, and only bare metal reports it. Independent of the
+  #              kernel's memory map entirely, so it also catches RAM the firmware never
+  #              handed the kernel in the first place.
+  #
+  # MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and
+  # not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2%
+  # k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state.
+  # The healthy figure is the kernel image, memmap and firmware reservations; it is a
+  # near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10%
+  # keeps clearance for that. There is no plausible tuning of a threshold in this band
+  # that changes any of those five verdicts.
+  #
+  # The DIMM shortfall runs a little higher than the unmanaged figure on the same node
+  # (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and
+  # the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the
+  # kernel's to manage. Same bands: the signal being caught is 83%, not 3%.
+  mem_unmanaged_pct_warn: 10
+  mem_unmanaged_pct_fail: 25
+  mem_dimm_shortfall_pct_warn: 10
+  mem_dimm_shortfall_pct_fail: 25
   # Temperatures (deg C, any sensor)
   temp_warn_c: 70
   temp_fail_c: 85

infra/ansible/roles/node_health/files/node_health_collect.py

@@ -97,10 +97,22 @@ def read_kv(path, sep=":"):
     return out
 
 
+def which(tool):
+    """Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits.
+
+    `dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the
+    unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed
+    but unreachable would be a false reason on a SKIP, which is the one thing this
+    collector may never do."""
+    return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+
+
 def run(argv, timeout=20):
     """Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent."""
-    if shutil.which(argv[0]) is None:
+    exe = which(argv[0])
+    if exe is None:
         return None, "", "%s not installed" % argv[0]
+    argv = [exe] + list(argv[1:])
     try:
         proc = subprocess.run(
             argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout
@@ -284,9 +296,300 @@ def collect_memory():
         edac["ce_total"] = ce_total
         edac["ue_total"] = ue_total
     mem["edac"] = edac
+    # Physically present vs actually usable — the cause-independent memory-loss signal.
+    mem["zones"] = collect_zoneinfo(mem["mem_total_kb"])
+    # Firmware's own inventory of what is physically installed. Root-only; attempted
+    # here anyway so the unprivileged pass records the exact reason it could not read.
+    mem["devices"] = collect_memory_devices()
     return mem
 
 
+# --- Memory the kernel has but never hands out --------------------------------------
+# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
+# no DMI. Per zone it reports:
+#   spanned  pages in the zone's address range, holes included
+#   present  spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says
+#            physically exists here
+#   managed  present MINUS everything memblock reserved before the buddy allocator was
+#            handed the rest (kernel image, memmap, initrd, crashkernel, firmware
+#            reservations). sum(managed) IS MemTotal, exactly, on every node measured.
+# So `present - managed` is, by construction, "RAM this machine has that the OS will
+# never allocate", whatever the reason. On a healthy node it is the couple of hundred
+# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and
+# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% —
+# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude
+# separation, which is why this needs no clever tuning to be trustworthy.
+#
+# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if
+# a future firmware, kernel or hypervisor lost the memory for an entirely different
+# reason, and it needs neither root nor a parseable kernel log.
+def collect_zoneinfo(mem_total_kb):
+    raw = read_text("/proc/zoneinfo", optional=False)
+    page_kb = os.sysconf("SC_PAGE_SIZE") // 1024
+    out = {
+        "available": raw is not None,
+        "reason": None if raw is not None else "/proc/zoneinfo could not be read",
+        "page_size_kb": page_kb,
+        "zones": [],
+        "present_kb": None,
+        "managed_kb": None,
+        "unmanaged_kb": None,
+        "unmanaged_pct": None,
+    }
+    if raw is None:
+        return out
+    node = zone = None
+    cur = {}
+    present_pages = managed_pages = 0
+    for line in raw.splitlines():
+        fields = line.split()
+        if line.startswith("Node ") and "zone" in line:
+            # "Node 0, zone   Normal"
+            if cur:
+                out["zones"].append(cur)
+            node, zone = fields[1].rstrip(","), fields[-1]
+            cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None}
+        elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"):
+            try:
+                cur[fields[0]] = int(fields[1])
+            except ValueError:
+                cur[fields[0]] = None
+    if cur:
+        out["zones"].append(cur)
+    # An empty parse is a PARSE failure, not "this node has no memory" — say so and stay
+    # null rather than emitting a confident 0.
+    if not out["zones"]:
+        out["available"] = False
+        out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it"
+        return out
+    for z in out["zones"]:
+        present_pages += z["present"] or 0
+        managed_pages += z["managed"] or 0
+    out["present_kb"] = present_pages * page_kb
+    out["managed_kb"] = managed_pages * page_kb
+    out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"]
+    if out["present_kb"] > 0:
+        out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1)
+    # Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the
+    # two readings disagree and the discrepancy itself is worth showing.
+    out["matches_mem_total"] = (
+        None if (mem_total_kb is None or out["managed_kb"] is None)
+        else abs(out["managed_kb"] - mem_total_kb) <= page_kb
+    )
+    return out
+
+
+# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) -----------------------------------
+# The independent answer to "how much RAM is physically in this box", from the firmware's
+# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads
+# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that
+# failure is recorded as the SKIP reason — never as zero DIMMs.
+#
+# On a VM there is usually either no type-17 structure at all or a single synthetic entry
+# that merely restates the configured guest RAM; both are handled by reporting what was
+# found and letting the check SKIP when nothing populated was reported.
+DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def collect_memory_devices():
+    out = {
+        "available": False,
+        "reason": None,
+        "devices": [],
+        "slots": 0,
+        "populated_slots": 0,
+        "installed_kb": None,
+    }
+    if which("dmidecode") is None:
+        out["reason"] = ("dmidecode is not installed on this node; it ships with "
+                         "roles/node_baseline (node_baseline_packages). "
+                         "This role never installs packages.")
+        return out
+    rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30)
+    if rc != 0:
+        out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % (
+            rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output")
+        return out
+    # Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which
+    # may or may not hold a module ("Size: No Module Installed").
+    total_kb = 0
+    saw_size = False
+    for block in stdout.split("\n\n"):
+        if "Memory Device" not in block:
+            continue
+        fields = {}
+        for line in block.splitlines():
+            key, sep, value = line.partition(":")
+            if sep:
+                fields[key.strip()] = value.strip()
+        if "Size" not in fields:
+            continue
+        out["slots"] += 1
+        size_kb = None
+        parts = fields["Size"].split()
+        if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS:
+            try:
+                size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()]
+            except ValueError:
+                size_kb = None
+        if size_kb:
+            saw_size = True
+            total_kb += size_kb
+            out["populated_slots"] += 1
+        out["devices"].append(
+            {
+                "locator": fields.get("Locator"),
+                "bank_locator": fields.get("Bank Locator"),
+                # Verbatim, so an unparsed spelling is visible rather than silently null.
+                "size_raw": fields["Size"],
+                "size_kb": size_kb,
+                "type": fields.get("Type"),
+                "speed": fields.get("Speed"),
+                "configured_speed": fields.get("Configured Memory Speed"),
+                "manufacturer": fields.get("Manufacturer"),
+                "part_number": fields.get("Part Number"),
+                "rank": fields.get("Rank"),
+                # Serial deliberately NOT collected: the artifact already carries drive
+                # serials and a DIMM serial adds identifying data no check reads.
+            }
+        )
+    out["available"] = saw_size
+    if not saw_size:
+        out["reason"] = ("dmidecode ran but reported no populated memory device "
+                         "(%d slot structure(s) found) — typical of a VM with no SMBIOS "
+                         "type 17 inventory" % out["slots"])
+    else:
+        out["installed_kb"] = total_kb
+    return out
+
+
+# --- Kernel ring buffer: the EFI Memory Attributes Table ----------------------------
+# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative
+# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
+# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
+# the product). The kernel prints one of two lines:
+#
+#   "Failed to map EFI Memory Attributes table @ 0x..."   the garbage version read <= 2, so
+#       the kernel TRUSTED the table and reserved memory from it. This is emitted later, by
+#       efi_memattr_apply_permissions()'s memremap(), and it fails for ANY sufficiently
+#       wrong tbl_size. So it proves the table was corrupt and trusted — it does NOT prove
+#       memory was lost: only a NEGATIVE tbl_size (bit 31 set, sign-extended and clamped to
+#       PHYS_ADDR_MAX) reserves everything above the table, while a garbage-but-positive
+#       tbl_size costs nothing measurable. Whether memory actually went missing is decided
+#       by MEASUREMENT (collect_zoneinfo below), not by this line: k3s-prod-2 lost 13.3 GiB
+#       twice at 0xbeeb0018 and then booted with all 15.1 GiB at the same address, same line.
+#   "Unexpected EFI Memory Attributes table version <N>"  the garbage version read > 2, so
+#       the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
+#       firmware is still publishing a corrupt table and the outcome is decided afresh by
+#       what the garbage reads on every boot. That is luck, not immunity.
+#
+# This collector reports the LINES and the addresses; it never rules on memory. Classifying
+# is roles/node_health/tasks/assess.yml, which rules on the measured shortfall.
+#
+# It is boot-to-boot NON-DETERMINISTIC, which is precisely why it is detected on every
+# health run instead of diagnosed once.
+#
+# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
+# every node here). The unprivileged pass therefore records WHY it could not read, and
+# roles/node_health re-runs this collector with `--privileged` when escalation works.
+BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:")
+EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
+MEMATTR_FAILED_RE = re.compile(
+    r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
+# NOTE the `-?`. The kernel prints this version with %d — SIGNED — so a garbage field with
+# bit 31 set is logged NEGATIVE (k3s-prod-2, 2026-09-04: "version -1569980416"). Without the
+# minus sign this regex missed those lines entirely and the state fell through to "clean",
+# i.e. a node whose kernel had just REJECTED a corrupt table was reported as one whose
+# kernel "accepted it without complaint" — a false PASS on the exact bug this check exists
+# to find. Anchor future edits on that: the field is signed, and every diagnostic that
+# quotes it must survive a negative value.
+MEMATTR_VERSION_RE = re.compile(
+    r"Unexpected EFI Memory Attributes table version\s+(-?\d+)", re.I)
+MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
+# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
+# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
+# captured separately, above, and quoting it as evidence would bury the one line that
+# matters in 200 characters of unrelated table addresses.
+MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I)
+# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line
+# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict.
+MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available")
+MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved")
+
+
+def collect_klog():
+    out = {
+        "available": False,
+        "reason": None,
+        "boot_covered": None,
+        "memattr_state": None,
+        "memattr_published": None,
+        "memattr_table_address": None,
+        "memattr_version": None,
+        "memattr_lines": [],
+        "memory_line": None,
+    }
+    rc, stdout, stderr = run(["dmesg"], timeout=30)
+    if rc is None:
+        out["reason"] = "dmesg is not installed on this node"
+        return out
+    if rc != 0:
+        detail = (stderr or stdout).strip().splitlines()
+        out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % (
+            rc, os.geteuid(), detail[-1] if detail else "no output",
+            " — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG"
+            if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "")
+        return out
+
+    out["available"] = True
+    # Does the buffer still reach back to early boot? The ring buffer WRAPS, and every
+    # memattr line is printed in the first fraction of a second. If the boot header is
+    # gone then so may be a memattr line that was printed — absence would prove nothing,
+    # and the check must SKIP rather than PASS. This is the difference between
+    # "unaffected" and "could not tell", and the role is not allowed to blur it.
+    out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout))
+
+    published = EFI_CONFIG_RE.search(stdout)
+    if published:
+        out["memattr_published"] = True
+        out["memattr_table_address"] = published.group(1)
+    elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M):
+        # UEFI booted and the config-table line is present, but it lists no MEMATTR=.
+        out["memattr_published"] = False
+    # else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None.
+
+    for line in stdout.splitlines():
+        if MEMATTR_LINE_RE.search(line):
+            out["memattr_lines"].append(line.strip())
+    joined = "\n".join(out["memattr_lines"])
+    if MEMATTR_CORRUPT_RE.search(joined):
+        # Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected.
+        out["memattr_state"] = "corrupt_rejected"
+    elif MEMATTR_FAILED_RE.search(joined):
+        out["memattr_state"] = "failed_to_map"
+        addr = MEMATTR_FAILED_RE.search(joined).group(1)
+        if addr:
+            out["memattr_table_address"] = addr
+    elif MEMATTR_VERSION_RE.search(joined):
+        out["memattr_state"] = "bad_version"
+        out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1))
+    else:
+        out["memattr_state"] = "clean"
+
+    mem_line = MEMORY_LINE_RE.search(stdout)
+    if mem_line:
+        full = [l for l in stdout.splitlines() if mem_line.group(0) in l]
+        reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None
+        out["memory_line"] = {
+            "available_kb": int(mem_line.group(1)),
+            "total_kb": int(mem_line.group(2)),
+            "reserved_kb": int(reserved.group(1)) if reserved else None,
+            "raw": full[0].strip() if full else None,
+        }
+    return out
+
+
 # --- Thermals -----------------------------------------------------------------------
 def collect_thermal():
     zones = []
@@ -708,15 +1011,34 @@ def collect_os():
 def collect_tools():
     tools = {}
     for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
-        tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+        tools[tool] = which(tool)
     return tools
 
 
 def main():
+    # --privileged: re-collect ONLY the fields that genuinely need root, for the second
+    # pass roles/node_health makes when (and only when) escalation is proven to work
+    # (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place —
+    # re-implementing the kernel-log grep in Jinja would be a second source of truth for
+    # the same fact, and the two would drift.
+    if "--privileged" in sys.argv[1:]:
+        doc = {
+            "schema": SCHEMA,
+            "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "collector_uid": os.geteuid(),
+            "klog": collect_klog(),
+            "memory": {"devices": collect_memory_devices()},
+        }
+        doc["unreadable"] = UNREADABLE
+        json.dump(doc, sys.stdout, indent=2, sort_keys=True)
+        sys.stdout.write("\n")
+        return
+
     doc = {
         "schema": SCHEMA,
         "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
         "collector_uid": os.geteuid(),
+        "klog": collect_klog(),
         "os": collect_os(),
         "dmi": collect_dmi(),
         "cpu": collect_cpu(),

infra/ansible/roles/node_health/tasks/assess.yml

@@ -387,6 +387,118 @@
                ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
                ~ avail_pct | string ~ '%)') }}
 
+# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
+# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
+# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
+# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
+# from the presence of a kernel-log line — and they duly contradicted each other on the same
+# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
+# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
+# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
+# measurement was right; the inference from the log line was wrong — the memattr block below
+# has the refined mechanism.
+#
+# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
+# place and both checks read this fact: neither derives a shortfall of its own, and the
+# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
+# disagree about whether memory was lost — there is only one verdict on that question, and
+# it is this one. Thresholds and their measured baselines: defaults/main.yml.
+- name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
+  ansible.builtin.set_fact:
+    node_health_mem_shortfall: "{{ {
+      'zones_pct': z_pct, 'zones_status': z_status,
+      'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
+      'status': worst_status } }}"
+  vars:
+    ansible_become: false
+    th: "{{ node_health_thresholds }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+
+    # 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
+    #    actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
+    #    tools, identical on metal and in a VM — so this half is available almost always.
+    z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
+    z_status: >-
+      {{ 'SKIP' if z_pct is not number
+         else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
+               else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
+
+    # 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
+    #    VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
+    #    never as a zero DIMM count. Independent of the kernel's memory map entirely.
+    inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
+    d_pct: >-
+      {{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
+         if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
+         else none }}
+    d_status: >-
+      {{ 'SKIP' if d_pct is not number
+         else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
+               else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
+
+    # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
+    # SKIPs must never add up to a PASS.
+    worst_status: >-
+      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
+         | default('SKIP', true) }}
+
+# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
+# firmware.efi_memattr names the one cause we have actually hit here; this one asks the
+# blunter question — "is this machine using the RAM it has?" — and answers it from two
+# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
+# losing memory for some other reason still shows up.
+- name: Assess — usable memory vs the memory this machine physically has
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'memory.physical_total',
+      'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
+      'status': phys_status,
+      'detail': phys_detail }] }}"
+  vars:
+    ansible_become: false
+    ms: "{{ node_health_mem_shortfall }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+    # Straight from the shared measurement — this check re-derives nothing.
+    z_pct: "{{ ms.zones_pct }}"
+    inst_kb: "{{ ms.installed_kb }}"
+    d_pct: "{{ ms.dimm_pct }}"
+    phys_status: "{{ ms.status }}"
+    # What the kernel log says, IF it was read — quoted here only as attribution for the
+    # measured shortfall. The verdict above is the shared measurement and never depends on
+    # the log; firmware.efi_memattr reads that same verdict, so the two always agree.
+    phys_cause: >-
+      {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
+           firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
+           if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
+         else ('. The kernel log was NOT read here, so this shortfall is unattributed —
+                see check firmware.efi_memattr')
+           if not (node_health_facts.klog.available | bool)
+         else '' }}
+    phys_detail: >-
+      {{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
+           ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
+           ~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
+             was never handed to the allocator (/proc/zoneinfo)')
+          if z_pct is number
+          else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
+         ~ '. '
+         ~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
+             ~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
+             ~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
+             ~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
+             ~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
+            if d_pct is number
+            else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
+         ~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
+              reservations and (on this platform) 64 MiB of graphics stolen memory.
+              A shortfall of this size is memory the machine HAS and cannot use')
+            if phys_status in ['WARN', 'FAIL'] else '')
+         ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
+
 - name: Assess — swap (kubelet requires it off)
   ansible.builtin.set_fact:
     node_health_checks: "{{ node_health_checks + [{
@@ -674,3 +786,245 @@
                ~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
                ~ (' — serial numbers not read (root-only DMI fields)'
                   if d.product_serial is none else '')) }}
+
+# --- EFI Memory Attributes Table ------------------------------------------------------
+# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
+# what follows is only what this check needs to be read correctly.
+#
+# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
+# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
+# are involved, and conflating them is what made this check wrong until 2026-09-03:
+#
+#   efi_memattr_init()              tbl_size = sizeof(*tbl) + num_entries * desc_size, then
+#                                   memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
+#                                   is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
+#                                   NEGATIVE: a garbage product with bit 31 set sign-extends
+#                                   through phys_addr_t and memblock_cap_size() clamps it to
+#                                   PHYS_ADDR_MAX — everything above the table is reserved.
+#                                   A garbage-but-POSITIVE tbl_size reserves a modest, wrong
+#                                   amount and costs nothing measurable.
+#   efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
+#                                   THIS is what prints "Failed to map". It fails for ANY
+#                                   sufficiently wrong tbl_size — negative OR merely large.
+#
+# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
+# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
+# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
+# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
+# is the counter-example that falsified the old inference.
+#
+# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
+#
+#   "Failed to map ... @ 0x..."   the bug FIRED: corrupt table, trusted by the kernel.
+#       FAIL when the shared measurement (node_health_mem_shortfall) says memory is
+#       catastrophically missing — memory is lost NOW.
+#       WARN otherwise — this boot drew a survivable garbage value; the next may not.
+#   "Unexpected ... version <N>"  version read > 2, so the kernel bailed out before
+#       reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
+#       safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
+#       gates and no draw can reach the fatal arithmetic any more. Same line, opposite
+#       meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
+#   FW_BUG "Corrupted EFI Memory Attributes Table detected!"  the >= 6.13 THIRD gate
+#       (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
+#       must not be treated as such: it is only reachable on a boot whose garbage leaves the
+#       version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
+#       line instead. All three prod nodes did exactly that on 2026-09-04.
+#
+# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
+# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
+# measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
+# firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
+# the draw still happens but every outcome is a rejection, so the same lines mean the
+# validation is working.
+#
+# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
+# so it cannot be diagnosed once. It has to be measured on every health run.
+#
+# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
+# node_health_mem_shortfall, computed once further up this file; this check does not measure
+# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
+# one verdict on any host and both checks quote it.
+#
+# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
+# ONLY if the kernel log was actually read AND still reaches back to early boot — every
+# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
+# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
+# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
+# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
+# log at all, so a node whose log is unreadable is still not unmeasured.
+- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'firmware.efi_memattr',
+      'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
+      'status': memattr_status,
+      'detail': memattr_detail }] }}"
+  vars:
+    ansible_become: false
+    k: "{{ node_health_facts.klog }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    # THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
+    # near memory.physical_total. This check contributes the kernel-log facts only.
+    ms: "{{ node_health_mem_shortfall }}"
+    ma_state: "{{ k.memattr_state | default(none) }}"
+    # DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
+    # carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
+    # THREE checks before it computes tbl_size at all:
+    #     tbl->version > 2                       -> "Unexpected ... table version %d"
+    #     desc_size implausible                  -> "Unexpected ... descriptor size %u ..."
+    #     num_entries > SZ_64K                   -> FW_BUG "Corrupted EFI Memory Attributes
+    #                                               Table detected! ..."
+    # ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
+    # given boot produces is decided by WHICH field that boot's garbage lands in — it is not
+    # a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
+    # version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
+    # memblock_reserve(). That is the whole difference, and it is why the verdict for
+    # `bad_version` below is opposite on the two kernels: same log line, completely different
+    # meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
+    # something the `version` test can parse.
+    memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
+    memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
+    # `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
+    # COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
+    # this check and memory.physical_total can no longer disagree about lost memory.
+    # `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
+    # boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
+    # WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
+    # forever would make the check unable to ever show the fix landing, which is exactly what
+    # it is for.
+    memattr_status: >-
+      {{ 'SKIP' if not (k.available | bool)
+         else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
+               else ('WARN' if ma_state == 'failed_to_map'
+                     else ('PASS' if ma_state == 'corrupt_rejected'
+                           else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
+                                 if ma_state == 'bad_version'
+                                 else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
+    # One remediation, quoted by both findings. NOTE what it must NOT say: there is no
+    # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
+    # could take effect. Suggesting one would be a fix that quietly does nothing.
+    memattr_fix: >-
+      FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
+      upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
+      bogus", first released in v6.13 and absent from v6.12 and from the current
+      linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
+      There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
+      bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
+      unplugging USB only change where the table lands and may APPEAR to fix it — they are
+      a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+    # The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
+    # to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
+    # kernel log has no part in; the verdict word comes from ms.status, so it always agrees
+    # with memory.physical_total. When the measurement is missing it says so, and says why.
+    memattr_figures: >-
+      {{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+          ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
+          ~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
+            allocator (/proc/zoneinfo)')
+         if ms.zones_pct is number
+         else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
+                reports as installed (/proc/zoneinfo NOT read — '
+                ~ (z.reason | default('no reason recorded', true)) ~ ')')
+               if ms.dimm_pct is number else '') }}
+    memattr_corroboration: >-
+      {{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
+          ~ ms.status ~ '): ' ~ memattr_figures
+          ~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
+             else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
+                   else ' — the normal few percent (kernel image, memmap, firmware
+                        reservations, graphics stolen memory). NO measurable memory was lost
+                        on this boot.')))
+         if (memattr_figures | trim | length) > 0
+         else (' Whether memory was lost on this boot could NOT be measured here: neither
+                present-vs-managed pages nor installed DIMM capacity was read — see check
+                memory.physical_total for the reasons.') }}
+    # The kernel log lines themselves, verbatim. A verdict the operator cannot check
+    # against the evidence is not much better than a wrong one.
+    memattr_evidence: >-
+      {{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
+          if (k.memattr_lines | default([]) | length) > 0 else '')
+         ~ ((' At boot the kernel reported '
+             ~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+             ~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
+             ~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
+             ~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
+            if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
+            else '') }}
+    memattr_detail: >-
+      {%- if not (k.available | bool) -%}
+      the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
+      Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
+      lives only in dmesg.{{ memattr_corroboration }}
+      {%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
+      THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
+      line. The kernel failed to map the EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }}, which proves the
+      table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
+      from its contents; the measurement below shows that reservation was the catastrophic
+      kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
+      through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
+      table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'failed_to_map' -%}
+      THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
+      loss either. The kernel failed to map the EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }}: the table was
+      corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
+      WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
+      the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
+      with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
+      tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
+      measurable and still fails the later memremap() that printed this line. This boot drew a survivable
+      value; the draw is repeated on EVERY boot, at the same address — this platform has
+      been seen to lose 13.3 GiB and to keep its full memory at one identical table address
+      (infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
+      {{ memattr_fix }}
+      {%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
+      the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
+      kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
+      VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
+      boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
+      outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
+      efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
+      WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
+      Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
+      (num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
+      version field <= 2. Both messages are rejections, and which one a boot prints is the
+      lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
+      path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
+      PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
+      {{ memattr_corroboration }}
+      {%- elif ma_state == 'bad_version' -%}
+      EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
+      CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
+      version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
+      kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
+      EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
+      tbl_size the garbage produces — nothing measurable if that value is positive,
+      everything above the table if it is negative.{{ memattr_evidence }}
+      {{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'corrupt_rejected' -%}
+      the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
+      validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
+      cannot cost this node memory on this or any future boot; no
+      action.{{ memattr_evidence }}{{ memattr_corroboration }}
+      {%- elif not (k.boot_covered | bool) -%}
+      NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
+      back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
+      the ring buffer, which wraps). Every memattr diagnostic is printed within the first
+      fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
+      pass. Read /var/log/kern.log for this boot, or re-run after a
+      reboot.{{ memattr_corroboration }}
+      {%- elif k.memattr_published is sameas true -%}
+      the firmware publishes an EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }} and the kernel
+      accepted it without complaint, in a log that still reaches back to early boot. Not
+      affected on this boot.
+      {%- elif k.memattr_published is sameas false -%}
+      this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
+      lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
+      guest.
+      {%- else -%}
+      no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
+      so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
+      boot, so that absence is measured, not assumed.
+      {%- endif -%}

infra/ansible/roles/node_health/tasks/collect_privileged.yml

@@ -0,0 +1,62 @@
+---
+# The SECOND pass of the collector, run with escalation, for the two facts that are
+# genuinely root-gated:
+#
+#   * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an
+#     unprivileged read gets EPERM. This is where the EFI Memory Attributes Table
+#     diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference
+#     between a node that has lost 13.3 GiB and one that has not.
+#   * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed
+#     DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem.
+#
+# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a
+# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line-
+# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND
+# source of truth for the same fact, and the two would drift. `--privileged` makes the
+# collector emit exactly those subtrees and nothing else, so the parsing stays in one
+# place and is identical whichever pass produced it.
+#
+# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass
+# actually failed to read one of them (main.yml checks both). On a node where
+# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule
+# is that everything readable WITHOUT root is read without root.
+#
+# If escalation is unavailable the checks that consume these SKIP with the exact reason
+# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line"
+# and "could not read the kernel log" are different answers and the role says which.
+
+- name: Re-collect the root-only facts with escalation
+  ansible.builtin.script:
+    cmd: node_health_collect.py --privileged
+    executable: /usr/bin/python3
+  become: true
+  vars:
+    ansible_become: true
+  register: node_health_privileged_raw
+  changed_when: false   # strictly read-only; it can never change the node
+
+- name: Assert the privileged payload is the schema this role understands
+  ansible.builtin.assert:
+    that:
+      - (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1'
+    fail_msg: >-
+      Privileged collector pass returned an unexpected schema; role and collector are
+      out of step.
+    quiet: true
+  vars:
+    ansible_become: false
+
+# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive
+# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine
+# would REPLACE it and throw away the reasons the first pass recorded.
+- name: Merge the privileged facts over the unprivileged ones
+  ansible.builtin.set_fact:
+    node_health_facts: >-
+      {{ node_health_facts | combine({
+           'klog': priv.klog,
+           'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}),
+           'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])),
+         }) }}
+  vars:
+    ansible_become: false
+    priv: "{{ node_health_privileged_raw.stdout | from_json }}"

infra/ansible/roles/node_health/tasks/main.yml

@@ -4,9 +4,11 @@
 # ORDER OF OPERATIONS
 #   1. Collect everything readable WITHOUT root (one script, one round trip).
 #   2. Probe whether privilege escalation actually works on this host.
-#   3. Only if it does AND smartmontools is installed: read SMART.
+#   3. Only if it does, and only for what the first pass could NOT read: re-run the same
-#   4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#      collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
-#   5. Render the human summary + write the JSON artifact.
+#   4. Only if it does AND smartmontools is installed: read SMART.
+#   5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#   6. Render the human summary + write the JSON artifact.
 #
 # BECOME HANDLING — the subtle bit, read before editing.
 #   `ansible_become` set as an inventory/group var (which is exactly what
@@ -43,6 +45,16 @@
   ansible.builtin.include_tasks: probe_privilege.yml
   when: node_health_try_privileged | bool
 
+# Only when the unprivileged pass actually failed to read one of them. On a node with
+# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
+# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
+- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
+  ansible.builtin.include_tasks: collect_privileged.yml
+  when:
+    - node_health_root_available | bool
+    - (not (node_health_facts.klog.available | bool))
+      or (not (node_health_facts.memory.devices.available | bool))
+
 - name: Read SMART data (privileged)
   ansible.builtin.include_tasks: collect_smart.yml
   when:

infra/ansible/roles/node_health/templates/summary.txt.j2

@@ -12,6 +12,17 @@
 {% macro gib(n) -%}
 {{ (n | int / 1073741824) | round(1) }} GiB
 {%- endmacro %}
+{# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not
+   report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of
+   its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised
+   "object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it
+   as `n/a` (the same token this template already uses for unreported readings) rather than
+   defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The
+   collector already guards this — tasks/collect_smart.yml uses selectattr('thresh',
+   'defined') and `a.thresh | default(0)` — so only the template was exposed. #}
+{% macro thr(a) -%}
+{{ a.thresh | default('n/a') }}
+{%- endmacro %}
 ================================================================================
  NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
  collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
@@ -45,6 +56,15 @@
 {% endif %}
  Memory       {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
  ECC/EDAC     {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
+{% set mz = node_health_facts.memory.zones %}
+{% set md = node_health_facts.memory.devices %}
+ Present RAM  {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }}
+ DIMMs        {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }}
+{% for dimm in md.devices | default([]) if dimm.size_kb %}
+              {{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }}
+{% endfor %}
+{% set klog = node_health_facts.klog %}
+ EFI memattr  {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
  Load         {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
 
 --------------------------------------------------------------------------------
@@ -64,19 +84,19 @@
               lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }}  temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
               pending(197)={{ sm.pending_sectors | default('?') }}  offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }}  reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }}  crc(199)={{ sm.crc_errors | default('?') }}
 {% if sm.failing_now or sm.failed_past %}
-              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %}
+              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %}
 
 {% elif sm.thresholds_present %}
-              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %}
+              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %}
 
 {% endif %}
               vendor context (NOT used for the verdict — raw units are vendor-defined):
               reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }}  grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }}  erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }}  program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }}  unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
 {% if sm.realloc_attr %}
-              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged)
+              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged)
 {% endif %}
 {% for a in sm.attributes | default([]) %}
-              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
+              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
 
 {% endfor %}
 {% else %}

platform/coredns/coredns-custom.yaml

@@ -29,9 +29,9 @@ metadata:
 data:
   # Filename `home.server` -> imported as a standalone server block for the whole zone.
   home.server: |
-    home.keithjackson.dev:53 {
+    ${BASE_DOMAIN}:53 {
         errors
-        # Answer EVERY name in the zone (apex + all `<svc>.home.keithjackson.dev`) with the Gateway
+        # Answer EVERY name in the zone (apex + all `<svc>.${BASE_DOMAIN}`) with the Gateway
         # VIP. The template plugin's default match is `.*`, so this covers current and FUTURE app
         # hostnames without editing this file per service. `{{ .Name }}` echoes the queried FQDN.
         template IN A {

platform/crafty/deployment.yaml

@@ -125,7 +125,7 @@ spec:
             - --http-address=0.0.0.0:4180
             - --reverse-proxy=true
             - --provider=oidc
-            - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+            - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
             # Gate the co-located Crafty UI. Crafty serves HTTPS with a SELF-SIGNED cert, so skip verify
             # on this in-pod (127.0.0.1) hop; the browser-facing TLS is the trusted wildcard at the Gateway.
             - --upstream=https://127.0.0.1:8443
@@ -138,9 +138,9 @@ spec:
             - --allowed-group=platform-admins
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
-            - --redirect-url=https://crafty.home.keithjackson.dev/oauth2/callback
+            - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.home.keithjackson.dev
+            - --cookie-domain=.${BASE_DOMAIN}
-            - --whitelist-domain=.home.keithjackson.dev
+            - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true
             - --set-xauthrequest=true

platform/crafty/httproute.yaml

@@ -25,7 +25,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - crafty.home.keithjackson.dev
+    - crafty.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/crafty/service-l4.yaml

@@ -42,7 +42,7 @@ metadata:
     lbipam.cilium.io/sharing-key: "crafty"
     lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
     # external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
-    external-dns.alpha.kubernetes.io/hostname: "crafty-mc.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

platform/externaldns/helmrelease.yaml

@@ -29,15 +29,46 @@ spec:
     sources:
       - gateway-httproute
       - service
-    # domain-filter MUST be the actual Cloudflare ZONE (keithjackson.dev), not the platform subdomain
+    # ---------------------------------------------------------------------------------------------
-    # (home.keithjackson.dev) — ExternalDNS filters the account's zones by this value, and a subdomain
+    # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
-    # filter drops the parent zone → "no hosted zone matching record DNS Name" and nothing is created.
+    #
-    # Scope stays effectively home.keithjackson.dev because that's the only suffix services request;
+    # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
-    # policy=sync + registry=txt only ever touch ExternalDNS-OWNED records, never the pre-existing
+    # `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because
-    # wildcard or other keithjackson.dev records. (docs/09; resolves the WS-NET TODO below.)
+    # NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore
+    # read dev's TXT registry entries as its own, found no local resource claiming those names, and
+    # garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
+    # Full write-up: infra/PROD-PROVISIONING.md §10.
+    #
+    # THE TWO GUARDS THAT ACTUALLY PREVENT A RECURRENCE — note that neither is domainFilters:
+    #   1. txtOwnerId  — per-environment identity; a cluster only ever touches records it owns.
+    #   2. DISJOINT HOSTNAME SETS — the two environments serve different suffixes, so they never
+    #      describe the same record in the first place. dev = *.dev.home.keithjackson.dev,
+    #      prod = *.home.keithjackson.dev (prod IS the apex; operator decision 2026-09-04).
+    # ---------------------------------------------------------------------------------------------
+    # Which names this instance may manage at all. BOTH ENVIRONMENTS RENDER THE CLOUDFLARE **ZONE**
+    # `keithjackson.dev` HERE, and this is NOT an oversight to be tightened.
+    #
+    # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for ZONE
+    # SELECTION as well as record matching, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+    # matches NO zone — external-dns generates the record and then skips it with "no hosted zone
+    # matching record DNS Name", publishing NOTHING. This has been "improved" and reverted more than
+    # once (an agent narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in
+    # review). It is not defence in depth; it is a total outage of DNS publication. Isolation comes
+    # from the two guards above, which a domain filter cannot provide anyway — both environments live
+    # in the ONE zone regardless of what this says. See clusters/base/cluster-vars.yaml.
     domainFilters:
-      - "${DNS_ZONE}"
+      - "${EXTERNALDNS_DOMAIN_FILTER}"
-    policy: sync            # create + update + delete records as services come and go
+    # The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
+    # caused the incident. dev renders `dev`, prod renders `prod`, and the BASE default is a
+    # non-colliding sentinel so a new environment that forgets the knob owns nothing and can delete
+    # nothing. NOTHING renders `default` any more (the external-dns chart default, and the value both
+    # clusters silently shared): dev held it only because its live apex records were registered under
+    # it, and dev's move to *.dev.home.keithjackson.dev retired that constraint along with those
+    # records. Prod must NOT be given `default` even though it inherits the apex NAMES — that would
+    # hand prod ownership of dev's old apex records and let `policy: sync` below delete them.
+    txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
+    policy: sync            # create + update + delete records as services come and go — but ONLY
+                            # records this owner id owns AND this domainFilter admits (see above).
     # Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007).
     env:
       - name: CF_API_TOKEN

platform/gateway/gateway.yaml

@@ -13,7 +13,7 @@ spec:
     - name: https
       protocol: HTTPS
       port: 443
-      hostname: "*.home.keithjackson.dev"
+      hostname: "*.${BASE_DOMAIN}"
       tls:
         mode: Terminate
         certificateRefs:
@@ -30,7 +30,7 @@ spec:
     - name: http-redirect
       protocol: HTTP
       port: 80
-      hostname: "*.home.keithjackson.dev"
+      hostname: "*.${BASE_DOMAIN}"
       allowedRoutes:
         namespaces:
           from: Selector
@@ -46,7 +46,7 @@ spec:
     - name: https-apex
       protocol: HTTPS
       port: 443
-      hostname: "home.keithjackson.dev"
+      hostname: "${BASE_DOMAIN}"
       tls:
         mode: Terminate
         certificateRefs:
@@ -61,7 +61,7 @@ spec:
     - name: http-apex-redirect
       protocol: HTTP
       port: 80
-      hostname: "home.keithjackson.dev"
+      hostname: "${BASE_DOMAIN}"
       allowedRoutes:
         namespaces:
           from: Selector

platform/gateway/wildcard-certificate.yaml

@@ -10,8 +10,8 @@ metadata:
 spec:
   secretName: wildcard-home-tls        # consumed by the Gateway HTTPS listener below
   dnsNames:
-    - "*.home.keithjackson.dev"
+    - "*.${BASE_DOMAIN}"
-    - "home.keithjackson.dev"          # apex for the C2 UI (docs/09)
+    - "${BASE_DOMAIN}"          # apex for the C2 UI (docs/09)
   issuerRef:
     kind: ClusterIssuer
     # ${ACME_CLUSTER_ISSUER}: dev -> letsencrypt-staging, prod -> letsencrypt-prod (cluster-vars).

platform/headlamp/httproute.yaml

@@ -24,7 +24,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - headlamp.home.keithjackson.dev
+    - headlamp.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/homepage/configmap.yaml

@@ -36,63 +36,63 @@ data:
     ---
     - Platform:
         - Headlamp:
-            href: https://headlamp.home.keithjackson.dev
+            href: https://headlamp.${BASE_DOMAIN}
             description: Kubernetes dashboard (per-user OIDC RBAC)
             icon: mdi-kubernetes
         - Grafana:
-            href: https://grafana.home.keithjackson.dev
+            href: https://grafana.${BASE_DOMAIN}
             description: Metrics, logs & dashboards (fronts Prometheus/Loki/Alertmanager)
             icon: mdi-chart-areaspline
         - Semaphore:
-            href: https://semaphore.home.keithjackson.dev
+            href: https://semaphore.${BASE_DOMAIN}
             description: Ansible automation & CI
             icon: mdi-cog-play
         - Gitea:
-            href: https://gitea.home.keithjackson.dev
+            href: https://gitea.${BASE_DOMAIN}
             description: Git hosting & Actions (GitOps source of truth)
             icon: mdi-git
         - Keycloak:
-            href: https://keycloak.home.keithjackson.dev
+            href: https://keycloak.${BASE_DOMAIN}
             description: Identity provider (SSO / OIDC)
             icon: mdi-account-key
 
     - Media:
         - Plex:
-            href: https://plex.home.keithjackson.dev
+            href: https://plex.${BASE_DOMAIN}
             description: Media streaming
             icon: mdi-plex
         - Seerr:
-            href: https://seerr.home.keithjackson.dev
+            href: https://seerr.${BASE_DOMAIN}
             description: Media requests
             icon: mdi-play-box-multiple
         - Sonarr:
-            href: https://sonarr.home.keithjackson.dev
+            href: https://sonarr.${BASE_DOMAIN}
             description: TV library manager
             icon: mdi-television-classic
         - Radarr:
-            href: https://radarr.home.keithjackson.dev
+            href: https://radarr.${BASE_DOMAIN}
             description: Movie library manager
             icon: mdi-movie-open
         - Jackett:
-            href: https://jackett.home.keithjackson.dev
+            href: https://jackett.${BASE_DOMAIN}
             description: Indexer proxy
             icon: mdi-magnify
         - qBittorrent:
-            href: https://qbittorrent.home.keithjackson.dev
+            href: https://qbittorrent.${BASE_DOMAIN}
             description: Download client (VPN egress)
             icon: mdi-download-network
 
     - Network & Infra:
         - Pi-hole:
-            href: https://pihole.home.keithjackson.dev
+            href: https://pihole.${BASE_DOMAIN}
             description: Network-wide DNS & ad-blocking
             icon: mdi-shield-check
         - UniFi:
-            href: https://unifi.home.keithjackson.dev:11443
+            href: https://unifi.${BASE_DOMAIN}:11443
             description: Network controller (UOS)
             icon: mdi-router-network
         - Nextcloud:
-            href: https://nextcloud.home.keithjackson.dev
+            href: https://nextcloud.${BASE_DOMAIN}
             description: Files, calendar & contacts
             icon: mdi-cloud
 
@@ -104,7 +104,7 @@ data:
               href: https://gitea.keithjackson.dev
         - Auth portal:
             - abbr: SSO
-              href: https://auth.home.keithjackson.dev
+              href: https://auth.${BASE_DOMAIN}
     - Reference:
         - Homepage docs:
             - abbr: HP

platform/homepage/deployment.yaml

@@ -54,7 +54,7 @@ spec:
             # REQUIRED by Homepage >= 1.0 — requests whose Host header is not in this list get 403.
             # The public apex host (served via the Gateway) + the pod IP (for kubelet probes).
             - name: HOMEPAGE_ALLOWED_HOSTS
-              value: "home.keithjackson.dev,$(MY_POD_IP):3000"
+              value: "${BASE_DOMAIN},$(MY_POD_IP):3000"
           resources:
             requests:
               cpu: 25m

platform/homepage/httproute.yaml

@@ -28,7 +28,7 @@ spec:
       namespace: gateway
       sectionName: https-apex     # the apex listener (wildcard `https` won't match the bare apex)
   hostnames:
-    - home.keithjackson.dev
+    - ${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/keycloak/httproute.yaml

@@ -16,7 +16,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - keycloak.home.keithjackson.dev
+    - keycloak.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/keycloak/keycloak-instance.yaml

@@ -30,7 +30,7 @@ spec:
     poolMaxSize: 20
   # --- Exposure: TLS at the Gateway; Keycloak serves HTTP and trusts the proxy headers. ---
   hostname:
-    hostname: https://keycloak.home.keithjackson.dev
+    hostname: https://keycloak.${BASE_DOMAIN}
   http:
     httpEnabled: true                      # TLS is terminated at the Gateway (edge), not in Keycloak
   proxy:

platform/keycloak/keycloak-realm-import.yaml

@@ -98,9 +98,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:GRAFANA_CLIENT_SECRET)"
         redirectUris:
-          - https://grafana.home.keithjackson.dev/login/generic_oauth
+          - https://grafana.${BASE_DOMAIN}/login/generic_oauth
         webOrigins:
-          - https://grafana.home.keithjackson.dev
+          - https://grafana.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -125,10 +125,10 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:OAUTH2_PROXY_CLIENT_SECRET)"
         redirectUris:
-          - https://*.home.keithjackson.dev/oauth2/callback
+          - https://*.${BASE_DOMAIN}/oauth2/callback
-          - https://auth.home.keithjackson.dev/oauth2/callback
+          - https://auth.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://*.home.keithjackson.dev
+          - https://*.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -209,9 +209,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:GITEA_CLIENT_SECRET)"
         redirectUris:
-          - https://gitea.home.keithjackson.dev/user/oauth2/keycloak/callback
+          - https://gitea.${BASE_DOMAIN}/user/oauth2/keycloak/callback
         webOrigins:
-          - https://gitea.home.keithjackson.dev
+          - https://gitea.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -240,9 +240,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:NEXTCLOUD_CLIENT_SECRET)"
         redirectUris:
-          - https://nextcloud.home.keithjackson.dev/apps/user_oidc/code
+          - https://nextcloud.${BASE_DOMAIN}/apps/user_oidc/code
         webOrigins:
-          - https://nextcloud.home.keithjackson.dev
+          - https://nextcloud.${BASE_DOMAIN}
         # Post-logout redirect allow-list (#logout "redirect failed"). On sign-out user_oidc redirects the
         # browser to Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud base '/'> and
         # client_id=nextcloud but NO id_token_hint (provider sendIdTokenHint=0), so Keycloak validates the
@@ -250,7 +250,7 @@ spec:
         # "redirect failed". The wildcard covers the base URL and any logout landing path. Keycloak stores
         # this as the client attribute `post.logout.redirect.uris` (## separates multiple values).
         attributes:
-          post.logout.redirect.uris: https://nextcloud.home.keithjackson.dev/*
+          post.logout.redirect.uris: https://nextcloud.${BASE_DOMAIN}/*
         defaultClientScopes:
           - email
           - profile
@@ -279,9 +279,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:SEMAPHORE_CLIENT_SECRET)"
         redirectUris:
-          - https://semaphore.home.keithjackson.dev/api/auth/oidc/keycloak/redirect
+          - https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect
         webOrigins:
-          - https://semaphore.home.keithjackson.dev
+          - https://semaphore.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -318,9 +318,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:HEADLAMP_CLIENT_SECRET)"
         redirectUris:
-          - https://headlamp.home.keithjackson.dev/oidc-callback
+          - https://headlamp.${BASE_DOMAIN}/oidc-callback
         webOrigins:
-          - https://headlamp.home.keithjackson.dev
+          - https://headlamp.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -363,9 +363,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:CRAFTY_CLIENT_SECRET)"
         redirectUris:
-          - https://crafty.home.keithjackson.dev/oauth2/callback
+          - https://crafty.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://crafty.home.keithjackson.dev
+          - https://crafty.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -398,9 +398,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:PIHOLE_CLIENT_SECRET)"
         redirectUris:
-          - https://pihole.home.keithjackson.dev/oauth2/callback
+          - https://pihole.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://pihole.home.keithjackson.dev
+          - https://pihole.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -433,9 +433,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:JACKETT_CLIENT_SECRET)"
         redirectUris:
-          - https://jackett.home.keithjackson.dev/oauth2/callback
+          - https://jackett.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://jackett.home.keithjackson.dev
+          - https://jackett.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -459,9 +459,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:QBITTORRENT_CLIENT_SECRET)"
         redirectUris:
-          - https://qbittorrent.home.keithjackson.dev/oauth2/callback
+          - https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://qbittorrent.home.keithjackson.dev
+          - https://qbittorrent.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -485,9 +485,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:RADARR_CLIENT_SECRET)"
         redirectUris:
-          - https://radarr.home.keithjackson.dev/oauth2/callback
+          - https://radarr.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://radarr.home.keithjackson.dev
+          - https://radarr.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile
@@ -511,9 +511,9 @@ spec:
         directAccessGrantsEnabled: false
         secret: "$(env:SONARR_CLIENT_SECRET)"
         redirectUris:
-          - https://sonarr.home.keithjackson.dev/oauth2/callback
+          - https://sonarr.${BASE_DOMAIN}/oauth2/callback
         webOrigins:
-          - https://sonarr.home.keithjackson.dev
+          - https://sonarr.${BASE_DOMAIN}
         defaultClientScopes:
           - email
           - profile

platform/kubevirt/alerts.yaml

@@ -30,7 +30,7 @@ spec:
           annotations:
             summary: "KubeVirt VM(s) failing to launch in {{ $labels.namespace }}"
             description: "One or more VirtualMachineInstances are in the Failed phase in namespace {{ $labels.namespace }} for 5m — a failed VM launch (image import, scheduling, or qemu error). Check `kubectl get vmi -n {{ $labels.namespace }}` and the virt-launcher pod events."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-launch-failed"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-launch-failed"
         # Crash-loop: the VM's virt-launcher pod is restarting repeatedly (guest kernel panic, OOM, or a
         # crashing qemu). >3 restarts in 15m is a crash-loop, not a one-off reboot.
         - alert: KubeVirtVMCrashLooping
@@ -42,7 +42,7 @@ spec:
           annotations:
             summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
             description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-crashloop"
         # Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle
         # operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod
         # 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole
@@ -59,7 +59,7 @@ spec:
           annotations:
             summary: "KubeVirt control plane degraded"
             description: "The KubeVirt control plane (virt-controller/virt-api) is not ready or its metrics are absent for 10m — VM lifecycle operations (create/start/stop) will fail (docs/06)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-control-plane"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-control-plane"
         # CDI import stuck: a DataVolume import/clone still in flight after 30m blocks the VM it backs.
         # kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a positive value
         # sustained means an import is not progressing (metric name confirmed against CDI at bring-up).
@@ -71,4 +71,4 @@ spec:
           annotations:
             summary: "CDI DataVolume import stuck: {{ $labels.namespace }}/{{ $labels.name }}"
             description: "A CDI DataVolume import/clone in {{ $labels.namespace }} has not completed after 30m — the VM disk it backs cannot start. Check the importer pod (often an egress/default-deny block on the image source, or a slow mirror)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cdi-import-stuck"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cdi-import-stuck"

platform/longhorn/README.md

@@ -25,7 +25,7 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
 | Requirement | Summary | How this component serves it |
 |---|---|---|
 | **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
-| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
+| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the node's data disk at `${LONGHORN_DATA_PATH}` (a `cluster-vars` knob: dev `/var/lib/longhorn`, a dedicated disk; prod `/data/longhorn`, a subdirectory of the partition it shares with k3s's data dir). `replicaAutoBalance: best-effort` re-spreads on node return. |
 | **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
 
 ## Dev-scoped choices (change for prod)
@@ -48,9 +48,19 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
 
 ## Node prerequisites (already handled, informational)
 
-Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
+Each node presents ext4 storage at `${LONGHORN_DATA_PATH}` and installs `open-iscsi` +
-`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
+`nfs-common` + kernel modules — all via `roles/node_baseline` (`infra/cloud-init/` is a README
-`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
+now; provisioning became API-only). This component assumes that floor. The shape differs per
+environment, which is why the path is a knob:
+
+| | |
+|---|---|
+| dev | a **dedicated** disk (`/dev/vdb`, `LABEL=longhorn`) mounted at `/var/lib/longhorn` — the mount root *is* the dataPath |
+| prod | `/data/longhorn`, a **subdirectory** of the ~424 GB partition (`LABEL=k3s-data`) mounted at `/data`, whose sibling `/data/k3s` holds k3s's entire data dir (etcd + the containerd image store, moved off the 11.8 GiB `/var` — `infra/PROD-PROVISIONING.md` §2) |
+
+`[parity]`: the value for an environment MUST equal `node_baseline_longhorn_data_path` in that
+environment's Ansible group_vars. Ansible creates and mounts the directory; this component tells
+Longhorn to use it.
 
 ## Contract (consumed by everyone with state)
 

platform/longhorn/helmrelease.yaml

@@ -42,10 +42,22 @@ spec:
       defaultClass: false
 
     defaultSettings:
-      # --- data path (R-STATE-02) ---
+      # --- data path (R-STATE-02, R-ENV-02) ---
-      # Dedicated Longhorn data disk (/dev/vdb → ext4) is mounted here by cloud-init on every node
+      # The on-node directory every replica is written into. A cluster-vars KNOB, not a literal,
-      # (infra/cloud-init/user-data.yaml.tftpl). Longhorn stores replicas on this disk only.
+      # because the two environments mount their data disk differently and a hardcoded path would
-      defaultDataPath: "/var/lib/longhorn"
+      # put one of them on the OS filesystem:
+      #   dev  /var/lib/longhorn  — a dedicated ext4 disk (/dev/vdb) mounted there by
+      #                             roles/node_baseline. The mount root IS the dataPath: one disk,
+      #                             one tenant. 18 live volumes; this value must never change.
+      #   prod /data/longhorn     — a SUBDIRECTORY of the claimed ~424 GB partition mounted at
+      #                             /data, whose sibling /data/k3s holds k3s's entire data dir
+      #                             (etcd + the containerd image store, moved off the 11.8 GiB
+      #                             /var). Two tenants, so Longhorn gets a subdirectory rather
+      #                             than the mount root. infra/PROD-PROVISIONING.md §2.
+      # [parity] MUST equal roles/node_baseline's node_baseline_longhorn_data_path for the same
+      # environment — Ansible creates and mounts the directory, Helm tells Longhorn to use it.
+      # Set in BOTH overlays (clusters/dev, clusters/prod); base default is the dev value.
+      defaultDataPath: "${LONGHORN_DATA_PATH}"
 
       # --- replication (R-STATE-02) ---
       # Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly

platform/longhorn/values.yaml

@@ -8,7 +8,9 @@
 #   defaultClass: false                              # we own StorageClasses (storageclasses.yaml)
 #
 # defaultSettings:
-#   defaultDataPath: /var/lib/longhorn               # dedicated data disk (infra/cloud-init)
+#   defaultDataPath: ${LONGHORN_DATA_PATH}           # cluster-vars knob: dev /var/lib/longhorn
+#                                                    # (dedicated /dev/vdb), prod /data/longhorn
+#                                                    # (a subdirectory of the shared data partition)
 #   defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT}   # overlay knob (=3 in dev)
 #   replicaSoftAntiAffinity: false                   # HARD anti-affinity: one replica per node
 #   replicaZoneSoftAntiAffinity: true                # single zone (3 mini-PCs, no failure zones)

platform/monitoring/alerts.yaml

@@ -38,7 +38,7 @@ spec:
           annotations:
             summary: "Node exporter down on {{ $labels.instance }}"
             description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
         - alert: NodeMemoryPressureHigh
           expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
           for: 10m
@@ -47,7 +47,7 @@ spec:
           annotations:
             summary: "Memory pressure high on {{ $labels.instance }}"
             description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-memory"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
         - alert: NodeDiskFillingUp
           expr: |
             (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
@@ -58,7 +58,7 @@ spec:
           annotations:
             summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
             description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-disk"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
         - alert: NodeCPUPressureHigh
           # PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
           # run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
@@ -71,7 +71,7 @@ spec:
           annotations:
             summary: "CPU pressure high on {{ $labels.instance }}"
             description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
         - alert: NodeCPULoadHigh
           # 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
           # runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
@@ -86,7 +86,7 @@ spec:
           annotations:
             summary: "Load per core high on {{ $labels.instance }}"
             description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
 
 # =====================================================================================
 # Storage — Longhorn
@@ -113,7 +113,7 @@ spec:
           annotations:
             summary: "Longhorn volume degraded: {{ $labels.volume }}"
             description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-degraded"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
         - alert: LonghornVolumeFaulted
           expr: longhorn_volume_robustness == 3
           for: 1m
@@ -122,7 +122,7 @@ spec:
           annotations:
             summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
             description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-faulted"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
         - alert: LonghornVolumeCapacityHigh
           expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
           for: 30m
@@ -131,7 +131,7 @@ spec:
           annotations:
             summary: "Longhorn volume near capacity: {{ $labels.volume }}"
             description: "Volume {{ $labels.volume }} is over 85% used for 30m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
         - alert: LonghornNodeStorageHigh
           expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
           for: 30m
@@ -140,7 +140,7 @@ spec:
           annotations:
             summary: "Longhorn node storage near capacity: {{ $labels.node }}"
             description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
 
 # =====================================================================================
 # GitOps — Flux
@@ -169,7 +169,7 @@ spec:
           annotations:
             summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/flux-reconcile"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
         - alert: FluxResourceSuspended
           expr: |
             max by (namespace, name, kind) (gotk_suspend_status) == 1
@@ -179,7 +179,7 @@ spec:
           annotations:
             summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
-            runbook_url: "https://home.keithjackson.dev/runbooks/flux-suspended"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
 
 # =====================================================================================
 # Certificates — cert-manager
@@ -205,7 +205,7 @@ spec:
           annotations:
             summary: "Certificate expiring soon: {{ $labels.name }}"
             description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-expiry"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
         - alert: CertificateExpiryImminent
           expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
           for: 1h
@@ -214,7 +214,7 @@ spec:
           annotations:
             summary: "Certificate expiry imminent: {{ $labels.name }}"
             description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-expiry"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
         - alert: CertificateNotReady
           expr: certmanager_certificate_ready_status{condition="False"} == 1
           for: 1h
@@ -223,7 +223,7 @@ spec:
           annotations:
             summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
             description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-issuance"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"
 
 # =====================================================================================
 # Networking — VPN / egress-gateway / denied flows (Hubble)
@@ -255,7 +255,7 @@ spec:
           annotations:
             summary: "Egress VPN tunnel down"
             description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/vpn-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
         # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
         # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
         # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
@@ -270,7 +270,7 @@ spec:
           annotations:
             summary: "Spike in policy-denied network flows"
             description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/denied-flows"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"
 
 # =====================================================================================
 # Backups — Velero
@@ -296,7 +296,7 @@ spec:
           annotations:
             summary: "Velero backup failed: schedule {{ $labels.schedule }}"
             description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-failure"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
         - alert: VeleroBackupPartialFailure
           expr: increase(velero_backup_partial_failure_total[1h]) > 0
           for: 5m
@@ -305,7 +305,7 @@ spec:
           annotations:
             summary: "Velero backup partially failed"
             description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-partial"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
         - alert: VeleroNoRecentBackup
           expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
           for: 1h
@@ -314,7 +314,7 @@ spec:
           annotations:
             summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
             description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-missed"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
 
 # =====================================================================================
 # Identity — Keycloak (SSO is a critical dependency — docs/15)
@@ -345,7 +345,7 @@ spec:
           annotations:
             summary: "Keycloak IdP down"
             description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
         - alert: KeycloakDatabaseDown
           # Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
           # DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
@@ -358,7 +358,7 @@ spec:
           annotations:
             summary: "Keycloak database unavailable"
             description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-db"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
 
 # =====================================================================================
 # The monitor's own capacity — don't let the monitor blind itself (docs/10)
@@ -386,7 +386,7 @@ spec:
           annotations:
             summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
             description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
-            runbook_url: "https://home.keithjackson.dev/runbooks/monitoring-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
         - alert: PrometheusTargetsMissing
           expr: |
             (count(up == 0) / count(up)) > 0.10
@@ -396,7 +396,7 @@ spec:
           annotations:
             summary: "More than 10% of Prometheus targets are down"
             description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
-            runbook_url: "https://home.keithjackson.dev/runbooks/targets-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"
 
 # =====================================================================================
 # Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
@@ -430,7 +430,7 @@ spec:
           annotations:
             summary: "Flux image-update-automation failing: {{ $labels.name }}"
             description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
-            runbook_url: "https://home.keithjackson.dev/runbooks/image-automation"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
         # An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
         # auth / bad range) → that service stops receiving updates even though automation is healthy.
         - alert: ImagePolicyStalled
@@ -444,7 +444,7 @@ spec:
           annotations:
             summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/image-policy-stalled"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
         # A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
         # holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
         # (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
@@ -461,4 +461,4 @@ spec:
           annotations:
             summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
             description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/update-rollback"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"

platform/monitoring/grafana-httproute.yaml

@@ -14,7 +14,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - grafana.home.keithjackson.dev
+    - grafana.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

platform/monitoring/kube-prometheus-stack.yaml

@@ -194,15 +194,15 @@ spec:
             optional: true
       grafana.ini:
         server:
-          root_url: https://grafana.home.keithjackson.dev
+          root_url: https://grafana.${BASE_DOMAIN}
         auth.generic_oauth:
           enabled: true
           name: Keycloak
           client_id: grafana
           scopes: "openid email profile"
-          auth_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/auth
+          auth_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/auth
-          token_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/token
+          token_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/token
-          api_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/userinfo
+          api_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/userinfo
           login_attribute_path: preferred_username
           email_attribute_path: email
           # Map Keycloak groups (the `groups` claim) to Grafana roles (R-IDP-06): platform-admins ->

platform/oauth2-proxy/helmrelease.yaml

@@ -32,7 +32,7 @@ spec:
       existingSecret: oauth2-proxy-credentials
     extraArgs:
       provider: oidc
-      oidc-issuer-url: https://keycloak.home.keithjackson.dev/realms/homelab
+      oidc-issuer-url: https://keycloak.${BASE_DOMAIN}/realms/homelab
       # Forward-auth (ext-auth) mode: no upstream app; return 202 for an authenticated session so the
       # Gateway ext-authz can allow/deny. Auth headers are surfaced to the backend.
       reverse-proxy: "true"
@@ -41,9 +41,9 @@ spec:
       pass-access-token: "true"
       email-domain: "*"
       # Single sign-in/callback host for all forward-auth-protected services on the platform zone.
-      redirect-url: https://auth.home.keithjackson.dev/oauth2/callback
+      redirect-url: https://auth.${BASE_DOMAIN}/oauth2/callback
-      whitelist-domain: ".home.keithjackson.dev"
+      whitelist-domain: ".${BASE_DOMAIN}"
-      cookie-domain: ".home.keithjackson.dev"
+      cookie-domain: ".${BASE_DOMAIN}"
       cookie-secure: "true"
       # Expose group membership so per-service group authorization (R-IDP-06) can be enforced. The
       # Keycloak `oauth2-proxy` client emits a `groups` claim (full.path=false => bare names like

platform/oauth2-proxy/httproute.yaml

@@ -14,7 +14,7 @@ spec:
       namespace: gateway
       sectionName: https
   hostnames:
-    - auth.home.keithjackson.dev
+    - auth.${BASE_DOMAIN}
   rules:
     - matches:
         - path:

schemas/servicedefinition.schema.json

@@ -9,7 +9,7 @@
   "properties": {
     "name": {
       "type": "string",
-      "description": "DNS label -> <name>.home.keithjackson.dev (docs/13, docs/09). Must be a valid RFC 1123 DNS label.",
+      "description": "DNS label -> <name>.${BASE_DOMAIN}, i.e. <name>.home.keithjackson.dev on dev and <name>.prod.home.keithjackson.dev on prod (docs/13, docs/09). Must be a valid RFC 1123 DNS label.",
       "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$",
       "minLength": 1,
       "maxLength": 63
@@ -223,8 +223,8 @@
         "properties": {
           "host": {
             "type": "string",
-            "description": "FQDN under the platform zone. Must be a <label>.home.keithjackson.dev name (docs/09).",
+            "description": "Hostname under the platform zone, as a single DNS label plus the ${BASE_DOMAIN} token — e.g. 'gitea.${BASE_DOMAIN}' (docs/09). ${BASE_DOMAIN} is the per-environment hostname suffix from clusters/base/cluster-vars.yaml, resolved by Flux postBuild substitution at reconcile time (dev: home.keithjackson.dev, prod: prod.home.keithjackson.dev). AUTHOR THE TOKEN, NOT A LITERAL: a literal suffix is what let prod be authored to serve dev's public hostnames out of the shared Cloudflare zone, and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10). The literal 'home.keithjackson.dev' form stays accepted so the L1 render fixtures — which are renderer INPUTS, not reconciled manifests — keep validating unchanged.",
-            "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.home\\.keithjackson\\.dev$"
+            "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.(\\$\\{BASE_DOMAIN\\}|home\\.keithjackson\\.dev)$"
           },
           "port": {
             "type": "integer",