feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite

OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach the cluster and provide fault tolerant HA. Without a change it will not work in HA." THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live before changing anything: all three prod cilium-agent pods carried KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated that stale comment as an open question when the evidence already settled it. SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs: k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join target and a tls-san. infra/tofu/outputs.tf GENERATES this name, so renaming would ripple into tofu and the runbooks. k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium consumes. Derived, not restated, so an environment overrides ONE line and the VIP, the SAN and the CNI endpoint move together. Prod needed no new override. THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran node-baseline -> k3s -> cilium, so Cilium would install against an address nothing announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value: site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier (kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so a cold run stops THERE with a named cause instead of surfacing as unreachable agents. Also fixed a pre-existing bug that only mattered once this play joined site.yml: the role's read-only interface probe ran without `check_mode: false`, so under --check it was skipped and the assert fired with a misleading "set kube_vip_interface explicitly". VERIFIED BY ME, not just reported: renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP) cilium template consumes k3s_apiserver_endpoint, not k3s_api_host cold join still `--server https://{{ k3s_api_host }}:6443` — the founder order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux gate full CI gate PASS Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders ['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved — .9 already answers under full CA verification; and --check --diff of kube-vip.yml against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the kube-vip prod already runs. ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine checks: the Cilium template must use the endpoint var; the endpoint must derive from kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal that environment's VIP and be none of its node IPs (the [parity] contract, now machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium. Every new assertion mutation-tested and bites. One honest negative recorded: reverting only the BASE default is not caught, because both overlays patch the key — the check reads effective per-environment values, so the base default is unguarded. APPLY ORDER, and this is the operator's step — nothing was applied: PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01 incident), it will not proceed past an unready pod, existing datapath survives an agent restart, and .9 is already live before the change lands. DEV SEPARATELY, operator present. Same mechanics but live dev agents dial 192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that. ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook. NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that test's gap list. Refs R-PLAT-01, R-NET-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>
fix(kyverno): unblock CDI imports by SATISFYING the label requirement, not exempting it
2026-09-06 20:25:44 -05:00 · 2026-09-06 08:41:18 -05:00 · 2026-09-06 08:36:42 -05:00 · 2026-09-04 22:43:54 -05:00 · 2026-09-04 18:53:17 -05:00 · 2026-09-04 00:39:10 -05:00
113 changed files with 6476 additions and 535 deletions
@@ -102,7 +102,7 @@ spec:
            userData: |-
              #cloud-config
              hostname: unifi
-              fqdn: unifi.home.keithjackson.dev
+              fqdn: unifi.${BASE_DOMAIN}
              package_update: true
              package_upgrade: false
              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
@@ -46,7 +46,7 @@ metadata:
    lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
    # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
    # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
-    external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
@@ -90,9 +90,9 @@ spec:
              name: gitea-bootstrap
              key: secret-key
        - name: GITEA__server__DOMAIN
-          value: gitea.home.keithjackson.dev
+          value: gitea.${BASE_DOMAIN}
        - name: GITEA__server__ROOT_URL
-          value: https://gitea.home.keithjackson.dev/
+          value: https://gitea.${BASE_DOMAIN}/
        - name: GITEA__actions__ENABLED
          value: 'true'
        - name: GITEA__actions__DEFAULT_ACTIONS_URL
@@ -127,7 +127,7 @@ spec:
              command:
              - /bin/sh
              - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
                \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
      volumes:
      - name: data
@@ -207,7 +207,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - gitea.home.keithjackson.dev
+  - gitea.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:9117
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=media-users
        - --skip-provider-button=true
-        - --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - jackett.home.keithjackson.dev
+  - jackett.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:80
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=platform-admins
        - --skip-provider-button=true
-        - --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -229,7 +229,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - pihole.home.keithjackson.dev
+  - pihole.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -183,7 +183,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - plex.home.keithjackson.dev
+  - plex.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8080
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=media-users
        - --skip-provider-button=true
-        - --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - qbittorrent.home.keithjackson.dev
+  - qbittorrent.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -165,7 +165,7 @@ spec:
              exec /sbin/tini -- /opt/act/run.sh
          env:
            - name: GITEA_INSTANCE_URL
-              value: https://gitea.home.keithjackson.dev
+              value: https://gitea.${BASE_DOMAIN}
            - name: GITEA_RUNNER_REGISTRATION_TOKEN
              valueFrom:
                secretKeyRef:
@@ -104,13 +104,13 @@ spec:
              name: nextcloud-bootstrap
              key: admin-password
        - name: NEXTCLOUD_TRUSTED_DOMAINS
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
        - name: OVERWRITEHOST
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
        - name: OVERWRITEPROTOCOL
          value: https
        - name: OVERWRITECLIURL
-          value: https://nextcloud.home.keithjackson.dev
+          value: https://nextcloud.${BASE_DOMAIN}
        - name: NEXTCLOUD_OIDC_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
@@ -125,9 +125,9 @@ spec:
              command:
              - /bin/sh
              - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc]\
-                [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
+                \ client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
-                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
+                ); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
      volumes:
      - name: data
        persistentVolumeClaim:
@@ -228,7 +228,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - nextcloud.home.keithjackson.dev
+  - nextcloud.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:7878
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=media-users
        - --skip-provider-button=true
-        - --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - radarr.home.keithjackson.dev
+  - radarr.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -110,7 +110,7 @@ spec:
        - name: SEMAPHORE_ADMIN_NAME
          value: Semaphore Admin
        - name: SEMAPHORE_ADMIN_EMAIL
-          value: admin@home.keithjackson.dev
+          value: admin@${BASE_DOMAIN}
        - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
          valueFrom:
            secretKeyRef:
@@ -231,7 +231,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - semaphore.home.keithjackson.dev
+  - semaphore.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8989
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=media-users
        - --skip-provider-button=true
-        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - sonarr.home.keithjackson.dev
+  - sonarr.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -223,7 +223,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - seerr.home.keithjackson.dev
+  - seerr.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #
-# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
+# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
 # it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
 # `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
 # same script bootstraps dev and prod with no fork (R-ENV-02).
 #
 # Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
 # =============================================================================
@@ -33,7 +36,7 @@
 #   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
+#   3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
 set -euo pipefail
 # --- config (env-overridable) ------------------------------------------------
-FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"     # dev is the ONLY Phase-1 build target
+# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
 # prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
 # every existing caller — override it, never edit it.
 FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
 FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
 SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
 # The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,7 +68,7 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
-echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
+echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
 for bin in flux kubectl; do
@@ -134,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
  exit 1
 fi
-# --- 3. apply the authored dev entrypoint ------------------------------------
+# --- 3. apply the authored environment entrypoint ----------------------------
 echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 # bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
-# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
+# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
-# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
+# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
-# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
+# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
-# CONTRIBUTING.md).
+# using it now.
 #
-# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
+# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
 # It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
 #
 #   PROD (bare metal, current use):  HOST_NAT_IP=192.168.2.132
 #       The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
 #       NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
 #       three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
 #       apply to prod, so do not "fix" prod by opening zones.
 #
 #   DEV (libvirt VMs, HISTORICAL):   HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
 #       *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
 #       bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
 #       recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
 #
 # WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
 # bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
 # author-only work and must never be fired off casually or "to see if it works": it starts a
 # long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
 # and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
 # it will then reconcile. Know which HEAD you are publishing before you run it.
 set -euo pipefail
 ENGINE="${ENGINE:-podman}"
 GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}"   # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
 # subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
 DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
 DATA_DIR="podman volume ${DATA_VOLUME}"   # display label only (data lives in the named volume)
-# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
+# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
 # clone/redirect URLs are correct from inside the cluster.
-# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
+#
 # The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
 # (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
 # publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
 # Pass it explicitly, e.g. for prod:   HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
 #
 # NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
 # VM->host:3000 by default and had to be opened once with
 #   sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
-HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
+# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
 if [ -z "${HOST_NAT_IP:-}" ]; then
  echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
  echo "       prod: HOST_NAT_IP=192.168.2.132   (same LAN as the nodes, no NAT, no firewall change)" >&2
  echo "       There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
  exit 1
 fi
 ORG="${GITEA_ORG:-Home-Network}"
 REPO="${GITEA_REPO:-home-network-k3s-impl}"
@@ -75,8 +107,17 @@ cat <<EOF
 Gitea bootstrap source ready (${DATA_DIR}).
  UI (host):                     ${BASE}/${ORG}/${REPO}
-  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git   (clusters/dev)
+  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
                                 (patch this url into the overlay being bootstrapped:
                                  clusters/prod today; clusters/dev has already cut over)
  Admin creds (host, 0600):      ${ADMIN_CRED_FILE}
-Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
+Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
 firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
 must allow VM->host:${HTTP_PORT} —
  ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
 REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
 at this source will reconcile THAT revision.
 TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
 any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
 procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
 EOF
@@ -24,9 +24,9 @@ env:
        name: gitea-bootstrap
        key: secret-key
  - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.home.keithjackson.dev
+    value: gitea.${BASE_DOMAIN}
  - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
  - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
    value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
  - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,7 +104,7 @@ lifecycle:
          # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
          # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
          # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
          gitea_oidc_fail() {
            echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
@@ -168,7 +168,7 @@ lifecycle:
          # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
          # the moment its OIDC guard was fixed and the block actually ran for the first time.
          exit 0
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
  - name: data
@@ -178,7 +178,7 @@ storage:
                                # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
    port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
@@ -17,7 +17,7 @@ name: jackett
 kind: container
 source:
  image: lscr.io/linuxserver/jackett:latest
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
  - name: config
@@ -25,7 +25,7 @@ storage:
    class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
    port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
  mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
@@ -18,7 +18,7 @@ dependencies:
    port: 5432
    protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
    port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
  mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
@@ -50,13 +50,13 @@ env:
  #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
  #     though it speaks plain HTTP behind the proxy. ---
  - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
  - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
  - name: OVERWRITEPROTOCOL
    value: https
  - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
  # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
  #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
  #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
          # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
          # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
          OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
          nextcloud_oidc_fail() {
            echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -228,7 +228,7 @@ dependencies:
    port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
    protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
    port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
@@ -14,7 +14,7 @@ storage:
    class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
    port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
  mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
@@ -72,7 +72,7 @@ storage:
                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
    port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
  mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
@@ -32,7 +32,7 @@ name: qbittorrent
 kind: container
 source:
  image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
  - name: config
@@ -40,7 +40,7 @@ storage:
    class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
    port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
  mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
@@ -9,7 +9,7 @@ name: radarr
 kind: container
 source:
  image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
    port: 9117
    protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
    port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
@@ -44,7 +44,15 @@ except ImportError:  # pragma: no cover - explicit, actionable failure
 # --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
 LBL = "homelab.keithjackson.dev"
-ZONE = "home.keithjackson.dev"
+# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
 # Every rendered manifest lands under a Flux path whose Kustomization declares
 # postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
 # reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
 # It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
 # the escape contract below reserves `$${...}` for expansions meant for a container's shell.
 # Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
 # and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
 ZONE = "${BASE_DOMAIN}"
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
@@ -57,7 +57,7 @@ dependencies:
    port: 32400
    protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
    port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
@@ -58,7 +58,7 @@ env:
  - name: SEMAPHORE_ADMIN_NAME
    value: Semaphore Admin
  - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
  # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
  #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
  - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -110,7 +110,7 @@ dependencies:
    port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
    protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
    port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
@@ -9,7 +9,7 @@ name: sonarr
 kind: container
 source:
  image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
    port: 9117
    protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
    port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
@@ -97,6 +97,27 @@ Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
 Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)
 ```
 ### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
 The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
 controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
 and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
 reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
 before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
 **deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
 `platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
 have existed for months); it cost 12h on prod's cold start.
 So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
 platform-config`), and a rule:
 > A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
 > **controller at runtime** rather than by a manifest in git.
 `apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
 `StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
 profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
 Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
 exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
 (`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
@@ -4,8 +4,11 @@
 # NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
 # import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
 # health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
-# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
+# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
-# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
+# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
 # platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
 # them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
 # adoption concern).
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
@@ -19,6 +22,28 @@ spec:
  wait: false
  dependsOn:
    - name: platform-config
    # LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
    # storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
    # decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
    # (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
    # live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
    # with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
    # to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
    # conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
    #
    # HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
    # derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
    # status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
    # it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
    #
    # This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
    # deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
    # dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
    #
    # The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
    # edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
    # cannot silently drop this tier's KubeVirt/StorageClass gate.
    - name: platform-config-cdi
  sourceRef:
    kind: GitRepository
    name: flux-system
@@ -17,9 +17,63 @@ data:
  CLUSTER_ENV: "base"                       # dev | prod  (overlays set this)
  # --- DNS / TLS ---
-  BASE_DOMAIN: "home.keithjackson.dev"      # platform hostname suffix (same both envs; split-horizon)
+  # THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
-  DNS_ZONE: "keithjackson.dev"              # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
+  # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
-                                            # ExternalDNS domain-filter must be the zone, not the subdomain
+  # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
  #
  # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
  # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
  # subdomain, so the two clusters can no longer describe the same name.
  #
  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
  #   prod  -> prod.home.keithjackson.dev
  #
  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
  BASE_DOMAIN: "home.keithjackson.dev"
  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
  # so it cannot be created, updated or deleted regardless of TXT ownership.
  #
  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
  #
  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
  # — it can never delete another environment's records — and the fallback is to set prod's value
  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
  # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
  # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
  #
  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
  #
  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
  ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
  GATEWAY_VIP: "192.168.1.53"                # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
                                            # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
@@ -47,10 +101,37 @@ data:
  # ship the literal string into the cluster.
  # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
-  # [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
+  #
-  # main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
+  # *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
-  # reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
+  # 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
-  K3S_API_HOST: "192.168.1.50"
+  # control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
  # live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
  # own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
  # Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
  # outage you would notice. A virtual address that follows the control plane is the fix, and it is
  # what kube-vip was stood up to provide.
  #
  #   dev  -> 192.168.1.49   (= kube_vip_address, group_vars/all.yml; base default, below)
  #   prod -> 192.168.2.9    (= kube_vip_address, group_vars/k3s_prod/vars.yml)
  #
  # THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
  # unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
  # move. Read it as "the API host Cilium dials", which is what it always meant.
  #
  # [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
  # (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
  # already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
  # the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
  # already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
  # block at the top of group_vars/all.yml.
  #
  # Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
  # in both envs, so it is NOT a knob.
  #
  # TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
  # both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
  # https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
  K3S_API_HOST: "192.168.1.49"
  # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
  # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
@@ -74,6 +155,17 @@ data:
  # --- storage (ADR-0003) ---
  LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
  LONGHORN_DATA_PATH: "/var/lib/longhorn"
  # --- observability sizing (docs/10) ---
  MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long
@@ -14,6 +14,15 @@ resources:
  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
  - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
  - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
  # manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
  # datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
  # platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
  # keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
  # CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
  # cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
  # as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
  - platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
  # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
  # Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
  # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
@@ -0,0 +1,135 @@
 ---
 # Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
 # Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
 #
 # WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
 #
 # `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
 # kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
 # `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
 # So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
 # platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
 # deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
 # have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
 #
 # `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
 # stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
 # graph order itself, so a cold cluster needs no human in the loop.
 #
 # THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
 # against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
 # version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
 # N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
 # not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
 # reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
 # kind. One edge, no pre-apply.
 #
 # WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
 # and everything else waits on: putting CDI's health on that critical path is the mistake the
 # oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
 # StorageProfiles that were in the wrong pass.
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: platform-config-cdi
  namespace: flux-system
 spec:
  interval: 10m
  # 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
  # very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
  # now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
  # about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
  # cache is one retryInterval. Keep it short.
  retryInterval: 2m
  # 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
  # image to pull and no controller to become Available, so a slow apply here means a real error, not
  # a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
  timeout: 2m
  # TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
  # component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
  # It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
  # second path also declares. See the transition note at the bottom of this file.
  prune: true
  # TRUE, but understand exactly what it buys and what it does NOT.
  #
  # CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
  # provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
  # `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
  # it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
  # is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
  # If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
  # `apps-vms` waits on it.
  #
  # WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
  # provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
  # storageprofile-controller computes that status from our spec override plus inference. So Ready
  # here means "the spec override is accepted by the API server", not "CDI has folded it into
  # status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
  # below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
  # a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
  wait: true
  dependsOn:
    # platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
    # causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
    # ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
    # flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
    - name: platform-config
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./platform-config-cdi
  # NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
  # no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
  # oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
  # clusters/dev or clusters/prod for this tier.
  #
  # substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
  # substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
  #   1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
  #      whose Kustomization declares substituteFrom. Omitting it would move
  #      cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
  #      could ship as a literal string, precisely the class of bug that check exists to stop.
  #   2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
  #      with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
  #      empty. Without the stanza there is no substitution at all and the literal ships silently.
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
  # ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
  # Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
  # Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
  # StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
  # the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
  # dual-ownership fight — the objects are pruned before they are re-applied under this tier's
  # `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
  #
  # Why that is acceptable here — and it is NOT the platform/cilium case:
  #   * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
  #     nothing else in the repo declares a StorageProfile. The stale set is exactly two
  #     cluster-scoped CDI CRs.
  #   * The objects do not actually go missing: CDI's storageprofile-controller creates a
  #     StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
  #     inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
  #     absent".
  #   * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
  #     existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
  #     is disturbed and no data path is touched.
  #   * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
  #     instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
  #     permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
  #     dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
  #     created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
  #     could.
  #
  # PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
  # happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
  # config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
  # healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
  # --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
  # StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
  #   `flux get kustomization platform-config` must be Ready BEFORE merging.
  # If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
  # temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
  # this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
  # `prune: true` in a second commit.
@@ -148,6 +148,33 @@ patches:
      - op: replace
        path: /data/ACME_CLUSTER_ISSUER
        value: "letsencrypt-prod"
      # --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
      # DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
      # like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
      # Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
      # Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
      # (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
      # prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
      # not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
      # not a variable change: infra/PROD-PROVISIONING.md §10.
      - op: replace
        path: /data/BASE_DOMAIN
        value: "home.keithjackson.dev"
      # ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
      # and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
      # the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
      - op: replace
        path: /data/EXTERNALDNS_DOMAIN_FILTER
        value: "keithjackson.dev"
      # DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
      # registry under owner `default` (the external-dns chart default, which is what both clusters
      # silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
      # BY this string: any other value and it stops recognising every record it manages, then
      # re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
      # whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
      - op: replace
        path: /data/EXTERNALDNS_TXT_OWNER_ID
        value: "default"
      # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
      # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
      # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -160,11 +187,25 @@ patches:
      - op: replace
        path: /data/GATEWAY_VIP
        value: "192.168.1.53"
-      # k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
+      # k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
-      # (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
+      # the address kube-vip floats, NOT the primary node .50.
      # [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
      # (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
      #
      # *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
      # Every other value in this block is what dev is running right now. This one is not:
      # the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
      # DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
      # AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
      # in parity for exactly this reason). Do not push this to dev-live casually, and not
      # while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
      # but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
      # already (k3s_tls_sans) and dev's own kubeconfig validates against it via
      # tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
      # same LAN as the nodes and ARP-announced by kube-vip on every server.
      - op: replace
        path: /data/K3S_API_HOST
-        value: "192.168.1.50"
+        value: "192.168.1.49"
      # Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
      # (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
      - op: replace
@@ -190,6 +231,16 @@ patches:
      - op: replace
        path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
        value: "3"
      # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
      # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
      # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
      # else would orphan every one of them (they are not recoverable from the old path by
      # rescheduling). Restated explicitly even though it equals the base default, exactly like the
      # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
      # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
      - op: replace
        path: /data/LONGHORN_DATA_PATH
        value: "/var/lib/longhorn"
      # smaller retention / volumes to keep the dev workstation footprint light.
      - op: replace
        path: /data/MONITORING_PROMETHEUS_RETENTION
@@ -115,8 +115,20 @@ patches:
            name: sops-age
  # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
  #
-  # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
+  # ############################################################################################
  # ###                                                                                      ###
  # ###   *** T E M P O R A R Y   B O O T S T R A P   S O U R C E  —  NOT PERMANENT ***       ###
  # ###                                                                                      ###
  # ###   The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside   ###
  # ###   any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST   ###
  # ###   be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this  ###
  # ###   and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is  ###
  # ###   a live single point of failure, not a settled design. See "CUTOVER" below.          ###
  # ###                                                                                      ###
  # ############################################################################################
  #
  # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
  # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
  # Two independent reasons, both consequences of the split-horizon design:
  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
@@ -126,13 +138,46 @@ patches:
  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
  #      gitea.home.keithjackson.dev to itself and never reach a repository.
  #
-  # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
+  # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
-  # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
+  #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
-  # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
+  #   operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
  #   reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
  #   change was needed. This is the same "local bootstrap source" pattern dev itself used before
  #   its Phase-6 cutover (docs/05 "Source-of-truth").
  #
  # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
  # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
  # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
  # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
  #
  # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
  # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
  #   curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
  #   git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git  -> refs/heads/main
  # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
  # THAT FIRST: a private repo here fails exactly that way.
  #
  # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
  #   WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
  #         through prod's own Gateway VIP with a publicly-trusted cert.
  #   DO:   1. push/mirror this repo into the prod in-cluster Gitea;
  #         2. change the url below to
  #            https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
  #            and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
  #            clusters/dev/kustomization.yaml, is what "done" looks like);
  #         3. confirm `flux get sources git -n flux-system` reconciles the new revision;
  #         4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
  #            exactly as dev retained its bootstrap Gitea.
  #   UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
  #   powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
  #   it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
  #
  # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
  # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
  # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
  # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
  # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
  # is the revision you intend prod to run (`git ls-remote` above, against local `main`).
  - patch: |-
      apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
@@ -140,7 +185,9 @@ patches:
        name: flux-system
        namespace: flux-system
      spec:
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
        ref:
          branch: main
  - target:
@@ -154,9 +201,80 @@ patches:
      - op: replace
        path: /data/ACME_CLUSTER_ISSUER
        value: "letsencrypt-prod"
      # ==========================================================================================
      # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
      # ==========================================================================================
      # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
      # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
      # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
      # of which already happened:
      #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
      #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
      #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
      #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
      #      have raced each other's `_acme-challenge` TXT records on that one zone.
      # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
      # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
      #
      #   dev   -> home.keithjackson.dev         (UNCHANGED — dev is live; phase 2 moves it)
      #   prod  -> prod.home.keithjackson.dev    (this knob)
      #
      # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
      # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
      # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
      # `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
      # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
      #
      # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
      # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
      # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
      # trusts a different Keycloak than the one prod actually runs.
      - op: replace
        path: /data/BASE_DOMAIN
        value: "prod.home.keithjackson.dev"
      # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
      # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
      # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
      # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
      # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
      # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
      # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
      # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
      - op: replace
        path: /data/EXTERNALDNS_DOMAIN_FILTER
        # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
        # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
        # record then skipped it with "no hosted zone matching record DNS Name", so prod would
        # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
        # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
        # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
        # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
        value: "keithjackson.dev"
      # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
      # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
      # its own and, under policy: sync, deleted the records behind them. This value MUST differ from
      # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
      # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
      - op: replace
        path: /data/EXTERNALDNS_TXT_OWNER_ID
        value: "prod"
      - op: replace
        path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
        value: "3"
      # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
      # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
      # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
      # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
      # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
      # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
      #
      # [parity] MUST equal node_baseline_longhorn_data_path in
      # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
      # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
      # path on the OS filesystem instead of the data partition and /var fills up.
      - op: replace
        path: /data/LONGHORN_DATA_PATH
        value: "/data/longhorn"
      # longer retention / larger volumes on prod NVMe.
      - op: replace
        path: /data/MONITORING_PROMETHEUS_RETENTION
@@ -222,14 +340,32 @@ patches:
      - op: replace
        path: /data/CILIUM_LB_POOL_STOP
        value: "192.168.2.30"
      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
      # k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
-      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
+      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
-      # Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
+      #
-      # platform/cilium/helmrelease.yaml. Not yet decided.
+      # *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
      # read "Not yet decided" while shipping .10 — the founder's own address — and that is what
      # the LIVE prod cluster is running: all three cilium-agents carry
      # KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
      # 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
      # the datapath keeps running off Cilium's cache while no service or policy update lands, with
      # nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
      # control plane is, which is the whole reason kube-vip exists here.)
      #
      # It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
      # this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
      # equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
      # that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
      # `--server` join needs a host that is up before any VIP is floated. Two addresses, two
      # variables, deliberately.
      #
      # .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
      # LB-IPAM can never be handed the same IP — asserted by
      # conformance/tests/l0-static/ha-control-plane-as-code.
      - op: replace
        path: /data/K3S_API_HOST
-        value: "192.168.2.10"
+        value: "192.168.2.9"
      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
      # Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
      # prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
      # they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
@@ -68,7 +68,7 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8080
        - --email-domain=*
        - --scope=openid email profile
@@ -76,8 +76,8 @@ spec:
        - --allowed-group=platform-admins
        - --skip-provider-button=true
        - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -91,7 +91,7 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8989
        - --email-domain=*
        - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
        - --allowed-group=media-admins
        - --skip-provider-button=true
        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -91,7 +91,7 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8989
        - --email-domain=*
        - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
        - --allowed-group=media-admins
        - --skip-provider-button=true
        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOP
 > ```
 > (Both take the same target-group override as the rest — see **Targeting prod** below.)
 > **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
 > ```
 > ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
 > ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey
 > ```
 > `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
 > until `-e memattr_survey_reboots_authorised=true` is passed. See
 > `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
 ## Re-provisioning a node (tofu -replace / fresh OS)
 A node rebuild wipes **everything on the host filesystem** — including host-level config
 that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
@@ -70,11 +79,30 @@ Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
 afterward. The parity-critical values **must match** between this role's
 `templates/cilium-values.yaml.j2` and that HelmRelease:
 - `kubeProxyReplacement: true`
- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
+- `k8sServiceHost: {{ k3s_apiserver_endpoint }}` — **the kube-vip CONTROL-PLANE VIP**
- `k8sServicePort: 6443`
+  (`= kube_vip_address`: dev `192.168.1.49`, prod `192.168.2.9`), **not** `k3s_api_host`
 - `k8sServicePort: {{ k3s_api_port }}` (6443 in both environments)
-The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
+### Two API addresses, split 2026-09-06 — do not re-merge them
-to the same value this role uses (surfaced by `tofu output primary_node_ip`).
+`k3s_api_host` and `k3s_apiserver_endpoint` were one variable, and the conflation put every
 prod `cilium-agent` on `192.168.2.10` (`k3s-prod-1`'s own address) on a cluster whose point is
 HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no
 service or policy update ever lands, which is a silent failure, not an outage.
 | variable | value | consumed by | why it must be that address |
 |---|---|---|---|
 | `k3s_api_host` | the founder's own node IP (dev `192.168.1.50`, prod `192.168.2.10`) | `roles/k3s_server`: the secondary `--server https://…:6443` join URL, and one `k3s_tls_sans` entry | on a **cold** build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran `--cluster-init` and is up |
 | `k3s_apiserver_endpoint` | `{{ kube_vip_address }}` (dev `192.168.1.49`, prod `192.168.2.9`) | `roles/cilium` → `k8sServiceHost`; any steady-state client | it must survive losing any single server |
 Consequences already handled: `playbooks/site.yml` imports `kube-vip.yml` **between** `k3s.yml`
 and `cilium.yml` (the VIP has to be floating before the CNI dials it), and `roles/kube_vip`'s
 last task blocks until the VIP answers on 6443. `k3s_tls_sans` already carried both addresses,
 so no cert reissue and no k3s restart are involved.
 The Flux side of the same contract is `${K3S_API_HOST}` in `platform/cilium/helmrelease.yaml`,
 set per overlay in `clusters/<env>/kustomization.yaml`; `[parity]` means it must equal
 `k3s_apiserver_endpoint` for the SAME environment. Asserted by
 `conformance/tests/l0-static/ha-control-plane-as-code`.
 ## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
 `roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
@@ -109,6 +137,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
 `playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
 keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
 bring-up chain.
 `playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
 for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
 role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
 can be mistyped; inventory membership cannot.
 ```bash
 # dev — completely unchanged, no extra vars
@@ -135,7 +167,9 @@ ansible-playbook playbooks/site.yml \
 What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
 `k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
 target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
-node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
+node-IP list), `k3s_api_host=192.168.2.10` (the founder, bootstrap-join only),
 `kube_vip_address=192.168.2.9` — and therefore `k3s_apiserver_endpoint=192.168.2.9`, which is what
 Cilium dials — and
 `cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
 `.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
@@ -27,19 +27,85 @@ k3s_servers_group: k3s_servers
 # a kubeconfig is a credential.)
 cluster_kubeconfig_name: dev.kubeconfig
-# The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP.
+# --- TWO API ADDRESSES, AND THEY ARE NOT INTERCHANGEABLE (split 2026-09-06) ----------
 # This used to be ONE variable, k3s_api_host, consumed by BOTH the cold-bootstrap join and
 # Cilium's k8sServiceHost. That conflation is what pointed every cilium-agent on the LIVE
 # prod cluster at 192.168.2.10 — k3s-prod-1's own address — on a cluster whose entire
 # purpose is HA: verified 2026-09-06, all three agents carried
 # KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held 192.168.2.9. Lose that one node
 # and no service/policy update reaches the CNI until it returns (the existing datapath
 # survives on Cilium's cache, so it fails SILENTLY). The two consumers have genuinely
 # different requirements, so they now have two variables.
 #
 #   k3s_api_host            THE FOUNDER'S OWN ADDRESS. Bootstrap-time only.
 #   k3s_apiserver_endpoint  THE HA ENDPOINT (the kube-vip VIP). Steady-state.
 #
 # DO NOT "simplify" these back into one. See each block below for why each half must be
 # the address it is.
 # THE FOUNDER'S OWN NODE IP — a BOOTSTRAP-TIME address, deliberately NOT the VIP.
 # NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node
-# IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san /
+# IP — so we read the real 192.168.1.x from k3s_node_ip instead.
-# secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux
+#
-# HelmRelease stay in PARITY (docs/02 contract).
+# EXACTLY TWO CONSUMERS, and both of them need the founder specifically:
 #   1. roles/k3s_server — the secondary `--server https://<host>:<port>` JOIN URL. On a COLD
 #      build the VIP DOES NOT EXIST YET at join time: kube-vip is a k3s auto-deploy manifest
 #      that only lands once a cluster exists, so a join aimed at the VIP is a join aimed at
 #      an unanswered address and the cold build is unjoinable. The founder is up (it just ran
 #      --cluster-init), which is the whole reason this variable is the founder.
 #   2. roles/k3s_server — one entry of k3s_tls_sans, so the founder's own address is a SAN
 #      alongside the VIP and every other server's IP.
 #
 # ANYTHING THAT RUNS AGAINST A CLUSTER THAT ALREADY EXISTS MUST NOT USE THIS. It is a
 # single point of failure by construction. Use k3s_apiserver_endpoint below.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}"
 # Shared by both halves: k3s serves the API on 6443 in every environment, so the port is
 # not a per-environment knob and is not duplicated per address.
 k3s_api_port: 6443
 # THE HA kube-apiserver ENDPOINT — a VIRTUAL address that reaches the cluster wherever the
 # control plane currently is, not a machine. OPERATOR DECISION 2026-09-06: in-cluster
 # components dial the VIP, as kube-vip was always intended to provide.
 #
 # DERIVED from kube_vip_address (below), never restated: an environment overrides the VIP in
 # ONE place (group_vars/k3s_prod/vars.yml sets kube_vip_address: 192.168.2.9) and this
 # follows automatically, so prod needs no override of its own and a fourth environment
 # cannot set one of the two and forget the other.
 #
 # NAMED FOR THE ROLE, NOT THE MECHANISM. Consumers want "the address that reaches the
 # apiserver, HA"; that it is currently floated by kube-vip is an implementation detail. If
 # the mechanism is ever something else (an external LB, a DNS name), this one line moves and
 # no consumer changes.
 #
 # CONSUMERS:
 #   - roles/cilium -> cilium-values.yaml.j2 -> k8sServiceHost.  [parity] with the Flux side:
 #     clusters/base/cluster-vars.yaml K3S_API_HOST (and each overlay's patch) MUST equal
 #     THIS value for that environment, or Flux and the bootstrap fight over the CNI's
 #     control-plane path. See platform/cilium/helmrelease.yaml's PARITY CONTRACT header.
 #
 # TLS IS ALREADY COVERED — verified, not assumed: k3s_tls_sans (roles/k3s_server/defaults)
 # has always included kube_vip_address, and on 2026-09-06 the prod apiserver was reached
 # through https://192.168.2.9:6443 with FULL CA verification and tls-server-name=192.168.2.9
 # (infra/ansible/.cache/prod.kubeconfig, no insecure-skip-tls-verify). So the serving cert
 # needs no change and no k3s restart for this switch.
 #
 # ORDERING CONSEQUENCE, and it is why playbooks/site.yml changed too: Cilium now needs the
 # VIP to be UP before it installs. site.yml therefore imports kube-vip.yml BETWEEN k3s.yml
 # and cilium.yml, and roles/kube_vip waits for the VIP to answer before returning. kube-vip
 # can precede the CNI because its DaemonSet is hostNetwork with
 # KUBERNETES_SERVICE_HOST=127.0.0.1 — it needs neither a CNI nor kube-proxy.
 k3s_apiserver_endpoint: "{{ kube_vip_address }}"
 # Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover
 # endpoint for the kube-apiserver, fronted by an operator pfSense port-forward at
-# 192.168.1.49:6443. Shared here so BOTH kube_vip AND the k3s_server tls-san list
+# 192.168.1.49:6443. Shared here so THREE things agree on the one address:
-# agree on the address (the serving cert must carry this IP or TLS via the VIP
+#   - roles/kube_vip          — what to ARP and lease-elect for;
-# fails to validate).
+#   - roles/k3s_server        — the tls-san entry (the serving cert must carry this IP or
 #                               TLS via the VIP fails to validate);
 #   - k3s_apiserver_endpoint  — above; what in-cluster components actually dial.
 # THIS is the one line an environment overrides (prod: 192.168.2.9, in
 # group_vars/k3s_prod/vars.yml); all three follow.
 kube_vip_address: "192.168.1.49"
 # Controller (dev container) reaches the kube-API DIRECTLY via the pfSense NAT
@@ -51,12 +51,18 @@ k3s_servers_group: k3s_prod
 # than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential.
 cluster_kubeconfig_name: prod.kubeconfig
-# --- k3s API host — safe to set now, the primary's real IP is already known -------
+# --- k3s API host — THE FOUNDER'S ADDRESS, bootstrap-time only --------------------
 # Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
 # dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
 # has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
 # address (see the KEY DIFFERENCE note there) — so point straight at it instead of
 # needing a k3s_node_ip split.
 #
 # WHAT THIS IS *NOT*, since 2026-09-06: it is not the endpoint anything steady-state
 # dials. That is k3s_apiserver_endpoint (group_vars/all.yml), which derives from
 # kube_vip_address below and therefore needs NO override here. k3s_api_host is consumed
 # only by the secondary `--server` cold-join URL and by one tls-san entry. Read the
 # TWO API ADDRESSES block at the top of group_vars/all.yml before touching either.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
 # by template reference, so it needs no separate override here — it picks up the real
@@ -75,8 +81,15 @@ k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # round (control-plane .49, ingress pool .53-.60).
 #
 # This is the address kube-vip floats for the API and that k3s tls-san's onto its
-# serving cert; secondaries use it as their --server join target. Dev's equivalent is
+# serving cert. Dev's equivalent is kube_vip_address: 192.168.1.49 in group_vars/all.yml.
-# kube_vip_address: 192.168.1.49 in group_vars/all.yml.
+#
 # CORRECTION 2026-09-06: an earlier version of this comment claimed "secondaries use it as
 # their --server join target". They do not, and they must not — on a COLD build the VIP is
 # not up when the secondaries join (kube-vip is a k3s auto-deploy manifest and needs a
 # cluster first), so the join uses k3s_api_host, the founder. What DOES use this address is
 # k3s_apiserver_endpoint (group_vars/all.yml, derived from this line), i.e. Cilium's
 # k8sServiceHost and any other steady-state client. Overriding kube_vip_address here moves
 # BOTH the VIP and prod's Cilium endpoint together, which is the point of deriving it.
 #
 # HISTORY: an earlier pass here wrongly set this to 192.168.2.20 — that is the INGRESS
 # VIP, not the control-plane VIP. Corrected. Keep the two distinct.
@@ -87,24 +100,155 @@ kube_vip_address: "192.168.2.9"
 # — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
 cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
-# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
+# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
-# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
+# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
-# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
+# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
-# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
+# backports kernel as a side effect of sharing the role), and this line switches it on for
-# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
+# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
-# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
+# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
-# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
+# rather than from anyone remembering.
 #
-# The overrides below are deliberately present-but-broken: leaving them UNSET would
+# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
-# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
+# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
-# hardware — a confusing failure that looks like a wiring bug. This value fails just
+# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
-# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
+# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
-# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
+# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
-# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
+# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
-# updates these two lines. All three nodes must end up with an identical layout
+# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
-# (Longhorn replica placement assumes symmetry) before any of this is used for real.
+# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
-node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
-node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# in §3.
 #
 # COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
 # guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
 # the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
 # back to false, roll back per the note the role writes to
 # /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
 # roles/node_baseline.
 #
 # The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
 # FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
 node_baseline_backports_kernel: true
 # --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
 # Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
 #
 #   DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
 #                     disk, claimed and mounted at the NEUTRAL mountpoint `/data`
 #                     (LABEL=k3s-data), then split between TWO tenants:
 #                       /data/longhorn   Longhorn's dataPath
 #                       /data/k3s        k3s --data-dir (etcd AND the containerd image store)
 #   ETCD           -> NO dedicated device. It lives inside /data/k3s on that partition.
 #
 # ================================ THE 2026-09-04 REVISION ================================
 # The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
 # /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
 # data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
 # deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
 # would therefore have filled /var during its FIRST image pulls — and with etcd on the same
 # partition, a full /var means etcd cannot write its WAL. That is cluster death, not
 # degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
 # mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
 # tenants. Nothing was installed on prod when this was decided, which is exactly why it was
 # decided then — it is cheap now and very expensive after bring-up.
 #
 # THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
 # were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
 # contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
 # MEASURED, not assumed — exit condition in §2.
 # =========================================================================================
 #
 # Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
 # not as a device path and a blank. That distinction is the point of this block:
 #
 # 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
 #    k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
 #    (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
 #    Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
 #    freezes a fact about today's hardware into inventory, where it goes stale silently the
 #    first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
 #    nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
 #    So inventory declares the INTENT ("the partition currently mounted at /srv") and the
 #    role discovers the device on the node, then proves it is safe to claim (a partition or
 #    disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
 #    unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
 #
 # 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
 #    describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
 #    layout etcd is inside /data/k3s on the claimed data partition, which is neither a
 #    dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
 #    Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
 #    and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
 #    on every run, that the filesystem backing the etcd data dir really is /data. If the
 #    mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
 #    WAL would land back on the OS filesystem, invisibly, until the disk filled. The
 #    `root-filesystem` value makes no such claim and gets no such assert — which is exactly
 #    why the two are distinct enum values rather than one with a comment.
 #
 # 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
 #    the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
 #    the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
 #    this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
 #    k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
 #    Longhorn volumes are untouched.
 node_baseline_longhorn_source: claim-mountpoint
 node_baseline_longhorn_claim_mountpoint: /srv
 node_baseline_etcd_source: shared-data-partition
 # --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
 # `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
 # CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
 # /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
 # as the destination would make "claimed" and "not claimed" indistinguishable.
 # `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
 # partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
 # else's named volume — so labelling ours `data` would make a stranger's `data` volume look
 # already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
 # `k3s-etcd` label convention already in this repo.
 node_baseline_data_mount: /data
 node_baseline_data_fs_label: k3s-data
 # Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
 # equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
 # side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
 # default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
 node_baseline_longhorn_data_path: /data/longhorn
 # --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
 # The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
 # `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
 # label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
 # would walk into the greenfield claim path — which fails closed at "the claim mountpoint
 # must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
 # is right but it is not convergence, so the transition is DECLARED here: the role unmounts
 # the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
 # the partition is empty (nothing orphaned) and is the device the retired label names.
 # Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
 # SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
 # then skip on the retired-label lookup and nothing else depends on them.
 node_baseline_data_retired_fs_label: longhorn
 node_baseline_data_retired_mount: /var/lib/longhorn
 # --- k3s --data-dir (§2) -----------------------------------------------------------
 # THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
 # claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
 # `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
 # which defers to this variable) creates the directory 0700 on the data partition and derives
 # etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
 # Ansible side cannot disagree with itself.
 #
 # Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
 # it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
 # dir would orphan that mount under a running cluster.
 k3s_data_dir: /data/k3s
 # Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
 # source uses a device path any more, but leaving these UNSET would let prod silently
 # inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
 # metal) the moment anyone flipped a source back to `device`. These values can never parse
 # as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
 # before mkfs is ever reached.
 node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
 node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
 # --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
 # roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
@@ -158,15 +302,58 @@ node_baseline_dns_servers:
  #   LAN clients       -> PiHole. Not a cluster concern.
  #   nodes             -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
  #
-  # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
+  # THE k3s API SERVER is a node process and carries `oidc-issuer-url=...` (built from
-  # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
+  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
-  # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
+  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
  # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
  # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
  # with the dev Gateway VIP.
  #
-  # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
+  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
+  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
+  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
+  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
  #
  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
  # external-dns has actually published the record. Until then the issuer does not resolve
  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
 # kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
 # k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
 # k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
 # postBuild.substituteFrom never sees it. Full rationale in
 # roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
 #
 # WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
 # SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
 # deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
 # prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
 # https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
 # in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
 k3s_base_domain: "prod.home.keithjackson.dev"
 # --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
 # roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
 # and group_vars/all.yml carries no override, so dev keeps resolving exactly that).
 # Prod bootstraps its OWN overlay. Before this seam existed the role could bootstrap
 # clusters/dev ONLY — bootstrap/flux/bootstrap.sh's FLUX_ENV_PATH default was never
 # exported by the role — so `flux bootstrap ... --path=clusters/prod` (§8 step 4 of
 # infra/PROD-PROVISIONING.md) was simply not reachable without editing the role.
 #
 # Same seam as kube_vip_address / k3s_api_host / k3s_data_dir / cluster_kubeconfig_name /
 # node_baseline_backports_kernel above: set once for the [k3s_prod] umbrella group, so a
 # FOURTH prod node inherits it from group membership with no per-host or per-run flag.
 #
 # WHAT clusters/prod STILL NEEDS before this is safe to run — do NOT invoke the
 # flux_bootstrap role until these are true (PROD-PROVISIONING.md §7):
 #   - the remaining TBD-PROD-* placeholders in clusters/prod/kustomization.yaml are real
 #     (S3_BACKUP_ENDPOINT); they are INVALID on purpose so an early apply fails loudly;
 #   - the GitRepository url there is the TEMPORARY workstation Gitea
 #     (http://192.168.2.132:3000/...) and that container is actually running and reachable
 #     from the prod nodes — it is outside every cluster and can simply be powered off;
 #   - the `sops-age` Secret is installed out of band on prod (bootstrap/sops).
 flux_bootstrap_env_path: "clusters/prod"
@@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 # =============================================================================
 # PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
-# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
+# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
-# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
+# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
-# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
+# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
-# nothing has been run against node 1, and the operator has decided the CLUSTER stays
+# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
-# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
+# check and the node-baseline PACKAGE floor has been run, and the operator has decided
-# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
+# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
-# later member-add dance.
+# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
 # single-server-etcd window, no later member-add dance.
 #
 # KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
 # hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
@@ -92,9 +93,18 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
 # node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
 # partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
 # nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
-# node 3 remains commented until it is racked.
+# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
 # Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
 # 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
 # memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
 # happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
 # needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
 # SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
 # three-way, not a node-1 outlier.
 # NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
 # the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
 k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
-# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
+k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
 # pattern; named to match the group_vars directory so Ansible auto-loads it).
@@ -3,6 +3,13 @@
 # Control-plane ONLY (svc_enable=false) — Cilium owns Service LoadBalancing.
 # Writes the manifest into each server's k3s auto-deploy dir; k3s applies it.
 #
 # PART OF THE ONE-COMMAND BRING-UP SINCE 2026-09-06, not a separate operator step:
 # playbooks/site.yml imports this play BETWEEN k3s.yml and cilium.yml, because Cilium's
 # k8sServiceHost is now this VIP (k3s_apiserver_endpoint, group_vars/all.yml) and the CNI
 # cannot be installed against an endpoint nothing announces yet. The role's final task waits
 # for the VIP to answer on 6443, so a failure to float it stops the run here with a named
 # cause instead of surfacing later as unreachable cilium-agents.
 #
 # TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
 #   ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
 # (the VIP itself comes from kube_vip_address, already scoped per environment in
@@ -0,0 +1,137 @@
 ---
 # EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
 # R-ENV-03, R-TEST-02).
 #
 # WHAT QUESTION THIS ANSWERS
 #   infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
 #   EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
 #   Linux 6.12 reserves everything from the table address to the top of RAM without
 #   validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
 #   playbook measures them:
 #
 #     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
 #        failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
 #        fleet-wide were at other addresses. If bad outcomes map onto a narrow address
 #        range, the causal chain closes with nothing touched.
 #     2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
 #        boots? Nodes 1 and 3 have each been observed exactly twice.
 #
 #   §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
 #   is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
 #   a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
 #   boots per arm; detecting a mere odds shift needs ~50.
 #
 # THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
 # package installs, no hardware moves, no node_baseline run. Its job is to establish real
 # p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
 # a Fast Boot flip, a 6.13 kernel) compares against it.
 #
 # HOW TO RUN IT
 #   Dry run first — every stage except the reboots, so the pipeline is proven before 45
 #   minutes of wall clock are committed:
 #
 #     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
 #         ansible-playbook playbooks/memattr-survey.yml'
 #
 #   The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
 #
 #     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
 #         ansible-playbook playbooks/memattr-survey.yml \
 #           -e memattr_survey_reboots_authorised=true'
 #
 #   Fewer rounds, or one node:
 #     -e memattr_survey_rounds=5        --limit k3s-prod-2
 #
 #   RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
 #   rounds are never lost or double-counted. Analysis alone, over existing artifacts:
 #     ansible-playbook playbooks/memattr-survey.yml --tags analyse
 #
 # TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
 # It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
 # not a member of memattr_survey_required_group — the dev cluster is live, and a survey
 # that reboots it 20 times because of a mistyped --limit is exactly the accident worth
 # engineering against.
 #
 # HOW TO READ THE OUTPUT
 #   infra/ansible/_artifacts/memattr-survey/  (gitignored, regenerated every run)
 #     <host>.rounds.jsonl  one JSON line per round, mirrored off the node after EVERY round
 #     <host>.boots.json    every boot the persistent journal holds, re-derived in one pass
 #                          at the end — the AUTHORITATIVE record
 #     summary.md           the analysis, also printed at the end of the run
 #   See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
 #   with NO memattr log line is a healthy third state and not a missing measurement.
 # =====================================================================================
 # PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
 # before a single reboot. That is the right asymmetry: the cost of a false stop is a
 # re-run, the cost of a false start is rebooting a machine that was doing something.
 # =====================================================================================
 - name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
  gather_facts: false
  become: true
  any_errors_fatal: true
  tags: [survey]
  tasks:
    - name: Preflight and baseline probe
      ansible.builtin.include_role:
        name: memattr_survey
        tasks_from: preflight
 # =====================================================================================
 # PLAY 2 — THE SURVEY.
 #
 # any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
 # ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
 # keep collecting.
 #
 # `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
 # assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
 # that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
 # then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
 # blocks belongs to exactly one host, so the other two sit idle through it and the survey
 # runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
 # completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
 # appeared in the output. `free` lets each host walk its own task list at its own pace,
 # which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
 #
 # Consequences worth knowing before editing: output INTERLEAVES between hosts (every
 # progress line is host-tagged for exactly this reason), and `run_once` is unreliable
 # under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
 # directory create on every host instead. Concurrency is also capped by `forks`
 # (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
 # than letting a bigger fleet silently serialise again.
 # =====================================================================================
 - name: memattr survey — reboot rounds and journal harvest
  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
  gather_facts: false
  become: true
  strategy: free
  tags: [survey]
  tasks:
    - name: Run the reboot rounds
      ansible.builtin.include_role:
        name: memattr_survey
        tasks_from: rounds
    - name: Harvest every retained boot from the persistent journal
      ansible.builtin.include_role:
        name: memattr_survey
        tasks_from: harvest
 # =====================================================================================
 # PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
 # even if every surveyed node died, because the artifacts are already on disk and the
 # analysis counts rounds from THOSE rather than from Ansible's host state. A node that
 # vanished shows up as a short arm with a wider interval, not as a missing report.
 # =====================================================================================
 - name: memattr survey — analyse
  hosts: localhost
  gather_facts: false
  become: false
  tags: [analyse]
  tasks:
    - name: Produce the analysis
      ansible.builtin.include_role:
        name: memattr_survey
        tasks_from: analyse
@@ -1,7 +1,10 @@
 ---
 # Full dev/test cluster bring-up, in order (docs/02 sequencing; run AFTER
 # `tofu apply` has created and cloud-init has baselined the VMs):
-#   k3s HA install -> Cilium CNI -> verify readiness -> Flux + SOPS hand-off.
+#   k3s HA install -> kube-vip control-plane VIP -> Cilium CNI -> verify readiness
 #   -> Flux + SOPS hand-off.
 # (kube-vip joined the chain 2026-09-06 — see the note at its import below; it is a
 # prerequisite of the CNI now, not a separate operator step.)
 #
 # TARGET (R-ENV-02): every imported play takes its host pattern from a var that
 # defaults to the dev group, so `ansible-playbook playbooks/site.yml` with no extra
@@ -20,6 +23,29 @@
 - name: Import - install k3s HA control plane
  ansible.builtin.import_playbook: k3s.yml
 # ADDED 2026-09-06, AND ITS POSITION IS LOAD-BEARING: kube-vip MUST come up between k3s and
 # Cilium. Cilium's k8sServiceHost is now the control-plane VIP (k3s_apiserver_endpoint =
 # kube_vip_address) rather than the founder's own node IP, so installing Cilium before the
 # VIP is announced would point every agent at an address nothing answers on — the exact
 # shape of the 2026-09-01 incident, where agents that could not reach the API server
 # deadlocked on a cold cluster.
 #
 # IT CANNOT COME EARLIER. kube-vip ships as a k3s auto-deploy manifest
 # (<data-dir>/server/manifests), so a cluster must exist before it can be applied — hence
 # after k3s.yml, not before.
 #
 # IT DOES NOT NEED THE CNI, which is what makes this ordering possible at all: the kube-vip
 # DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 (roles/kube_vip's template
 # explains why), so it reaches the node-local apiserver with no CNI and no kube-proxy, and
 # DaemonSet pods carry the default not-ready/network-unavailable tolerations that let them
 # schedule onto a still-NotReady node. roles/kube_vip blocks until the VIP actually answers
 # on 6443, so cilium.yml below cannot start against a VIP that is not up.
 #
 # Re-running this against the LIVE dev cluster is a no-op for this play: dev already runs
 # kube-vip with the same rendered manifest, and the wait finds the VIP already answering.
 - name: Import - float the control-plane HA VIP (kube-vip) BEFORE the CNI dials it
  ansible.builtin.import_playbook: kube-vip.yml
 - name: Import - install Cilium CNI
  ansible.builtin.import_playbook: cilium.yml
@@ -2,7 +2,17 @@
 # cilium role defaults (ADR-0002, docs/02 Cilium-ownership contract).
 # This role performs the INITIAL Cilium install; the Flux platform/cilium
 # HelmRelease adopts config afterward. The parity-critical values
-# (kubeProxyReplacement, k8sServiceHost/Port) are templated from k3s_api_host.
+# (kubeProxyReplacement, k8sServiceHost/Port) are templated from
 # k3s_apiserver_endpoint / k3s_api_port (group_vars/all.yml).
 #
 # k8sServiceHost IS THE HA VIP, NOT THE FOUNDER (changed 2026-09-06). It reads
 # k3s_apiserver_endpoint — which derives from kube_vip_address, so it is dev 192.168.1.49 /
 # prod 192.168.2.9 — and deliberately NOT k3s_api_host, the founder's own node IP. Those two
 # were one variable until 2026-09-06, and the conflation is what left all three prod
 # cilium-agents dialing k3s-prod-1 on an HA cluster. Do not re-merge them.
 #
 # THIS ROLE THEREFORE DEPENDS ON THE VIP EXISTING. playbooks/site.yml runs kube-vip.yml
 # before cilium.yml and roles/kube_vip blocks until the VIP answers on 6443.
 # Exact pin — MUST equal the platform/cilium HelmRelease chart.spec.version (1.20.1, an
 # exact pin, not a range). Renovate bumps both together.
@@ -15,9 +25,20 @@ cilium_namespace: "kube-system"
 # Cilium 1.20.1 targets Gateway API v1.6.1.
 gateway_api_version: "v1.6.1"
 gateway_api_crd_urls:
  # ONE URL, deliberately. As of v1.6.0 the STANDARD channel bundle already contains
  # TLSRoute (plus TCPRoute, UDPRoute, ListenerSet and BackendTLSPolicy), so the separate
  # experimental TLSRoute manifest this list used to carry is redundant — and worse than
  # redundant: v1.6 also ships a ValidatingAdmissionPolicy,
  # `safe-upgrades.gateway.networking.k8s.io`, which REJECTS an experimental-channel CRD
  # applied over a standard-channel one. Applying the old second URL therefore fails hard:
  #   ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' ... denied request:
  #   Installing experimental CRDs on top of standard channel CRDs is prohibited by default.
  # Verified 2026-09-04 on the prod bootstrap: standard-install.yaml v1.6.1 alone produces the
  # exact 10 gateway.networking.k8s.io CRDs the LIVE dev cluster is running, tlsroutes included
  # and annotated channel=standard, bundle-version=v1.6.1 — so this is parity with dev, not a
  # reduction. Dev needs no reapply; it already has them. If gateway_api_version is ever moved
  # BACK below v1.6.0, the experimental TLSRoute URL has to come back with it.
  - "https://github.com/kubernetes-sigs/gateway-api/releases/download/{{ gateway_api_version }}/standard-install.yaml"
  # TLSRoute (experimental) is required by Cilium's Gateway API support.
  - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
 # Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
 # comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev
@@ -89,7 +89,7 @@
  run_once: true
  changed_when: false
- name: Render the Cilium values file (parity-critical settings from k3s_api_host)
+- name: Render the Cilium values file (parity-critical settings from k3s_apiserver_endpoint)
  ansible.builtin.template:
    src: cilium-values.yaml.j2
    dest: "{{ cilium_values_path }}"
@@ -4,9 +4,24 @@
 ---
 # [parity] k3s kube-proxy is disabled → Cilium replaces it (ADR-0001/0002).
 kubeProxyReplacement: true
-# [parity] = the k3s API endpoint (primary node IP). The Flux HelmRelease's
+# [parity] = k3s_apiserver_endpoint, THE HA CONTROL-PLANE VIP (kube_vip_address) — dev
-# k8sServiceHost "REPLACE-ME" must be set to this same value.
+# 192.168.1.49, prod 192.168.2.9. NOT k3s_api_host, which is the FOUNDER'S OWN node IP:
-k8sServiceHost: "{{ k3s_api_host }}"
+# pointing the CNI's kube-proxy replacement at one machine on a 3-server HA cluster means
 # losing that machine stops every service/policy update reaching the datapath, silently
 # (the existing datapath keeps working from Cilium's cache, so nothing looks broken). That
 # was the live state of prod until 2026-09-06 — all three cilium-agents carried
 # KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held .9. See group_vars/all.yml.
 #
 # [parity] The Flux side is platform/cilium/helmrelease.yaml's k8sServiceHost:
 # "${K3S_API_HOST}", whose per-environment value comes from clusters/<env>/kustomization.yaml.
 # THAT VALUE MUST EQUAL k3s_apiserver_endpoint for the SAME environment. If it does not,
 # Flux and this bootstrap render different control-plane paths and fight over the CNI.
 #
 # THE VIP MUST BE UP BEFORE THIS ROLE RUNS. playbooks/site.yml imports kube-vip.yml between
 # k3s.yml and cilium.yml for exactly this reason, and roles/kube_vip blocks until the VIP
 # answers. Rendering this against a VIP that nothing announces yet reproduces the
 # 2026-09-01 shape: agents that cannot reach the API server on a cold cluster.
 k8sServiceHost: "{{ k3s_apiserver_endpoint }}"
 k8sServicePort: {{ k3s_api_port }}
 # Ingress via Gateway API — no Traefik (docs/09, ADR-0002).
@@ -19,5 +19,22 @@ flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
 flux_bootstrap_sops_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/sops/install-age-key.sh"
 # Assumed interface of bootstrap/flux (GitOps Engineer, T-1.4):
-#   runs `flux bootstrap ... --path=clusters/dev` against this cluster.
+#   runs `flux install` + `kubectl apply -k <env path>` against this cluster.
 flux_bootstrap_flux_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/flux/bootstrap.sh"
 # --- WHICH ENVIRONMENT OVERLAY THIS BOOTSTRAPS (R-ENV-02) -------------------------
 # The one thing that makes this role environment-agnostic. bootstrap/flux/bootstrap.sh
 # reads FLUX_ENV_PATH (default clusters/dev) and ends with `kubectl apply -k <that>`;
 # tasks/main.yml exports this value into that variable, so the role NEVER names an
 # environment and the task graph is identical for dev and prod.
 #
 # THE DEFAULT IS DEV ON PURPOSE — dev is LIVE, and group_vars/all.yml carries no
 # override, so a dev run resolves exactly the value it always had (clusters/dev) and
 # behaves byte-for-byte as before.
 #
 # PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same group_vars
 # seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name /
 # node_baseline_backports_kernel. No role fork, no per-run extra-var, no second
 # playbook: group membership decides. Before this existed the role could bootstrap
 # ONLY clusters/dev, so prod bring-up was impossible without editing the role.
 flux_bootstrap_env_path: "clusters/dev"
@@ -4,11 +4,17 @@
 # — it calls the entrypoints the Security + GitOps Engineers author there.
 #
 # Order matters: install the age decryption key FIRST (so Flux can decrypt on its
-# very first reconcile), THEN run `flux bootstrap` at clusters/dev.
+# very first reconcile), THEN bootstrap Flux at the environment's overlay.
 #
 # The two commands are variables (see defaults/main.yml). If the bootstrap workers
 # expose different entrypoints, override flux_bootstrap_sops_cmd / _flux_cmd — the
 # task graph does not change.
 #
 # ENVIRONMENT-AGNOSTIC (R-ENV-02): this role names NO environment. Which overlay gets
 # bootstrapped comes from `flux_bootstrap_env_path` (defaults/main.yml, default
 # clusters/dev), exported below as FLUX_ENV_PATH — the variable bootstrap/flux/
 # bootstrap.sh already reads. Prod overrides it in group_vars/k3s_prod/vars.yml, the
 # same seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name.
 - name: Assert the bootstrap/ subtree exists (authored by the other worker)
  ansible.builtin.stat:
@@ -29,11 +35,14 @@
  run_once: true
  changed_when: true
- name: Bootstrap Flux against clusters/dev (bootstrap/flux)
+- name: Bootstrap Flux against this environment's overlay (bootstrap/flux)
  ansible.builtin.command:
    cmd: "{{ flux_bootstrap_flux_cmd }}"
  environment:
    KUBECONFIG: "{{ flux_bootstrap_kubeconfig }}"
    # The ONLY per-environment input. clusters/dev by default (dev unchanged);
    # clusters/prod for [k3s_prod] hosts via group_vars.
    FLUX_ENV_PATH: "{{ flux_bootstrap_env_path }}"
  delegate_to: localhost
  become: false
  run_once: true
@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
 # the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
 k3s_disable: "traefik,servicelb,local-storage"
 # --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
 # EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
 # WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
 # image store (agent/containerd) — which is by far the largest tenant.
 #
 # THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
 # LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
 # /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
 # server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
 # running cluster's entire state directory out from under it and orphan that etcd mount.
 #
 # PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
 # kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
 # node inherits it from group membership rather than from anyone remembering. It is set
 # there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
 # turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
 # data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
 # this same variable). One line moves both.
 #
 # WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
 # separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
 # alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
 # FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
 # not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
 #
 # NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
 # INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
 # config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
 # Restart handler for a value that is not changing. The install flags are the correct and
 # sufficient place, and prod has no k3s installed yet.
 k3s_data_dir: "/var/lib/rancher/k3s"
 # Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
 k3s_server_mode: "secondary"
@@ -27,6 +59,17 @@ k3s_server_mode: "secondary"
 # serving cert on restart when this set changes. Deduplicated (the primary IP is
 # also k3s_api_host).
 #
 # THIS LIST COVERS BOTH API ADDRESSES, WHICH IS WHY THE 2026-09-06 SPLIT NEEDED NO CERT
 # CHANGE AND NO k3s RESTART. group_vars/all.yml now distinguishes k3s_api_host (the
 # founder, for the cold join) from k3s_apiserver_endpoint (= kube_vip_address, what Cilium
 # and every steady-state client dial). Both are already in this set, so repointing a
 # consumer from one to the other cannot produce a TLS failure. VERIFIED LIVE 2026-09-06,
 # not assumed: prod's apiserver answered through https://192.168.2.9:6443 with full CA
 # verification and tls-server-name=192.168.2.9. Do not remove either entry — dropping
 # kube_vip_address breaks every client of the VIP; dropping k3s_api_host breaks nothing
 # today only because it is also in the k3s_servers_group node-IP list below, and that
 # coincidence is not a contract.
 #
 # k3s_servers_group — NOT a literal group name — is what makes this role environment-
 # agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
 # group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
@@ -34,3 +77,33 @@ k3s_tls_sans: >-
  {{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
      + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
     | unique }}
 # --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
 # THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
 #
 # The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
 # /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
 # sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
 # suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
 #
 #   Flux side     clusters/base/cluster-vars.yaml   BASE_DOMAIN      (per-overlay)
 #   Ansible side  THIS variable                     k3s_base_domain  (per-inventory-group)
 #
 # [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
 # If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
 # one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
 # while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
 # not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
 #
 # THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
 # convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
 # base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
 # what it already is and the Restart handler does NOT fire on a re-run.
 #
 # PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
 # k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
 # from group membership rather than from anyone remembering.
 k3s_base_domain: "home.keithjackson.dev"
 # Derived, so the realm path is stated once. Override k3s_base_domain, not this.
 k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
@@ -22,6 +22,7 @@
      --write-kubeconfig-mode=644
      {% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
      --node-ip={{ k3s_node_ip }}
      --data-dir={{ k3s_data_dir }}
 # On an ALREADY-installed cluster the install task above is idempotent (creates:
 # k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
@@ -74,13 +75,19 @@
      # OIDC (Keycloak SSO for kubectl). The API server trusts ID tokens from the homelab realm;
      # the `kubernetes` PUBLIC client (PKCE) + kubelogin mint them, and RBAC binds the `groups`
      # claim (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins->cluster-admin,
-      # oidc:viewers->view). Nodes resolve keycloak.home.keithjackson.dev to the Cilium gateway VIP
+      # oidc:viewers->view). Nodes resolve keycloak.<k3s_base_domain> to the Cilium gateway VIP
-      # (192.168.1.53) and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
+      # and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
      # prefixes namespace federated identities away from cert / service-account subjects.
      # ADDITIVE + break-glass-safe: the cert-based admin kubeconfig keeps full access with or
      # without Keycloak, so a bad realm/OIDC state can never lock you out of the cluster.
      #
      # NOT a Flux ${BASE_DOMAIN} token: this is a k3s SERVER FLAG, not a manifest, so
      # postBuild.substituteFrom never touches it. k3s_oidc_issuer_url derives from
      # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
      # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
      # config.yaml is unchanged and the Restart handler does not fire.
      kube-apiserver-arg:
-        - "oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab"
+        - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
        - "oidc-client-id=kubernetes"
        - "oidc-username-claim=email"
        - "oidc-username-prefix=oidc:"
@@ -97,15 +104,20 @@
    INSTALL_K3S_VERSION: "{{ k3s_version }}"
  when: k3s_server_mode == 'primary'
 # PATH FOLLOWS --data-dir. k3s writes the node-token to <data-dir>/server/token, NOT to
 # the literal /var/lib/rancher/k3s. On dev k3s_data_dir IS /var/lib/rancher/k3s (the role
 # default, restated explicitly above) so this is byte-for-byte the old path; on prod
 # --data-dir is /data/k3s (group_vars/k3s_prod) and the hardcoded path did not exist at all
 # — this task would have timed out after 180s and the two secondaries could never join.
 - name: Wait for the k3s node-token to exist (primary)
  ansible.builtin.wait_for:
-    path: /var/lib/rancher/k3s/server/token
+    path: "{{ k3s_data_dir }}/server/token"
    timeout: 180
  when: k3s_server_mode == 'primary'
 - name: Read the k3s node-token (primary)
  ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
  register: k3s_token_raw
  when: k3s_server_mode == 'primary'
@@ -125,9 +137,11 @@
 # "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
 # dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
 # delegates to the PROD primary using this same role, unchanged (R-ENV-02).
 # k3s_data_dir is a CLUSTER-WIDE value (group_vars for prod, role default for dev), so the
 # secondary's own copy of it is also the primary's data dir — correct for the delegated read.
 - name: Fetch the k3s node-token directly from the primary (secondary)
  ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
  delegate_to: "{{ groups[k3s_primary_group][0] }}"
  register: k3s_primary_token_raw
  when:
@@ -141,6 +155,18 @@
         | default(k3s_primary_token_raw.content | b64decode | trim, true) }}
  when: k3s_server_mode == 'secondary'
 # THE JOIN TARGET IS k3s_api_host — THE FOUNDER — AND MUST NOT BECOME THE VIP.
 # group_vars/all.yml splits the two API addresses (k3s_api_host = the founder's own node IP,
 # k3s_apiserver_endpoint = the kube-vip HA VIP) and this is the canonical founder-only
 # consumer. On a COLD build the VIP does not exist at this point in the sequence: kube-vip
 # is a k3s auto-deploy manifest, so it cannot be applied until a cluster exists, and
 # playbooks/site.yml only imports kube-vip.yml AFTER k3s.yml. A `--server` pointed at the
 # VIP would therefore dial an address nothing announces and the cold build would be
 # unjoinable — with no diagnostic beyond a connection timeout.
 # On a WARM single-node rejoin (`--limit k3s-dev-N`) the founder is up and answering too, so
 # the founder works in both cases and the VIP works in only one. Cilium's k8sServiceHost is
 # the opposite trade (steady state, must survive losing the founder) and uses the endpoint
 # variable instead.
 - name: Install k3s secondary (join as server)
  ansible.builtin.command:
    cmd: >-
@@ -24,6 +24,16 @@ kube_vip_port: 6443
 # only if detection is wrong.
 kube_vip_interface: ""
 # How long to wait for the VIP to actually ANSWER on kube_vip_port after the manifest is
 # written (seconds). This is a real gate, not cosmetics: since 2026-09-06 Cilium's
 # k8sServiceHost is this VIP (k3s_apiserver_endpoint), and playbooks/site.yml runs this role
 # immediately before cilium.yml — so if the VIP never comes up, the run must stop HERE with
 # a clear message rather than proceed to install a CNI pointed at an address nothing
 # answers. On a cold cluster the budget covers k3s's deploy controller applying the
 # manifest, the ghcr.io image pull and leader election; on an already-running cluster the
 # check returns immediately.
 kube_vip_wait_timeout: 300
 # Leader-election tuning (seconds) — a snappy failover without thrashing.
 kube_vip_lease_duration: 15
 kube_vip_renew_deadline: 10
@@ -31,4 +41,14 @@ kube_vip_retry_period: 2
 # k3s auto-deploy manifests dir: k3s applies anything here on start, so kube-vip
 # is reproducible and independent of Flux/Helm.
-kube_vip_manifest_path: "/var/lib/rancher/k3s/server/manifests/kube-vip.yaml"
+#
 # THE PATH FOLLOWS --data-dir. k3s reads auto-deploy manifests from
 # <data-dir>/server/manifests, so this must be derived from the cluster-wide
 # k3s_data_dir (roles/k3s_server turns the same variable into --data-dir), not from the
 # literal /var/lib/rancher/k3s. Dev leaves k3s_data_dir unset outside roles/k3s_server's
 # own defaults, so the fallback below reproduces dev's existing path exactly; prod sets
 # k3s_data_dir: /data/k3s in group_vars/k3s_prod, where the old hardcoded path was simply
 # not the directory k3s watches — the manifest would have been written and silently never
 # applied, and the control-plane VIP would never have come up.
 kube_vip_manifest_path: >-
  {{ k3s_data_dir | default('/var/lib/rancher/k3s') }}/server/manifests/kube-vip.yaml
@@ -1,17 +1,34 @@
 ---
 # Deploy kube-vip (control-plane HA VIP) via the k3s auto-deploy manifests dir.
-# k3s applies /var/lib/rancher/k3s/server/manifests/*.yaml on start, so this is
+# k3s applies <data-dir>/server/manifests/*.yaml on start, so this is
-# reproducible and independent of Flux. The same manifest is written on every
+# reproducible and independent of Flux (see kube_vip_manifest_path in defaults/main.yml —
 # the dir follows k3s's --data-dir, which prod moves to /data/k3s). The same manifest is written on every
 # server so the DaemonSet survives the loss of any single node.
- name: Detect the LAN interface holding the node's 192.168.1.5x address
+# WHICH INTERFACE kube-vip ARPs THE VIP ON. Detected from this node's OWN k3s_node_ip
 # (inventory, per host) rather than a hardcoded subnet: the previous awk matched the
 # literal 192.168.1.5[0-2], i.e. DEV's node addresses, so on prod (192.168.2.10/.11/.12)
 # it returned nothing and the assert below aborted the play. k3s_node_ip is the same
 # address k3s advertises with --node-ip and is defined for dev and prod alike, so this
 # resolves eno1 on prod and the identical dev interface it always resolved before.
 # `ip addr show to <ip>/32` does the matching in iproute2 instead of a regex over dotted
 # quads (where `.` would match any character).
 - name: Detect the LAN interface holding this node's k3s_node_ip
  ansible.builtin.shell: >-
    set -o pipefail;
-    ip -o -4 addr show | awk '/192\.168\.1\.5[0-2]\//{print $2; exit}'
+    ip -o -4 addr show to {{ k3s_node_ip }}/32 | awk '{print $2; exit}'
  args:
    executable: /bin/bash
  register: kube_vip_iface_detect
  changed_when: false
  # RUN THIS EVEN UNDER --check. It is `ip -o -4 addr show`: it reads and changes nothing.
  # Without check_mode: false the shell module is SKIPPED in check mode, kube_vip_iface_detect
  # has no stdout, and the assert two tasks down aborts the play with "Could not detect the
  # interface ... set kube_vip_interface explicitly" — a misleading failure that says nothing
  # about the node. That mattered less while this play was an out-of-band operator step; it
  # matters now that playbooks/site.yml imports it, because `--check` on the bring-up before a
  # cold prod build is exactly the rehearsal an operator should be able to run.
  check_mode: false
 - name: Resolve the kube-vip VIP interface (override wins, else detected)
  ansible.builtin.set_fact:
@@ -25,7 +42,7 @@
      - kube_vip_vip_interface is defined
      - kube_vip_vip_interface | length > 0
    fail_msg: >-
-      Could not detect the 192.168.1.5x LAN interface for kube-vip on
+      Could not detect the interface holding {{ k3s_node_ip }} for kube-vip on
      {{ inventory_hostname }} — set kube_vip_interface explicitly.
 - name: Ensure the k3s auto-deploy manifests directory exists
@@ -43,3 +60,34 @@
    owner: root
    group: root
    mode: "0600"
 # PROVE THE VIP IS ACTUALLY UP BEFORE THIS PLAY RETURNS (added 2026-09-06).
 #
 # Templating the manifest is not the same as having a floating endpoint: k3s's deploy
 # controller still has to apply it, the image still has to pull, and leader election still
 # has to pick an owner. Nothing used to check, which was tolerable while the VIP was only an
 # operator convenience. It is not tolerable now — Cilium's k8sServiceHost IS this VIP
 # (k3s_apiserver_endpoint) and playbooks/site.yml runs cilium.yml straight after this play,
 # so an unannounced VIP would mean installing a CNI that cannot reach the API server. That
 # failure is the 2026-09-01 shape and it is very expensive to diagnose from the far side;
 # failing here instead names the actual cause.
 #
 # run_once: the VIP is ONE address for the whole environment, so one probe answers for the
 # environment. Probing from a server host (not the controller) is deliberate: the servers
 # are on the VIP's own LAN in both environments, whereas the dev controller reaches dev's
 # API only through a pfSense NAT — a controller-side probe would have been a property of
 # where the play was launched from. READ-ONLY: it opens a TCP connection and changes nothing.
 - name: Wait for the control-plane VIP to answer (the CNI's endpoint depends on it)
  ansible.builtin.wait_for:
    host: "{{ kube_vip_address }}"
    port: "{{ kube_vip_port }}"
    timeout: "{{ kube_vip_wait_timeout }}"
  run_once: true
  changed_when: false
  vars:
    ansible_become: false
  # DELIBERATELY NOT check_mode: false, unlike the interface probe above — so a `--check`
  # rehearsal SKIPS this and does not sit on a socket for kube_vip_wait_timeout. wait_for
  # declares no check-mode support, so the skip is the module's own behaviour and this comment
  # is here only so the skip does not read as an oversight. CONSEQUENCE, stated plainly: a
  # `--check` run does NOT verify the VIP is up. Only a real run gates the CNI on it.
@@ -0,0 +1,201 @@
 # `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery
 **Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02
 **Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism
 is settled there and this role does **not** re-derive it.
 ## The question
 §0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes
 Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it,
 and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from
 the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.
 Two things §0b leaves unmeasured, and this role measures:
 1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed
   failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes
   fleet-wide were at other addresses. If bad outcomes map onto a narrow address range,
   the causal chain closes without touching hardware or firmware.
 2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each.
 ## Why 20 rounds
 Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson
 interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is
 worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a
 mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds`
 is a variable; lowering it below 20 produces an arm that cannot answer the question it was
 built for, and the analysis will say so.
 ## This is the BASELINE arm
 **Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware
 moves, no `node_baseline` run. The role installs nothing and configures nothing; its only
 write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's
 own allocation footprint — initrd size, module set, boot device — is *part of the causal
 chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that
 the kernel release and cmdline were constant on every node, for exactly this reason.
 ## Running it
 ```bash
 # DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
 ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
    ansible-playbook playbooks/memattr-survey.yml'
 # The real thing. Reboots are OPT-IN; the interlock is deliberate.
 ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
    ansible-playbook playbooks/memattr-survey.yml \
      -e memattr_survey_reboots_authorised=true'
 # Re-analyse existing artifacts, no hardware touched.
 ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
    ansible-playbook playbooks/memattr-survey.yml --tags analyse'
 ```
 Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`,
 `-e memattr_survey_hosts=<group>`.
 20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are
 independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That
 parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count;
 preflight asserts the latter. See the playbook comment for why the default linear
 strategy silently serialises a looped `include_tasks` (it was measured, not assumed).
 ## Safety — verified, not assumed
 The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That
 premise is re-checked on **every** run, because a document and a machine drift apart:
 | Check | Failure mode it prevents |
 |---|---|
 | Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. |
 | `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. |
 | `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. |
 | `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. |
 | `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. |
 Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the
 whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a
 false start is a rebooted production machine.
 ## Fail-safe and resume
 * **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node
  that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the
  observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that
  host only** and the others keep collecting. The abort is announced immediately and again
  in the report — which counts rounds from the **artifacts**, not from Ansible's host
  state, so even a node that goes fully unreachable (which `rescue` cannot catch) still
  appears as a short arm with a wider interval, never as a silently missing report.
 * **Resumable.** Round numbers come from the on-node ledger, so re-running after an
  interruption continues where it stopped and can neither lose nor duplicate a round. The
  boot found on a resume is *not* re-appended: it is already round N of the interrupted
  run, and double-counting it would bias the sample toward whichever outcome survived the
  interruption.
 * **The abort is durable, not scrollback.** `<host>.aborted.json` records the round, the
  failing task and the reason, so the report can say *why* an arm is short instead of
  leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every
  run and re-written only if that run aborts. Its *absence* is not proof of success — a
  node can be lost abruptly enough that Ansible calls it unreachable before the rescue
  runs — so completeness is still judged on round count, and the marker only supplies the
  reason when there is one.
 * **Analysis survives everything.** It is a separate play on `localhost`, so it runs even
  if every surveyed node died. With a node down, `--tags analyse` is the way in: a full
  run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host.
 ## Belt and braces on the data
 Two independent records, joined on `boot_id`:
 | | written by | authoritative for |
 |---|---|---|
 | `<host>.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** |
 | `<host>.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** |
 | `<host>.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** |
 The journal does not care whether the driver was alive: the boot happened, the kernel
 logged it, journald persisted it. So the harvest can only ever be a *superset* of the
 ledger, and a round the driver failed to append still shows up.
 ## Reading the output
 Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as
 `roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never
 committed. `summary.md` is also printed at the end of the run.
 ### The verdict is measured, never inferred from a log line
 This is the subtlest part of the design. There are **three** kernel readings, not two:
 | `kernel_read` | What the kernel logged | Memory |
 |---|---|---|
 | `valid` | **nothing at all** — it parsed a table it considered valid | intact |
 | `bail-version` | `Unexpected EFI Memory Attributes table version <garbage>` | intact — the *lucky* path, and it looks alarming |
 | `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** |
 Silence is therefore *evidence of health*, not a missing measurement — which is why the
 record carries an explicit `memattr_diagnostic_present: false` rather than an empty field.
 And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`,
 *later* than the `memblock_reserve()` that did the damage: it is a consequence, not the
 cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and
 `kernel_read` is recorded alongside it as an independent observation. §5a of the report
 cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.
 Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed
 clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything
 landing between them is reported as `indeterminate` and excluded from `n` — a new third
 thing deserves human eyes, not a coin flip into a bucket.
 ### What is and is not pooled
 Boots the survey did **not cause** are shown in the per-round table for context and
 excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot
 each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so
 pooling it would double-count prior evidence into a run whose point is fresh trials).
 ### The statistics
 `files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is.
 * **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the
  Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly
  instead of needing a special case. A bare fraction is what this exercise exists to stop
  reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
 * **The address↔outcome question is tested three ways**, because the tempting version is
  circular:
  * **4b determinism** — was any single address seen with *both* outcomes? One such
    address falsifies "the address determines the outcome" outright.
  * **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed
    up as a p-value: the bad set is defined after seeing the data.
  * **4d the pre-registered prediction** — §0b, written before this run, names
    `0xbeeb0018`. Fisher exact against this run's boots only. This is the one
    non-circular test available.
  * **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a
    run produces no bad boot, `verdict` has no variance to explain, but `kernel_read`
    does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in
    §0b's mechanism. An address that reproduces the same *parse* boot after boot is the
    address doing causal work, even when none of those parses was catastrophic.
 ## Files
 | Path | What it is |
 |---|---|
 | `tasks/preflight.yml` | Safety gate, resume point, baseline probe |
 | `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock |
 | `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue |
 | `tasks/harvest.yml` | The authoritative journal pass |
 | `tasks/analyse.yml` | Controller-side analysis |
 | `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable |
 | `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report |
 ## What this cannot tell you
 It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute
 the difference between nodes, because node 2 is simultaneously the only single-DIMM node
 and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled
 rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS
 settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this
 role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all
 three nodes, and that is not in dispute.
@@ -0,0 +1,92 @@
 ---
 # memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
 # described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
 # settled and is NOT re-derived here.
 #
 # THE QUESTION THIS ROLE ANSWERS
 #   §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
 #   Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
 #     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
 #        observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
 #        successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
 #        address range, the causal chain is closed with no hardware or firmware touched.
 #     2. Is the fault node-2-specific, or does any node reach the bad path given enough
 #        boots? Nodes 1 and 3 have each been observed exactly twice.
 #
 # WHY 20 ROUNDS AND NOT FIVE
 #   Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
 #   interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
 #   boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
 #   needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
 #   not a round number.
 #
 # THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
 #   No BIOS writes, no kernel change, no package installs, no hardware moves. The role
 #   installs nothing and configures nothing; its only write to a node is its own append-
 #   only ledger under /var/lib. Anything that alters the boot chain's allocation
 #   footprint — and per §0b that includes the initrd, the module set and the boot device —
 #   changes the variable under test and voids the arm.
 # --- Sample size --------------------------------------------------------------------
 # Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
 # machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
 memattr_survey_rounds: 20
 # --- Safety interlock ----------------------------------------------------------------
 # REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
 # preflight, a baseline probe of the current boot, the journal harvest and the analysis.
 # That is a genuinely useful dry run and it is also how you prove the pipeline works
 # before committing 45 minutes of wall clock to it.
 memattr_survey_reboots_authorised: false
 # The role refuses to run against a host that is not in this group. The prod boxes carry
 # no k3s and nothing is running on them, which is the entire reason rebooting them is
 # sanctioned; the dev cluster is live and must never be caught by a stray --limit.
 memattr_survey_required_group: k3s_prod
 # Unit names whose presence or activity means something IS running on this node. Any hit
 # aborts the whole playbook before the first reboot (see tasks/preflight.yml).
 memattr_survey_forbidden_units:
  - k3s
  - k3s-agent
  - kubelet
  - containerd
  - docker
 # Paths whose existence means k3s has been installed here, even if not currently running.
 memattr_survey_forbidden_paths:
  - /etc/rancher/k3s
  - /var/lib/rancher/k3s
  - /usr/local/bin/k3s
 # --- Reboot behaviour -----------------------------------------------------------------
 # A node that does not answer within this is declared lost: the role stops rebooting THAT
 # node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
 # observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
 # slow boot.
 memattr_survey_reboot_timeout: 600
 memattr_survey_connect_timeout: 20
 # Let journald settle before reading the new boot's kernel log. The kernel log itself is
 # complete long before SSH is up; this is purely margin.
 memattr_survey_post_reboot_delay: 15
 # --- On-node ledger (belt) -------------------------------------------------------------
 # Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
 # the controller dies. This is what makes the survey resumable: re-running skips rounds
 # already recorded here rather than starting over.
 memattr_survey_state_dir: /var/lib/memattr-survey
 memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
 # --- Controller artifacts (braces) ------------------------------------------------------
 # Same convention as roles/node_health and conformance/_artifacts/: generated output under
 # a `_artifacts/` directory that .gitignore excludes, never committed.
 #   <host>.rounds.jsonl  the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
 #                        mid-survey still leaves every completed round on the controller
 #   <host>.boots.json    the journal harvest — every retained boot re-derived in one
 #                        pass. THIS is the authoritative record; the ledger is the backup.
 #   summary.md           the analysis (also printed)
 memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
 memattr_survey_write_artifacts: true
 # Print each round's one-line result as it happens. A 45-minute unattended run with no
 # output is indistinguishable from a hung one.
 memattr_survey_show_progress: true
@@ -0,0 +1,521 @@
 #!/usr/bin/env python3
 """memattr_analyse.py — turn a memattr survey's artifacts into the answer.
 Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads
  <host>.boots.json    the journal harvest — authoritative, one entry per retained boot
  <host>.rounds.jsonl  the per-round ledger — authoritative for ROUND NUMBERS
 and joins them on boot_id.
 WHAT IT ANSWERS, in order:
  1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated
     loudly and up front, because every number below it is weaker than it looks.
  2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing
     this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and
     completely different evidence. Clopper-Pearson is the exact (conservative) binomial
     interval — no normal approximation, correct at 0/n and n/n where the approximation is
     degenerate and where a clean arm is most likely to land.
  3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three
     ways, deliberately, because the tempting version of this test is circular:
       (a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such
           address falsifies "the address determines the outcome" outright, and no amount
           of correlation elsewhere repairs it.
       (b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported
           as sorted min/max per outcome plus the count of good addresses falling inside
           the bad range. This is descriptive; it is NOT dressed up as a p-value, because
           the bad set was chosen after seeing the data.
       (c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts
           that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots
           this run observed, which is the one non-circular test available.
  4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory
     ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell.
 Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is.
 """
 from __future__ import annotations
 import argparse
 import glob
 import json
 import math
 import os
 import sys
 from collections import Counter, defaultdict
 # §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad
 # both times, while all five successes fleet-wide were at other addresses.
 PREREGISTERED_BAD_ADDRESS = "0xbeeb0018"
 # --- Exact binomial interval ----------------------------------------------------------
 def binom_tail_ge(n: int, k: int, p: float) -> float:
    """P(X >= k) for X ~ Bin(n, p)."""
    if k <= 0:
        return 1.0
    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1))
 def binom_tail_le(n: int, k: int, p: float) -> float:
    """P(X <= k) for X ~ Bin(n, p)."""
    if k >= n:
        return 1.0
    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1))
 def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
    """Exact 95% (by default) binomial confidence interval for k successes in n trials.
    Solved by bisection on the exact tail probabilities rather than via the Beta/F
    quantile identity, so it needs nothing outside the standard library and the
    boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing:
      lower bound solves P(X >= k | p) = alpha/2,  and is 0 when k = 0
      upper bound solves P(X <= k | p) = alpha/2,  and is 1 when k = n
    """
    if n == 0:
        return (0.0, 1.0)
    def bisect(fn, target, lo, hi):
        for _ in range(200):
            mid = (lo + hi) / 2
            if fn(mid) < target:
                lo = mid
            else:
                hi = mid
        return (lo + hi) / 2
    lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0)
    upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0)
    return (lower, upper)
 def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float:
    """Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only.
    Used ONLY for the pre-registered prediction (address == the §0b address vs outcome),
    never for a bad set defined after the fact — that would be a garden of forking paths
    with a p-value stapled on.
    """
    n = a + b + c + d
    if n == 0:
        return 1.0
    row1, col1 = a + b, a + c
    def prob(x):
        return (
            math.comb(row1, x)
            * math.comb(n - row1, col1 - x)
            / math.comb(n, col1)
        )
    observed = prob(a)
    lo = max(0, col1 - (n - row1))
    hi = min(row1, col1)
    # 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating
    # float noise in the equality case.
    return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9)))
 # --- Loading ---------------------------------------------------------------------------
 def load_host(art_dir: str, host: str) -> dict:
    """Harvest (authoritative outcomes) joined to ledger (authoritative round numbers)."""
    with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle:
        harvest = json.load(handle)
    rounds_by_boot: dict[str, int] = {}
    ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl")
    ledger: list[dict] = []
    if os.path.exists(ledger_path):
        with open(ledger_path, encoding="utf-8") as handle:
            for line in handle:
                line = line.strip()
                if not line:
                    continue
                try:
                    record = json.loads(line)
                except json.JSONDecodeError:
                    continue
                ledger.append(record)
                if record.get("boot_id") and isinstance(record.get("round"), int):
                    rounds_by_boot[record["boot_id"]] = record["round"]
    boots = []
    for boot in harvest.get("boots", []):
        entry = dict(boot)
        entry["round"] = rounds_by_boot.get(boot.get("boot_id"))
        boots.append(entry)
    boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0)))
    # Written by the per-round rescue when a node stops coming back. Its absence is not
    # proof of success — a node can be lost so abruptly that Ansible calls it unreachable
    # before the rescue runs — so completeness is still judged on round COUNT, and this
    # only supplies the reason when one exists.
    abort = None
    abort_path = os.path.join(art_dir, f"{host}.aborted.json")
    if os.path.exists(abort_path):
        try:
            with open(abort_path, encoding="utf-8") as handle:
                abort = json.load(handle)
        except (OSError, json.JSONDecodeError):
            abort = {"reason": "abort marker present but unreadable"}
    return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort}
 def discover_hosts(art_dir: str) -> list[str]:
    return sorted(
        os.path.basename(p)[: -len(".boots.json")]
        for p in glob.glob(os.path.join(art_dir, "*.boots.json"))
    )
 # --- Rendering ---------------------------------------------------------------------------
 def gib(kb) -> str:
    return "?" if kb is None else f"{kb / 1048576:.2f}"
 def pct(x) -> str:
    return "?" if x is None else f"{100 * x:.0f}%"
 def render(hosts: list[dict], expected_rounds: int) -> str:
    out: list[str] = []
    w = out.append
    all_boots = [b for h in hosts for b in h["boots"]]
    # Only boots this survey CAUSED count toward any statistic. Excluded, deliberately:
    #   * pre-existing journal boots (round None) — observed under an unknown selection
    #     process, and already counted once in §0b's table;
    #   * round 0, the boot each node was found sitting on — it is a boot nobody chose,
    #     but for node 2 it is literally §0b's "boot 0" row, so pooling it would double-
    #     count prior evidence into a run whose entire point is fresh, independent trials.
    # Both are still SHOWN in the per-round table, because context is not contamination.
    surveyed = [b for b in all_boots if b.get("round") not in (None, 0)]
    w("# EFI Memory Attributes Table — baseline reboot survey")
    w("")
    w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no")
    w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this")
    w("measures p(bad) per node and tests whether the outcome is predicted by the address")
    w("the firmware published the table at.")
    w("")
    # --- 1. Completeness -----------------------------------------------------------------
    w("## 1. Completeness — read this before any number below it")
    w("")
    w("| node | rounds recorded | target | journal boots harvested | short? |")
    w("|---|---:|---:|---:|---|")
    short_arms = []
    for h in hosts:
        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
        n = len(rounds)
        is_short = n < expected_rounds
        if is_short:
            short_arms.append((h["host"], n))
        w(
            f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | "
            f"{'**YES — arm incomplete**' if is_short else 'no'} |"
        )
    w("")
    if short_arms:
        w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms))
        w("> Every interval below is correspondingly wider than planned.")
        w("")
        for h in hosts:
            if h.get("abort"):
                a = h["abort"]
                w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of "
                  f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}")
                w(f"> (failing task: {a.get('failed_task', 'unknown')})")
                w("")
    # --- 2. Per-round table ---------------------------------------------------------------
    w("## 2. Per-round results")
    w("")
    for h in hosts:
        installed = h["harvest"].get("smbios_installed_kb")
        w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)")
        w("")
        w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |")
        w("|---:|---|---|---|---:|---:|---:|---|")
        for b in h["boots"]:
            rnd = b.get("round")
            label = "pre" if rnd is None else str(rnd)
            usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb")
            w(
                f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | "
                f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | "
                f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | "
                f"{gib(usable)} | {pct(b.get('usable_fraction'))} | "
                f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |"
            )
        w("")
        w("`round` = survey round. `pre` = a boot already in the journal before the survey;")
        w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED")
        w("from every statistic below. `kernel read`: `valid` = table")
        w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage")
        w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =")
        w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from")
        w("the presence of a log line.")
        w("")
    # --- 3. p(bad) -------------------------------------------------------------------------
    w("## 3. p(bad) per node — 95% Clopper-Pearson")
    w("")
    w("| node | bad | n | p(bad) | 95% CI | reading |")
    w("|---|---:|---:|---:|---|---|")
    for h in hosts:
        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
        n = len(rounds)
        bad = sum(1 for b in rounds if b.get("verdict") == "bad")
        lo, hi = clopper_pearson(bad, n)
        if n == 0:
            reading = "no data"
        elif bad == 0:
            reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}"
        elif bad == n:
            reading = f"bad every time in {n}; p(bad) at least {lo:.2f}"
        else:
            reading = f"p(bad) between {lo:.2f} and {hi:.2f}"
        w(
            f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | "
            f"{lo:.2f} – {hi:.2f} | {reading} |"
        )
    w("")
    indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")]
    if indet:
        w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are")
        w("> reported as `indeterminate` rather than forced into a bucket. They are")
        w("> excluded from n above. Inspect them by hand.")
        w("")
    # --- 4. Address vs outcome -----------------------------------------------------------
    w("## 4. Is the outcome predicted by where the table landed?")
    w("")
    by_addr: dict[str, Counter] = defaultdict(Counter)
    addr_hosts: dict[str, set] = defaultdict(set)
    for b in surveyed:
        addr = b.get("memattr_address") or "none"
        by_addr[addr][b.get("verdict")] += 1
        addr_hosts[addr].add(b.get("host") or "?")
    w("### 4a. Every address observed in the survey")
    w("")
    w("| MEMATTR | good | bad | other | nodes | verdict for this address |")
    w("|---|---:|---:|---:|---|---|")
    ambiguous = []
    for addr in sorted(by_addr, key=lambda a: (a == "none", a)):
        counts = by_addr[addr]
        good, bad = counts.get("good", 0), counts.get("bad", 0)
        other = sum(v for k, v in counts.items() if k not in ("good", "bad"))
        if good and bad:
            note = "**BOTH — address does not determine outcome**"
            ambiguous.append(addr)
        elif bad:
            note = "always bad"
        elif good:
            note = "always good"
        else:
            note = "—"
        w(
            f"| `{addr}` | {good} | {bad} | {other} | "
            f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |"
        )
    w("")
    w("### 4b. Determinism")
    w("")
    if not by_addr:
        w("No surveyed boots — nothing to test.")
    elif ambiguous:
        w(f"**FALSIFIED for {len(ambiguous)} address(es):** "
          + ", ".join(f"`{a}`" for a in ambiguous)
          + " was seen with BOTH outcomes. The address alone cannot determine the outcome.")
    else:
        w("**Not falsified.** No address in this survey was ever seen with both outcomes —")
        w("consistent with the address determining the outcome, though with this many")
        w("distinct addresses most were seen only once, which makes the test weak.")
        singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1)
        w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)")
    w("")
    w("### 4c. Separation of the address ranges")
    w("")
    bad_addrs = sorted({b["memattr_address"] for b in surveyed
                        if b.get("verdict") == "bad" and b.get("memattr_address")})
    good_addrs = sorted({b["memattr_address"] for b in surveyed
                         if b.get("verdict") == "good" and b.get("memattr_address")})
    if bad_addrs and good_addrs:
        blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16)
        inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi]
        w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`")
        w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`")
        w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**"
          + (" — the ranges interleave, so 'a narrow bad range' does not hold"
             if inside else " — the ranges are disjoint"))
        w("")
        w("Descriptive only. The bad set was defined after seeing the data, so this is a")
        w("picture, not a test.")
    elif not bad_addrs:
        w("No bad boot was observed in this survey, so there is no bad range to separate.")
    else:
        w("No good boot was observed in this survey.")
    w("")
    w("### 4d. The pre-registered prediction")
    w("")
    w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the")
    w("bad address (both prior failures were there; all five prior successes elsewhere).")
    w("Tested against this survey's boots only:")
    w("")
    a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
    b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
    c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
    d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
    w(f"| | bad | good |")
    w(f"|---|---:|---:|")
    w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |")
    w(f"| elsewhere | {c} | {d} |")
    w("")
    if a + b_ == 0:
        w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the")
        w("prediction had no opportunity to be right or wrong. That is itself informative:")
        w("the address moves every boot, so a specific address is not a reproducible handle.")
    else:
        p = fisher_exact_2x2(a, b_, c, d)
        w(f"Fisher exact (two-sided) p = **{p:.4g}**"
          + ("  — the prediction holds." if p < 0.05 and a and not b_
             else "  — not significant at 0.05." if p >= 0.05
             else "  — significant, but read the cell counts before believing it."))
    w("")
    w("### 4e. Does the address predict how the KERNEL READ the table?")
    w("")
    w("The strongest test this data can support when no bad boot occurred. `verdict` has")
    w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both")
    w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:")
    w("the kernel reads whatever bytes happen to be at the address the firmware published.")
    w("If the address determines the reading, the address is doing causal work, one step")
    w("short of the catastrophic path.")
    w("")
    reads_by_addr: dict[str, set] = defaultdict(set)
    n_by_addr: Counter = Counter()
    for b in surveyed:
        addr = b.get("memattr_address") or "none"
        reads_by_addr[addr].add(b.get("kernel_read"))
        n_by_addr[addr] += 1
    repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1}
    inconsistent = {a: r for a, r in repeated.items() if len(r) > 1}
    distinct_reads = {r for rs in reads_by_addr.values() for r in rs}
    w(f"- distinct addresses: **{len(reads_by_addr)}**, of which "
      f"**{len(repeated)}** were observed more than once")
    w(f"- distinct kernel readings across the survey: "
      + ", ".join(f"`{r}`" for r in sorted(distinct_reads)))
    w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**")
    w("")
    if len(distinct_reads) < 2:
        w("Only one reading occurred in the whole survey, so there is nothing to predict.")
    elif inconsistent:
        w("**The address does NOT determine the reading:** "
          + "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items()))
    elif not repeated:
        w("No address recurred, so the test could not be run.")
    else:
        w(f"**The address determines the reading, on every address observed more than once**")
        w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both")
        w("readings present in the sample). This is direct evidence for the causal chain")
        w("§0b describes: the same address reproduces the same parse, boot after boot.")
        w("It does NOT establish which addresses are catastrophic — no bad boot occurred —")
        w("only that WHERE the table lands is what decides what the kernel finds there.")
    w("")
    w("| MEMATTR | boots | kernel reading(s) |")
    w("|---|---:|---|")
    for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)):
        marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else ""
        w(f"| `{addr}` | {n_by_addr[addr]} | "
          f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |")
    w("")
    # --- 5. Corroboration -------------------------------------------------------------------
    w("## 5. Corroborating evidence")
    w("")
    w("### 5a. Kernel's own reading vs the measured outcome")
    w("")
    cross: dict[tuple, int] = Counter()
    for b in surveyed:
        cross[(b.get("kernel_read"), b.get("verdict"))] += 1
    w("| kernel read | verdict | boots |")
    w("|---|---|---:|")
    for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))):
        w(f"| {read} | {verdict} | {count} |")
    w("")
    disagreements = [
        (read, verdict, count)
        for (read, verdict), count in cross.items()
        if (read == "failed-map") != (verdict == "bad")
    ]
    if disagreements:
        w("> **The log line and the measurement DISAGREE** in "
          f"{sum(c for _, _, c in disagreements)} boot(s): "
          + "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements)
          + ". That is a finding in its own right — §0b's model says `failed-map` and only")
        w("> `failed-map` costs memory.")
    else:
        w("`failed-map` and only `failed-map` cost memory, in every boot observed —")
        w("exactly as §0b's mechanism predicts, and independent of the address question.")
    w("")
    w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)")
    w("")
    w("| verdict | NODE_DATA end address | boots |")
    w("|---|---|---:|")
    nd: Counter = Counter()
    for b in surveyed:
        line = b.get("node_data_line") or ""
        end = line.split("-")[-1].rstrip("]") if "-" in line else "?"
        nd[(b.get("verdict"), end)] += 1
    for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
        w(f"| {verdict} | `{end}` | {count} |")
    w("")
    # --- 6. Invariants ------------------------------------------------------------------
    w("## 6. Nothing under test changed")
    w("")
    w("| node | kernels seen | distinct cmdlines | SMBIOS installed |")
    w("|---|---|---:|---|")
    for h in hosts:
        kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")})
        cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")}
        w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | "
          f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |")
    w("")
    w("More than one kernel or more than one cmdline on a node would void that node's arm.")
    w("")
    return "\n".join(out) + "\n"
 def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--artifact-dir", required=True)
    parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted")
    parser.add_argument("--out", default=None, help="also write the report here")
    args = parser.parse_args()
    names = discover_hosts(args.artifact_dir)
    if not names:
        print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr)
        return 1
    hosts = [load_host(args.artifact_dir, name) for name in names]
    report = render(hosts, args.rounds)
    if args.out:
        with open(args.out, "w", encoding="utf-8") as handle:
            handle.write(report)
    sys.stdout.write(report)
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,489 @@
 #!/usr/bin/env python3
 """memattr_probe.py — read one boot's EFI Memory Attributes Table outcome off a node.
 Part of roles/memattr_survey (infra/ansible). See that role's README.md for the
 question this answers; see infra/PROD-PROVISIONING.md §0b for the settled mechanism.
 WHY A SCRIPT AND NOT A HANDFUL OF `command:` TASKS
  The record for one boot needs six sources cross-referenced (kernel log config-table
  line, kernel log memattr diagnostic, kernel log `Memory:` line, kernel log
  `NODE_DATA` line, /proc/meminfo, SMBIOS) and one derived verdict. Doing that in
  Jinja across six registered results is unreadable and would put the classification
  logic in a template instead of in one testable place. It also has to be exactly the
  same logic for a live boot and for a boot harvested out of the journal, which a
  single --all-boots mode gives for free.
 MODES
  (default)      one JSON record for the CURRENT boot, on stdout
  --all-boots    one JSON object with a `boots` list: EVERY boot the persistent
                 journal still holds, re-derived from scratch. This is the
                 AUTHORITATIVE record — it survives the driver script dying mid-run.
  --status       {"completed_rounds": N, "records": M} from the on-node ledger, so the
                 playbook can resume without re-deriving anything.
  --round N      stamp the record with the survey round number
  --append PATH  ALSO append the record to PATH as one JSON line (the on-node ledger),
                 created 0644 with parents. Appended with O_APPEND in one write() so a
                 concurrent reader never sees a half line.
 VERDICT IS MEASURED, NOT INFERRED FROM A LOG LINE
  A node whose kernel reads a VALID table logs no memattr diagnostic at all and is
  perfectly healthy; a node whose kernel bails out on a garbage version > 2 logs a
  scary warning and is ALSO perfectly healthy. Only the third state — "Failed to map"
  — costs memory, and even that message is emitted by a LATER function than the one
  that did the damage. So the verdict comes from usable RAM versus SMBIOS installed
  RAM, and `kernel_read` is recorded ALONGSIDE it as an independent observation. If
  the two ever disagree, that disagreement is itself the finding.
 EXIT STATUS is 0 whenever a record could be produced, even a mostly-null one: a field
 this script could not read comes back as null with a reason in `notes`, never as an
 empty string and never as a hard failure. A survey that aborts on one unreadable field
 loses the other 19 rounds.
 """
 from __future__ import annotations
 import argparse
 import json
 import os
 import re
 import subprocess
 import sys
 import time
 SCHEMA = "memattr-survey/1"
 # --- Verdict thresholds -------------------------------------------------------------
 # Observed on this fleet: a good boot yields usable/installed of 0.95-0.98 (kernel code,
 # reserved firmware regions and struct page overhead account for the rest); the bad path
 # yields 0.15-0.17 (everything above the table address is reserved). Two orders of
 # separation, so the thresholds are nowhere near either cluster. Anything landing in the
 # gap is deliberately NOT called: it is a new third thing and wants human eyes.
 VERDICT_GOOD_MIN_FRACTION = 0.85
 VERDICT_BAD_MAX_FRACTION = 0.60
 # `Memory: 2568544K/16546088K available (... 13958980K reserved ...)`
 RE_MEMORY_LINE = re.compile(
    r"^Memory:\s+(?P<available>\d+)K/(?P<present>\d+)K available\b.*?"
    r"(?:(?P<reserved>\d+)K reserved)?"
 )
 # `efi: ACPI=0x... SMBIOS=0x... MEMATTR=0xbeeb0018 MOKvar=0x... INITRD=0x...`
 RE_MEMATTR_ADDR = re.compile(r"\bMEMATTR=(?P<addr>0x[0-9a-fA-F]+)")
 # `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749`
 RE_TABLE_VERSION = re.compile(r"table version\s+(?P<version>-?\d+)")
 # `NODE_DATA(0) allocated [mem 0xbee85680-0xbeeaffff]` — per §0b the cleanest A/B tell:
 # on the bad path pgdat is forced immediately BELOW the table address instead of landing
 # at the top of RAM. Captured because it is free and it is independent evidence.
 RE_NODE_DATA = re.compile(r"^NODE_DATA\(\d+\) allocated \[mem (?P<range>[0-9a-fx]+-[0-9a-fx]+)\]")
 RE_CMDLINE = re.compile(r"^Command line:\s*(?P<cmdline>.*)$")
 # dmidecode type 17 `Size: 16 GB` / `Size: 8192 MB` / `Size: No Module Installed`
 RE_DMI_SIZE = re.compile(r"^\s*Size:\s+(?P<n>\d+)\s+(?P<unit>[kKmMgGtT]B)\s*$")
 DMI_UNIT_KB = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
 def run(argv: list[str], timeout: int = 60) -> tuple[int, str, str]:
    """Run a command, never raise. Returns (rc, stdout, stderr); rc 127 if absent."""
    try:
        proc = subprocess.run(
            argv, capture_output=True, text=True, timeout=timeout, check=False
        )
    except FileNotFoundError:
        return 127, "", f"{argv[0]}: not found"
    except subprocess.TimeoutExpired:
        return 124, "", f"{' '.join(argv)}: timed out after {timeout}s"
    except OSError as exc:  # pragma: no cover - defensive
        return 126, "", f"{' '.join(argv)}: {exc}"
    return proc.returncode, proc.stdout, proc.stderr
 def read_text(path: str) -> str | None:
    try:
        with open(path, "r", encoding="utf-8", errors="replace") as handle:
            return handle.read()
    except OSError:
        return None
 def kernel_log(boot: str) -> tuple[list[str], str | None]:
    """Kernel ring buffer for `boot` ('0' = current, '-1' = previous, or a boot ID).
    `-o cat` is deliberate: it yields the message VERBATIM with no host/facility
    decoration, which is what the survey wants to record, and it is stable across
    systemd versions. `-k` restricts to _TRANSPORT=kernel.
    """
    rc, out, err = run(["journalctl", "-k", "-b", boot, "-o", "cat", "--no-pager"], timeout=180)
    if rc != 0:
        return [], f"journalctl -k -b {boot} failed (rc={rc}): {err.strip()[:200]}"
    return out.splitlines(), None
 def list_boots() -> tuple[list[dict], str | None]:
    """Every boot the journal still holds, oldest first.
    Prefers `--list-boots -o json` (systemd >= 250, so always on Debian 13) and falls
    back to parsing the table, because a survey that cannot enumerate boots has lost
    its authoritative record and must say so rather than silently reporting one boot.
    """
    rc, out, err = run(["journalctl", "--list-boots", "-o", "json", "--no-pager"], timeout=120)
    if rc == 0:
        try:
            parsed = json.loads(out)
        except json.JSONDecodeError as exc:
            return [], f"--list-boots -o json unparseable: {exc}"
        boots = [
            {
                "index": int(entry.get("index", 0)),
                "boot_id": entry.get("boot_id"),
                "first_entry_usec": entry.get("first_entry"),
                "last_entry_usec": entry.get("last_entry"),
            }
            for entry in parsed
        ]
        return sorted(boots, key=lambda b: b["index"]), None
    rc, out, err = run(["journalctl", "--list-boots", "--no-pager"], timeout=120)
    if rc != 0:
        return [], f"journalctl --list-boots failed (rc={rc}): {err.strip()[:200]}"
    boots = []
    for line in out.splitlines():
        fields = line.split()
        if len(fields) < 2 or not re.fullmatch(r"-?\d+", fields[0]):
            continue  # header row
        boots.append(
            {
                "index": int(fields[0]),
                "boot_id": fields[1],
                "first_entry_usec": None,
                "last_entry_usec": None,
            }
        )
    return sorted(boots, key=lambda b: b["index"]), None
 def journal_storage() -> dict:
    """Is the journal PERSISTENT? The harvest is only authoritative if it is.
    /var/log/journal existing is what makes journald write to disk; /run/log/journal is
    the volatile fallback and would mean every reboot erases the record.
    """
    return {
        "var_log_journal": os.path.isdir("/var/log/journal"),
        "run_log_journal": os.path.isdir("/run/log/journal"),
        "persistent": os.path.isdir("/var/log/journal"),
    }
 def smbios_installed_kb() -> tuple[int | None, str | None]:
    """Installed RAM per SMBIOS type-17 records — the GROUND TRUTH for the verdict.
    Deliberately not /proc/meminfo (that is the number under test) and not dmesg's
    `Memory: x/yK` denominator (which is already post-e820 and post-reservation).
    Hardware does not change between boots, so one read applies to harvested boots too.
    """
    rc, out, err = run(["dmidecode", "-t", "17"], timeout=60)
    if rc != 0:
        return None, f"dmidecode -t 17 failed (rc={rc}): {err.strip()[:200]}"
    total = 0
    seen = 0
    for line in out.splitlines():
        match = RE_DMI_SIZE.match(line)
        if not match:
            continue  # "No Module Installed" and every non-Size line
        total += int(match.group("n")) * DMI_UNIT_KB[match.group("unit").lower()]
        seen += 1
    if seen == 0:
        return None, "dmidecode -t 17 reported no populated memory devices"
    return total, None
 def meminfo_total_kb() -> tuple[int | None, str | None]:
    text = read_text("/proc/meminfo")
    if text is None:
        return None, "/proc/meminfo unreadable"
    for line in text.splitlines():
        if line.startswith("MemTotal:"):
            return int(line.split()[1]), None
    return None, "/proc/meminfo has no MemTotal"
 def classify_kernel_read(memattr_addr, diagnostics: list[str]) -> tuple[str, int | None]:
    """What did the kernel make of the table? THREE outcomes, not two (§0b).
    Returns (classification, table_version_if_reported).
      no-table      firmware published no MEMATTR pointer at all (the dev nodes)
      valid         a MEMATTR pointer AND no diagnostic => the kernel parsed a table it
                    considered valid. This is SILENCE AS EVIDENCE, and it is a healthy
                    state, not a missing measurement.
      bail-version  garbage version > 2 => kernel warned and bailed. Memory intact.
                    The LUCKY path, and it looks alarming in the log.
      failed-map    the catastrophic path. The message comes from
                    efi_memattr_apply_permissions() and is therefore PROOF the bogus
                    memblock_reserve() already happened upstream of it.
      bogus-size    a kernel >= 6.13 rejecting the table. Should never appear on 6.12;
                    if it does, the kernel under test changed and the arm is void.
    """
    version = None
    for line in diagnostics:
        match = RE_TABLE_VERSION.search(line)
        if match:
            version = int(match.group("version"))
    joined = " | ".join(diagnostics)
    if "Failed to map" in joined:
        return "failed-map", version
    if "Corrupted EFI Memory Attributes Table" in joined:
        return "bogus-size", version
    if "Unexpected EFI Memory Attributes table version" in joined:
        return "bail-version", version
    if diagnostics:
        return "diagnostic-unrecognised", version
    if memattr_addr is None:
        return "no-table", version
    return "valid", version
 def derive_verdict(usable_kb, installed_kb) -> tuple[str, float | None]:
    if usable_kb is None or not installed_kb:
        return "unknown", None
    fraction = usable_kb / installed_kb
    if fraction >= VERDICT_GOOD_MIN_FRACTION:
        return "good", fraction
    if fraction <= VERDICT_BAD_MAX_FRACTION:
        return "bad", fraction
    return "indeterminate", fraction
 def build_record(
    *,
    boot_selector: str,
    boot_meta: dict | None,
    installed_kb,
    installed_note,
    round_number,
    live: bool,
 ) -> dict:
    notes: list[str] = []
    if installed_note:
        notes.append(installed_note)
    lines, log_note = kernel_log(boot_selector)
    if log_note:
        notes.append(log_note)
    config_table_line = None
    memattr_addr = None
    diagnostics: list[str] = []
    memory_line = None
    node_data_line = None
    cmdline = None
    for line in lines:
        stripped = line.strip()
        if memattr_addr is None:
            match = RE_MEMATTR_ADDR.search(stripped)
            if match and stripped.startswith("efi:"):
                memattr_addr = match.group("addr").lower()
                config_table_line = stripped
        # Every memattr-related diagnostic, verbatim. Matched on the pr_fmt prefix the
        # kernel actually uses (`efi: memattr: `) AND on the message text, so a prefix
        # change upstream cannot silently drop the line.
        if stripped.startswith("efi: memattr:") or "EFI Memory Attributes" in stripped:
            if stripped != config_table_line:
                diagnostics.append(stripped)
        if memory_line is None and stripped.startswith("Memory:"):
            if RE_MEMORY_LINE.match(stripped):
                memory_line = stripped
        if node_data_line is None:
            match = RE_NODE_DATA.match(stripped)
            if match:
                node_data_line = stripped
        if cmdline is None:
            match = RE_CMDLINE.match(stripped)
            if match:
                cmdline = match.group("cmdline").strip()
    available_kb = present_kb = reserved_kb = None
    if memory_line:
        match = RE_MEMORY_LINE.match(memory_line)
        available_kb = int(match.group("available"))
        present_kb = int(match.group("present"))
        reserved = match.group("reserved")
        reserved_kb = int(reserved) if reserved else None
    else:
        notes.append("no `Memory: .../... available` line found in this boot's kernel log")
    kernel_read, table_version = classify_kernel_read(memattr_addr, diagnostics)
    # /proc/meminfo describes the CURRENT boot only. For a harvested past boot the
    # equivalent measurement is the kernel's own `Memory: available` figure, which
    # tracks MemTotal to within ~7% on both paths. Which one was used is recorded, so
    # nobody has to guess later.
    mem_total_kb, meminfo_note = (None, None)
    if live:
        mem_total_kb, meminfo_note = meminfo_total_kb()
        if meminfo_note:
            notes.append(meminfo_note)
    usable_kb = mem_total_kb if mem_total_kb is not None else available_kb
    basis = "MemTotal" if mem_total_kb is not None else "dmesg Memory: available"
    verdict, fraction = derive_verdict(usable_kb, installed_kb)
    record = {
        "schema": SCHEMA,
        "round": round_number,
        "host": os.uname().nodename,
        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
        "recorded_at_epoch": int(time.time()),
        "live_boot": live,
        "boot_index": (boot_meta or {}).get("index"),
        "boot_id": (boot_meta or {}).get("boot_id"),
        "boot_first_entry_usec": (boot_meta or {}).get("first_entry_usec"),
        "kernel_release": os.uname().release,
        "kernel_cmdline": cmdline,
        # --- the independent variable -------------------------------------------------
        "efi_config_table_line": config_table_line,
        "memattr_address": memattr_addr,
        "memattr_diagnostics": diagnostics,
        "memattr_diagnostic_count": len(diagnostics),
        # An explicit marker, never an empty field: "the kernel said nothing" is a
        # measurement (it read a VALID table), not a gap in the data.
        "memattr_diagnostic_present": bool(diagnostics),
        "memattr_table_version": table_version,
        "kernel_read": kernel_read,
        # --- the dependent variable ---------------------------------------------------
        "memory_line": memory_line,
        "memory_available_kb": available_kb,
        "memory_present_kb": present_kb,
        "memory_reserved_kb": reserved_kb,
        "mem_total_kb": mem_total_kb,
        "smbios_installed_kb": installed_kb,
        "usable_kb": usable_kb,
        "usable_fraction": round(fraction, 5) if fraction is not None else None,
        "verdict": verdict,
        "verdict_basis": f"{basis} vs SMBIOS installed",
        # --- corroborating evidence ---------------------------------------------------
        "node_data_line": node_data_line,
        "notes": notes,
    }
    return record
 def append_record(path: str, record: dict) -> str | None:
    """One O_APPEND write of one line, so a reader never sees a torn record."""
    try:
        parent = os.path.dirname(path) or "."
        os.makedirs(parent, mode=0o755, exist_ok=True)
        payload = (json.dumps(record, sort_keys=True) + "\n").encode()
        fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
        try:
            os.write(fd, payload)
            os.fsync(fd)  # the point of the ledger is surviving the next reboot
        finally:
            os.close(fd)
    except OSError as exc:
        return f"could not append to {path}: {exc}"
    return None
 def ledger_status(path: str) -> dict:
    """Resume support: how far did a previous run get?
    Round numbers come from the ledger itself rather than a line count, so the
    baseline round 0 record and any hand-added line cannot shift the numbering.
    """
    rounds: list[int] = []
    records = 0
    malformed = 0
    if os.path.exists(path):
        with open(path, "r", encoding="utf-8", errors="replace") as handle:
            for line in handle:
                line = line.strip()
                if not line:
                    continue
                records += 1
                try:
                    parsed = json.loads(line)
                except json.JSONDecodeError:
                    malformed += 1
                    continue
                value = parsed.get("round")
                if isinstance(value, int):
                    rounds.append(value)
    return {
        "schema": SCHEMA,
        "ledger": path,
        "records": records,
        "malformed_records": malformed,
        "completed_rounds": max(rounds) if rounds else 0,
        "rounds_recorded": sorted(rounds),
    }
 def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--round", type=int, default=None, help="survey round number to stamp")
    parser.add_argument("--append", default=None, help="on-node ledger to append to")
    parser.add_argument("--all-boots", action="store_true", help="harvest every retained boot")
    parser.add_argument("--status", default=None, metavar="LEDGER", help="report resume state")
    args = parser.parse_args()
    if args.status:
        print(json.dumps(ledger_status(args.status), indent=2, sort_keys=True))
        return 0
    installed_kb, installed_note = smbios_installed_kb()
    boots, boots_note = list_boots()
    storage = journal_storage()
    if args.all_boots:
        records = []
        for boot in boots:
            # Address boots by BOOT ID, not by relative index: indices shift under the
            # harvester's feet if journald rotates mid-harvest, and the ID is what the
            # per-round ledger recorded, so the two records join cleanly.
            selector = boot["boot_id"] or str(boot["index"])
            records.append(
                build_record(
                    boot_selector=selector,
                    boot_meta=boot,
                    installed_kb=installed_kb,
                    installed_note=installed_note,
                    round_number=None,
                    live=(boot["index"] == 0),
                )
            )
        payload = {
            "schema": SCHEMA,
            "host": os.uname().nodename,
            "harvested_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
            "journal_storage": storage,
            "smbios_installed_kb": installed_kb,
            "boots_listed": len(boots),
            "notes": [n for n in [boots_note, installed_note] if n],
            "boots": records,
        }
        print(json.dumps(payload, indent=2, sort_keys=True))
        return 0
    current = next((b for b in boots if b["index"] == 0), None)
    record = build_record(
        boot_selector="0",
        boot_meta=current,
        installed_kb=installed_kb,
        installed_note=installed_note,
        round_number=args.round,
        live=True,
    )
    record["journal_storage"] = storage
    if boots_note:
        record["notes"].append(boots_note)
    if args.append:
        note = append_record(args.append, record)
        if note:
            record["notes"].append(note)
    print(json.dumps(record, indent=2, sort_keys=True))
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,42 @@
 ---
 # Controller-side analysis. Runs on localhost over the artifact directory, so it is
 # independent of whether any surveyed node is still alive — and so it can be re-run on
 # its own (`--tags analyse`) without touching hardware.
 #
 # The statistics live in files/memattr_analyse.py, not in Jinja: Clopper-Pearson and
 # Fisher belong somewhere testable, and a confidence interval assembled out of template
 # filters is a confidence interval nobody will ever check.
 - name: Locate the survey artifacts
  ansible.builtin.find:
    paths: "{{ memattr_survey_artifact_dir }}"
    patterns: "*.boots.json"
  register: memattr_survey_artifacts
 - name: Report that there is nothing to analyse
  ansible.builtin.debug:
    msg: >-
      No *.boots.json under {{ memattr_survey_artifact_dir }} — no survey has produced a
      harvest yet. Run the playbook with -e memattr_survey_reboots_authorised=true.
  when: memattr_survey_artifacts.matched | int == 0
 - name: Analyse the survey
  ansible.builtin.command:
    argv:
      - /usr/bin/env
      - python3
      - "{{ role_path }}/files/memattr_analyse.py"
      - "--artifact-dir"
      - "{{ memattr_survey_artifact_dir }}"
      - "--rounds"
      - "{{ memattr_survey_rounds }}"
      - "--out"
      - "{{ memattr_survey_artifact_dir }}/summary.md"
  register: memattr_survey_analysis
  changed_when: false
  when: memattr_survey_artifacts.matched | int > 0
 - name: Show the analysis
  ansible.builtin.debug:
    msg: "{{ memattr_survey_analysis.stdout.split('\n') }}"
  when: memattr_survey_analysis is not skipped
@@ -0,0 +1,66 @@
 ---
 # The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from
 # scratch in one pass by the same code that produced the per-round ledger.
 #
 # WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and
 # the append, that round exists on the machine but not in the ledger. The journal does not
 # care whether the driver was alive — the boot happened, the kernel logged it, journald
 # persisted it. So the harvest can only ever be a superset of the ledger, and the analysis
 # joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and
 # any boot the ledger missed still shows up.
 #
 # This runs even for a node that aborted mid-survey, which is exactly when it matters most.
 - name: Harvest every retained boot from the persistent journal
  ansible.builtin.script:
    cmd: memattr_probe.py --all-boots
    executable: /usr/bin/python3
  register: memattr_survey_harvest_raw
  changed_when: false   # strictly a read of the journal
 - name: Parse the harvest
  ansible.builtin.set_fact:
    memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}"
 - name: Ensure the artifact directory exists (on the controller)
  ansible.builtin.file:
    path: "{{ memattr_survey_artifact_dir }}"
    state: directory
    mode: "0755"
  delegate_to: localhost
  become: false
  # NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and
  # run_once is unreliable there. `state: directory` is idempotent, so three hosts each
  # asserting the directory exists is cheaper than a conditional that might not fire.
  changed_when: false
  when: memattr_survey_write_artifacts | bool
 - name: Write the harvested boot record
  ansible.builtin.copy:
    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json"
    mode: "0644"
    content: "{{ memattr_survey_harvest | to_nice_json }}"
  delegate_to: localhost
  become: false
  changed_when: false
  when: memattr_survey_write_artifacts | bool
 - name: Mirror the final ledger to the controller
  ansible.builtin.fetch:
    src: "{{ memattr_survey_state_file }}"
    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
    flat: true
  changed_when: false
  when: memattr_survey_write_artifacts | bool
 - name: Harvest summary
  ansible.builtin.debug:
    msg: >-
      {{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots
      from the journal
      ({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad,
      {{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good,
      {{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other)
      {%- if memattr_survey_aborted | default(false) | bool %}
      — NOTE: this node ABORTED ({{ memattr_survey_abort_reason }})
      {%- endif %}
@@ -0,0 +1,23 @@
 ---
 # memattr_survey — see defaults/main.yml for the question, README.md for how to run and
 # read it, and infra/PROD-PROVISIONING.md §0b for the settled mechanism this measures.
 #
 # ORDER OF OPERATIONS
 #   1. preflight.yml — refuse to touch a node that is not prod, not idle, or has no
 #      persistent journal; read the resume point; record the boot we found it on.
 #   2. round.yml x N — reboot, measure, append, mirror. Per-node fail-safe.
 #   3. harvest.yml   — re-derive every retained boot from the journal (authoritative).
 #
 # The playbook splits these across plays so that preflight is any_errors_fatal (one unsafe
 # node stops everything) while the rounds are not (one dead node stops only itself), and
 # so that the controller-side analysis runs even if every node dies. This file is the
 # whole-role entry point kept for completeness; the playbook calls the stages directly.
 - name: Preflight and baseline
  ansible.builtin.include_tasks: preflight.yml
 - name: Run the survey rounds
  ansible.builtin.include_tasks: rounds.yml
 - name: Harvest the authoritative record
  ansible.builtin.include_tasks: harvest.yml
@@ -0,0 +1,149 @@
 ---
 # Safety gate. Runs with any_errors_fatal at the play level (playbooks/memattr-survey.yml),
 # so a violation on ANY node stops the entire playbook before a single reboot — the
 # correct posture when the failure mode is "rebooted a machine that was doing something".
 #
 # VERIFY, DO NOT ASSUME. The premise that makes these reboots safe is that prod carries no
 # k3s and runs nothing. That premise is checked here on every run rather than trusted from
 # a document, because the document and the machine drift independently.
 - name: Assert the target host is in the prod group
  # The dev cluster is LIVE. This is the one check that a mistyped --limit or a copied
  # command line cannot slip past: membership is read from the inventory, not the pattern.
  ansible.builtin.assert:
    that:
      - memattr_survey_required_group in (group_names | default([]))
    fail_msg: >-
      REFUSING TO SURVEY {{ inventory_hostname }}: it is not a member of
      '{{ memattr_survey_required_group }}' (groups: {{ group_names | default([]) | join(', ') }}).
      This role reboots its targets. Point it at prod, or override
      memattr_survey_required_group deliberately.
    quiet: true
 - name: Check whether any cluster or container runtime unit is active
  # `is-active` returns non-zero for inactive/unknown units, which is the expected answer
  # here, so the rc is data and not a failure.
  ansible.builtin.command:
    argv: "{{ ['systemctl', 'is-active'] + memattr_survey_forbidden_units }}"
  register: memattr_survey_units
  changed_when: false
  failed_when: false
 - name: Check whether k3s has been installed on this node
  ansible.builtin.stat:
    path: "{{ item }}"
  loop: "{{ memattr_survey_forbidden_paths }}"
  register: memattr_survey_paths
  changed_when: false
 - name: Assert nothing is running on this node
  ansible.builtin.assert:
    that:
      - memattr_survey_units.stdout_lines | reject('equalto', 'inactive') | reject('equalto', 'unknown') | list | length == 0
      - memattr_survey_paths.results | selectattr('stat.exists') | list | length == 0
    fail_msg: >-
      REFUSING TO REBOOT {{ inventory_hostname }} — it is not idle.
      units {{ memattr_survey_forbidden_units | zip(memattr_survey_units.stdout_lines) | map('join', '=') | join(' ') }};
      present paths: {{ memattr_survey_paths.results | selectattr('stat.exists') | map(attribute='item') | list | join(', ') | default('none', true) }}.
      The survey is only sanctioned because these boxes carry no k3s and run nothing.
    quiet: true
 - name: Clear any abort marker from a previous run
  # A marker left by a run that has since been resumed successfully would make this run's
  # report describe the LAST failure rather than this one. The marker means "this run
  # aborted", so it is cleared at the start of every run and re-written only if it does.
  ansible.builtin.file:
    path: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
    state: absent
  delegate_to: localhost
  become: false
  changed_when: false
  when: memattr_survey_write_artifacts | bool
 - name: Assert forks allows every node to be surveyed concurrently
  # The survey play runs `strategy: free` so the nodes reboot in parallel, but forks is a
  # hard cap on that: with fewer forks than hosts the survey quietly serialises and takes
  # a multiple of the wall clock it should. Cheap to satisfy (`-f <n>`), so assert rather
  # than discover it three hours later.
  ansible.builtin.assert:
    that:
      - (lookup('ansible.builtin.config', 'DEFAULT_FORKS') | int) >= (ansible_play_hosts_all | length)
    fail_msg: >-
      forks={{ lookup('ansible.builtin.config', 'DEFAULT_FORKS') }} but this survey targets
      {{ ansible_play_hosts_all | length }} hosts. The rounds would serialise. Re-run with
      -f {{ ansible_play_hosts_all | length }} (or higher).
    quiet: true
  run_once: true
 - name: Read the node's resume state and baseline boot
  # One call does double duty: it reports how far a previous run got (so this one resumes)
  # and, with --round 0, records the boot we found the node on. Round 0 is a free data
  # point — it is a boot nobody chose, so it is not selected for by the survey.
  ansible.builtin.script:
    cmd: >-
      memattr_probe.py --status {{ memattr_survey_state_file | quote }}
    executable: /usr/bin/python3
  register: memattr_survey_status_raw
  changed_when: false
 - name: Record the resume point
  ansible.builtin.set_fact:
    memattr_survey_status: "{{ memattr_survey_status_raw.stdout | from_json }}"
    memattr_survey_aborted: false
    memattr_survey_abort_reason: ""
 - name: Assert the probe schema is the one this role understands
  ansible.builtin.assert:
    that:
      - memattr_survey_status.schema == 'memattr-survey/1'
    fail_msg: >-
      Probe returned schema {{ memattr_survey_status.schema | default('<none>') }};
      this role understands memattr-survey/1. Role and probe are out of step.
    quiet: true
 - name: Probe the boot the node is currently on
  # On a FIRST run this is stamped round 0 and appended: the boot the node was already
  # sitting on is a free, unselected data point. On a RESUME it is read but NOT appended,
  # because that boot is already round N of the interrupted run and double-counting it
  # would bias the sample toward whatever outcome happened to survive the interruption.
  ansible.builtin.script:
    cmd: >-
      memattr_probe.py
      {% if memattr_survey_status.completed_rounds | int == 0 %}
      --round 0 --append {{ memattr_survey_state_file | quote }}
      {% endif %}
    executable: /usr/bin/python3
  register: memattr_survey_baseline_raw
  # The append is the experiment's own ledger, not a change to the node under test: this
  # role reports `changed` only for things that could affect a boot, and a file written
  # under /var/lib long after the kernel parsed the table cannot.
  changed_when: false
 - name: Parse the baseline record
  ansible.builtin.set_fact:
    memattr_survey_baseline: "{{ memattr_survey_baseline_raw.stdout | from_json }}"
 - name: Assert the journal is PERSISTENT
  # The end-of-run harvest is the authoritative record precisely because it re-derives
  # every boot from the journal. On a volatile journal (/run/log/journal only) each
  # reboot erases the evidence and the harvest would silently return one boot — a survey
  # that looks complete and holds nothing. Verified, not assumed.
  ansible.builtin.assert:
    that:
      - memattr_survey_baseline.journal_storage.persistent | bool
    fail_msg: >-
      {{ inventory_hostname }} has NO persistent journal (/var/log/journal missing).
      Every reboot would erase the record this survey depends on. Refusing to run.
    quiet: true
 - name: Report the starting state
  ansible.builtin.debug:
    msg: >-
      {{ inventory_hostname }}: idle, prod, persistent journal.
      kernel {{ memattr_survey_baseline.kernel_release | default('?') }},
      installed {{ (memattr_survey_baseline.smbios_installed_kb | default(0) | int / 1048576) | round(1) }} GiB.
      baseline boot MEMATTR={{ memattr_survey_baseline.memattr_address | default('none') }}
      kernel_read={{ memattr_survey_baseline.kernel_read | default('?') }}
      verdict={{ memattr_survey_baseline.verdict | default('?') | upper }}.
      resuming from round {{ memattr_survey_status.completed_rounds }}
      of {{ memattr_survey_rounds }}.
@@ -0,0 +1,98 @@
 ---
 # ONE round on ONE node: reboot, wait for it back, measure the new boot, record it twice.
 # Included in a loop from tasks/main.yml with loop_var `memattr_survey_round`.
 #
 # FAIL-SAFE, PER NODE. The block/rescue below is what keeps one dead node from ending the
 # survey: on any failure this host sets memattr_survey_aborted and every later round is
 # skipped FOR THIS HOST ONLY, while the other nodes keep going (Ansible runs the loop for
 # all hosts in lockstep, so the reboots are genuinely concurrent). The abort is reported
 # loudly here AND again in the final analysis, which counts rounds from the artifacts
 # rather than from Ansible's host state — so a node that vanishes so hard that it is
 # marked unreachable (which rescue cannot catch) still shows up as a short arm.
 - name: "Round {{ memattr_survey_round }} — survey step"
  block:
    - name: "Round {{ memattr_survey_round }} — reboot and wait for the node to return"
      ansible.builtin.reboot:
        reboot_timeout: "{{ memattr_survey_reboot_timeout | int }}"
        connect_timeout: "{{ memattr_survey_connect_timeout | int }}"
        post_reboot_delay: "{{ memattr_survey_post_reboot_delay | int }}"
        msg: "memattr survey round {{ memattr_survey_round }} of {{ memattr_survey_rounds }}"
    - name: "Round {{ memattr_survey_round }} — measure the new boot and append to the ledger"
      ansible.builtin.script:
        cmd: >-
          memattr_probe.py --round {{ memattr_survey_round }}
          --append {{ memattr_survey_state_file | quote }}
        executable: /usr/bin/python3
      register: memattr_survey_round_raw
      changed_when: false   # see the note in preflight.yml — the ledger is not the node
    - name: "Round {{ memattr_survey_round }} — parse the record"
      ansible.builtin.set_fact:
        memattr_survey_record: "{{ memattr_survey_round_raw.stdout | from_json }}"
    - name: "Round {{ memattr_survey_round }} — result"
      ansible.builtin.debug:
        msg: >-
          [{{ inventory_hostname }} r{{ memattr_survey_round }}/{{ memattr_survey_rounds }}]
          MEMATTR={{ memattr_survey_record.memattr_address | default('none') }}
          read={{ memattr_survey_record.kernel_read }}
          {%- if memattr_survey_record.memattr_table_version is not none %}
          (version {{ memattr_survey_record.memattr_table_version }})
          {%- endif %}
          MemTotal={{ (memattr_survey_record.mem_total_kb | default(0) | int / 1048576) | round(2) }}G
          of {{ (memattr_survey_record.smbios_installed_kb | default(0) | int / 1048576) | round(0) | int }}G
          => {{ memattr_survey_record.verdict | upper }}
          boot={{ memattr_survey_record.boot_id | default('?') | truncate(9, true, '') }}
      when: memattr_survey_show_progress | bool
    # BRACES. The on-node ledger already survives a reboot; re-fetching it every round
    # means the controller also holds every completed round if the DRIVER dies, without
    # waiting for the end-of-run harvest. One tiny file per host, per-host destination
    # paths, so three concurrent fetches cannot collide.
    - name: "Round {{ memattr_survey_round }} — mirror the ledger to the controller"
      ansible.builtin.fetch:
        src: "{{ memattr_survey_state_file }}"
        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
        flat: true
      changed_when: false
      when: memattr_survey_write_artifacts | bool
  rescue:
    - name: "Round {{ memattr_survey_round }} — ABORT this node, continue the others"
      ansible.builtin.set_fact:
        memattr_survey_aborted: true
        memattr_survey_abort_reason: >-
          round {{ memattr_survey_round }} failed:
          {{ ansible_failed_result.msg | default('unknown error') | trim }}
    # The abort must survive the run, not live only in terminal scrollback. Without
    # this the report can see that an arm is SHORT but not WHY, and "0 rounds" reads
    # like a targeting mistake instead of a node that failed to come back.
    - name: "Round {{ memattr_survey_round }} — record the abort as an artifact"
      ansible.builtin.copy:
        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
        mode: "0644"
        content: "{{ memattr_survey_abort_record | to_nice_json }}"
      vars:
        memattr_survey_abort_record:
          schema: "memattr-survey/1"
          host: "{{ inventory_hostname }}"
          aborted_at_round: "{{ memattr_survey_round }}"
          target_rounds: "{{ memattr_survey_rounds }}"
          reason: "{{ memattr_survey_abort_reason | trim }}"
          failed_task: "{{ ansible_failed_task.name | default('unknown') }}"
      delegate_to: localhost
      become: false
      changed_when: false
      when: memattr_survey_write_artifacts | bool
    - name: "Round {{ memattr_survey_round }} — announce the abort"
      ansible.builtin.debug:
        msg: >-
          *** SURVEY ABORTED ON {{ inventory_hostname }} at round
          {{ memattr_survey_round }}/{{ memattr_survey_rounds }}:
          {{ memattr_survey_abort_reason }}
          *** The other nodes continue. This node's arm is SHORT — the analysis will
          say so, and its p(bad) interval will be correspondingly wider.
@@ -0,0 +1,40 @@
 ---
 # The loop itself, kept separate from round.yml so the loop bounds (which encode the
 # RESUME logic) are readable in one place.
 #
 # RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a
 # re-run after an interruption continues where it stopped and can never lose or duplicate
 # an earlier round. Already at or past the target count => nothing to do, and the run
 # collapses to a harvest.
 #
 # NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which
 # makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest,
 # analysis) still runs, so a dry run exercises the entire pipeline without touching a
 # single machine.
 - name: Announce the plan
  ansible.builtin.debug:
    msg: >-
      {{ inventory_hostname }}:
      {%- if not (memattr_survey_reboots_authorised | bool) %}
      DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm).
      {%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %}
      already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do.
      {%- else %}
      rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }}
      to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots).
      {%- endif %}
 - name: Survey round
  ansible.builtin.include_tasks: round.yml
  loop: >-
    {{ range(memattr_survey_status.completed_rounds | int + 1,
             memattr_survey_rounds | int + 1) | list }}
  loop_control:
    loop_var: memattr_survey_round
    label: "{{ inventory_hostname }} round {{ memattr_survey_round }}"
  # Evaluated per iteration, so the abort flag set by a rescue in one round skips every
  # round after it on THIS host and no others.
  when:
    - memattr_survey_reboots_authorised | bool
    - not (memattr_survey_aborted | bool)
@@ -18,22 +18,193 @@ node_baseline_packages:
  # node from first baseline — package installation is the role's job, never a manual step
  # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
  - smartmontools
 # --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
 # OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
 # baseline — not a one-off `apt install` someone has to remember on the next node.
 #
 # WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
 # Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
 # from its contents, so an unlucky boot reserves everything above the table and the node
 # comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
 # 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
 # first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
 # no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
 # kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
 # >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
 #
 # WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
 # publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
 # backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
 # overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
 # encode the environment into the ROLE, would silently miss a fourth prod node named
 # differently, and could not be flipped for a one-off test host. So it is a plain boolean,
 # default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
 # kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
 #
 # SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
 # covered by the Debian Security Team's guarantees for the main archive — no security-tracker
 # commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
 # is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
 #
 # EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
 # 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
 # delete this block. Nothing else in the repo depends on it.
 node_baseline_backports_kernel: false
 # The suite, mirror and components the source line is built from. Split out so a mirror
 # change is a variable, not an edit to a heredoc in tasks/main.yml.
 node_baseline_backports_suite: trixie-backports
 node_baseline_backports_mirror: "http://deb.debian.org/debian"
 # Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
 # contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
 # surface beyond what the node already trusts is a separate decision nobody has taken.
 node_baseline_backports_components:
  - main
  - non-free-firmware
 # The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
 # would stop receiving even backports' own updates the moment the ABI rolls, leaving the
 # node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
 # The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
 # far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
 # means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
 # is recorded on the node itself — see node_baseline_backports_kernel_stamp.
 node_baseline_backports_kernel_packages:
  - linux-image-amd64
 # Provenance/rollback note written on the node. An operator standing in front of a prod box
 # with no repo checkout must be able to answer "why is this machine not on the distro
 # kernel, and how do I undo it?" from the machine itself.
 node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
 node_baseline_kernel_modules:
  - overlay
  - br_netfilter
  - iscsi_tcp
  - dm_crypt
  - nfs
 # --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
 # The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
 # a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
 #
 #   node_baseline_longhorn_source:
 #     device           dedicated block device at node_baseline_longhorn_device (DEV: the
 #                      tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
 #     claim-mountpoint re-purpose the partition the node ALREADY has mounted at
 #                      node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
 #                      path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
 #                      and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
 #                      is correct and a hardcoded one would eventually mkfs the wrong
 #                      partition on a future node. Destructive; heavily guarded — read
 #                      tasks/disks.yml before changing anything here.
 #
 #   node_baseline_etcd_source:
 #     device                dedicated block device at node_baseline_etcd_device (DEV:
 #                           /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
 #                           isolated from Longhorn rebuild I/O after dev was burned by HDD
 #                           contention).
 #     root-filesystem       etcd lives on the OS filesystem, with NO dedicated device and
 #                           NO claimed data partition under it. Kept as a valid, supported
 #                           shape; NOT what prod uses any more (see below).
 #     shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
 #                           CLAIMED data partition (PROD). No dedicated device, but also NOT
 #                           the OS filesystem: the whole point of the 2026-09-04 layout
 #                           revision is that /var (11.8 GiB) can hold neither the container
 #                           image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
 #
 # Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
 # recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
 # var, which still fails the `disks` assert.
 node_baseline_longhorn_source: device
 node_baseline_longhorn_device: /dev/vdb
-node_baseline_longhorn_mount: /var/lib/longhorn
+# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
-node_baseline_longhorn_fs_label: longhorn
+# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
 # tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
 # store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
 # both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
 #
 # THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
 # Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
 # /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
 # at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
 # partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
 node_baseline_data_mount: /var/lib/longhorn
 node_baseline_data_fs_label: longhorn
 # Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
 # root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
 # sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
 # in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
 # same fact, and the two are checked against each other by nothing but this comment.
 node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
 # k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
 # one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
 # NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
 # install flags, this role reads the same value to create the directory and to place etcd,
 # and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
 # value (higher precedence than any role default) is what actually resolves here, even
 # though roles/k3s_server is not in the node-baseline play; the default() keeps
 # node-baseline.yml runnable standalone.
 node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
 # Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
 # default so a dev host cannot enter the claim path even by accident.
 node_baseline_longhorn_claim_mountpoint: ""
 # --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
 # A node claimed by an EARLIER version of this role carries the OLD label at the OLD
 # mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
 # "not claimed yet" and walk into the greenfield claim path — which then fails closed at
 # "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
 # two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
 # are NOT changing, so dev has nothing to transition and cannot enter the transition path.
 node_baseline_data_retired_fs_label: ""
 node_baseline_data_retired_mount: ""
 # Sanity floor for the claim path: a data partition smaller than this is not "a small
 # store", it is the wrong partition. Prod's /srv is 415-425 GiB.
 node_baseline_longhorn_min_size_gb: 100
 # Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
 # so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
 # See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
-# device renames (matches the Longhorn LABEL= approach).
+# device renames (matches the data partition's LABEL= approach).
 node_baseline_etcd_source: device
 node_baseline_etcd_device: /dev/vdc
-node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
+# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
 # the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
 # /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
 # to /data/k3s/server/db, inside the claimed partition.
 node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
 node_baseline_etcd_fs_label: k3s-etcd
 # Mounts whose backing devices the claim path must NEVER touch. Used to build the
 # protected-device set from the node's own gathered facts (plus live swap), so a node
 # partitioned differently from the fleet is judged on its own layout rather than against
 # a hardcoded list of partition names.
 node_baseline_protected_mounts:
  - /
  - /boot
  - /boot/efi
  - /usr
  - /var
  - /home
  - /tmp
 # --- WIRELESS (tasks/main.yml, tag `wireless`) ---------------------------------------
 # A CLUSTER NODE GETS EXACTLY ONE PATH ONTO THE NETWORK. ON by default — unlike the
 # backports kernel above, which is a prod-only deviation, this is a cluster-wide policy and
 # it is a provable NO-OP where it does not apply: the tasks act only on interfaces the node
 # reports as real 802.11 devices (a `phy80211` link in sysfs), and dev's Proxmox VMs have
 # none, so on dev nothing is written, nothing is unloaded and nothing changes. Defaulting it
 # true is what makes a FOURTH node with a wifi card inherit the policy from being a cluster
 # node, rather than from someone remembering to set a flag.
 #
 # WHY IT EXISTS: k3s-prod-3 came up on both eno1 (192.168.2.12, the inventory address) and
 # wlp1s0 (192.168.2.195, DHCP on the same /24). The wifi lease's dhcpcd run rewrote
 # /etc/resolv.conf three minutes after this role wrote it, and a second address on the LAN
 # where kube-vip floats 192.168.2.9 and Cilium L2-announces .20-.30 is a problem in its own
 # right. See the `wireless` block in tasks/main.yml for the full reasoning, including why the
 # DRIVER MODULE is the layer that gets blocked rather than the link, a service, or rfkill.
 #
 # SET IT FALSE for a host that must genuinely keep wifi (there is none today). That does NOT
 # undo an already-applied block — delete the file below on the node as well.
 node_baseline_disable_wireless: true
 # Where the persistent block lives. 99- so it is read last and overrides any earlier
 # permissive rule; the name says what it does, on a node with no repo checkout to hand.
 node_baseline_wireless_modprobe_file: /etc/modprobe.d/99-k3s-no-wireless.conf
 # --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
 # These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
@@ -0,0 +1,642 @@
 ---
 # =====================================================================================
 # node_baseline — DATA DISKS.  Every task here carries the `disks` tag (applied once by
 # the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
 # Split into its own file rather than living in main.yml because it is the one part of
 # the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
 #
 # TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
 #
 #   dev  (group_vars/all.yml, role defaults)
 #     node_baseline_longhorn_source: device          -> mkfs /dev/vdb, mount LABEL=longhorn
 #                                                       at /var/lib/longhorn (== dataPath)
 #     node_baseline_etcd_source:     device          -> mkfs /dev/vdc, mount LABEL=k3s-etcd
 #     k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
 #     THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
 #     exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
 #
 #   prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
 #     node_baseline_longhorn_source: claim-mountpoint  (claim mountpoint /srv)
 #     node_baseline_data_mount:      /data            (LABEL=k3s-data)
 #     node_baseline_etcd_source:     shared-data-partition
 #     k3s_data_dir:                  /data/k3s
 #     ONE physical disk per node, already fully partitioned by the Debian installer. The
 #     ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
 #     between TWO tenants:  /data/longhorn (Longhorn replicas)  +  /data/k3s (the whole k3s
 #     data dir: etcd AND the containerd image store).
 #
 # WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
 # §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
 # BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
 # cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
 # have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
 # WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
 # partition is the fix; the partition therefore serves two tenants, and neither its
 # mountpoint nor its filesystem label may name only one of them.
 #
 # WHY /data AND LABEL k3s-data.
 #   * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
 #     Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
 #     at, and the idempotency contract is that /srv is gone from /proc/mounts and
 #     /etc/fstab once the claim has happened. Re-using it as the destination would make
 #     "claimed" and "not yet claimed" indistinguishable.
 #   * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
 #     accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
 #     `data` as an example of somebody else's named volume. Labelling ours `data` would make
 #     a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
 #     cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
 #     convention in this repo.
 #
 # WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
 # "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
 # empty string, and the second one must never silently succeed. `root-filesystem` and
 # `shared-data-partition` are explicit, greppable statements of intent that the assert
 # below accepts; an unset or placeholder device is still a hard failure. The two differ in
 # a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
 # `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
 # until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
 # than printing a claim it never checked (see the etcd section).
 #
 # WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
 # nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
 # correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
 # that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
 # arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
 # point at the wrong partition on some future node and mkfs it. So the role asks the NODE
 # which block device is mounted at the claim mountpoint, and then proves that answer is
 # safe (see the assert wall below) before touching it. The inventory declares INTENT
 # ("the partition currently mounted at /srv"); the node supplies the FACT.
 # =====================================================================================
 # Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
 # its data-disk layout. Two failure shapes are caught:
 #   * an undeclared/typo'd source enum, and
 #   * source: device with a device path that is a PLACEHOLDER rather than a real device.
 # group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
 # vars now that prod uses neither: if anyone ever flips a source back to `device` without
 # also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
 # not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
 # stays in the token list because that was the pre-decision placeholder and may still be
 # carried by other environments/branches.
 - name: Assert this environment's data-disk layout has actually been declared
  ansible.builtin.assert:
    that:
      - node_baseline_longhorn_source in ['device', 'claim-mountpoint']
      - node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
      # `shared-data-partition` is only coherent when there IS a claimed data partition to
      # share, and only when the k3s data dir actually sits INSIDE it. Catch the
      # contradiction here, in inventory terms, rather than three sections later when etcd
      # has silently landed back on the OS filesystem.
      - >-
        node_baseline_etcd_source != 'shared-data-partition'
        or node_baseline_longhorn_source == 'claim-mountpoint'
      - >-
        node_baseline_etcd_source != 'shared-data-partition'
        or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
      # The transition knobs are a matched PAIR: a retired label with no retired mountpoint
      # (or vice versa) cannot describe a state to migrate from.
      - >-
        (node_baseline_data_retired_fs_label | length > 0)
        == (node_baseline_data_retired_mount | length > 0)
      - >-
        node_baseline_longhorn_source != 'device'
        or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
      - >-
        node_baseline_etcd_source != 'device'
        or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
      - >-
        node_baseline_longhorn_source != 'claim-mountpoint'
        or (node_baseline_longhorn_claim_mountpoint | length) > 0
    fail_msg: >-
      This host's data-disk layout is not declared.
      longhorn_source={{ node_baseline_longhorn_source }}
      (device={{ node_baseline_longhorn_device }},
      claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
      etcd_source={{ node_baseline_etcd_source }}
      (device={{ node_baseline_etcd_device }}).
      Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
      `claim-mountpoint` (re-purpose the partition already mounted at
      node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
      `device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
      it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
      data partition — prod's shape since 2026-09-04, which additionally requires
      longhorn_source `claim-mountpoint` and a k3s data dir under
      node_baseline_data_mount={{ node_baseline_data_mount }};
      k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
      operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
      Declare it in the environment's group_vars and record the reasoning in
      §2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
      `--tags packages`, which skips every disk task including this assert.
    quiet: true
 # =====================================================================================
 # THE DATA PARTITION  (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
 # =====================================================================================
 # THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
 # rather than a reformat. Once the claim has happened the partition carries the ext4
 # label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
 # mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
 # mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
 # "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
 # path means converged, rc != 0 means the claim has not been made. Everything destructive
 # below is gated on rc != 0.
 - name: "Data partition: look for a filesystem already carrying the data-partition label"
  ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
  register: node_baseline_lh_labelled
  changed_when: false
  failed_when: false
  check_mode: false
 # --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
 # WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
 # label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
 # takes over — which then fails closed at "the claim mountpoint must actually be a
 # mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
 # behaviour but it is not convergence, and the prod nodes are in exactly that state today:
 # claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
 # So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
 #
 # WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
 # partition with the new label. It never touches the DATA and never reformats: e2label is a
 # metadata write, exactly as in the claim path below. The common mount task then mounts
 # LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
 # `blkid -L <new label>` succeeds and none of this is reachable.
 #
 # THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
 # volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
 # unmounting it would orphan every one of them:
 #   1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
 #      group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
 #   2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
 #      the same structural gate that keeps dev out of the claim path keeps it out of this one.
 #   3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
 #      replica directories on it fails closed and nothing is unmounted.
 # Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
 - name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
  ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
  register: node_baseline_lh_retired_labelled
  changed_when: false
  failed_when: false
  check_mode: false
  when:
    - node_baseline_longhorn_source == 'claim-mountpoint'
    - node_baseline_data_retired_fs_label | length > 0
    - node_baseline_lh_labelled.rc != 0
 - name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
  ansible.builtin.set_fact:
    # A node claimed under the RETIRED label. Mutually exclusive with the claim below.
    node_baseline_lh_transition_needed: >-
      {{ node_baseline_longhorn_source == 'claim-mountpoint'
         and node_baseline_data_retired_fs_label | length > 0
         and node_baseline_lh_labelled.rc != 0
         and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
    # True only for a claim-mountpoint environment that has NOT been claimed under EITHER
    # label. A `device` environment (dev) can never enter the claim path at all — that is
    # the structural guarantee that no generalisation of prod's behaviour can reach the
    # LIVE dev nodes and their /dev/vdb Longhorn replicas.
    node_baseline_lh_claim_needed: >-
      {{ node_baseline_longhorn_source == 'claim-mountpoint'
         and node_baseline_lh_labelled.rc != 0
         and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
 - name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
  ansible.builtin.command: >-
    findmnt --noheadings --first-only --output SOURCE
    --mountpoint {{ node_baseline_data_retired_mount }}
  register: node_baseline_lh_retired_findmnt
  changed_when: false
  failed_when: false
  check_mode: false
  when: node_baseline_lh_transition_needed | bool
 # Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
 # not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
 # an unmount of a populated Longhorn store would orphan every replica on the node.
 - name: "Data partition (transition): read the contents of the retired mountpoint"
  ansible.builtin.find:
    paths: "{{ node_baseline_data_retired_mount }}"
    file_type: any
    hidden: true
    recurse: false
    excludes: ['lost+found']
  register: node_baseline_lh_retired_contents
  when: node_baseline_lh_transition_needed | bool
 - name: "Data partition (transition): prove this node is safe to re-label"
  ansible.builtin.assert:
    that:
      # 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
      #    retired label names. The only state this role knows how to transition is the one
      #    the previous version of it produced; anything else is an operator decision.
      - (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
      - >-
        (node_baseline_lh_retired_findmnt.stdout | default('') | trim)
        == (node_baseline_lh_retired_labelled.stdout | default('') | trim)
      # 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
      #    not the node we think it is, or a tenant has already written data — in both cases
      #    unmounting and re-labelling under it is destructive and the role must stop.
      - (node_baseline_lh_retired_contents.files | length) == 0
    fail_msg: >-
      REFUSING to transition this node's data partition from
      LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
      device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
      device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
      (findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
      non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
      {{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
      If the partition holds DATA, this transition would orphan it — Longhorn replicas are
      not recoverable from an unmounted, re-labelled filesystem. Migrate the data
      deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
      Do NOT relax this assert.
    quiet: true
  when: node_baseline_lh_transition_needed | bool
 - name: "Data partition (transition): report exactly what is about to change"
  ansible.builtin.debug:
    msg: >-
      Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
      from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
      unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
      No filesystem is created and no data is moved.
  when: node_baseline_lh_transition_needed | bool
 # e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
 # breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
 - name: "Data partition (transition): unmount the retired mountpoint"
  ansible.posix.mount:
    path: "{{ node_baseline_data_retired_mount }}"
    state: unmounted
  when: node_baseline_lh_transition_needed | bool
 - name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
  ansible.posix.mount:
    path: "{{ node_baseline_data_retired_mount }}"
    state: absent_from_fstab
    backup: true
  when: node_baseline_lh_transition_needed | bool
 - name: "Data partition (transition): re-stamp the partition with the new filesystem label"
  ansible.builtin.command: >-
    e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
    {{ node_baseline_data_fs_label }}
  changed_when: true
  when: node_baseline_lh_transition_needed | bool
 # --- The claim (prod, first run only) -------------------------------------------------
 # Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
 # mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
 # returning the device behind `/` — which is precisely the mistake that would hand the
 # root partition to e2label.
 - name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
  ansible.builtin.command: >-
    findmnt --noheadings --first-only --output SOURCE
    --mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
  register: node_baseline_lh_claim_findmnt
  changed_when: false
  failed_when: false
  check_mode: false
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
  ansible.builtin.assert:
    that:
      - node_baseline_lh_claim_findmnt.rc == 0
      - (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
    fail_msg: >-
      node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
      yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
      this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
      stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
      The role will NOT go looking for "a big partition" to claim instead. Either the
      node is partitioned differently from the fleet — in which case fix the node or give
      this host its own declaration — or the claim already happened and the label was
      removed by hand.
    quiet: true
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): inspect the candidate partition"
  ansible.builtin.command: >-
    lsblk --json --nodeps --bytes
    --output PATH,TYPE,FSTYPE,LABEL,SIZE
    {{ node_baseline_lh_claim_findmnt.stdout | trim }}
  register: node_baseline_lh_claim_lsblk
  changed_when: false
  check_mode: false
  when: node_baseline_lh_claim_needed | bool
 # The devices this role must never, under any circumstances, hand to e2label. Built from
 # gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
 # NODE's account of what is load-bearing, not a hardcoded list of partition names.
 - name: "Longhorn (claim): list the devices currently backing active swap"
  ansible.builtin.command: swapon --noheadings --show=NAME
  register: node_baseline_lh_swapdevs
  changed_when: false
  failed_when: false
  check_mode: false
  when: node_baseline_lh_claim_needed | bool
 # Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
 # ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
 - name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
  ansible.builtin.find:
    paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
    file_type: any
    hidden: true
    recurse: false
    excludes: ['lost+found']
  register: node_baseline_lh_claim_contents
  when: node_baseline_lh_claim_needed | bool
 # THE GUARD WALL. Five independent conditions, every one of which must hold before the
 # role modifies a filesystem it did not create. Any single failure stops the play with
 # the evidence in the message — nothing is unmounted, relabelled or mounted first.
 - name: "Longhorn (claim): prove the candidate partition is safe to claim"
  ansible.builtin.assert:
    that:
      # 1. It is a real block device — a partition or a whole disk. Catches a claim
      #    mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
      #    which would have produced a non-/dev SOURCE or an lsblk row of another type.
      - node_baseline_lh_candidate.type in ['part', 'disk']
      # 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
      #    so a differently-partitioned node is caught by its own facts.
      - node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
      # 3. It already holds ext4. The claim path RELABELS rather than reformats (see
      #    below), which is only meaningful for the filesystem type we are about to
      #    declare in fstab. Anything else is an operator decision, not a role's.
      - node_baseline_lh_candidate.fstype == 'ext4'
      # 4. It carries no filesystem label, or already carries ours. A partition labelled
      #    `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
      #    it would make it unfindable by its own fstab entry.
      - (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
      # 5. It is big enough to be the partition we meant. A data partition of a few GB is
      #    not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
      - (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
      # 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
      #    ext4 5% root reserve; this is where that premise is verified per node instead
      #    of trusted. If any node has real data on it, the play stops here — the role
      #    never wipes data it cannot account for.
      - (node_baseline_lh_claim_contents.files | length) == 0
    fail_msg: >-
      REFUSING to claim {{ node_baseline_lh_candidate.path }}
      (mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
      type={{ node_baseline_lh_candidate.type }},
      fstype={{ node_baseline_lh_candidate.fstype }},
      label='{{ node_baseline_lh_candidate.label | default('', true) }}',
      size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
      (minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
      non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
      {{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
      devices this node needs and that must never be claimed:
      {{ node_baseline_lh_protected_devices }}.
      This is the destructive step of the role and it fails CLOSED. If the partition
      holds data, back it up or pick a different one; do NOT relax this assert.
    quiet: true
  vars:
    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
    # The node's OWN account of what it cannot afford to lose: the devices behind the
    # mounts in node_baseline_protected_mounts (from gathered facts, so a node
    # partitioned differently is judged on its own layout) plus every active swap device.
    # Built here rather than hardcoded so no partition-name list can go stale.
    node_baseline_lh_protected_devices: >-
      {{ (ansible_facts.mounts | default([])
          | selectattr('mount', 'in', node_baseline_protected_mounts)
          | map(attribute='device') | list)
         + (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): report exactly what is about to be claimed"
  ansible.builtin.debug:
    msg: >-
      Claiming {{ node_baseline_lh_candidate.path }}
      ({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
      currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
      as the data partition: unmount -> drop its fstab entry ->
      relabel LABEL={{ node_baseline_data_fs_label }} ->
      mount at {{ node_baseline_data_mount }}.
  vars:
    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
  when: node_baseline_lh_claim_needed | bool
 # --- Why the role must unmount and relabel, not just point fstab at the device ---------
 # The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
 # /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
 # path does not. The installer-created /srv filesystem has NO label (fstab pins it by
 # UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
 # the unmount first — and the fstab entry must go in the same breath, or the next boot
 # re-mounts the same partition at /srv underneath Longhorn.
 #
 # RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
 # mkfs would buy nothing and would destroy anything the emptiness check got wrong.
 # e2label is a metadata write: if some assert above was too permissive, the worst case is
 # a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
 # cannot do this job: it declines to touch a device that already has a filesystem, so it
 # would silently no-op and then the LABEL= mount would fail with no useful message.)
 - name: "Longhorn (claim): unmount the claim mountpoint"
  ansible.posix.mount:
    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
    state: unmounted
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
  ansible.posix.mount:
    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
    state: absent_from_fstab
    backup: true
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
  ansible.builtin.command: >-
    e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
    {{ node_baseline_data_fs_label }}
  changed_when: true
  when: node_baseline_lh_claim_needed | bool
 # --- The dedicated-device path (dev) --------------------------------------------------
 # Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
 # never reformats a device that already carries a filesystem.
 - name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
  community.general.filesystem:
    fstype: ext4
    dev: "{{ node_baseline_longhorn_device }}"
    opts: "-L {{ node_baseline_data_fs_label }}"
  when: node_baseline_longhorn_source == 'device'
 # --- Common to ALL paths (device, claim, transition) ----------------------------------
 # Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
 # mount task. Idempotent: on a converged node this reports ok, which is what makes a
 # re-run a no-op rather than a reformat. It is also what completes the transition above:
 # the retired mountpoint is gone from fstab and the partition now answers to the new label,
 # so this task mounts it at the new path.
 - name: Mount the data partition
  ansible.posix.mount:
    path: "{{ node_baseline_data_mount }}"
    src: "LABEL={{ node_baseline_data_fs_label }}"
    fstype: ext4
    opts: "defaults,nofail,x-systemd.device-timeout=5s"
    state: mounted
 # --- The per-tenant subdirectories ---------------------------------------------------
 # WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
 # tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
 # subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
 # /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
 # either there would at best be a no-op and at worst re-mode the running dev cluster's data
 # dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
 # mounted — i.e. only the directories it is actually responsible for. That test is derived
 # from the paths themselves, so no environment has to remember to switch it off.
 #
 # MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
 # the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
 # etcd database, the node token and the containerd image store, and k3s would tighten it to
 # 0700 itself on first start — doing it here means it is never briefly world-readable.
 - name: Create the per-tenant data directories on the data partition
  ansible.builtin.file:
    path: "{{ item.path }}"
    state: directory
    owner: root
    group: root
    mode: "{{ item.mode }}"
  loop:
    - { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
    - { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
  loop_control:
    label: "{{ item.path }} ({{ item.tenant }})"
  when: item.path.startswith(node_baseline_data_mount ~ '/')
 # =====================================================================================
 # ETCD DATA DIR
 # =====================================================================================
 # --- The dedicated-device path (dev) --------------------------------------------------
 # Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
 # mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
 # etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
 # a disk that already carries a filesystem, so re-running against an already-migrated
 # node leaves the etcd data intact. On EXISTING nodes the one-time data migration
 # (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
 # role would mount over a populated directory.
 - name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
  community.general.filesystem:
    fstype: ext4
    dev: "{{ node_baseline_etcd_device }}"
    opts: "-L {{ node_baseline_etcd_fs_label }}"
  when: node_baseline_etcd_source == 'device'
 - name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
  ansible.posix.mount:
    path: "{{ node_baseline_etcd_mount }}"
    src: "LABEL={{ node_baseline_etcd_fs_label }}"
    fstype: ext4
    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
    opts: "defaults,noatime"
    dump: "0"
    passno: "2"
    state: mounted
  when: node_baseline_etcd_source == 'device'
 # --- The no-dedicated-device paths (prod) ---------------------------------------------
 # Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
 # populated — so etcd gets no device of its own. That is an operator decision
 # (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
 # UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
 # on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
 # latency rather than collapsing it. Accepted to be MEASURED, not assumed.
 #
 # WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
 # filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
 # 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
 # 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
 # could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
 # partition and etcd moved with it: `shared-data-partition`. Both enum values are still
 # supported, and the role now PROVES which one is true instead of printing a claim.
 #
 # There is nothing to CREATE here for either shape — the data-partition section above
 # already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
 # role's job is to verify the assumption rather than act on it.
 - name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
  ansible.builtin.command: >-
    findmnt --noheadings --first-only --output SOURCE
    --mountpoint {{ node_baseline_etcd_mount }}
  register: node_baseline_etcd_existing_mount
  changed_when: false
  failed_when: false
  check_mode: false
  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
 - name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
  ansible.builtin.assert:
    that:
      - node_baseline_etcd_existing_mount.rc != 0
    fail_msg: >-
      node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
      NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
      ALREADY a separate mount, backed by
      {{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
      contradiction the role will not resolve on its own: either this node was previously
      built with a dedicated etcd device (in which case unmount it deliberately, after
      accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
      the node in a state where the declared layout and the real one disagree.
    quiet: true
  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
 # Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
 # WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
 # existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
 # that does exist — the same filesystem the data dir will be created on. TARGET is captured
 # separately from the human-readable line because the assert below judges on it.
 - name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
  ansible.builtin.shell: |
    set -eu
    p={{ node_baseline_etcd_mount | quote }}
    while [ ! -d "$p" ]; do p=$(dirname "$p"); done
    findmnt --noheadings --first-only --output TARGET --target "$p"
    findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
  register: node_baseline_etcd_backing_fs
  changed_when: false
  check_mode: false
  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
 # THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
 # `shared-data-partition` is a distinct enum value rather than a comment. Declaring
 # k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
 # above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
 # of container images would land back on the OS filesystem — the exact failure this change
 # exists to prevent, and completely invisible until the disk filled. So the role checks that
 # the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
 # run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
 # which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
 # whole record for that shape.
 - name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
  ansible.builtin.assert:
    that:
      - >-
        (node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
        == node_baseline_data_mount
    fail_msg: >-
      node_baseline_etcd_source is `shared-data-partition`, which asserts that
      {{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
      {{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
      '{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
      (full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
      The data partition is therefore NOT mounted where this layout requires, and k3s would
      put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
      filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
      image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
      this node until the mount is correct.
    quiet: true
  when: node_baseline_etcd_source == 'shared-data-partition'
 - name: "etcd (no dedicated device): show which filesystem etcd will live on"
  ansible.builtin.debug:
    msg: >-
      etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
      (source={{ node_baseline_etcd_source }}); it lands on
      {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
      (SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
      measured risk — §2.
  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
@@ -1,6 +1,6 @@
 ---
-# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
+# TAGS — every task here carries one of `packages`, `kernel`, `system`, `wireless`, `disks` or
-# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
+# `dns`. An untagged run is unchanged (all six run, in this order). The split exists because the PACKAGE floor
 # and the DATA-DISK work have different readiness: packages are safe on any node from
 # first contact, while the disk tasks need block devices that a new environment may not
 # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -12,6 +12,16 @@
 # mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
 # at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
 #   ansible-playbook playbooks/node-baseline.yml --tags dns
 # `kernel` (the backports-kernel block below) is separated for the same reason `packages`
 # is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
 # it must be runnable without touching a disk decision that is still open:
 #   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
 # `wireless` (a cluster node gets exactly ONE path onto the network) is separated for a third
 # reason: it must be runnable BEFORE `dns` and without touching packages, the kernel or a disk
 # decision, because a node with a second DHCP-managed interface reverts the `dns` write. It is
 # placed before the disk work for the same reason, and the pair is the whole remedy for the
 # k3s-prod-3 incident:
 #   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags wireless,dns
 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
  ansible.builtin.apt:
@@ -21,6 +31,237 @@
    cache_valid_time: 3600
  tags: [packages]
 # =====================================================================================
 # BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
 # =====================================================================================
 # A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
 # FUTURE prod node gets it from provisioning instead of from someone remembering.
 #
 # WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
 # Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
 # `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
 # signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
 # through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
 # address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
 # Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
 # (first released in v6.13) validates the size and rejects the table instead. It is absent
 # from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
 # stable feed will ever pick it up. There is NO kernel cmdline workaround —
 # `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
 # take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
 # same lottery — node 1 and node 3 have merely drawn safe tickets so far.
 #
 # WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
 # defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
 # Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
 # LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
 # change for no benefit. See defaults/main.yml for why this is a variable rather than a
 # `k3s-prod-*` hostname test.
 #
 # IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
 # deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
 # off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
 # therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
 # before touching the next, never node 2 first — and a role cannot enforce that. The last
 # task here reports the pending reboot loudly instead of letting a green play imply the node
 # is already fixed.
 - name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
  when: node_baseline_backports_kernel | bool
  tags: [kernel]
  block:
    # The suite name and the running release must agree, or apt resolves no candidate and
    # `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
    # a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
    # dist-upgraded to forky while group_vars still says trixie-backports).
    - name: Assert the configured backports suite matches this node's Debian release
      ansible.builtin.assert:
        that:
          - ansible_facts['distribution'] == 'Debian'
          - node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
        fail_msg: >-
          node_baseline_backports_kernel is enabled but node_baseline_backports_suite
          ({{ node_baseline_backports_suite }}) is not the backports suite for this node's
          release ({{ ansible_facts['distribution'] }}
          {{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
          candidate and this play would report success while leaving the node on the
          vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
        quiet: true
    # deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
    # prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
    # rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
    # idempotent in a way `apt_repository` — which rewrites and reformats — is not.
    #
    # THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
    # backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
    # apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
    # 500 for a package that was explicitly installed from there so it keeps getting updates.
    # Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
    # HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
    # be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
    - name: Enable the Debian backports apt source (deb822)
      ansible.builtin.copy:
        dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
        owner: root
        group: root
        mode: "0644"
        content: |
          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
          # A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
          # machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
          # Table its firmware publishes (upstream 8fbe4c49c0cc). See
          # infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
          # on this node.
          #
          # Backports is NotAutomatic, so nothing is taken from here unless asked for by name
          # with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
          # ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
          # Pin-Priority would switch that off.
          Types: deb
          URIs: {{ node_baseline_backports_mirror }}
          Suites: {{ node_baseline_backports_suite }}
          Components: {{ node_baseline_backports_components | join(' ') }}
          Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
      register: node_baseline_backports_source
    - name: Refresh the apt cache after enabling backports
      ansible.builtin.apt:
        update_cache: true
      when: node_baseline_backports_source is changed
    # `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
    # nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
    # nothing at all — the single most likely way to write this task and have it silently fix
    # nothing forever. `latest` + `default_release` resolves the candidate inside backports
    # and upgrades the metapackage to it. Once the node is current a re-run is a genuine
    # no-op: no change, no churn, no spurious reboot flag. Scope is exactly
    # node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
    # is dragged out of the main archive.
    - name: Install the fixed kernel from backports (>= 6.13)
      ansible.builtin.apt:
        name: "{{ node_baseline_backports_kernel_packages }}"
        state: latest
        default_release: "{{ node_baseline_backports_suite }}"
        update_cache: true
        cache_valid_time: 3600
      register: node_baseline_backports_kernel_apt
    # WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
    # metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
    # answer anybody can roll back from six months later.
    - name: Read back the installed backports kernel metapackage version
      ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
      register: node_baseline_backports_kernel_version
      changed_when: false
      check_mode: false
    # `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
    # neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
    # list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
    # GRUB's "Advanced options" without uninstalling anything.
    - name: List every kernel image installed on this node, newest first
      ansible.builtin.shell:
        cmd: set -o pipefail; linux-version list | linux-version sort --reverse
        executable: /bin/bash
      register: node_baseline_installed_kernels
      changed_when: false
      check_mode: false
    # The node must be able to explain itself to an operator standing in front of it with no
    # repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
    # role is a real no-op rather than a permanent daily "changed".
    - name: Record the deviation, the exact version, and the rollback path on the node
      ansible.builtin.copy:
        dest: "{{ node_baseline_backports_kernel_stamp }}"
        owner: root
        group: root
        mode: "0644"
        content: |
          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
          #
          # THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
          #
          # WHY
          #   This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
          #   Linux < 6.13 reserves memory from it without validating its size, and an unlucky
          #   boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
          #   k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
          #   is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
          #   v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
          #   Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
          #
          # PROOF THE FIX IS ACTIVE (not merely a lucky boot)
          #   BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
          #   >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
          #     zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
          #   (roles/node_health does the version check for you; see the health report's
          #   firmware.efi_memattr finding.)
          #
          #   FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
          #   and each prints a DIFFERENT line — all three are rejections:
          #     tbl->version > 2         'Unexpected EFI Memory Attributes table version %d'
          #                              (NOTE: printed with %d, so garbage may read NEGATIVE)
          #     bad desc_size            'Unexpected EFI Memory Attributes descriptor size ...'
          #     num_entries > 64K        '[Firmware Bug]: Corrupted EFI Memory Attributes
          #                               Table detected! (version == ..., num_entries == ...)'
          #   Which one appears depends on WHICH field this boot's garbage corrupted, so do
          #   NOT treat the absence of the FW_BUG line as the fix being absent — all three
          #   prod nodes printed the *version* line on 2026-09-04.
          #   The one line that means the fix is NOT active:
          #     'Failed to map EFI Memory Attributes table @ 0x...'  — the kernel trusted the
          #     corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
          #     escalate, do not shrug.
          #
          # INSTALLED
          #   source   /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
          #   package  {{ node_baseline_backports_kernel_packages | join(' ') }}
          #   version  {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
          #   kernels installed on this node, newest first:
          {% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
          #     {{ kv }}
          {% endfor %}
          #
          # SECURITY COST
          #   Packages from *-backports are provided AS IS. They are NOT covered by the Debian
          #   Security Team's guarantees for the main archive: no security-tracker commitment,
          #   no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
          #
          # ROLLBACK — no repo checkout needed
          #   1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
          #      and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
          #      GNU/Linux". That answers "is the new kernel the problem?" without
          #      uninstalling anything. Note it reinstates the memattr bug for that boot.
          #   2. PERMANENT. Set node_baseline_backports_kernel: false in
          #      inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
          #      puts the backports kernel straight back — then, on this node:
          #        apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
          #        apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
          #        rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
          #        apt-get update && update-grub && reboot
          #
          # EXIT CONDITION
          #   If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
          #   rebase), this deviation can be removed entirely: flip the variable to false,
          #   roll back as above, and delete the `kernel` block from roles/node_baseline.
    # Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
    # it means the fixed kernel is on disk. See the block header for why the role will not
    # reboot on its own.
    - name: Report whether this node is running the kernel it now has installed
      ansible.builtin.debug:
        msg: >-
          {{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
              ~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
              prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
              node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
              with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
             if newest != ansible_facts['kernel']
             else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
                   ~ '); no reboot pending for the memattr fix.') }}
      vars:
        newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
 - name: Persist kernel modules for k3s/Cilium/Longhorn
  ansible.builtin.copy:
    dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
@@ -79,70 +320,330 @@
    state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
  tags: [system]
-# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
+# =====================================================================================
-# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
+# WIRELESS — tag `wireless`: a cluster node gets exactly ONE path onto the network
-# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
+# =====================================================================================
-# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
+# WHAT WENT WRONG (k3s-prod-3, reported by the operator 2026-09-04). The node was up on BOTH
-# which do not exist on bare metal either — a confusing failure that looks like a wiring
+# eno1 (192.168.2.12 — the inventory address) and wlp1s0 (192.168.2.195, DHCP, the SAME /24).
-# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
+# The wifi lease carried a dhcpcd run with it, and dhcpcd's `20-resolv.conf` hook REWROTE
-# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
+# /etc/resolv.conf, undoing the resolvers this role had just written: the file read
- name: Assert this environment's data-disk devices have actually been decided
+# "# Generated by dhcpcd from wlp1s0.dhcp, wlp1s0.dhcp6, wlp1s0.ra" with nameserver
 # 192.168.2.1 plus two Google IPv6 servers, timestamped THREE MINUTES AFTER the role's own
 # write (01:32 -> 01:35). The role had reported `changed` and green. That is exactly the
 # "successful run that fixed nothing" the `contended` branch of the DNS block below exists to
 # prevent, and the detection blind spot that let it through is closed there.
 #
 # WHY THE INTERFACE IS REMOVED RATHER THAN JUST REPORTED. Closing the detection hole alone
 # would have turned a silent revert into a loud failure — an improvement, but it leaves the
 # node with a second, DHCP-managed path onto the LAN, and on a k3s server that is three
 # distinct problems beyond DNS:
 #   * a SECOND DEFAULT ROUTE, whose metric decides which source address the node uses;
 #   * a SECOND ADDRESS ON THE SAME L2 — the segment where kube-vip floats 192.168.2.9 and
 #     Cilium L2-announces 192.168.2.20-.30 (§4). One host answering ARP on one subnet from
 #     two NICs is the exact shape neither mechanism can reason about;
 #   * an UNMANAGED address: nothing in this repo knows about .195, and the DHCP server is
 #     free to hand out a different one after any lease expiry.
 # None of that is a resolver bug, and none of it is fixed by writing resolv.conf harder.
 #
 # WHY THE DRIVER LAYER — the four alternatives, and why each is weaker. What has to hold is:
 # survives a REBOOT, survives a LEASE RENEWAL, and cannot be undone by a routine `ifup`.
 # Measured on the nodes themselves — all three are HP EliteDesk 800 G5 minis, eno1 is
 # `e1000e`, the wifi NIC is an Intel AX200 on `iwlwifi`, and the OS is Debian 13 trixie with
 # ifupdown + `networking.service`, NO NetworkManager, NO systemd-networkd, NO systemd-resolved:
 #   `ip link set <wifi> down` — NOT persistent. Any `ifup`, any hotplug event, any reboot
 #       undoes it. It is what had already been done by hand on node 3, and the stale dhcpcd
 #       /etc/resolv.conf outlived it — the link was down and the wrong resolvers were still
 #       in the file, across a reboot.
 #   mask the service that manages it — THERE IS NO SUCH SERVICE HERE. dhcpcd on these nodes
 #       is `dhcpcd-base`: no dhcpcd.service exists at all (`systemctl is-enabled dhcpcd` =
 #       not-found), it is spawned per-interface by ifupdown. The only unit that can bring
 #       wireless up is `networking.service` — which ALSO owns eno1's static address, so
 #       masking it would cut the one SSH path to a machine whose operator is not on site.
 #   `rfkill` soft-block — weaker on every axis. The netdev still EXISTS, so ifupdown and
 #       dhcpcd can still be pointed at it; persistence depends on systemd-rfkill restoring
 #       saved state (the kernel's own `persistent` flag reads 0 on these devices, and the
 #       `rfkill` CLI is not even installed); and one `rfkill unblock wlan` undoes it.
 #   the BIOS `M.2 WLAN/BT` toggle — genuinely the deepest layer, and it reads `Enable` on all
 #       three nodes (hp-bioscfg survey, §1b). NOT taken here: it needs a REBOOT to take
 #       effect, and a prod reboot is a serialized, supervised operator step (§0b "SECOND
 #       FAULT" — k3s-prod-2 hung on a warm reboot and needed the power button). It also
 #       disables Bluetooth on the same M.2 card. Left as an operator recommendation in §4.
 #   THE DRIVER MODULE — chosen. `blacklist` plus `install <module> /bin/false` in
 #       /etc/modprobe.d means kmod refuses to load the wifi driver on this boot and every
 #       future one, so THE NETDEV NEVER EXISTS. No interface means no DHCP lease, no
 #       resolv.conf hook, no second route, no second address — and every re-activation path
 #       above (udev hotplug, a hand-run `ifup`, wpa_supplicant, a hand-run dhcpcd) needs a
 #       device that is not there. `blacklist` alone would still allow a dependency- or
 #       hand-triggered `modprobe`; the `install ... /bin/false` line is what makes it refuse
 #       outright. It is ONE declarative file, readable on the node, and undone by deleting
 #       it. Paired with the runtime unload below, it needs NO reboot to take effect.
 #
 # SAFE AND IDEMPOTENT ON ALL THREE PROD NODES *AND* ON THE LIVE DEV CLUSTER. Nothing here is
 # hostname- or hardware-specific: the block acts only on interfaces the node itself reports as
 # real 802.11 devices (a `phy80211` link in sysfs) and on the driver modules behind them.
 #   * dev (Proxmox VMs, virtio NICs) has no such interface, so the discovery comes back empty,
 #     every task below is skipped, and NOT EVEN THE modprobe.d FILE IS WRITTEN. The block
 #     cannot touch dev's networking, which is why it is safe to have on by default.
 #   * k3s-prod-1 (wlp1s0) and k3s-prod-2 (wlp2s0) already have the link DOWN, so the link-down
 #     task is skipped by its own `when` and only the modprobe.d file and the module unload
 #     remain — and both are no-ops on a re-run.
 #   * k3s-prod-3 (wlp1s0) is the reported node.
 # A re-run changes nothing on any of them: the file is content-identical, `ip link set down`
 # is gated on the link not already being down, and the unload is gated on the module still
 # being loaded.
 #
 # AND IT REFUSES TO CUT ITS OWN PATH IN. The assert below stops the play if the address
 # Ansible is connected to is bound to a wireless interface, or if the default route leaves
 # over one, or if the wireless driver module is also the driver of the default-route
 # interface. That is the one case where doing this would lock everybody out of a machine
 # nobody can walk up to, so it is checked per node, on every run, from that node's own facts.
 - name: Survey this node's 802.11 devices, their drivers and any live DHCP client
  ansible.builtin.shell:
    executable: /bin/sh
    cmd: |
      set -u
      # Flat KEY|value lines so the Ansible side needs one single-group regex per fact and
      # no parsing cleverness. A wireless interface is one with a phy80211 link in sysfs —
      # that is the kernel's own answer to "is this 802.11", not a name pattern like wlp*.
      wifi=''
      for d in /sys/class/net/*; do
        [ -e "$d/phy80211" ] || continue
        i=$(basename "$d")
        wifi="$wifi $i"
        m=$(basename "$(readlink -f "$d/device/driver" 2>/dev/null || echo unknown)")
        state=$(cat "$d/operstate" 2>/dev/null || echo unknown)
        printf 'IFACE|%s\n' "$i"
        printf 'DETAIL|%s driver=%s operstate=%s\n' "$i" "$m" "$state"
        # LIVE means "not already down" — the link-down task keys off this, so a node whose
        # radio is already down reports no change.
        [ "$state" = down ] || printf 'IFACE_LIVE|%s\n' "$i"
        ip -o addr show dev "$i" 2>/dev/null | awk '{print "IFACE_ADDR|" $4}'
        [ "$m" = unknown ] || printf 'MODULE|%s\n' "$m"
      done
      for m in $(printf '%s\n' $wifi | while read -r i; do
            [ -n "$i" ] && basename "$(readlink -f "/sys/class/net/$i/device/driver" 2>/dev/null || echo unknown)"
          done | sort -u); do
        [ "$m" = unknown ] && continue
        [ -d "/sys/module/$m" ] || continue
        printf 'MODULE_LOADED|%s\n' "$m"
        # Holders must be unloaded FIRST: `modprobe -r iwlwifi` fails while iwlmvm still
        # holds it. Read from sysfs rather than assumed, so a different wifi driver stack
        # (ath*, rtw*, mt76*) works without editing this role.
        for h in $(ls "/sys/module/$m/holders" 2>/dev/null); do printf 'MODULE_HOLDER|%s\n' "$h"; done
      done
      dr=$(ip -o route show default 2>/dev/null | awk '{for (i = 1; i < NF; i++) if ($i == "dev") print $(i + 1)}' | head -1)
      printf 'DEFROUTE_IFACE|%s\n' "${dr:-none}"
      if [ -n "${dr:-}" ] && [ -e "/sys/class/net/$dr" ]; then
        printf 'DEFROUTE_MODULE|%s\n' "$(basename "$(readlink -f "/sys/class/net/$dr/device/driver" 2>/dev/null || echo unknown)")"
      fi
      # Any DHCP client process, with the interfaces it was started for. Used only to stop a
      # client that holds a WIRELESS interface and nothing else.
      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
        printf 'DHCPCLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
      done
      exit 0
  register: node_baseline_wifi_scan
  changed_when: false
  check_mode: false
  when: node_baseline_disable_wireless | bool
  tags: [wireless]
 - name: Interpret the wireless survey
  ansible.builtin.set_fact:
    node_baseline_wifi_ifaces: "{{ scan | regex_findall('(?m)^IFACE\\|(.+)$') }}"
    node_baseline_wifi_live: "{{ scan | regex_findall('(?m)^IFACE_LIVE\\|(.+)$') }}"
    node_baseline_wifi_addrs: "{{ scan | regex_findall('(?m)^IFACE_ADDR\\|(\\S+)') | map('regex_replace', '/.*$', '') | list }}"
    node_baseline_wifi_modules: "{{ scan | regex_findall('(?m)^MODULE\\|(.+)$') | unique }}"
    node_baseline_wifi_modules_loaded: "{{ scan | regex_findall('(?m)^MODULE_LOADED\\|(.+)$') | unique }}"
    node_baseline_wifi_module_holders: "{{ scan | regex_findall('(?m)^MODULE_HOLDER\\|(.+)$') | unique }}"
    node_baseline_wifi_detail: "{{ scan | regex_findall('(?m)^DETAIL\\|(.+)$') }}"
    node_baseline_wifi_dhcp_clients: "{{ scan | regex_findall('(?m)^DHCPCLIENT\\|(.+)$') }}"
    node_baseline_defroute_iface: "{{ (scan | regex_findall('(?m)^DEFROUTE_IFACE\\|(.+)$') | first) | default('none', true) }}"
    node_baseline_defroute_module: "{{ (scan | regex_findall('(?m)^DEFROUTE_MODULE\\|(.+)$') | first) | default('unknown', true) }}"
  vars:
    scan: "{{ node_baseline_wifi_scan.stdout | default('') }}"
  when: node_baseline_disable_wireless | bool
  tags: [wireless]
 - name: Report this node's wireless hardware
  ansible.builtin.debug:
    msg: >-
      wireless devices = {{ node_baseline_wifi_detail | default([]) | join('; ') | default('(none)', true) }}
      | driver modules = {{ node_baseline_wifi_modules | default([]) | join(', ') | default('(none)', true) }}
      (loaded: {{ node_baseline_wifi_modules_loaded | default([]) | join(', ') | default('(none)', true) }},
      held by: {{ node_baseline_wifi_module_holders | default([]) | join(', ') | default('(none)', true) }})
      | default route via {{ node_baseline_defroute_iface | default('?') }}
      (driver {{ node_baseline_defroute_module | default('?') }})
  when: node_baseline_disable_wireless | bool
  tags: [wireless]
 # The one way this block could do real harm is by removing the interface it is being run
 # over. Checked from the node's own facts, per node, every run — not assumed from the fact
 # that prod "should" be wired.
 - name: Refuse to disable wireless if the path to this node runs OVER the wireless interface
  ansible.builtin.assert:
    that:
-      - "'BLOCKED' not in node_baseline_longhorn_device"
+      - node_baseline_defroute_iface not in node_baseline_wifi_ifaces
-      - "'BLOCKED' not in node_baseline_etcd_device"
+      - node_baseline_defroute_module not in node_baseline_wifi_modules
      - (ansible_host | default(inventory_hostname)) not in node_baseline_wifi_addrs
    fail_msg: >-
-      node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
+      REFUSING to disable wireless on this node: the connection to it appears to BE the
-      placeholders for this host's environment
+      wireless path. ansible_host={{ ansible_host | default(inventory_hostname) }}, wireless
-      (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
+      interfaces={{ node_baseline_wifi_ifaces | join(', ') }} carrying
-      The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
+      {{ node_baseline_wifi_addrs | join(', ') | default('no addresses', true) }}, default route
-      and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
+      via {{ node_baseline_defroute_iface }} (driver {{ node_baseline_defroute_module }}).
-      the SMART read that decision waits on can be taken) re-run this play with
+      Disabling the driver here would drop this node off the network with no remote way back.
-      `--tags packages`, which skips every disk task including this assert.
+      Give the node a wired address, point the inventory at it, and re-run.
    quiet: true
-  tags: [disks]
+  when:
    - node_baseline_disable_wireless | bool
    - node_baseline_wifi_ifaces | length > 0
  tags: [wireless]
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
+# THE PERSISTENT HALF. This file is the whole reboot story: kmod consults /etc/modprobe.d on
-  community.general.filesystem:
+# every load attempt, so from here on the driver cannot be loaded by udev at boot, as a
-    fstype: ext4
+# dependency, or by hand. Written ONLY when the node actually has a wireless driver, so a
-    dev: "{{ node_baseline_longhorn_device }}"
+# node without wifi hardware (every dev VM) gets no file and no change at all.
-    opts: "-L {{ node_baseline_longhorn_fs_label }}"
+# `update-initramfs` is deliberately NOT run: no wifi module is loaded from the initramfs on
-  tags: [disks]
+# these nodes (root is a local SATA/NVMe device), the driver is autoloaded by udev after
 # switch-root, and re-generating an initramfs on a prod node is a bigger risk than the one it
 # would cover.
 - name: Block the wireless driver module(s) from ever loading again
  ansible.builtin.copy:
    dest: "{{ node_baseline_wireless_modprobe_file }}"
    owner: root
    group: root
    mode: "0644"
    content: |
      # Managed by the Ansible node_baseline role (tags: wireless) — do not edit by hand.
      #
      # WHY THIS NODE HAS NO WIRELESS. A k3s server gets exactly ONE path onto the network.
      # A second, DHCP-managed interface on the same LAN gives the node a second default
      # route, a second address nothing in the cluster's config knows about, and a DHCP
      # client whose resolv.conf hook silently overwrites the resolvers Ansible manages
      # (that is what happened on k3s-prod-3, 2026-09-04). The link is also brought down and
      # the module unloaded at the same time, so no reboot is needed for this to take effect.
      #
      # `blacklist` stops udev's alias-driven autoload; the `install ... /bin/false` line is
      # what also refuses a dependency-triggered or hand-run `modprobe`. Both are needed.
      #
      # TO RE-ENABLE WIRELESS ON THIS NODE: delete this file and `modprobe` the module(s)
      # below (or reboot). To stop Ansible putting it back, set
      # node_baseline_disable_wireless: false for this host first — otherwise the next
      # node-baseline run recreates it. Full rationale: roles/node_baseline/tasks/main.yml,
      # the `wireless` block, and infra/PROD-PROVISIONING.md §4.
      {% for module in node_baseline_wifi_modules %}
      blacklist {{ module }}
      install {{ module }} /bin/false
      {% endfor %}
  when:
    - node_baseline_disable_wireless | bool
    - node_baseline_wifi_modules | length > 0
  tags: [wireless]
- name: Mount the Longhorn data disk
+# THE RUNTIME HALF — everything below makes the change true NOW, so the node does not have to
-  ansible.posix.mount:
+# be rebooted to stop carrying a second address (this role never reboots: §0b "SECOND FAULT").
-    path: "{{ node_baseline_longhorn_mount }}"
+#
-    src: "LABEL={{ node_baseline_longhorn_fs_label }}"
+# Stop a DHCP client that is holding a wireless interface first, so it releases its lease
-    fstype: ext4
+# instead of being left holding a device that is about to disappear. HEAVILY GUARDED: it runs
-    opts: "defaults,nofail,x-systemd.device-timeout=5s"
+# only for clients whose command line names a wireless interface and names NO non-wireless
-    state: mounted
+# interface, because `dhcpcd -x` against a shared master process would also drop a wired
-  tags: [disks]
+# lease. On these prod nodes eno1 is a static ifupdown address with no DHCP client at all, so
 # this task has nothing to do; it exists so the ordering is correct on a node where wifi is
 # still actively leasing.
 - name: Stop the dhcpcd instance that holds a wireless interface (and nothing else)
  ansible.builtin.command:
    argv: [dhcpcd, --exit, "{{ item }}"]
  loop: "{{ node_baseline_wifi_ifaces }}"
  register: node_baseline_wifi_dhcpcd_exit
  changed_when: rc == 0
  failed_when: false
  when:
    - node_baseline_disable_wireless | bool
    - node_baseline_wifi_dhcp_wifi_only | length > 0
    - node_baseline_wifi_dhcp_wifi_only | select('search', 'dhcpcd') | list | length > 0
  vars:
    # A client is "wireless-only" when its argv mentions a wireless interface and mentions no
    # other interface this node has. Empty interface sets are turned into a pattern that
    # cannot match, so an odd node fails towards doing nothing rather than towards guessing.
    node_baseline_non_wifi_ifaces: "{{ ansible_facts['interfaces'] | default([]) | difference(node_baseline_wifi_ifaces) | difference(['lo']) }}"
    node_baseline_wifi_dhcp_wifi_only: >-
      {{ node_baseline_wifi_dhcp_clients
         | select('search', '\b(' ~ ((node_baseline_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
         | reject('search', '\b(' ~ ((node_baseline_non_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
         | list }}
  tags: [wireless]
-# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
+- name: Take each wireless link down now
-# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
+  ansible.builtin.command:
-# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
+    argv: [ip, link, set, dev, "{{ item }}", down]
-# a disk that already carries a filesystem, so re-running against an already-migrated
+  loop: "{{ node_baseline_wifi_live }}"
-# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
+  changed_when: true
-# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
+  when: node_baseline_disable_wireless | bool
-# role would mount over a populated directory.
+  tags: [wireless]
 - name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
  community.general.filesystem:
    fstype: ext4
    dev: "{{ node_baseline_etcd_device }}"
    opts: "-L {{ node_baseline_etcd_fs_label }}"
  tags: [disks]
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
+# Unload the driver so the netdev is gone immediately — holders (iwlmvm on these nodes) first,
-  ansible.posix.mount:
+# then the driver itself, because `modprobe -r` refuses a module that is still held.
-    path: "{{ node_baseline_etcd_mount }}"
+# NOT fatal on failure: the persistent block above is already in place, so a module that
-    src: "LABEL={{ node_baseline_etcd_fs_label }}"
+# cannot be unloaded right now is gone at the next boot regardless. The report below says so
-    fstype: ext4
+# out loud rather than letting a green play imply the radio is already off.
-    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
+- name: Unload the wireless driver module(s) now (no reboot needed)
-    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
+  community.general.modprobe:
-    opts: "defaults,noatime"
+    name: "{{ item }}"
-    dump: "0"
+    state: absent
-    passno: "2"
+  loop: "{{ node_baseline_wifi_module_holders + node_baseline_wifi_modules_loaded }}"
-    state: mounted
+  register: node_baseline_wifi_unload
  failed_when: false
  when: node_baseline_disable_wireless | bool
  tags: [wireless]
 - name: Report the wireless state this node is left in
  ansible.builtin.debug:
    msg: >-
      {{ 'No 802.11 device on this node — nothing to disable (no modprobe.d file written).'
         if (node_baseline_wifi_ifaces | length) == 0
         else ((('WIRELESS DISABLED: ' ~ (node_baseline_wifi_modules | join(', ')) ~ ' blocked in '
                 ~ node_baseline_wireless_modprobe_file ~ ' (survives reboot) and unloaded now; '
                 ~ (node_baseline_wifi_ifaces | join(', ')) ~ ' is down and the netdev is gone.')
                if (failed_unloads | length) == 0
                else ('WIRELESS BLOCKED FOR THE NEXT BOOT BUT STILL LOADED NOW: could not unload '
                      ~ (failed_unloads | join(', ')) ~ '. ' ~ node_baseline_wireless_modprobe_file
                      ~ ' is in place, so the driver will not load again after a reboot, but the '
                      ~ 'radio is live until then — check `ip link` and whether something still '
                      ~ 'holds the device.'))) }}
  vars:
    failed_unloads: >-
      {{ (node_baseline_wifi_unload.results | default([]))
         | selectattr('failed', 'defined') | selectattr('failed') | map(attribute='item') | list }}
  when: node_baseline_disable_wireless | bool
  tags: [wireless]
 # =====================================================================================
 # DATA DISKS — tag `disks`, in tasks/disks.yml
 # =====================================================================================
 # Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
 # (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
 # should be readable end-to-end on its own; and both environments' storage shapes — dev's
 # two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
 # the OS filesystem — are declared there as an explicit enum rather than inferred from
 # which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
 #
 # The `disks` tag is applied ONCE here, in BOTH places it is needed:
 #   * `tags:` on the include itself, so `--tags packages` never even reaches the include
 #     and a placeholder device path is not so much as templated; and
 #   * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
 #     optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
 #     its own tags to its children the way `import_tasks` does, so without `apply` the
 #     included tasks are untagged and `--tags disks` selects the include and then runs
 #     NOTHING inside it. That failure is silent and reports success, which is the worst
 #     shape a guard can have.
 # A FULL run against a node whose layout is undeclared stops on the assert at the top of
 # that file instead of failing obscurely at mkfs time.
 - name: Provision the data disks (Longhorn store, etcd data dir)
  ansible.builtin.include_tasks:
    file: disks.yml
    apply:
      tags: [disks]
  tags: [disks]
 - name: Disable swap now
@@ -180,6 +681,11 @@
 # node and the matching seam is written. Adding a third mechanism means adding a branch
 # here, not rewriting the role.
 #
 # AND DETECTION IS ONLY AS GOOD AS THE SET OF OWNERS IT KNOWS ABOUT. It originally looked for
 # resolvconf/openresolv and NetworkManager, missed dhcpcd, and therefore reported a green,
 # `changed` run on k3s-prod-3 whose resolvers a wifi lease reverted three minutes later. dhcpcd
 # is now part of that set — see the detection task below and the `contended` failure.
 #
 # All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
 # documented at the top of this file) still installs nothing but packages.
@@ -199,6 +705,55 @@
  check_mode: false
  tags: [dns]
 # THE dhcpcd BLIND SPOT, CLOSED (k3s-prod-3, 2026-09-04). dhcpcd's `20-resolv.conf` hook
 # rewrites /etc/resolv.conf on every lease, renewal and carrier change, and it is a THIRD
 # way to own that file — neither resolvconf nor NetworkManager. On node 3 it silently reverted
 # this role's write three minutes later, and because nothing below looked for it the role had
 # already reported `changed` and green: the precise failure the `contended` branch exists to
 # make impossible.
 #
 # WHY THIS LOOKS FOR A LIVE CLIENT AND NOT FOR AN INSTALLED PACKAGE. `dhcpcd-base` is
 # installed on all three prod nodes and always will be — it is the DHCP client ifupdown
 # reaches for — and there is NO dhcpcd.service on them at all (`systemctl is-active dhcpcd` =
 # not-found; it is spawned per-interface by ifupdown). Keying off the package, or off a unit
 # that does not exist, would put every prod node permanently in `contended` and leave the DNS
 # block unable to configure a node it is perfectly safe to configure. What actually matters is
 # whether a DHCP client CAN still write the file, so all three real signals are checked:
 #   CLIENT| a dhcpcd/dhclient process running right now (with the argv that started it);
 #   UNIT|   a dhcpcd/dhclient unit that IS active or IS enabled, on a distro that has one;
 #   STANZA| an UNCOMMENTED `iface <x> inet[6] dhcp` in /etc/network/interfaces{,.d} — the
 #           configuration that makes ifupdown spawn a client at the next boot or `ifup`, even
 #           when none is running at this instant.
 # Any one of them means "another manager is in play" and the play stops below. None of them
 # means no client can run, and the file is genuinely ours to own.
 - name: Determine whether a DHCP client (dhcpcd/dhclient) can rewrite /etc/resolv.conf
  ansible.builtin.shell:
    executable: /bin/sh
    cmd: |
      set -u
      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
        printf 'CLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
      done
      for u in dhcpcd dhcpcd5 dhclient isc-dhcp-client; do
        a=$(systemctl is-active "$u" 2>/dev/null || true)
        e=$(systemctl is-enabled "$u" 2>/dev/null || true)
        # Compared exactly, never with a substring match: `is-active` answers "inactive",
        # which contains the word "active".
        if [ "$a" = active ] || [ "$e" = enabled ] || [ "$e" = enabled-runtime ]; then
          printf 'UNIT|%s active=%s enabled=%s\n' "$u" "$a" "$e"
        fi
      done
      # `^[[:space:]]*iface` cannot match a commented-out stanza, which is how the wireless
      # stanza is left on these nodes.
      grep -rhE '^[[:space:]]*iface[[:space:]]+[^[:space:]]+[[:space:]]+inet6?[[:space:]]+dhcp' \
        /etc/network/interfaces /etc/network/interfaces.d/ 2>/dev/null \
        | sed -e 's/^[[:space:]]*/STANZA|/'
      exit 0
  register: node_baseline_dhcp_client_scan
  changed_when: false
  check_mode: false
  tags: [dns]
 # follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
 # points, which is the strongest single signal of who owns the file.
 - name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
@@ -229,9 +784,10 @@
 #       resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
 #       then the ONLY thing that changes real behaviour; writing resolv.conf would be
 #       overwritten by resolved (or, worse, land in /run and vanish at reboot).
-#   contended — nobody's stub, but resolvconf or NetworkManager is present/running and
+#   contended — nobody's stub, but resolvconf, NetworkManager or a DHCP client (dhcpcd /
-#       would regenerate the file underneath us. We do NOT write. The role fails loudly
+#       dhclient) is present/running/configured and would regenerate the file underneath us.
-#       below rather than leaving an operator believing a change stuck when it did not.
+#       We do NOT write. The role fails loudly below rather than leaving an operator
 #       believing a change stuck when it did not.
 #   resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
 #       Safe to own outright.
 # resolved running but NOT owning resolv.conf correctly falls through to the file branch:
@@ -241,13 +797,21 @@
    node_baseline_dns_mechanism: >-
      {{
        'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
-        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
+        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present
                              or node_baseline_dhcp_client_live)
              else 'resolv-conf-file')
      }}
    # Kept as facts (not just task vars) so the failure message, the report and the
    # reclaim note below all quote the SAME evidence this decision was made on.
    node_baseline_dhcp_client_evidence: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) | regex_findall('(?m)^(?:CLIENT|UNIT|STANZA)\\|(.+)$') }}"
    # A file dhcpcd WROTE, versus a dhcpcd that can still write it — two different facts, and
    # only the second one is contention. See the reclaim note below.
    node_baseline_resolv_dhcpcd_written: "{{ 'Generated by dhcpcd' in (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
  vars:
    node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
    node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
    node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
    node_baseline_dhcp_client_live: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) is search('(?m)^(?:CLIENT|UNIT|STANZA)\\|') }}"
    node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
    node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
    node_baseline_resolved_owns_resolv: >-
@@ -264,7 +828,9 @@
      NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
      /etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
      target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
-      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
+      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }},
      DHCP client evidence={{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }},
      resolv.conf written by dhcpcd={{ node_baseline_resolv_dhcpcd_written | default(false) }})
  tags: [dns]
 # Same shape, and the same reason, as the data-disk assert above: this environment's
@@ -297,16 +863,45 @@
 - name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
  ansible.builtin.fail:
    msg: >-
-      /etc/resolv.conf on this node is not the systemd-resolved stub, but
+      /etc/resolv.conf on this node is not the systemd-resolved stub, but something else would
-      resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
+      regenerate the file underneath this role: resolvconf/openresolv
-      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
+      (/etc/resolvconf/resolv.conf.d present:
-      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
+      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}), NetworkManager
-      would regenerate the file underneath this role. Writing it here would look like it
+      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}), or a DHCP client
-      worked and silently revert. Add the matching branch to roles/node_baseline (the
+      that can still write it — dhcpcd/dhclient evidence:
-      `dns`-tagged block) for whichever manager this node really uses — do not force it.
+      {{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }}.
      Writing it here would look like it worked and silently revert — which is exactly what
      happened on k3s-prod-3 on 2026-09-04, where a wifi lease's dhcpcd hook reverted this
      role's resolvers three minutes after a green run. If the evidence above is a DHCP client
      on a WIRELESS interface, the fix is the `wireless` block earlier in this role (a cluster
      node gets one network path) — run it, then re-run this. If it is a DHCP client on the
      node's real interface, or another manager entirely, add the matching branch to
      roles/node_baseline for the manager this node really uses — do not force it.
  when: node_baseline_dns_mechanism == 'contended'
  tags: [dns]
 # THE CORPSE CASE, and why it is NOT a hole in the fail-closed rule above. After the
 # `wireless` block disables the radio, k3s-prod-3 is left with a /etc/resolv.conf that dhcpcd
 # WROTE (its "# Generated by dhcpcd from wlp1s0.dhcp..." header, the gateway as sole IPv4
 # resolver, two Google IPv6 servers) and NO dhcpcd that can ever run again. `contended` means
 # "another manager is live and will undo this write"; a file left behind by a client that no
 # longer exists is not a manager, it is the damage — and refusing to write here would leave the
 # node permanently mis-resolved, which is the very outcome the operator reported. So the file
 # branch below reclaims it, and says so.
 - name: Note that a dhcpcd-written /etc/resolv.conf is being reclaimed
  ansible.builtin.debug:
    msg: >-
      /etc/resolv.conf on this node was written by dhcpcd, but no DHCP client can write it any
      more (no process, no active/enabled unit, no `inet dhcp` stanza). Taking ownership and
      replacing it with the declared upstreams
      ({{ node_baseline_dns_servers | join(', ') }}). If a DHCP client ever comes back on this
      node, the `contended` check above will stop the play instead of letting it revert
      silently.
  when:
    - node_baseline_dns_mechanism == 'resolv-conf-file'
    - node_baseline_resolv_dhcpcd_written | default(false)
  tags: [dns]
 # --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
 # A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
 # never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses
@@ -54,7 +54,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
 | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
-| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
+| `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
 Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
 in task logic.
@@ -65,33 +65,81 @@ in task logic.
 `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
 HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
-does not validate it before reserving memory from its contents —
+does not validate it before reserving memory from its contents. **Two different kernel
-`drivers/firmware/efi/memattr.c` multiplies two never-validated fields
+functions matter, and the check used to conflate them:**
 (`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the
 table address to the top of RAM. Which of two kernel-log lines appears decides whether the
 node keeps its memory, and it is **decided afresh on every boot**:
-| kernel log | meaning | verdict |
+- `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both
  fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so
  **catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through
  `phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything
  above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and
  costs nothing measurable.
 - `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints
  "Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or
  merely large.
 So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not*
 prove memory was lost: that depends on the garbage **value** at that address on that boot,
 not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03)
 booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on
 measured memory, never on the log line alone:**
 | kernel log | measured shortfall | verdict |
 |---|---|---|
-| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* |
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
-| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table |
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
-| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path |
-| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic |
-| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected |
+| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working |
-| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected |
+| no memattr line, log reaches back to boot | (not consulted) | PASS |
 | no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
 | `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
-The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and
+The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt
-differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before
+table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt
-it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing.
+table is not passing a health check even when it kept its RAM.
 ### The FW_BUG line is proof, but its ABSENCE proves nothing
 `efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG
 line:
 | gate | message | reachable when |
 |---|---|---|
 | `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
 | implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not |
 | `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not |
 **All three are rejections.** Which one a boot prints is decided by *which field* that boot's
 garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very
 often prints the *first* message, and looking only for the FW_BUG line will make a working
 fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed
 the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version
 alongside the log line, and why the check must never demand the FW_BUG string as its
 evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in
 the kernel image itself, not a line in `dmesg`.
 **The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged
 *negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that
 until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected
 a corrupt table was reported as one that "accepted it without complaint" — a false PASS on
 the exact bug this check exists to find.
 **Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
 `trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
 workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
 rebooting/CMOS/BIOS/USB changes only move where the table lands.
-`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node
+### The two memory checks cannot contradict each other
-whose kernel log cannot be read is still not unmeasured — and a future memory loss with a
+
-different cause is still caught.
+`memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine
 lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus
 the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in
 `tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads
 it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of
 its own, so there is exactly one verdict on lost memory per host and both checks quote it.
 A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a
 future memory loss with a different cause is still caught.
 ## Output
@@ -468,19 +468,27 @@ def collect_memory_devices():
 # write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
 # Table, and Linux < 6.13 does not validate it before reserving memory from its contents
 # (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
-# the product). The kernel prints exactly one of two lines, and WHICH one decides whether
+# the product). The kernel prints one of two lines:
 # the node keeps its RAM:
 #
-#   "Failed to map EFI Memory Attributes table @ 0x..."   the table's garbage version read
+#   "Failed to map EFI Memory Attributes table @ 0x..."   the garbage version read <= 2, so
-#       <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW.
+#       the kernel TRUSTED the table and reserved memory from it. This is emitted later, by
 #       efi_memattr_apply_permissions()'s memremap(), and it fails for ANY sufficiently
 #       wrong tbl_size. So it proves the table was corrupt and trusted — it does NOT prove
 #       memory was lost: only a NEGATIVE tbl_size (bit 31 set, sign-extended and clamped to
 #       PHYS_ADDR_MAX) reserves everything above the table, while a garbage-but-positive
 #       tbl_size costs nothing measurable. Whether memory actually went missing is decided
 #       by MEASUREMENT (collect_zoneinfo below), not by this line: k3s-prod-2 lost 13.3 GiB
 #       twice at 0xbeeb0018 and then booted with all 15.1 GiB at the same address, same line.
 #   "Unexpected EFI Memory Attributes table version <N>"  the garbage version read > 2, so
 #       the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
 #       firmware is still publishing a corrupt table and the outcome is decided afresh by
-#       where the table lands on every boot. That is luck, not immunity.
+#       what the garbage reads on every boot. That is luck, not immunity.
 #
-# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots
+# This collector reports the LINES and the addresses; it never rules on memory. Classifying
-# before it lost 13.3 GiB), which is precisely why it is detected on every health run
+# is roles/node_health/tasks/assess.yml, which rules on the measured shortfall.
-# instead of diagnosed once.
+#
 # It is boot-to-boot NON-DETERMINISTIC, which is precisely why it is detected on every
 # health run instead of diagnosed once.
 #
 # dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
 # every node here). The unprivileged pass therefore records WHY it could not read, and
@@ -489,8 +497,15 @@ BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Co
 EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
 MEMATTR_FAILED_RE = re.compile(
    r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
 # NOTE the `-?`. The kernel prints this version with %d — SIGNED — so a garbage field with
 # bit 31 set is logged NEGATIVE (k3s-prod-2, 2026-09-04: "version -1569980416"). Without the
 # minus sign this regex missed those lines entirely and the state fell through to "clean",
 # i.e. a node whose kernel had just REJECTED a corrupt table was reported as one whose
 # kernel "accepted it without complaint" — a false PASS on the exact bug this check exists
 # to find. Anchor future edits on that: the field is signed, and every diagnostic that
 # quotes it must survive a negative value.
 MEMATTR_VERSION_RE = re.compile(
-    r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I)
+    r"Unexpected EFI Memory Attributes table version\s+(-?\d+)", re.I)
 MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
 # The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
 # line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
@@ -387,18 +387,28 @@
               ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
               ~ avail_pct | string ~ '%)') }}
-# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the
+# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
-# one cause we have actually hit here; this one asks the blunter question — "is this
+# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
-# machine using the RAM it has?" — and answers it from two sources that share NOTHING
+# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
-# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some
+# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
-# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml.
+# from the presence of a kernel-log line — and they duly contradicted each other on the same
- name: Assess — usable memory vs the memory this machine physically has
+# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
 # 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
 # measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
 # measurement was right; the inference from the log line was wrong — the memattr block below
 # has the refined mechanism.
 #
 # THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
 # place and both checks read this fact: neither derives a shortfall of its own, and the
 # memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
 # disagree about whether memory was lost — there is only one verdict on that question, and
 # it is this one. Thresholds and their measured baselines: defaults/main.yml.
 - name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
  ansible.builtin.set_fact:
-    node_health_checks: "{{ node_health_checks + [{
+    node_health_mem_shortfall: "{{ {
-      'id': 'memory.physical_total',
+      'zones_pct': z_pct, 'zones_status': z_status,
-      'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
+      'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
-      'status': phys_status,
+      'status': worst_status } }}"
      'detail': phys_detail }] }}"
  vars:
    ansible_become: false
    th: "{{ node_health_thresholds }}"
@@ -430,11 +440,36 @@
    # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
    # SKIPs must never add up to a PASS.
-    phys_status: >-
+    worst_status: >-
      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
         | default('SKIP', true) }}
-    # What the kernel log says, IF it was read — quoted here only as attribution for a
+
-    # shortfall this check measured on its own. The verdict above never depends on it.
+# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
 # firmware.efi_memattr names the one cause we have actually hit here; this one asks the
 # blunter question — "is this machine using the RAM it has?" — and answers it from two
 # sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
 # losing memory for some other reason still shows up.
 - name: Assess — usable memory vs the memory this machine physically has
  ansible.builtin.set_fact:
    node_health_checks: "{{ node_health_checks + [{
      'id': 'memory.physical_total',
      'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
      'status': phys_status,
      'detail': phys_detail }] }}"
  vars:
    ansible_become: false
    ms: "{{ node_health_mem_shortfall }}"
    z: "{{ node_health_facts.memory.zones }}"
    dev: "{{ node_health_facts.memory.devices }}"
    m: "{{ node_health_facts.memory }}"
    # Straight from the shared measurement — this check re-derives nothing.
    z_pct: "{{ ms.zones_pct }}"
    inst_kb: "{{ ms.installed_kb }}"
    d_pct: "{{ ms.dimm_pct }}"
    phys_status: "{{ ms.status }}"
    # What the kernel log says, IF it was read — quoted here only as attribution for the
    # measured shortfall. The verdict above is the shared measurement and never depends on
    # the log; firmware.efi_memattr reads that same verdict, so the two always agree.
    phys_cause: >-
      {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
           firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
@@ -757,24 +792,59 @@
 # what follows is only what this check needs to be read correctly.
 #
 # HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
-# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c
+# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
-# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s
+# are involved, and conflating them is what made this check wrong until 2026-09-03:
 # the product — everything from the table address to the top of RAM. The kernel prints
 # exactly one of two lines, and WHICH one is decided by whether the garbage `version` field
 # happens to read > 2:
 #
-#   "Failed to map EFI Memory Attributes table @ 0x..."  version read <= 2, the kernel
+#   efi_memattr_init()              tbl_size = sizeof(*tbl) + num_entries * desc_size, then
-#       TRUSTED the table. The node has lost the memory. FAIL.
+#                                   memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
-#   "Unexpected EFI Memory Attributes table version <N>"  version read > 2, the kernel
+#                                   is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
-#       bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a
+#                                   NEGATIVE: a garbage product with bit 31 set sign-extends
-#       corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS:
+#                                   through phys_addr_t and memblock_cap_size() clamps it to
-#       k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself
+#                                   PHYS_ADDR_MAX — everything above the table is reserved.
-#       booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one
+#                                   A garbage-but-POSITIVE tbl_size reserves a modest, wrong
-#       unlucky boot from losing 83% of its RAM is not passing a health check.
+#                                   amount and costs nothing measurable.
 #   efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
 #                                   THIS is what prints "Failed to map". It fails for ANY
 #                                   sufficiently wrong tbl_size — negative OR merely large.
 #
 # SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
 # prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
 # that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
 # 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
 # is the counter-example that falsified the old inference.
 #
 # THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
 #
 #   "Failed to map ... @ 0x..."   the bug FIRED: corrupt table, trusted by the kernel.
 #       FAIL when the shared measurement (node_health_mem_shortfall) says memory is
 #       catastrophically missing — memory is lost NOW.
 #       WARN otherwise — this boot drew a survivable garbage value; the next may not.
 #   "Unexpected ... version <N>"  version read > 2, so the kernel bailed out before
 #       reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
 #       safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
 #       gates and no draw can reach the fatal arithmetic any more. Same line, opposite
 #       meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
 #   FW_BUG "Corrupted EFI Memory Attributes Table detected!"  the >= 6.13 THIRD gate
 #       (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
 #       must not be treated as such: it is only reachable on a boot whose garbage leaves the
 #       version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
 #       line instead. All three prod nodes did exactly that on 2026-09-04.
 #
 # The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
 # a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
 # measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
 # firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
 # the draw still happens but every outcome is a rejection, so the same lines mean the
 # validation is working.
 #
 # WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
 # so it cannot be diagnosed once. It has to be measured on every health run.
 #
 # IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
 # node_health_mem_shortfall, computed once further up this file; this check does not measure
 # memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
 # one verdict on any host and both checks quote it.
 #
 # THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
 # ONLY if the kernel log was actually read AND still reaches back to early boot — every
 # memattr diagnostic is printed in the first fraction of a second, and the ring buffer
@@ -793,13 +863,43 @@
    ansible_become: false
    k: "{{ node_health_facts.klog }}"
    z: "{{ node_health_facts.memory.zones }}"
    # THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
    # near memory.physical_total. This check contributes the kernel-log facts only.
    ms: "{{ node_health_mem_shortfall }}"
    ma_state: "{{ k.memattr_state | default(none) }}"
    # DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
    # carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
    # THREE checks before it computes tbl_size at all:
    #     tbl->version > 2                       -> "Unexpected ... table version %d"
    #     desc_size implausible                  -> "Unexpected ... descriptor size %u ..."
    #     num_entries > SZ_64K                   -> FW_BUG "Corrupted EFI Memory Attributes
    #                                               Table detected! ..."
    # ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
    # given boot produces is decided by WHICH field that boot's garbage lands in — it is not
    # a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
    # version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
    # memblock_reserve(). That is the whole difference, and it is why the verdict for
    # `bad_version` below is opposite on the two kernels: same log line, completely different
    # meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
    # something the `version` test can parse.
    memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
    memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
    # `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
    # COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
    # this check and memory.physical_total can no longer disagree about lost memory.
    # `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
    # boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
    # WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
    # forever would make the check unable to ever show the fix landing, which is exactly what
    # it is for.
    memattr_status: >-
      {{ 'SKIP' if not (k.available | bool)
-         else ('FAIL' if ma_state == 'failed_to_map'
+         else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
-               else ('WARN' if ma_state == 'bad_version'
+               else ('WARN' if ma_state == 'failed_to_map'
                     else ('PASS' if ma_state == 'corrupt_rejected'
-                           else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
+                           else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
                                 if ma_state == 'bad_version'
                                 else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
    # One remediation, quoted by both findings. NOTE what it must NOT say: there is no
    # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
    # could take effect. Suggesting one would be a fix that quietly does nothing.
@@ -812,14 +912,32 @@
      bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
      unplugging USB only change where the table lands and may APPEAR to fix it — they are
      a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
-    # The symptom, measured independently of this log (see memory.physical_total).
+    # The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
-    memattr_corroboration: >-
+    # to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
-      {{ (' Measured independently of this log: MemTotal '
+    # kernel log has no part in; the verdict word comes from ms.status, so it always agrees
-          ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+    # with memory.physical_total. When the measurement is missing it says so, and says why.
    memattr_figures: >-
      {{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
          ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
-          ~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away
+          ~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
-            from the allocator (/proc/zoneinfo) — check memory.physical_total.')
+            allocator (/proc/zoneinfo)')
-         if ((z.available | bool) and z.unmanaged_pct is number) else '' }}
+         if ms.zones_pct is number
         else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
                reports as installed (/proc/zoneinfo NOT read — '
                ~ (z.reason | default('no reason recorded', true)) ~ ')')
               if ms.dimm_pct is number else '') }}
    memattr_corroboration: >-
      {{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
          ~ ms.status ~ '): ' ~ memattr_figures
          ~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
             else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
                   else ' — the normal few percent (kernel image, memmap, firmware
                        reservations, graphics stolen memory). NO measurable memory was lost
                        on this boot.')))
         if (memattr_figures | trim | length) > 0
         else (' Whether memory was lost on this boot could NOT be measured here: neither
                present-vs-managed pages nor installed DIMM capacity was read — see check
                memory.physical_total for the reasons.') }}
    # The kernel log lines themselves, verbatim. A verdict the operator cannot check
    # against the evidence is not much better than a wrong one.
    memattr_evidence: >-
@@ -837,23 +955,58 @@
      the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
      Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
      lives only in dmesg.{{ memattr_corroboration }}
      {%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
      THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
      line. The kernel failed to map the EFI Memory Attributes table at
      {{ k.memattr_table_address | default('an unreported address') }}, which proves the
      table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
      from its contents; the measurement below shows that reservation was the catastrophic
      kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
      through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
      table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
      {%- elif ma_state == 'failed_to_map' -%}
-      THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory
+      THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
-      Attributes table at {{ k.memattr_table_address | default('an unreported address') }}
+      loss either. The kernel failed to map the EFI Memory Attributes table at
-      and reserved memory from the corrupt table anyway — everything from that address
+      {{ k.memattr_table_address | default('an unreported address') }}: the table was
-      upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
+      corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
-      {%- elif ma_state == 'bad_version' -%}
+      WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
-      EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table
+      the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
-      here too; this boot survived only because its garbage version field read
+      with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
-      {{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out
+      tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
-      before reserving anything. The outcome is re-decided on EVERY boot: if that field
+      measurable and still fails the later memremap() that printed this line. This boot drew a survivable
-      reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right
+      value; the draw is repeated on EVERY boot, at the same address — this platform has
-      now and this node is one unlucky boot from the failure state.{{ memattr_evidence }}
+      been seen to lose 13.3 GiB and to keep its full memory at one identical table address
      (infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
      {{ memattr_fix }}
      {%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
      the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
      kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
      VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
      boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
      outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
      efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
      WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
      Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
      (num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
      version field <= 2. Both messages are rejections, and which one a boot prints is the
      lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
      path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
      PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
      {{ memattr_corroboration }}
      {%- elif ma_state == 'bad_version' -%}
      EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
      CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
      version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
      kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
      EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
      tbl_size the garbage produces — nothing measurable if that value is positive,
      everything above the table if it is negative.{{ memattr_evidence }}
      {{ memattr_corroboration }} {{ memattr_fix }}
      {%- elif ma_state == 'corrupt_rejected' -%}
      the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
-      validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is
+      validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
-      intact and stays intact across reboots; no action.{{ memattr_evidence }}
+      cannot cost this node memory on this or any future boot; no
      action.{{ memattr_evidence }}{{ memattr_corroboration }}
      {%- elif not (k.boot_covered | bool) -%}
      NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
      back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
@@ -12,6 +12,17 @@
 {% macro gib(n) -%}
 {{ (n | int / 1073741824) | round(1) }} GiB
 {%- endmacro %}
 {# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not
   report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of
   its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised
   "object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it
   as `n/a` (the same token this template already uses for unreported readings) rather than
   defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The
   collector already guards this — tasks/collect_smart.yml uses selectattr('thresh',
   'defined') and `a.thresh | default(0)` — so only the template was exposed. #}
 {% macro thr(a) -%}
 {{ a.thresh | default('n/a') }}
 {%- endmacro %}
 ================================================================================
 NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
 collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
@@ -73,19 +84,19 @@
              lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }}  temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
              pending(197)={{ sm.pending_sectors | default('?') }}  offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }}  reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }}  crc(199)={{ sm.crc_errors | default('?') }}
 {% if sm.failing_now or sm.failed_past %}
-              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %}
+              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %}
 {% elif sm.thresholds_present %}
-              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %}
+              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %}
 {% endif %}
              vendor context (NOT used for the verdict — raw units are vendor-defined):
              reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }}  grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }}  erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }}  program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }}  unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
 {% if sm.realloc_attr %}
-              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged)
+              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged)
 {% endif %}
 {% for a in sm.attributes | default([]) %}
-              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
+              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
 {% endfor %}
 {% else %}
@@ -0,0 +1,41 @@
 ---
 # platform-config-cdi — the SECOND-ORDER config wave: custom resources whose CRD is created at
 # RUNTIME by a controller that a `platform-config` custom resource switches on.
 #
 # ── THE BUG THIS DIRECTORY EXISTS TO FIX (observed on PROD, 2026-09-05) ───────────────────────────
 # The repo already splits CONTROLLERS (`platform`) from CONFIGS (`platform-config`) because a CR
 # cannot be dry-run before its CRD is registered. That split assumes a CRD is registered by the
 # controller INSTALL. CDI breaks the assumption: `cdi-operator.yaml` registers exactly ONE CRD
 # (`cdis.cdi.kubevirt.io`). Every OTHER CDI CRD — `storageprofiles`, `datavolumes`, `datasources`,
 # `dataimportcrons`, … — is created by cdi-operator only AFTER it has reconciled the `CDI` CR.
 #
 # So `cdi-cr.yaml` is a FIRST-order CR (its CRD comes from the install) and belongs in
 # platform-config, while `cdi-storageprofiles.yaml` is a SECOND-order CR (its CRD comes from
 # reconciling the first-order CR). Having both in ONE Flux Kustomization is a true DEADLOCK, not a
 # race: kustomize-controller server-side dry-runs EVERY object before applying ANY of them, so the
 # unknown `StorageProfile` kind aborts the whole pass → the `CDI` CR is never applied → the
 # StorageProfile CRD is never created → platform-config can NEVER converge, and every tier that
 # dependsOn it stays blocked forever. On prod, cdi-operator sat Available for 12h with no CR:
 #
 #   StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile"
 #   in version "cdi.kubevirt.io/v1beta1"
 #
 # Invisible on dev (whose CDI CRDs have existed for months) and fatal on any COLD cluster.
 #
 # ── THE RULE, so this is not rediscovered a third time ────────────────────────────────────────────
 # A CR belongs HERE (not in platform-config) when the CRD it needs is created by a CONTROLLER AT
 # RUNTIME rather than by a manifest in git. Grep test: if `grep -rn "kind: CustomResourceDefinition"`
 # over ./platform (following remote bases) does not produce the CRD, and no chart installs it, then
 # some operator creates it after being switched on — and this is the tier for it.
 #
 # Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
 # manifests stay beside the component they configure). Build/verify locally with:
 #   kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config-cdi
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  # CDI StorageProfiles — snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
  # `longhorn-vm` virt-default class (clears CDIDefaultStorageClassDegraded). `StorageProfile`'s CRD
  # is created by cdi-operator only once the `CDI` CR (applied by platform-config) is reconciled,
  # which is exactly why these cannot live in the same pass as that CR.
  - ../platform/kubevirt/cdi-storageprofiles.yaml
@@ -25,6 +25,14 @@ resources:
  - ../platform/kyverno/policies/restrict-host-namespaces.yaml
  - ../platform/kyverno/policies/require-egress-class-label.yaml
  - ../platform/kyverno/policies/require-auth-mode-label.yaml
  # COLD-START FIX (2026-09-06, prod): a MUTATE policy that must travel WITH the two label
  # policies above. CDI generates its own short-lived worker pods (importer-*, uploader-*,
  # cloner-*) inside app namespaces; nothing in git can label them, so both policies DENIED them
  # and `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled with no disk. This policy adds
  # the two labels to pods created by ServiceAccount cdi/cdi-sa, so the requirement is SATISFIED
  # rather than bypassed. Applying it here (not later) matters: if it lands after the validating
  # policies, the next import is blocked in the gap.
  - ../platform/kyverno/policies/label-cdi-worker-pods.yaml
  - ../platform/kyverno/admission/secret-provenance.yaml
  # cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
  - cilium/l2-ip-pool.yaml
@@ -41,9 +49,23 @@ resources:
  # ConfigMap kept beside them for locality.
  - ../platform/kubevirt/kubevirt-cr.yaml
  - ../platform/kubevirt/cdi-cr.yaml
-  # CDI StorageProfiles: snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
+  # *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
-  # `longhorn-vm` virt-default class (consume cdi.kubevirt.io; clears CDIDefaultStorageClassDegraded).
+  # The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
-  - ../platform/kubevirt/cdi-storageprofiles.yaml
+  # (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
  #
  # `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
  # `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
  # only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
  # in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
  # `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
  # created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
  # A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
  #   StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
  #   "cdi.kubevirt.io/v1beta1"
  # Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
  #
  # THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
  # is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
  - ../platform/kubevirt/servicemonitor.yaml
  - ../platform/kubevirt/alerts.yaml
  - ../platform/kubevirt/dashboard.yaml
@@ -10,7 +10,8 @@
 #
 # SOURCE OF TRUTH (do not diverge without updating both):
 #   infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
-#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version, k3s_api_host/port)
+#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version)
 #   infra/ansible/inventory/group_vars/all.yml     (k3s_apiserver_endpoint, k3s_api_port)
 #
 # Every value tagged [parity] below is asserted identical to the Ansible render. If you change one
 # here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.)
@@ -45,11 +46,35 @@ spec:
    # [parity] MUST equal Ansible: kubeProxyReplacement: true
    kubeProxyReplacement: true
    # [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint
-    # (k3s_api_host/k3s_api_port). This is the primary control-plane node IP.
+    # (**k3s_apiserver_endpoint** / k3s_api_port, infra/ansible/inventory/group_vars/all.yml).
    # Setting these wrong breaks Cilium's connection to the API server on adopt.
-    # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.50 | prod set in clusters/prod.
+    #
-    # The `platform` Flux Kustomization substitutes it; the Ansible side is already parameterized
+    # THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS — operator decision 2026-09-06.
-    # from inventory (k3s_api_host), so the parity contract holds per-environment, not per-literal.
+    # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.49 | prod 192.168.2.9, each equal to
    # that environment's kube_vip_address. The knob's NAME still says "host"; read it as "the API
    # host Cilium dials", and see clusters/base/cluster-vars.yaml for why it was not renamed.
    #
    # WHY, in one line: this value used to be the PRIMARY NODE's IP, so on a 3-server HA cluster the
    # CNI's control-plane path depended on one machine. Verified on live prod 2026-09-06 — every
    # cilium-agent carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1) while kube-vip held .9.
    # Losing that node does not break the datapath (Cilium keeps serving from cache); it silently
    # stops every service and policy update from ever landing, which is worse to detect.
    #
    # [parity] IS NOW AGAINST k3s_apiserver_endpoint AND NOT k3s_api_host. Those were one Ansible
    # variable until 2026-09-06 and the conflation is the defect. k3s_api_host still exists and
    # still holds the FOUNDER's own address, because roles/k3s_server's secondary
    # `--server https://<host>:6443` join must target a host that is already up — on a COLD build
    # no VIP exists yet. So: founder for the join, VIP for the CNI. If you change this line, change
    # cilium-values.yaml.j2 in the same PR, and point it at the SAME variable.
    #
    # The `platform` Flux Kustomization substitutes it; the Ansible side is parameterized from
    # inventory (k3s_apiserver_endpoint derives from kube_vip_address), so the parity contract holds
    # per-environment, not per-literal.
    #
    # NEEDS NO CERT WORK: k3s_tls_sans has always carried kube_vip_address alongside every node IP,
    # so both addresses are already SANs on the serving cert (verified live 2026-09-06 — prod
    # answers through https://192.168.2.9:6443 under full CA verification, no insecure skip). This
    # switch reissues nothing and restarts no apiserver.
    k8sServiceHost: "${K3S_API_HOST}"
    k8sServicePort: 6443
    # --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 ---
@@ -6,16 +6,20 @@
 #
 # SOURCE OF TRUTH (all [parity] fields MUST equal these):
 #   infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
-#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version: 1.20.1, k3s_api_host/port)
+#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version: 1.20.1)
 #   infra/ansible/inventory/group_vars/all.yml     (k3s_apiserver_endpoint = the kube-vip VIP,
 #                                                   k3s_api_port) — NOT k3s_api_host, which is the
 #                                                   founder's own address and bootstrap-only
 #
 # chart version (helmrelease.yaml chart.spec.version): 1.20.1   # [parity] = cilium_version
 # [parity] k3s kube-proxy disabled → Cilium replaces it (ADR-0001/0002)
 kubeProxyReplacement: true
-# [parity] = the k3s API endpoint (primary control-plane node IP) + port.
+# [parity] = the k3s API endpoint + port. THE CONTROL-PLANE VIP (kube-vip), not the primary node —
 # changed 2026-09-06; the Ansible half is k3s_apiserver_endpoint, NOT k3s_api_host.
 # Mirrors the HelmRelease's ${K3S_API_HOST} cluster-vars token (R-ENV-02). This file is NOT a
 # kustomize resource, so nothing substitutes it — the token is shown with its dev value beside it.
-k8sServiceHost: "${K3S_API_HOST}"   # dev: 192.168.1.50
+k8sServiceHost: "${K3S_API_HOST}"   # the CONTROL-PLANE VIP — dev: 192.168.1.49 | prod: 192.168.2.9
 k8sServicePort: 6443
 # [parity] ingress via Gateway API, no Traefik (docs/09)
 gatewayAPI:
@@ -29,9 +29,9 @@ metadata:
 data:
  # Filename `home.server` -> imported as a standalone server block for the whole zone.
  home.server: |
-    home.keithjackson.dev:53 {
+    ${BASE_DOMAIN}:53 {
        errors
-        # Answer EVERY name in the zone (apex + all `<svc>.home.keithjackson.dev`) with the Gateway
+        # Answer EVERY name in the zone (apex + all `<svc>.${BASE_DOMAIN}`) with the Gateway
        # VIP. The template plugin's default match is `.*`, so this covers current and FUTURE app
        # hostnames without editing this file per service. `{{ .Name }}` echoes the queried FQDN.
        template IN A {
@@ -125,7 +125,7 @@ spec:
            - --http-address=0.0.0.0:4180
            - --reverse-proxy=true
            - --provider=oidc
-            - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+            - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
            # Gate the co-located Crafty UI. Crafty serves HTTPS with a SELF-SIGNED cert, so skip verify
            # on this in-pod (127.0.0.1) hop; the browser-facing TLS is the trusted wildcard at the Gateway.
            - --upstream=https://127.0.0.1:8443
@@ -138,9 +138,9 @@ spec:
            - --allowed-group=platform-admins
            # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
            - --skip-provider-button=true
-            - --redirect-url=https://crafty.home.keithjackson.dev/oauth2/callback
+            - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.home.keithjackson.dev
+            - --cookie-domain=.${BASE_DOMAIN}
-            - --whitelist-domain=.home.keithjackson.dev
+            - --whitelist-domain=.${BASE_DOMAIN}
            - --cookie-secure=true
            - --pass-access-token=true
            - --set-xauthrequest=true
@@ -25,7 +25,7 @@ spec:
      namespace: gateway
      sectionName: https
  hostnames:
-    - crafty.home.keithjackson.dev
+    - crafty.${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -42,7 +42,7 @@ metadata:
    lbipam.cilium.io/sharing-key: "crafty"
    lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
    # external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
-    external-dns.alpha.kubernetes.io/hostname: "crafty-mc.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
 spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
@@ -20,8 +20,12 @@ Neither k3s nor the Longhorn chart ships this. It is a prerequisite for:
 ## Wiring
 - **Pass:** `platform` (controllers). Registers the CRDs.
- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the
+- The `VolumeSnapshotClass` CR that consumes the CRDs lives in the `platform-config` pass
-  `platform-config` pass (`dependsOn: platform`), per the controllers/configs split.
+  (`dependsOn: platform`), per the controllers/configs split.
 - The CDI `StorageProfile`s live one pass further out, in `platform-config-cdi`
  (`dependsOn: platform-config`) — `storageprofiles.cdi.kubevirt.io` is created by cdi-operator only
  after it reconciles the `CDI` CR, so they cannot share a pass with it. See
  `clusters/base/platform-config-cdi.yaml`.
 ## Version
@@ -9,8 +9,10 @@
 #   * no VolumeSnapshotClass can be created (item 2), and
 #   * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
 #     which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
-# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI
+# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass that CONSUMES them
-# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform).
+# is applied by platform-config (dependsOn: platform), and the CDI StorageProfiles that set
 # `cloneStrategy: snapshot` by platform-config-cdi (dependsOn: platform-config — their own CRD is
 # created by cdi-operator only after the `CDI` CR reconciles; see clusters/base/platform-config-cdi.yaml).
 #
 # CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
 # CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment
@@ -29,15 +29,37 @@ spec:
    sources:
      - gateway-httproute
      - service
-    # domain-filter MUST be the actual Cloudflare ZONE (keithjackson.dev), not the platform subdomain
+    # ---------------------------------------------------------------------------------------------
-    # (home.keithjackson.dev) — ExternalDNS filters the account's zones by this value, and a subdomain
+    # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
-    # filter drops the parent zone → "no hosted zone matching record DNS Name" and nothing is created.
+    #
-    # Scope stays effectively home.keithjackson.dev because that's the only suffix services request;
+    # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
-    # policy=sync + registry=txt only ever touch ExternalDNS-OWNED records, never the pre-existing
+    # `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because
-    # wildcard or other keithjackson.dev records. (docs/09; resolves the WS-NET TODO below.)
+    # NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore
    # read dev's TXT registry entries as its own, found no local resource claiming those names, and
    # garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
    # Full write-up: infra/PROD-PROVISIONING.md §10.
    #
    # TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence:
    #   1. txtOwnerId       — per-environment identity; a cluster only ever touches records it owns.
    #   2. domainFilters    — per-environment name scope; prod cannot even see names outside its
    #                         own subtree. Defence in depth, added after the incident.
    # ---------------------------------------------------------------------------------------------
    # Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit
    # 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is
    # a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record
    # DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the
    # apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no
    # matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive)
    # and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving
    # txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml.
    domainFilters:
-      - "${DNS_ZONE}"
+      - "${EXTERNALDNS_DOMAIN_FILTER}"
-    policy: sync            # create + update + delete records as services come and go
+    # The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
    # caused the incident. Dev renders `default` (its live records are already registered under that
    # owner; changing it would orphan them and duplicate every record). Prod renders `prod`.
    txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
    policy: sync            # create + update + delete records as services come and go — but ONLY
                            # records this owner id owns AND this domainFilter admits (see above).
    # Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007).
    env:
      - name: CF_API_TOKEN
@@ -13,7 +13,7 @@ spec:
    - name: https
      protocol: HTTPS
      port: 443
-      hostname: "*.home.keithjackson.dev"
+      hostname: "*.${BASE_DOMAIN}"
      tls:
        mode: Terminate
        certificateRefs:
@@ -30,7 +30,7 @@ spec:
    - name: http-redirect
      protocol: HTTP
      port: 80
-      hostname: "*.home.keithjackson.dev"
+      hostname: "*.${BASE_DOMAIN}"
      allowedRoutes:
        namespaces:
          from: Selector
@@ -46,7 +46,7 @@ spec:
    - name: https-apex
      protocol: HTTPS
      port: 443
-      hostname: "home.keithjackson.dev"
+      hostname: "${BASE_DOMAIN}"
      tls:
        mode: Terminate
        certificateRefs:
@@ -61,7 +61,7 @@ spec:
    - name: http-apex-redirect
      protocol: HTTP
      port: 80
-      hostname: "home.keithjackson.dev"
+      hostname: "${BASE_DOMAIN}"
      allowedRoutes:
        namespaces:
          from: Selector
@@ -10,8 +10,8 @@ metadata:
 spec:
  secretName: wildcard-home-tls        # consumed by the Gateway HTTPS listener below
  dnsNames:
-    - "*.home.keithjackson.dev"
+    - "*.${BASE_DOMAIN}"
-    - "home.keithjackson.dev"          # apex for the C2 UI (docs/09)
+    - "${BASE_DOMAIN}"          # apex for the C2 UI (docs/09)
  issuerRef:
    kind: ClusterIssuer
    # ${ACME_CLUSTER_ISSUER}: dev -> letsencrypt-staging, prod -> letsencrypt-prod (cluster-vars).
@@ -24,7 +24,7 @@ spec:
      namespace: gateway
      sectionName: https
  hostnames:
-    - headlamp.home.keithjackson.dev
+    - headlamp.${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -36,63 +36,63 @@ data:
    ---
    - Platform:
        - Headlamp:
-            href: https://headlamp.home.keithjackson.dev
+            href: https://headlamp.${BASE_DOMAIN}
            description: Kubernetes dashboard (per-user OIDC RBAC)
            icon: mdi-kubernetes
        - Grafana:
-            href: https://grafana.home.keithjackson.dev
+            href: https://grafana.${BASE_DOMAIN}
            description: Metrics, logs & dashboards (fronts Prometheus/Loki/Alertmanager)
            icon: mdi-chart-areaspline
        - Semaphore:
-            href: https://semaphore.home.keithjackson.dev
+            href: https://semaphore.${BASE_DOMAIN}
            description: Ansible automation & CI
            icon: mdi-cog-play
        - Gitea:
-            href: https://gitea.home.keithjackson.dev
+            href: https://gitea.${BASE_DOMAIN}
            description: Git hosting & Actions (GitOps source of truth)
            icon: mdi-git
        - Keycloak:
-            href: https://keycloak.home.keithjackson.dev
+            href: https://keycloak.${BASE_DOMAIN}
            description: Identity provider (SSO / OIDC)
            icon: mdi-account-key
    - Media:
        - Plex:
-            href: https://plex.home.keithjackson.dev
+            href: https://plex.${BASE_DOMAIN}
            description: Media streaming
            icon: mdi-plex
        - Seerr:
-            href: https://seerr.home.keithjackson.dev
+            href: https://seerr.${BASE_DOMAIN}
            description: Media requests
            icon: mdi-play-box-multiple
        - Sonarr:
-            href: https://sonarr.home.keithjackson.dev
+            href: https://sonarr.${BASE_DOMAIN}
            description: TV library manager
            icon: mdi-television-classic
        - Radarr:
-            href: https://radarr.home.keithjackson.dev
+            href: https://radarr.${BASE_DOMAIN}
            description: Movie library manager
            icon: mdi-movie-open
        - Jackett:
-            href: https://jackett.home.keithjackson.dev
+            href: https://jackett.${BASE_DOMAIN}
            description: Indexer proxy
            icon: mdi-magnify
        - qBittorrent:
-            href: https://qbittorrent.home.keithjackson.dev
+            href: https://qbittorrent.${BASE_DOMAIN}
            description: Download client (VPN egress)
            icon: mdi-download-network
    - Network & Infra:
        - Pi-hole:
-            href: https://pihole.home.keithjackson.dev
+            href: https://pihole.${BASE_DOMAIN}
            description: Network-wide DNS & ad-blocking
            icon: mdi-shield-check
        - UniFi:
-            href: https://unifi.home.keithjackson.dev:11443
+            href: https://unifi.${BASE_DOMAIN}:11443
            description: Network controller (UOS)
            icon: mdi-router-network
        - Nextcloud:
-            href: https://nextcloud.home.keithjackson.dev
+            href: https://nextcloud.${BASE_DOMAIN}
            description: Files, calendar & contacts
            icon: mdi-cloud
@@ -104,7 +104,7 @@ data:
              href: https://gitea.keithjackson.dev
        - Auth portal:
            - abbr: SSO
-              href: https://auth.home.keithjackson.dev
+              href: https://auth.${BASE_DOMAIN}
    - Reference:
        - Homepage docs:
            - abbr: HP
@@ -54,7 +54,7 @@ spec:
            # REQUIRED by Homepage >= 1.0 — requests whose Host header is not in this list get 403.
            # The public apex host (served via the Gateway) + the pod IP (for kubelet probes).
            - name: HOMEPAGE_ALLOWED_HOSTS
-              value: "home.keithjackson.dev,$(MY_POD_IP):3000"
+              value: "${BASE_DOMAIN},$(MY_POD_IP):3000"
          resources:
            requests:
              cpu: 25m
@@ -28,7 +28,7 @@ spec:
      namespace: gateway
      sectionName: https-apex     # the apex listener (wildcard `https` won't match the bare apex)
  hostnames:
-    - home.keithjackson.dev
+    - ${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -16,7 +16,7 @@ spec:
      namespace: gateway
      sectionName: https
  hostnames:
-    - keycloak.home.keithjackson.dev
+    - keycloak.${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -30,7 +30,7 @@ spec:
    poolMaxSize: 20
  # --- Exposure: TLS at the Gateway; Keycloak serves HTTP and trusts the proxy headers. ---
  hostname:
-    hostname: https://keycloak.home.keithjackson.dev
+    hostname: https://keycloak.${BASE_DOMAIN}
  http:
    httpEnabled: true                      # TLS is terminated at the Gateway (edge), not in Keycloak
  proxy:
@@ -98,9 +98,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:GRAFANA_CLIENT_SECRET)"
        redirectUris:
-          - https://grafana.home.keithjackson.dev/login/generic_oauth
+          - https://grafana.${BASE_DOMAIN}/login/generic_oauth
        webOrigins:
-          - https://grafana.home.keithjackson.dev
+          - https://grafana.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -125,10 +125,10 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:OAUTH2_PROXY_CLIENT_SECRET)"
        redirectUris:
-          - https://*.home.keithjackson.dev/oauth2/callback
+          - https://*.${BASE_DOMAIN}/oauth2/callback
-          - https://auth.home.keithjackson.dev/oauth2/callback
+          - https://auth.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://*.home.keithjackson.dev
+          - https://*.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -209,9 +209,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:GITEA_CLIENT_SECRET)"
        redirectUris:
-          - https://gitea.home.keithjackson.dev/user/oauth2/keycloak/callback
+          - https://gitea.${BASE_DOMAIN}/user/oauth2/keycloak/callback
        webOrigins:
-          - https://gitea.home.keithjackson.dev
+          - https://gitea.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -240,9 +240,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:NEXTCLOUD_CLIENT_SECRET)"
        redirectUris:
-          - https://nextcloud.home.keithjackson.dev/apps/user_oidc/code
+          - https://nextcloud.${BASE_DOMAIN}/apps/user_oidc/code
        webOrigins:
-          - https://nextcloud.home.keithjackson.dev
+          - https://nextcloud.${BASE_DOMAIN}
        # Post-logout redirect allow-list (#logout "redirect failed"). On sign-out user_oidc redirects the
        # browser to Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud base '/'> and
        # client_id=nextcloud but NO id_token_hint (provider sendIdTokenHint=0), so Keycloak validates the
@@ -250,7 +250,7 @@ spec:
        # "redirect failed". The wildcard covers the base URL and any logout landing path. Keycloak stores
        # this as the client attribute `post.logout.redirect.uris` (## separates multiple values).
        attributes:
-          post.logout.redirect.uris: https://nextcloud.home.keithjackson.dev/*
+          post.logout.redirect.uris: https://nextcloud.${BASE_DOMAIN}/*
        defaultClientScopes:
          - email
          - profile
@@ -279,9 +279,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:SEMAPHORE_CLIENT_SECRET)"
        redirectUris:
-          - https://semaphore.home.keithjackson.dev/api/auth/oidc/keycloak/redirect
+          - https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect
        webOrigins:
-          - https://semaphore.home.keithjackson.dev
+          - https://semaphore.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -318,9 +318,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:HEADLAMP_CLIENT_SECRET)"
        redirectUris:
-          - https://headlamp.home.keithjackson.dev/oidc-callback
+          - https://headlamp.${BASE_DOMAIN}/oidc-callback
        webOrigins:
-          - https://headlamp.home.keithjackson.dev
+          - https://headlamp.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -363,9 +363,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:CRAFTY_CLIENT_SECRET)"
        redirectUris:
-          - https://crafty.home.keithjackson.dev/oauth2/callback
+          - https://crafty.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://crafty.home.keithjackson.dev
+          - https://crafty.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -398,9 +398,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:PIHOLE_CLIENT_SECRET)"
        redirectUris:
-          - https://pihole.home.keithjackson.dev/oauth2/callback
+          - https://pihole.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://pihole.home.keithjackson.dev
+          - https://pihole.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -433,9 +433,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:JACKETT_CLIENT_SECRET)"
        redirectUris:
-          - https://jackett.home.keithjackson.dev/oauth2/callback
+          - https://jackett.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://jackett.home.keithjackson.dev
+          - https://jackett.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -459,9 +459,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:QBITTORRENT_CLIENT_SECRET)"
        redirectUris:
-          - https://qbittorrent.home.keithjackson.dev/oauth2/callback
+          - https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://qbittorrent.home.keithjackson.dev
+          - https://qbittorrent.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -485,9 +485,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:RADARR_CLIENT_SECRET)"
        redirectUris:
-          - https://radarr.home.keithjackson.dev/oauth2/callback
+          - https://radarr.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://radarr.home.keithjackson.dev
+          - https://radarr.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -511,9 +511,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:SONARR_CLIENT_SECRET)"
        redirectUris:
-          - https://sonarr.home.keithjackson.dev/oauth2/callback
+          - https://sonarr.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://sonarr.home.keithjackson.dev
+          - https://sonarr.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -30,7 +30,7 @@ spec:
          annotations:
            summary: "KubeVirt VM(s) failing to launch in {{ $labels.namespace }}"
            description: "One or more VirtualMachineInstances are in the Failed phase in namespace {{ $labels.namespace }} for 5m — a failed VM launch (image import, scheduling, or qemu error). Check `kubectl get vmi -n {{ $labels.namespace }}` and the virt-launcher pod events."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-launch-failed"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-launch-failed"
        # Crash-loop: the VM's virt-launcher pod is restarting repeatedly (guest kernel panic, OOM, or a
        # crashing qemu). >3 restarts in 15m is a crash-loop, not a one-off reboot.
        - alert: KubeVirtVMCrashLooping
@@ -42,7 +42,7 @@ spec:
          annotations:
            summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
            description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-crashloop"
        # Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle
        # operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod
        # 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole
@@ -59,7 +59,7 @@ spec:
          annotations:
            summary: "KubeVirt control plane degraded"
            description: "The KubeVirt control plane (virt-controller/virt-api) is not ready or its metrics are absent for 10m — VM lifecycle operations (create/start/stop) will fail (docs/06)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-control-plane"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-control-plane"
        # CDI import stuck: a DataVolume import/clone still in flight after 30m blocks the VM it backs.
        # kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a positive value
        # sustained means an import is not progressing (metric name confirmed against CDI at bring-up).
@@ -71,4 +71,4 @@ spec:
          annotations:
            summary: "CDI DataVolume import stuck: {{ $labels.namespace }}/{{ $labels.name }}"
            description: "A CDI DataVolume import/clone in {{ $labels.namespace }} has not completed after 30m — the VM disk it backs cannot start. Check the importer pod (often an egress/default-deny block on the image source, or a slow mirror)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cdi-import-stuck"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cdi-import-stuck"
@@ -1,9 +1,19 @@
 ---
 # CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs
-# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so —
+# (ADR-0005).
-# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization
+#
-# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in
+# APPLIED BY `platform-config-cdi`, NOT `platform-config` (moved 2026-09-05). These are SECOND-ORDER
-# place from platform-config/kustomization.yaml.
+# CRs: `storageprofiles.cdi.kubevirt.io` is NOT registered by the CDI operator install (cdi-operator.yaml
 # ships only `cdis.cdi.kubevirt.io`) — cdi-operator creates it only AFTER it has reconciled the `CDI` CR
 # that platform-config applies. Flux server-side dry-runs every object in a pass before applying any of
 # them, so co-locating these with that CR was a TRUE DEADLOCK: unknown kind -> pass aborts -> CR never
 # applied -> CRD never created -> platform-config never converges -> every tier below it blocked. It was
 # invisible on dev (CDI CRDs long present) and cost 12h on prod's cold start, 2026-09-05.
 # They live here beside the KubeVirt/CDI component and are referenced in place from
 # platform-config-cdi/kustomization.yaml (dependsOn: platform-config, revision-gated).
 #
 # `apps-vms` dependsOn platform-config-cdi for these: a DataVolume created before the longhorn-vm
 # profile below exists is stamped RWO/Filesystem and is permanently not live-migratable.
 #
 # CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities).
 # The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it
@@ -8,6 +8,16 @@
 # (dependsOn: platform, wait) so the CRDs exist BEFORE the CRs apply. Applying both in one atomic pass
 # deadlocks ("no matches for kind KubeVirt"), exactly the failure F-MON1 documents.
 #
 # THERE IS A THIRD PASS FOR CDI, and it is not optional (2026-09-05). The two operator manifests below
 # register exactly ONE CDI CRD, `cdis.cdi.kubevirt.io`. Every other CDI CRD — `storageprofiles`,
 # `datavolumes`, `datasources`, `dataimportcrons` — is created by cdi-operator only AFTER it reconciles
 # the `CDI` CR. So `cdi-storageprofiles.yaml` is a SECOND-ORDER CR and CANNOT share a pass with
 # `cdi-cr.yaml`: the dry-run of the unknown `StorageProfile` kind aborts the pass, the CR is never
 # applied, the CRD is never created, and platform-config deadlocks (cost 12h on prod's cold start).
 # It is therefore applied by a third Kustomization, `platform-config-cdi` (dependsOn platform-config).
 # Before adding any new CDI CR here, ask which pass owns it: CRD in git -> platform-config;
 # CRD created by cdi-operator at runtime -> platform-config-cdi.
 #
 # Namespaces: the operator manifests each create their own namespace (`kubevirt`, `cdi`) with the
 # pod-security profile they need (kubevirt is privileged) — so we do NOT redeclare them here (that
 # would be a duplicate-resource kustomize error). These are PLATFORM namespaces: they intentionally
@@ -22,14 +22,19 @@ platform/kyverno/
 │   ├── disallow-plaintext-secrets.yaml      # R-SEC-01 (CI-only: sops block required)
 │   ├── restrict-host-namespaces.yaml        # R-SEC-02 (+R-NET-01 bypass guard)
 │   ├── require-egress-class-label.yaml      # R-NET-05
-│   └── require-auth-mode-label.yaml         # R-IDP-02
+│   ├── require-auth-mode-label.yaml         # R-IDP-02
 │   └── label-cdi-worker-pods.yaml           # R-NET-05+R-IDP-02 (mutate: CDI's own worker pods)
 ├── servicedefinition/            # ServiceDefinition shape validation (kyverno-json)
 │   ├── validate-servicedefinition.yaml      # R-GITOPS-02
 │   ├── payloads/ (valid-* / invalid-*)      # pass & fail fixtures
 │   └── README.md
 ├── tests/                        # `kyverno test` suite — runnable L0 tests NOW, no cluster
 │   ├── kyverno-test.yaml                     # policies × fixtures × expected results
-│   └── resources/ (good-* / bad-* fixtures + generated golden)
+│   ├── resources/ (good-* / bad-* fixtures + generated golden)
 │   └── controller-worker-pods/               # the CDI-worker mutation, proven BOTH ways
 │       ├── kyverno-test.yaml                 #   2 Tests: cdi-sa requester vs ordinary requester
 │       ├── userinfo-cdi-sa.yaml / userinfo-ordinary.yaml
 │       └── resources/ + patched/             #   real + FORGED importer pods; expected patch
 ├── admission/                    # in-cluster enforcing bundle (scaffolding, T-2.7)
 │   ├── namespace.yaml · helmrelease.yaml · kustomization.yaml
 │   ├── secret-provenance.yaml                # R-SEC-01 admission variant (provenance)
@@ -47,6 +52,7 @@ platform/kyverno/
 | `restrict-host-namespaces` | `disallow-host-namespaces`, `disallow-privileged-containers` | **R-SEC-02** (+R-NET-01) | docs/07 least reach | App pods can't use hostNetwork/PID/IPC or privileged (would bypass default-deny) |
 | `require-egress-class-label` | `require-egress-class` | **R-NET-05** | docs/03, docs/13 | App pods must declare `egress-class` ∈ {internet,vpn} |
 | `require-auth-mode-label` | `require-auth-mode` | **R-IDP-02** | docs/13, docs/15 | App pods must declare `auth-mode` ∈ {oidc,forward-auth,native,none} |
 | `label-cdi-worker-pods` (mutate) | `add-worker-pod-labels` | **R-NET-05**, **R-IDP-02** | docs/03, docs/13 | CDI's **controller-generated** worker pods (`importer-*`/`uploader-*`/`cloner-*`, created by SA `cdi/cdi-sa`) get `egress-class: internet` + `auth-mode: none` **added**, so they satisfy the two policies above instead of being exempted from them. Matched on the **generating ServiceAccount**, never a pod-name prefix |
 | `validate-servicedefinition` (kyverno-json) | shape / dependencies / storage | **R-GITOPS-02** | docs/13 (+03,15) | ServiceDefinition is schema/invariant valid before render; public⇒not `none` auth |
 | `admission/secret-provenance` | `secrets-must-be-flux-managed` | **R-SEC-01** | docs/07 | **Admission-only** variant: app-ns Secrets must come from Flux/SOPS (see below) |
@@ -90,6 +96,19 @@ self-contained — no cluster, no external state (reproducibility contract, R-TE
 - **Label keys are a proposed contract** (`labels.md`) shared with WS-NET / WS-SEC /
  WS-GITOPS. Values are spec-fixed; keys need owner sign-off (coordination-protocol §2).
 - **Controller-generated pods are a cold-start hazard.** A policy that matches *any* Pod in an
  app namespace also matches pods that no manifest in git authors. Prod hit this on 2026-09-06:
  `require-auth-mode-label` + `require-egress-class-label` denied CDI's own `importer-prime-*`
  pod, so `DataVolume unifi/unifi-rootdisk` stuck in `ImportScheduled` and the UniFi VM had no
  disk. Fix: `policies/label-cdi-worker-pods.yaml` **mutates** those pods to carry truthful
  labels (matched on the generating ServiceAccount `cdi/cdi-sa`), rather than excluding them.
  `virt-launcher-*` pods need nothing — they inherit the VM pod-template labels. KubeVirt
  hotplug (`hp-volume-*`) and export (`virt-export-*`) pods do **not** inherit them and are
  unused here; extend the same policy if either is adopted.
 - **`PolicyException` is NOT usable in this install**: the Kyverno admission/background/reports
  controllers all run `--enablePolicyException=false` (chart default, verified on prod
  2026-09-06). The CRD exists (from the pre-applied CRD bundle) so a `PolicyException` would
  APPLY cleanly and be silently ignored — a policy hole that looks like a fix and isn't.
 - **Secrets CI vs admission divergence** is deliberate and documented in
  `admission/README.md` and `policies/disallow-plaintext-secrets.yaml`.
 - The ServiceDefinition JSON Schema (WS-GITOPS T-0.3) is the complementary structural
@@ -13,6 +13,10 @@ resources:
  - ../policies/restrict-host-namespaces.yaml
  - ../policies/require-egress-class-label.yaml
  - ../policies/require-auth-mode-label.yaml
  # mutate: gives CDI's OWN generated worker pods (importer-*/uploader-*/cloner-*) the two
  # labels the policies above require -- they are controller-generated, so nothing in git can
  # label them. Ships with them, never separately.
  - ../policies/label-cdi-worker-pods.yaml
  # --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
  #     (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
  - secret-provenance.yaml
@@ -15,6 +15,17 @@ Shared contract with WS-NET (egress rendering), WS-SEC (auth), and WS-GITOPS (re
 | `homelab.keithjackson.dev/baseline` | NetworkPolicy | `default-deny` | (convention) identifies the baseline NP | `require-default-deny-baseline` (generate), `validate-default-deny-shape` |
 | `homelab.keithjackson.dev/secret-managed-by` | Secret | `controller` | (convention) exempts controller-generated Secrets | `disallow-plaintext-secrets` (exclude), admission `secret-provenance` |
 ## Controller-generated pods (who sets the labels when git cannot)
 `egress-class` / `auth-mode` are normally projected onto pod templates by the renderer (below).
 Some pods are generated by a **controller at runtime** and therefore have no manifest in git to
 label — CDI's `importer-*` / `uploader-*` / `cloner-*` worker pods are the live example: both
 label policies denied CDI's importer on prod (2026-09-06) and the UniFi VM got no disk. For those,
 `policies/label-cdi-worker-pods.yaml` **adds** the labels at admission (`+(…)` add-if-absent),
 matched on the **generating ServiceAccount** (`cdi/cdi-sa`) — never on a pod-name prefix or on
 CDI's own labels, both of which any pod author in the namespace can copy. `virt-launcher-*` needs
 nothing: KubeVirt propagates the VM pod-template labels onto it (verified on dev).
 ## Rendering expectation (for WS-GITOPS)
 The ServiceDefinition (`docs/13`) declares intent — `egress:` and `auth.mode:` — at the
@@ -0,0 +1,136 @@
 # Serves: R-NET-05 (egress is a declared property) + R-IDP-02 (auth mode is declared) —
 # this policy does NOT weaken either requirement. It makes the one class of pod that
 # CANNOT carry the labels from git SATISFY them, instead of being exempted from them.
 #
 # THE DEFECT IT FIXES (observed on LIVE prod, 2026-09-05/06):
 #   `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled for hours holding a detached
 #   40Gi `prime-…` scratch PVC. `kubectl -n cdi logs deploy/cdi-deployment` showed:
 #     admission webhook "validate.kyverno.svc-fail" denied the request: resource
 #     Pod/unifi/importer-prime-<uid> was blocked due to the following policies
 #       require-auth-mode-label / require-egress-class-label
 #   CDI generates its own short-lived worker pods (importer-*, uploader-*, cloner-*/
 #   clone-source-*) from inside the cdi-deployment controller. Those pods are not in git,
 #   so nothing can put the platform labels on them, and both label policies match ANY Pod
 #   in a namespace labelled homelab.keithjackson.dev/app-namespace=true. Result: the VM
 #   never gets a disk. This is a COLD-START defect, not a prod quirk — dev's identical
 #   import Succeeded before the policies were enforcing, so dev has never exercised it and
 #   would hit it on any rebuild.
 #
 # WHY MUTATE (and not an exception/exclusion):
 #   The honest reading of R-NET-05/R-IDP-02 is "every pod's egress path and auth posture is
 #   DECLARED". A CDI importer pod has truthful values for both:
 #     egress-class: internet  — it pulls the image from the public internet over https
 #                               (cloud-images.ubuntu.com / quay.io) and must take the
 #                               default WAN path, which is exactly what Cilium's egress
 #                               gateway does for `internet` (ADR-0013/ADR-0015). It is NOT
 #                               a `vpn` workload.
 #     auth-mode: none         — the pod serves no UI and terminates when the import ends;
 #                               there is nothing to put an IdP in front of. docs/13 permits
 #                               `none` for internal, non-public workloads (the
 #                               ServiceDefinition policy forbids only public + none).
 #   So the requirement is SATISFIED rather than bypassed: the invariant "every pod in an app
 #   namespace carries both labels" stays literally true cluster-wide, both label policies
 #   stay untouched (no `exclude`, no PolicyException, background scanning still on), and the
 #   labels are functional — the pod is now selectable by the Cilium egress fabric instead of
 #   being an unlabelled pod that silently falls through to the default path.
 #
 # THE MATCH IS THE GENERATING ServiceAccount, NOT A NAME PREFIX:
 #   `subjects: ServiceAccount cdi/cdi-sa` is the identity in the AdmissionReview's userInfo —
 #   the controller that issued the CREATE. Verified on prod from the Kyverno admission log of
 #   the blocked request:
 #     user={"username":"system:serviceaccount:cdi:cdi-sa", ...} roles=["cdi:cdi-deployment"]
 #     name=importer-prime-be3f9d1c-… namespace=unifi operation=CREATE
 #   A pod NAME (`importer-*`) or CDI's own labels (`cdi.kubevirt.io`) are attacker-CHOOSABLE:
 #   anyone who can create a pod in the namespace can name it `importer-prime-x` and copy the
 #   labels, and would then inherit the exemption. The requesting ServiceAccount cannot be
 #   chosen by the pod author — only by someone who already holds cdi-sa's token, which already
 #   confers "create any pod in any namespace" cluster-wide. So this adds no reachable
 #   privilege. It is also VERSION-ROBUST: it covers every CDI worker kind (importer, uploader,
 #   cloner/clone-source) without enumerating names that change between CDI releases —
 #   confirmed on prod that cdi-sa is the ONLY cdi-namespace ServiceAccount that can create
 #   pods (`kubectl auth can-i create pods -n unifi --as …`: cdi-sa yes; cdi-apiserver no;
 #   cdi-uploadproxy no).
 #
 # DELIBERATELY NOT COVERED — KubeVirt's own generated pods:
 #   virt-launcher-* pods INHERIT the VM's pod-template labels, so a VM authored in this repo
 #   already passes (verified on dev: virt-launcher-unifi-22dg7 carries egress-class=internet,
 #   auth-mode=native from apps/vms/unifi/30-virtualmachine.yaml). Exempting
 #   kubevirt-controller would REMOVE that useful enforcement on VM authors, so it is not done.
 #   KubeVirt's hotplug (`hp-volume-*`) and export (`virt-export-*`) pods do NOT inherit those
 #   labels, but nothing in this repo uses volume hotplug or VirtualMachineExport; if either is
 #   adopted, extend this policy then — with the same ServiceAccount-scoped shape.
 #
 # OPERATOR NOTE — this is the cluster's FIRST resource-MUTATING policy. Verified on prod
 #   2026-09-06 that `kyverno-resource-mutating-webhook-cfg` currently holds 0 webhooks; with the
 #   chart's `autoUpdateWebhooks=true` Kyverno registers `mutate.kyverno.svc-fail` for Pods within
 #   seconds of this policy being applied. Prod runs `forceFailurePolicyIgnore=false`, so that
 #   webhook is failurePolicy: Fail — the same posture as the validating webhook that is ALREADY in
 #   the path for these pods, so no new fail-open surface and no new blocking surface. CDI retries
 #   the importer pod on a loop, so the stuck import proceeds once the policy lands (the detached
 #   `prime-…` scratch PVC left behind by the blocked attempts is a separate operator cleanup).
 #
 # RESIDUAL TRADE-OFF (state it plainly, it is reversible by deleting this file):
 #   Anything created by `cdi:cdi-sa` in an app namespace now gets `egress-class: internet` +
 #   `auth-mode: none` supplied for it instead of being rejected for not declaring them. Whoever
 #   holds cdi-sa's token can therefore run a pod in an app namespace without declaring its egress
 #   class. That identity can already create arbitrary pods in every namespace (ClusterRole `cdi`:
 #   pods create/delete cluster-wide), so it is not a new capability — but the declaration is now
 #   assumed rather than demanded for that one identity. Nothing else changes: the pod is still
 #   subject to restrict-host-namespaces, the namespace default-deny baseline, and every other
 #   policy, and `internet` is the LEAST-reach of the two egress classes (it does not grant the VPN
 #   path).
 #
 # PROOF (offline, no cluster — platform/kyverno/tests/controller-worker-pods/):
 #   (a) a CDI importer pod created by cdi:cdi-sa is mutated and then PASSES both policies;
 #   (b) the SAME namespace's ordinary pod — including one that FORGES the importer name and
 #       CDI's labels — is NOT mutated and is still DENIED by both policies.
 apiVersion: kyverno.io/v1
 kind: ClusterPolicy
 metadata:
  name: label-cdi-worker-pods
  annotations:
    policies.kyverno.io/title: Label CDI-generated worker pods
    policies.kyverno.io/category: Network Isolation
    policies.kyverno.io/severity: medium
    homelab.keithjackson.dev/requirements: R-NET-05,R-IDP-02
    policies.kyverno.io/description: >-
      Adds homelab.keithjackson.dev/egress-class=internet and auth-mode=none to the
      short-lived worker Pods that the CDI controller (ServiceAccount cdi/cdi-sa)
      generates in application namespaces, so they satisfy require-egress-class-label
      and require-auth-mode-label instead of being exempted from them.
 spec:
  # No validationFailureAction: this is a MUTATE policy, there is nothing to fail.
  # background MUST be false: `match.subjects` is userInfo, which exists only on an
  # AdmissionReview. Kyverno rejects a userInfo match with background scanning enabled
  # ("invalid variable used at path ... policy contains 'userInfo'"), and a background scan
  # of an already-created pod has no requester to match anyway.
  background: false
  rules:
    - name: add-worker-pod-labels
      match:
        # One filter entry: the resource conditions AND the subject must both match.
        all:
          - resources:
              kinds:
                - Pod
              operations:
                - CREATE
              namespaceSelector:
                matchLabels:
                  homelab.keithjackson.dev/app-namespace: "true"
            subjects:
              # The CDI controller (cdi-deployment) ServiceAccount — the identity that
              # CREATEs importer-*/uploader-*/cloner-* pods. Not the pods' own SA.
              - kind: ServiceAccount
                namespace: cdi
                name: cdi-sa
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              # `+(key)` is Kyverno's ADD-IF-ABSENT anchor: an explicit value already on the
              # pod is never overwritten (so a future CDI worker that legitimately needs
              # egress-class: vpn can be given it, and virt-launcher-style inherited labels
              # would survive).
              +(homelab.keithjackson.dev/egress-class): internet
              +(homelab.keithjackson.dev/auth-mode): none
@@ -0,0 +1,91 @@
 # L0 policy-as-code suite for the CDI worker-pod label mutation — runs in CI WITHOUT a cluster
 # (discovered recursively by the gate's `kyverno test platform/kyverno`; ci/l0.sh).
 #
 # WHY A SEPARATE SUITE from ../kyverno-test.yaml: a `kyverno test` manifest carries ONE
 # `userinfo`, and the whole point here is that the SAME pods get different outcomes for
 # different requesters. Two Test documents, two identities, one file:
 #   1. cdi-controller-worker-pods-admitted  (requester = cdi/cdi-sa)     -> mutated, ADMITTED
 #   2. cdi-worker-exemption-is-narrow       (requester = ordinary user)  -> not mutated, DENIED
 # Keeping the mutate policy OUT of ../kyverno-test.yaml is deliberate: that suite declares no
 # userinfo, and with no userinfo the offline engine does not discriminate on `subjects`, which
 # would silently turn its negative (result: fail) rows into passes.
 apiVersion: cli.kyverno.io/v1alpha1
 kind: Test
 metadata:
  name: cdi-controller-worker-pods-admitted
 # Namespace labels for offline namespaceSelector resolution (shared with the main suite).
 variables: ../values.yaml
 userinfo: userinfo-cdi-sa.yaml
 policies:
  - ../../policies/label-cdi-worker-pods.yaml
  - ../../policies/require-egress-class-label.yaml
  - ../../policies/require-auth-mode-label.yaml
 resources:
  - ../resources/namespaces.yaml
  - resources/cdi-worker-pods.yaml
 results:
  # ---- the mutation fires for the CDI controller and adds exactly the two labels ----
  - policy: label-cdi-worker-pods
    rule: add-worker-pod-labels
    kind: Pod
    resources:
      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
    patchedResources: patched/importer-mutated.yaml
    result: pass
  # ---- R-NET-05 + R-IDP-02: the mutated CDI worker pod now SATISFIES both policies, so the
  #      import CDI could not start is admitted. (The engine validates the mutated resource.)
  - policy: require-egress-class-label
    rule: require-egress-class
    kind: Pod
    resources:
      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
    result: pass
  - policy: require-auth-mode-label
    rule: require-auth-mode
    kind: Pod
    resources:
      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
    result: pass
 ---
 apiVersion: cli.kyverno.io/v1alpha1
 kind: Test
 metadata:
  name: cdi-worker-exemption-is-narrow
 variables: ../values.yaml
 userinfo: userinfo-ordinary.yaml
 policies:
  - ../../policies/label-cdi-worker-pods.yaml
  - ../../policies/require-egress-class-label.yaml
  - ../../policies/require-auth-mode-label.yaml
 resources:
  - ../resources/namespaces.yaml
  - resources/cdi-worker-pods.yaml
 results:
  # ---- the mutation does NOT fire for an ordinary requester, not even for a pod that forges
  #      the importer's name and CDI's labels ----
  - policy: label-cdi-worker-pods
    rule: add-worker-pod-labels
    kind: Pod
    resources:
      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
      - importer-prime-forged-by-an-app
      - plain-app-pod-no-labels
    result: skip
  # ---- so every unlabelled pod in the SAME app namespace is still DENIED (R-NET-05) ----
  - policy: require-egress-class-label
    rule: require-egress-class
    kind: Pod
    resources:
      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
      - importer-prime-forged-by-an-app
      - plain-app-pod-no-labels
    result: fail
  # ---- ... and by R-IDP-02 ----
  - policy: require-auth-mode-label
    rule: require-auth-mode
    kind: Pod
    resources:
      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
      - importer-prime-forged-by-an-app
      - plain-app-pod-no-labels
    result: fail
@@ -0,0 +1,16 @@
 # Expected result of the mutation: CDI's own labels untouched, the two platform labels ADDED.
 apiVersion: v1
 kind: Pod
 metadata:
  name: importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
  namespace: app-ns
  labels:
    app: containerized-data-importer
    cdi.kubevirt.io: importer
    homelab.keithjackson.dev/egress-class: internet
    homelab.keithjackson.dev/auth-mode: none
 spec:
  restartPolicy: OnFailure
  containers:
    - name: importer
      image: quay.io/kubevirt/cdi-importer:v1.62.0