feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite

OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -1496,3 +1496,265 @@ thing that ever closes a requirement.
   chart default rather than set explicitly — see the follow-up in `platform/longhorn/`.
 - L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
   the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.
+
+---
+
+## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.**
+
+Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put
+`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes**
+and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down).
+Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold
+prod bootstrap than it was here.
+
+### What happened
+
+The HelmRelease config digest tells the whole story:
+
+```
+v8   05:49:39  digest e83bf695…      <- before
+v9   05:50:11  digest eebd6c88…      <- the bad render
+v10  05:55:23  digest e83bf695…      <- identical to v8 again
+```
+
+The config changed and then changed **back to byte-identical**. Nothing in git differed
+semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus
+`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets
+`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady
+state is identical. The damage happened in a transient intermediate state.
+
+### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations
+
+- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization.
+- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`**
+  Kustomization, via `postBuild.substituteFrom: cluster-vars`.
+- There is **no dependency ordering between defining a variable and using it.**
+
+So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`,
+which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an
+undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm
+upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` —
+which only the Cilium agent itself can translate. The agent was the thing restarting, so its own
+`config` init container could not reach the API server:
+
+```
+level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443
+  error="dial tcp 10.43.0.1:443: i/o timeout"
+level=fatal msg="Build config failed"
+```
+
+Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the
+correct value, digest returned to `e83bf695…`, and all three agents came up.
+
+`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires
+the ConfigMap to *exist*, not that every referenced key is defined.
+
+### Why this matters much more for prod
+
+On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the
+race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**,
+there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may
+never come up at all — with no healthy peer to recover from. This is a plausible
+bring-up-blocker for `clusters/prod`.
+
+### The fix — make substitution FAIL CLOSED
+
+`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict
+substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail
+the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine
+("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts
+this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile
+failure*.
+
+**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization
+that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in
+embedded scripts. The safe sequence, before flipping it:
+1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in
+   `cluster-vars` or correctly escaped as `$${VAR}`.
+2. Then enable the gate, and let CI + a dev reconcile prove it.
+
+Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and
+reference it in a later one**, so the definition is always reconciled before its first use.
+
+### Operational note worth keeping
+
+A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the
+`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is
+set, and the agent that translates that ClusterIP is the one being replaced. Keep
+`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time.
+
+---
+
+## 2026-09-01 — RESEARCH: Flux substitution + HelmRelease best practice. **Measured semantics + ranked plan.**
+
+Three parallel research passes (substitution/ordering, HelmRelease/CNI, and a full repo audit),
+prompted by the incident above. Two of my own earlier conclusions were WRONG and are corrected
+here. Everything below marked *measured* was run against flux 2.4.0 / the live cluster; everything
+else is cited inference.
+
+### THE SUBSTITUTION SEMANTICS TABLE (measured — trust this over prose elsewhere)
+
+| form | behaviour |
+|---|---|
+| `${DEFINED}` | substituted |
+| `${UNDEFINED}` | **silently empty** (non-strict) · fails loudly (strict) |
+| `${UNDEFINED:-d}` | **default applied at reconcile time — `--strict` does NOT catch it** |
+| `${VAR:?msg}` | renders **the message as the value** — a non-empty WRONG value. ANTI-PATTERN. |
+| `${VAR:=192.168.1.50}` | silently ships the dev value to prod. ANTI-PATTERN. |
+| `$${VAR}` / `$${VAR:-d}` / `$${ARR[0]}` | literal — **escaping is TOTAL** |
+| `$VAR` / `$(cmd)` | untouched, no escaping needed |
+| `${...}` / `${a b}` / `${ARR[0]}` unescaped | **hard parse error — fails the whole Kustomization** |
+
+Flux exempts default-providing operators (`-`, `:-`, `+`, `:+`, `=`, `:=`, `?`, `:?`) from strict
+mode by design ("they intentionally handle the unset case"). **That is why strict mode would never
+have caught the gitea/nextcloud dead-guard bug**, and why a targeted check was required.
+
+**Doctrine corrected:** `clusters/base/apps-vms.yaml` claimed an indexed expansion is fatal
+"escaped or not". Measured: escaping is total; only the UNESCAPED form is fatal. Fixed in place.
+
+**Comment nuance:** `kustomize build` strips *structural* YAML comments before envsubst sees them,
+so those are usually harmless. But `#` lines INSIDE a block scalar are data and DO reach envsubst,
+and a `$`-carrying body comment was observed surviving into built `apps/wave-1`. Do not rely on
+stripping — escape in comments too. (An earlier claim in this log that a YAML comment "nearly broke
+every merge" was overstated for the structural case; the hazard is real for block-scalar bodies.)
+
+### CORRECTION 1 (mine) — `StrictPostBuildSubstitutions` alone is INSUFFICIENT
+
+It fixes nothing about ordering; it converts "render a corrupt value and ship it" into "fail loudly
+and keep last-good". **The actual ordering fix is one line**, verified against
+`kustomization_controller.go :: checkDependencies` at our exact version (v1.4.0): `dependsOn` is
+**revision-gated**, not merely Ready-gated — when the dependency shares a GitRepository, the
+dependent cannot reconcile revision N until the dependency has APPLIED revision N.
+
+`platform` is the only Kustomization in the graph with no `dependsOn`; every other consumer chains
+transitively through it. So:
+
+```yaml
+# clusters/base/platform.yaml
+spec:
+  dependsOn:
+    - name: flux-system     # cluster-vars must be applied at THIS revision first
+```
+
+**Cost, recorded honestly:** the root becomes a gate — a malformed manifest anywhere under
+`clusters/dev` now stops the whole graph where today it would not. Fail-closed, consistent with
+doctrine, but a real blast-radius change. No deadlock risk: the root runs `wait: false`.
+
+Rejected alternative: moving `cluster-vars` into the consuming Kustomization is **strictly worse** —
+`SubstituteVariables` reads the ConfigMap from the API at build time, *before* the Kustomization
+applies its own resources, so it would always read the PREVIOUS value. That converts an occasional
+race into a permanent one-reconcile lag.
+
+### CORRECTION 2 (an agent's) — the safety machinery DID fire
+
+A research pass concluded that Helm considered the bad release healthy (because
+`updateStrategy.rollingUpdate.maxUnavailable: 2` on 3 nodes means `expectedReady = 1`), and
+therefore remediation never engaged. **Measured on the live cluster: that is wrong.** The Helm
+release secrets show **v9 = `failed`**; Helm's wait DID fail and the release was remediated, which
+is why it self-healed. `maxUnavailable: 2 of desired 3` IS confirmed live — so the real harm was
+blast radius (2 of 3 agents down at once), not a missed detection.
+
+### Verified, and worth knowing
+
+- **Flux 2.4.0 is OUT OF SUPPORT** — the project supports the last three minors; 2.9.x is current.
+- **Strict substitution became DEFAULT-ON** in kustomize-controller v1.9.0. So upgrading Flux flips
+  this behaviour whether or not we opt in — the variable audit is a prerequisite for the upgrade
+  regardless of today's decision.
+- **Cilium's `values.schema.json` permits `k8sServiceHost: ""`** — no `minLength`, no `pattern`.
+  Helm was never going to reject the bad render.
+- **Cilium docs make `k8sServiceHost` REQUIRED under kube-proxy replacement**, precisely because the
+  environment exports the API ClusterIP and there is no kube-proxy to provision it.
+- **The Cilium chart ships zero test templates**, so `spec.test` is useless for it. A post-upgrade
+  check must live outside Helm.
+- `valuesFrom` fails HARD on a missing key (verified in `chartutil/values.go`) unlike postBuild's
+  silent empty — but only on a MISSING key, not an empty value, and it cannot cross namespaces
+  (`cluster-vars` is in `flux-system`; the Cilium HelmRelease is in `kube-system`).
+- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** It compares the cluster against the
+  Helm storage manifest — during the incident that manifest WAS the bad render, so drift correction
+  would have actively re-asserted the broken value.
+
+### RANKED PLAN
+
+| P | Action | State |
+|---|---|---|
+| **P0** | `platform.dependsOn: [flux-system]` — closes the race completely | **PENDING — operator call** (blast-radius change) |
+| **P0** | Escape the dead OIDC guards; renderer now refuses the class | ✅ DONE |
+| **P1** | Substitution-safety L0 gate, unconditional in `ci/l0.sh` | ✅ DONE — 102 assertions |
+| **P1** | Enable `StrictPostBuildSubstitutions=true` | **READY — proven safe**, all 11 paths exit 0 under `flux envsubst --strict`. Sequence AFTER P0. |
+| **P2** | `updateStrategy.rollingUpdate.maxUnavailable: 1` on Cilium, as a new **[parity]** field in BOTH the HelmRelease and the Ansible j2 | PENDING — bounds blast radius to one node |
+| **P3** | Split Cilium into its own Kustomization (`wait: true`, `interval: 1h`, `timeout: 10m`); `platform` dependsOn it | PENDING — gives `flux suspend kustomization cilium` as an emergency valve |
+| **P4** | Upgrade Flux 2.4.0 → 2.9.x | PENDING — out of support; brings strict-by-default |
+| **P5** | Cilium MINOR bumps become manual (Renovate PR → human runs the documented pre-flight → merge). Never automerge a CNI minor. | PENDING — process |
+| — | Record `${VAR:?}` / `${VAR:=}` as anti-patterns | ✅ DONE (table above) |
+| — | *Not recommended*: drift detection on Cilium; `spec.test` on Cilium; `k8sServiceHost: auto` | — |
+
+### Follow-ups surfaced, not yet actioned
+- 6 dead `cluster-vars` knobs; `BASE_DOMAIN` and `CLUSTER_ENV` look like genuine rot.
+- `platform/{cilium,longhorn,velero,oauth2-proxy}/values.yaml` reference mirrors carry decorative
+  `${...}` that can silently drift from the real HelmRelease.
+- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
+  out while a legitimate Longhorn upgrade is still running.
+
+---
+
+## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
+
+All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
+
+### Landed and verified on the live cluster
+
+| Change | Verified |
+|---|---|
+| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
+| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
+| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
+| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
+| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
+
+### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
+
+Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
+`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
+
+```sh
+[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"   # LAST line of the hook
+```
+
+On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
+the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
+
+**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
+long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
+execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
+**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
+dead code is not a safe no-op — it is the first execution of never-tested code, and should be
+treated with the caution of new code, not of a one-character escape.
+
+Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
+on both paths, but only by accident of which statement happened to be last — made explicit there too
+so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
+Kustomizations Ready, both OIDC providers still registered.
+
+### STILL TO DO — strict substitution is committed but NOT ACTIVE
+
+`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
+`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
+ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
+controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
+standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
+~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
+
+**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
+With the gate on but the ordering edge absent, the same race would turn a transient empty render
+into a transient hard Kustomization failure.
+
+### Prod consequences recorded this session
+
+- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
+  soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
+  first" safety net that the last two days have leaned on heavily, twice.
+- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
+  currently points at Gitea running on the dev cluster.
+- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
+  with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
+  Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.

apps/vms/unifi/30-virtualmachine.yaml

@@ -92,14 +92,17 @@ spec:
             # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
             # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
             # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
-            # as a literal ${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
+            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
-            # by envsubst and need no escaping. NEVER write an unescaped ${...} here, and never write
+            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
-            # ${ARR[0]} even escaped-adjacent: envsubst hard-fails on `[` ("missing closing brace")
+            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
-            # and would break the whole apps-vms apply.
+            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
+            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
+            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
+            # wording in clusters/base/apps-vms.yaml overstated it.
             userData: |-
               #cloud-config
               hostname: unifi
-              fqdn: unifi.home.keithjackson.dev
+              fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
               packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]

apps/vms/unifi/50-service-l4.yaml

@@ -27,7 +27,7 @@
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
 # Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
-# exactly this token; every shell ${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
+# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
 # accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
 #
 # REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an
@@ -46,7 +46,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
     # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
     # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
-    external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Local

apps/wave-0/gitea.yaml

@@ -90,9 +90,9 @@ spec:
               name: gitea-bootstrap
               key: secret-key
         - name: GITEA__server__DOMAIN
-          value: gitea.home.keithjackson.dev
+          value: gitea.${BASE_DOMAIN}
         - name: GITEA__server__ROOT_URL
-          value: https://gitea.home.keithjackson.dev/
+          value: https://gitea.${BASE_DOMAIN}/
         - name: GITEA__actions__ENABLED
           value: 'true'
         - name: GITEA__actions__DEFAULT_ACTIONS_URL
@@ -127,8 +127,8 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1; break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
-                \ groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n"
+                \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -207,7 +207,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - gitea.home.keithjackson.dev
+  - gitea.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/jackett.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:9117
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - jackett.home.keithjackson.dev
+  - jackett.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/pihole.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:80
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -229,7 +229,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - pihole.home.keithjackson.dev
+  - pihole.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/plex.yaml

@@ -183,7 +183,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - plex.home.keithjackson.dev
+  - plex.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-0/qbittorrent.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - qbittorrent.home.keithjackson.dev
+  - qbittorrent.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/gitea-actions.yaml

@@ -165,7 +165,7 @@ spec:
               exec /sbin/tini -- /opt/act/run.sh
           env:
             - name: GITEA_INSTANCE_URL
-              value: https://gitea.home.keithjackson.dev
+              value: https://gitea.${BASE_DOMAIN}
             - name: GITEA_RUNNER_REGISTRATION_TOKEN
               valueFrom:
                 secretKeyRef:

apps/wave-1/nextcloud.yaml

@@ -104,13 +104,13 @@ spec:
               name: nextcloud-bootstrap
               key: admin-password
         - name: NEXTCLOUD_TRUSTED_DOMAINS
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEHOST
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEPROTOCOL
           value: https
         - name: OVERWRITECLIURL
-          value: https://nextcloud.home.keithjackson.dev
+          value: https://nextcloud.${BASE_DOMAIN}
         - name: NEXTCLOUD_OIDC_CLIENT_SECRET
           valueFrom:
             secretKeyRef:
@@ -125,8 +125,9 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra\
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc]\
-                \ positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
+                \ client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
+                ); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -227,7 +228,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - nextcloud.home.keithjackson.dev
+  - nextcloud.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/radarr.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:7878
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - radarr.home.keithjackson.dev
+  - radarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/semaphore.yaml

@@ -110,7 +110,7 @@ spec:
         - name: SEMAPHORE_ADMIN_NAME
           value: Semaphore Admin
         - name: SEMAPHORE_ADMIN_EMAIL
-          value: admin@home.keithjackson.dev
+          value: admin@${BASE_DOMAIN}
         - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
           valueFrom:
             secretKeyRef:
@@ -231,7 +231,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - semaphore.home.keithjackson.dev
+  - semaphore.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-1/sonarr.yaml

@@ -91,16 +91,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
         - --skip-provider-button=true
-        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - sonarr.home.keithjackson.dev
+  - sonarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

apps/wave-2/seerr.yaml

@@ -223,7 +223,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - seerr.home.keithjackson.dev
+  - seerr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:

bootstrap/flux/README.md

@@ -11,11 +11,69 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
 ## Files
 | File | Purpose |
 |---|---|
-| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
 | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
 | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
 | `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
 
+## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
+
+Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
+and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
+
+```
+$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
+{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
+ "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
+# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
+```
+
+No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
+`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
+documents for hand-installed clusters ([Customize Flux
+manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
+
+```
+flux install --export …  ->  <tmp>/gotk-components.yaml     # regenerated from the pinned flux CLI
+components/kustomization.yaml -> <tmp>/kustomization.yaml   # the only hand-authored part (in git)
+kubectl kustomize <tmp> | kubectl apply -f -
+```
+
+Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
+(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
+more feature gates) is another entry in `components/kustomization.yaml`.
+
+### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
+
+> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
+> controller Deployment is outside Flux's inventory. Someone must run step 1.
+
+Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
+
+```bash
+# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
+d=$(mktemp -d)
+flux install --export --namespace=flux-system \
+  --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
+cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
+kubectl kustomize "$d" | kubectl apply -f -      # add --server-side --force-conflicts on a field-manager conflict
+kubectl -n flux-system rollout status deploy/kustomize-controller
+```
+
+**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
+reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
+is lost. Confirm afterwards:
+
+```bash
+kubectl -n flux-system get deploy kustomize-controller \
+  -o jsonpath='{.spec.template.spec.containers[0].args}'   # must contain --feature-gates=StrictPostBuildSubstitutions=true
+flux get kustomizations -A                                  # all 11 must return to Ready
+```
+
+**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
+simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
+
 ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
 
 The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):

bootstrap/flux/bootstrap.sh

@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #
-# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
+# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
+# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
+# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
+# same script bootstraps dev and prod with no fork (R-ENV-02).
 #
 # Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
 # =============================================================================
@@ -33,7 +36,7 @@
 #   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
+#   3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
 set -euo pipefail
 
 # --- config (env-overridable) ------------------------------------------------
-FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"     # dev is the ONLY Phase-1 build target
+# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
+# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
+# every existing caller — override it, never edit it.
+FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
 FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
 SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
 # The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,23 +68,66 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
 
-echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
+echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
 
 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
 for bin in flux kubectl; do
   command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
 done
 
-# --- 1. install Flux controllers ---------------------------------------------
+# --- 1. install Flux controllers (rendered through ./components) --------------
 # Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
 # ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
 # image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
-if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then
+#
-  echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})"
+# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
-  flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}"
+# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
+# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
+# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
+#
+# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
+# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
+# documents for hand-installed clusters ("Customize Flux manifests"):
+#
+#     flux install --export  >  gotk-components.yaml     # exact manifests `flux install` would apply
+#     kustomize overlay      =  ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
+#     kubectl apply -k
+#
+# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
+# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
+# a temp dir — the repo tree is never written to.
+#
+# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
+# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
+echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
+flux_install_args=(--namespace="${FLUX_NAMESPACE}")
+[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
+
+if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
+  echo "    WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
+  echo "             StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
+  flux install "${flux_install_args[@]}"
 else
-  echo "==> [1/3] flux install (default controllers only)"
+  components_dir="${script_dir}/components"
-  flux install --namespace="${FLUX_NAMESPACE}"
+  [ -f "${components_dir}/kustomization.yaml" ] || {
+    echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
+    echo "       (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
+    exit 1
+  }
+  render_dir="$(mktemp -d)"
+  trap 'rm -rf "${render_dir}"' EXIT
+  echo "    rendering flux install --export through ${components_dir}"
+  flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
+  cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
+  # Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
+  # version bump) — better a hard stop than silently installing without the feature gate.
+  kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
+  grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
+    echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
+    echo "       Check ./components/kustomization.yaml against this flux CLI version." >&2
+    exit 1
+  }
+  kubectl apply -f "${render_dir}/rendered.yaml"
 fi
 
 # --- 2. verify the out-of-band age key Secret exists -------------------------
@@ -91,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
   exit 1
 fi
 
-# --- 3. apply the authored dev entrypoint ------------------------------------
+# --- 3. apply the authored environment entrypoint ----------------------------
 echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
 

bootstrap/flux/components/kustomization.yaml

@@ -0,0 +1,64 @@
+---
+# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments).
+#
+# WHY THIS DIRECTORY EXISTS
+# =========================
+# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the
+# reconciliation graph is authored by hand in clusters/base, and the controllers are installed
+# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01:
+#
+#     kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
+#     -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1
+#
+# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no
+# existing mechanism to customize them, and no amount of editing under clusters/** can reach them.
+#
+# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" —
+# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests
+# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result.
+# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported
+# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the
+# repo, so the manifests can never drift from the pinned FLUX_VERSION
+# (infra/devcontainer/versions.env) the way a vendored copy would.
+#
+# Every future controller customization (vertical scaling, tolerations, extra feature gates,
+# --concurrent) belongs HERE, as another entry under `patches:`.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  # Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in.
+  - gotk-components.yaml
+patches:
+  # -------------------------------------------------------------------------------------------
+  # StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED.
+  #
+  # Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the
+  # EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium
+  # HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this
+  # gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state.
+  #
+  # PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with
+  # `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` and
+  # exited 0, with a negative control reproducing the incident. The permanent guard that keeps it
+  # true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/.
+  #
+  # NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) —
+  # Flux exempts them by design. That class is what the L0 gate covers; the two are complementary.
+  #
+  # This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in
+  # kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in.
+  #
+  # JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic
+  # list, so a strategic-merge patch would REPLACE every arg `flux install` generates
+  # (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election)
+  # and silently drop any the CLI adds in a future version.
+  - target:
+      group: apps
+      version: v1
+      kind: Deployment
+      name: kustomize-controller
+      namespace: flux-system
+    patch: |-
+      - op: add
+        path: /spec/template/spec/containers/0/args/-
+        value: --feature-gates=StrictPostBuildSubstitutions=true

bootstrap/gitea/bootstrap-gitea.sh

@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 # bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
-# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
+# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
-# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
+# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
-# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
+# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
-# CONTRIBUTING.md).
+# using it now.
 #
-# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
+# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
+# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
+#
+#   PROD (bare metal, current use):  HOST_NAT_IP=192.168.2.132
+#       The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
+#       NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
+#       three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
+#       apply to prod, so do not "fix" prod by opening zones.
+#
+#   DEV (libvirt VMs, HISTORICAL):   HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
+#       *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
+#       bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
+#       recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
+#
+# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
+# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
+# author-only work and must never be fired off casually or "to see if it works": it starts a
+# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
+# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
+# it will then reconcile. Know which HEAD you are publishing before you run it.
 set -euo pipefail
 ENGINE="${ENGINE:-podman}"
 GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}"   # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
 # subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
 DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
 DATA_DIR="podman volume ${DATA_VOLUME}"   # display label only (data lives in the named volume)
-# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
+# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
 # clone/redirect URLs are correct from inside the cluster.
-# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
+#
+# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
+# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
+# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
+# Pass it explicitly, e.g. for prod:   HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
+#
+# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
+# VM->host:3000 by default and had to be opened once with
 #   sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
-HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
+# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
+if [ -z "${HOST_NAT_IP:-}" ]; then
+  echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
+  echo "       prod: HOST_NAT_IP=192.168.2.132   (same LAN as the nodes, no NAT, no firewall change)" >&2
+  echo "       There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
+  exit 1
+fi
 ORG="${GITEA_ORG:-Home-Network}"
 REPO="${GITEA_REPO:-home-network-k3s-impl}"
 
@@ -75,8 +107,17 @@ cat <<EOF
 
 Gitea bootstrap source ready (${DATA_DIR}).
   UI (host):                     ${BASE}/${ORG}/${REPO}
-  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git   (clusters/dev)
+  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
+                                 (patch this url into the overlay being bootstrapped:
+                                  clusters/prod today; clusters/dev has already cut over)
   Admin creds (host, 0600):      ${ADMIN_CRED_FILE}
-Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
+Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
+firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
+must allow VM->host:${HTTP_PORT} —
   ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
+REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
+at this source will reconcile THAT revision.
+TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
+any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
+procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
 EOF

catalog/README.md

@@ -28,6 +28,16 @@ inherit every guarantee automatically.
   `auth.groups`; `native`/`none` are documented exceptions only.
 - Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster
   schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`.
+- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a
+  `lifecycle` postStart script, an `env` value — lands under `apps/wave-<N>/`, and every
+  `apps-wave-<N>` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs
+  envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the
+  *shell* is replaced with the empty string before the container ever starts (this silently killed
+  gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and
+  need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and
+  written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml).
+  `render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode
+  cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms).
 
 ## Validate before commit — single command, no cluster (R-TEST-02)
 ```bash

catalog/gitea.yaml

@@ -24,9 +24,9 @@ env:
         name: gitea-bootstrap
         key: secret-key
   - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.home.keithjackson.dev
+    value: gitea.${BASE_DOMAIN}
   - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
   - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
     value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
   - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,12 +104,24 @@ lifecycle:
           # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
           # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
           # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           gitea_oidc_fail() {
             echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
           }
-          if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
+          # NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
+          # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
+          # time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
+          # a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
+          # — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
+          # every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
+          # already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
+          # Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
+          # `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
+          # Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
+          # `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
+          # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
+          if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
             echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
           else
             # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
@@ -150,7 +162,13 @@ lifecycle:
               gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
             fi
           fi
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+          # A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
+          # it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
+          # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
+          # the moment its OIDC guard was fixed and the block actually ran for the first time.
+          exit 0
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: data
@@ -160,7 +178,7 @@ storage:
                                 # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
     port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)

catalog/jackett.yaml

@@ -17,7 +17,7 @@ name: jackett
 kind: container
 source:
   image: lscr.io/linuxserver/jackett:latest
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
@@ -25,7 +25,7 @@ storage:
     class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
     port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)

catalog/keycloak.yaml

@@ -18,7 +18,7 @@ dependencies:
     port: 5432
     protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
     port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
   mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.

catalog/nextcloud.yaml

@@ -50,13 +50,13 @@ env:
   #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
   #     though it speaks plain HTTP behind the proxy. ---
   - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEPROTOCOL
     value: https
   - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
   # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
   #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
   #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
           # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
           # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
           OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           nextcloud_oidc_fail() {
             echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -111,7 +111,19 @@ lifecycle:
           # every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.
           OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \
             || echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2
-          if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
+          # NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild.
+          # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
+          # time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is
+          # not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS
+          # TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never
+          # installing/enabling user_oidc nor registering the provider. SSO kept working only because
+          # the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with
+          # NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator
+          # forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion
+          # for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why
+          # `--clientsecret-env` and the bare uses below always worked and only the guard broke.
+          # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
+          if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
             echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)"
             exit 0
           fi
@@ -191,7 +203,17 @@ lifecycle:
             [ "$missing" = 0 ] && break
             p=$((p + 1)); sleep 2
           done
-          [ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          if [ "$p" -ge 15 ]; then
+            nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          fi
+          # A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
+          # the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
+          # SUCCESS path the test returned 1 and became the script's exit status. That was latent for
+          # as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
+          # exposed it immediately. Every failure path here logs loudly and writes a marker file and
+          # deliberately does NOT crash the pod — the break-glass local admin must stay usable.
+          exit 0
 exposure: public                # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
 storage:
@@ -206,7 +228,7 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
     port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/pihole.yaml

@@ -14,7 +14,7 @@ storage:
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)

catalog/plex.yaml

@@ -72,7 +72,7 @@ storage:
                               # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
     port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
   mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/qbittorrent.yaml

@@ -32,7 +32,7 @@ name: qbittorrent
 kind: container
 source:
   image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
@@ -40,7 +40,7 @@ storage:
     class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
     port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)

catalog/radarr.yaml

@@ -9,7 +9,7 @@ name: radarr
 kind: container
 source:
   image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)

catalog/render.py

@@ -29,6 +29,7 @@ conformance harness stays stdlib-only and only shells out to this script + diffs
 from __future__ import annotations
 
 import argparse
+import re
 import sys
 from pathlib import Path
 
@@ -43,7 +44,15 @@ except ImportError:  # pragma: no cover - explicit, actionable failure
 
 # --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
 LBL = "homelab.keithjackson.dev"
-ZONE = "home.keithjackson.dev"
+# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
+# Every rendered manifest lands under a Flux path whose Kustomization declares
+# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
+# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
+# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
+# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
+# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
+# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
+ZONE = "${BASE_DOMAIN}"
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
@@ -936,6 +945,124 @@ def r_virtualmachine(s: dict) -> dict:
     }
 
 
+# =============================================================================
+# Substitution safety (R-ENV-02 / R-GITOPS-02) — refuse to emit a manifest that
+# Flux's envsubst would silently eat before the workload ever runs.
+# =============================================================================
+# Every manifest this renderer writes lands under apps/wave-<N>/, and EVERY apps-wave-<N>
+# Flux Kustomization declares `postBuild.substituteFrom: cluster-vars` (see
+# flux_wave_kustomization_text below). Flux therefore runs envsubst over the WHOLE rendered
+# document — including any shell script carried verbatim in a `lifecycle` hook or an env
+# value — at reconcile time, long before a container sees it. The escape contract is stated
+# in clusters/base/apps-vms.yaml and proven with `flux envsubst`:
+#
+#   `${VAR}`   -> SUBSTITUTED (empty when unset)   => a shell brace must be written `$${VAR}`,
+#                                                     which envsubst emits back as `${VAR}`.
+#   `$VAR`     -> left alone                       => bare shell vars need NO escaping.
+#   `$(cmd)`   -> left alone                       => command substitution needs NO escaping.
+#   `${A[0]}`  -> HARD ERROR ("missing closing brace"), failing the whole Kustomization.
+#
+# This bit us for real (fixed 2026-09-01): gitea/nextcloud guarded their whole Keycloak OIDC
+# registration with `if [ -z "${..._OIDC_CLIENT_SECRET:-}" ]`. Neither name is a cluster-var,
+# so envsubst rendered the guard as `if [ -z "" ]` — ALWAYS TRUE — and every boot skipped the
+# registration entirely. Flux's STRICT substitution mode cannot catch this class: it
+# deliberately EXEMPTS the default-operator forms (`${V:-}`, `${V:=x}`, `${V:?}`), which is
+# exactly the form a defensive shell script uses. So the renderer checks it itself.
+CLUSTER_VARS_PATH = (
+    Path(__file__).resolve().parent.parent / "clusters" / "base" / "cluster-vars.yaml"
+)
+
+# A run of `$` followed by `{...}`. envsubst consumes `$$` as a literal `$`, so a run of N
+# dollars leaves the brace SUBSTITUTED when N is odd (`${X}`, `$$${X}`) and LITERAL when N is
+# even (`$${X}` — the escape). Deliberately greedy about the body so malformed/indexed forms
+# (`${A[0]}`) are caught too rather than skipped.
+_BRACED_EXPANSION_RE = re.compile(r"(\$+)\{([^{}]*)\}")
+_PLAIN_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
+
+_CLUSTER_VARS_CACHE: dict = {}
+
+
+def cluster_vars_keys(path: Path | None = None) -> set:
+    """The declared cluster-vars keys — the ONLY names envsubst may legitimately expand.
+
+    Read from clusters/base/cluster-vars.yaml (the single knob surface, R-ENV-02) so the
+    renderer never carries a stale hardcoded copy. Unreadable/absent => empty set, i.e.
+    FAIL-CLOSED: with no known knobs, every braced expansion must be escaped for the shell.
+    """
+    key = str(path or CLUSTER_VARS_PATH)
+    if key not in _CLUSTER_VARS_CACHE:
+        try:
+            doc = yaml.safe_load(Path(key).read_text())
+            data = doc.get("data") if isinstance(doc, dict) else None
+            _CLUSTER_VARS_CACHE[key] = set(data) if isinstance(data, dict) else set()
+        except Exception:  # pragma: no cover - absent/malformed => fail closed
+            _CLUSTER_VARS_CACHE[key] = set()
+    return _CLUSTER_VARS_CACHE[key]
+
+
+def substitution_violations(text: str, known: set | None = None) -> list:
+    """Every unescaped `${...}` in `text` that is NOT a bare, declared cluster-vars key.
+
+    Returns `(line_number, token, reason)` triples in line order. An expansion is ALLOWED only
+    when it is exactly `${KNOWN_KEY}` — a deliberate R-ENV-02 knob. Anything else is a bug:
+    a shell expansion that envsubst will eat (`${V:-}`), a typo'd/undeclared knob that
+    envsubst will blank, or an indexed form that hard-fails the whole Kustomization.
+    """
+    keys = cluster_vars_keys() if known is None else known
+    out = []
+    for lineno, line in enumerate(text.splitlines(), start=1):
+        for m in _BRACED_EXPANSION_RE.finditer(line):
+            if len(m.group(1)) % 2 == 0:
+                continue  # even run of `$` => escaped; envsubst emits it back literally
+            expr = m.group(2)
+            if _PLAIN_IDENT_RE.match(expr):
+                if expr in keys:
+                    continue  # a declared cluster-vars knob, intentionally substituted
+                reason = (
+                    f"'{expr}' is not a declared cluster-vars key — envsubst will replace "
+                    "it with the EMPTY STRING"
+                )
+            elif "[" in expr:
+                reason = (
+                    "indexed expansion — envsubst hard-fails with 'missing closing brace' "
+                    "and the whole Kustomization stops reconciling"
+                )
+            else:
+                reason = (
+                    "shell expansion (default-operator/parameter form) — envsubst eats it "
+                    "at reconcile time and STRICT mode exempts this form, so nothing else "
+                    "will catch it"
+                )
+            out.append((lineno, m.group(0), reason))
+    return out
+
+
+def assert_substitution_safe(text: str, origin: str = "<render>") -> None:
+    """Refuse to emit a manifest carrying an unescaped braced expansion (see above).
+
+    Raises ValueError listing every offending token. The fix is almost always to escape the
+    shell brace as `$${...}` (envsubst emits a literal `${...}` for the shell), or — if the
+    value really is an environment knob — to declare it in clusters/base/cluster-vars.yaml
+    and write it as a bare `${KNOB}`.
+    """
+    bad = substitution_violations(text)
+    if not bad:
+        return
+    lines = [
+        f"{origin}: refusing to emit — unescaped `${{...}}` in a manifest that Flux's "
+        "postBuild.substituteFrom (cluster-vars) will substitute at reconcile time:"
+    ]
+    for lineno, token, reason in bad:
+        lines.append(f"  line {lineno}: {token}  -> {reason}")
+    lines.append(
+        "  FIX: a SHELL expansion must be escaped `$${...}` (envsubst emits a literal "
+        "`${...}`); bare `$VAR` / `$(cmd)` need no escaping. A genuine environment knob "
+        "must be declared in clusters/base/cluster-vars.yaml and written bare `${KNOB}`. "
+        "Contract: clusters/base/apps-vms.yaml."
+    )
+    raise ValueError("\n".join(lines))
+
+
 # =============================================================================
 # Top-level render + emit
 # =============================================================================
@@ -1026,8 +1153,14 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
     return docs
 
 
-def emit(docs: list) -> str:
+def emit(docs: list, origin: str = "<render>") -> str:
-    """Deterministic multi-document YAML text (golden-diffable)."""
+    """Deterministic multi-document YAML text (golden-diffable).
+
+    Every emitted document is checked against the substitution contract before it is handed
+    back (assert_substitution_safe): the rendered manifest lands on a Flux path that runs
+    envsubst over it, so an unescaped `${...}` meant for a SHELL is silently blanked. Raises
+    ValueError rather than emitting such a manifest.
+    """
     parts = []
     for comment, resource in docs:
         body = yaml.dump(
@@ -1037,7 +1170,9 @@ def emit(docs: list) -> str:
             width=4096,
         )
         parts.append(f"---\n# {comment}\n{body}".rstrip() + "\n")
-    return "".join(parts)
+    text = "".join(parts)
+    assert_substitution_safe(text, origin)
+    return text
 
 
 def load_sd(path: Path) -> dict:
@@ -1076,7 +1211,9 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N
     src_dir = catalog_dir if catalog_dir is not None else path.parent
     rev = build_reverse_map(_load_catalog(src_dir))
     inbound = rev.get(sd["name"], [])
-    return emit(render(sd, image_automation=image_automation, inbound=inbound))
+    return emit(
+        render(sd, image_automation=image_automation, inbound=inbound), origin=str(path)
+    )
 
 
 # =============================================================================
@@ -1729,6 +1866,90 @@ def _selftest() -> int:
     check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"],
           "wave kustomization must list exactly the wave's resources")
 
+    # 8) substitution safety (R-ENV-02): the renderer must REFUSE to emit a manifest whose
+    #    shell script carries an unescaped `${...}`. Every rendered manifest lands under a
+    #    Flux path with postBuild.substituteFrom cluster-vars, so envsubst rewrites it before
+    #    the container runs — this is the gitea/nextcloud OIDC-guard bug (`[ -z "${X:-}" ]`
+    #    -> `[ -z "" ]`, always true, whole registration block skipped on every boot).
+    known = {"BASE_DOMAIN", "GATEWAY_VIP"}
+    # allowed: a bare, DECLARED cluster-vars knob is a deliberate R-ENV-02 substitution.
+    check(
+        substitution_violations("host: app.${BASE_DOMAIN}", known) == [],
+        "a bare declared cluster-vars key must be allowed (that is what substitution is FOR)",
+    )
+    # allowed: escaped shell braces, and the forms envsubst leaves alone.
+    check(
+        substitution_violations('if [ -z "$${X:-}" ]; then', known) == [],
+        "an ESCAPED `$${...}` must be allowed (envsubst emits it back as a literal `${...}`)",
+    )
+    check(
+        substitution_violations('t="$(mktemp)"; echo "$HOME $1"', known) == [],
+        "bare `$VAR` / `$(cmd)` are left alone by envsubst and must NOT be flagged",
+    )
+    # `$$` is a literal `$` to envsubst, so an ODD run of dollars still leaves the brace live.
+    check(
+        len(substitution_violations("$$${X:-}", known)) == 1,
+        "an ODD run of `$` still substitutes ($$ -> literal $, then `${X:-}` is eaten)",
+    )
+    # flagged: the exact regression — a default-operator form STRICT mode deliberately exempts.
+    guard_bug = substitution_violations('if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then', known)
+    check(
+        len(guard_bug) == 1 and guard_bug[0][1] == "${GITEA_OIDC_CLIENT_SECRET:-}",
+        "an unescaped `${VAR:-}` shell guard must be flagged (strict mode exempts this form)",
+    )
+    # flagged: an undeclared bare knob (envsubst blanks it), and an indexed form (hard error).
+    check(
+        len(substitution_violations("value: ${NOT_A_KNOB}", known)) == 1,
+        "an undeclared bare `${VAR}` must be flagged (envsubst blanks it silently)",
+    )
+    idx = substitution_violations("echo ${PIPESTATUS[0]}", known)
+    check(
+        len(idx) == 1 and "missing closing brace" in idx[0][2],
+        "an indexed `${A[0]}` must be flagged (envsubst hard-fails the whole Kustomization)",
+    )
+    # the checker must be wired into emit(): rendering an SD whose lifecycle hook carries the
+    # bug RAISES rather than writing a manifest whose guard is dead on arrival.
+    bad_sd = {
+        "name": "subbad",
+        "source": {"image": "a:1"},
+        "lifecycle": {
+            "postStart": {
+                "exec": {
+                    "command": [
+                        "/bin/sh",
+                        "-c",
+                        'if [ -z "${SUBBAD_SECRET:-}" ]; then exit 0; fi',
+                    ]
+                }
+            }
+        },
+    }
+    try:
+        emit(render(bad_sd), origin="<selftest>")
+        check(False, "emit must REFUSE a manifest with an unescaped `${...}` shell expansion")
+    except ValueError as exc:
+        check(
+            "SUBBAD_SECRET" in str(exc) and "$${" in str(exc),
+            "the refusal must name the offending token AND state the `$${...}` fix",
+        )
+    # the ESCAPED form of the same hook renders normally (the fix must be emittable).
+    good_sd = dict(bad_sd, name="subgood")
+    good_sd["lifecycle"] = {
+        "postStart": {
+            "exec": {
+                "command": [
+                    "/bin/sh",
+                    "-c",
+                    'if [ -z "$${SUBBAD_SECRET:-}" ]; then exit 0; fi',
+                ]
+            }
+        }
+    }
+    check(
+        "$${SUBBAD_SECRET:-}" in emit(render(good_sd), origin="<selftest>"),
+        "the escaped `$${...}` form must render (and reach the shell as `${...}`)",
+    )
+
     if failures:
         for f in failures:
             sys.stderr.write(f"SELFTEST FAIL: {f}\n")

catalog/seerr.yaml

@@ -57,7 +57,7 @@ dependencies:
     port: 32400
     protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
     port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/semaphore.yaml

@@ -58,7 +58,7 @@ env:
   - name: SEMAPHORE_ADMIN_NAME
     value: Semaphore Admin
   - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
   # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
   #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
   - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -110,7 +110,7 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
     port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/sonarr.yaml

@@ -9,7 +9,7 @@ name: sonarr
 kind: container
 source:
   image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)

ci/l0.sh

@@ -100,6 +100,43 @@ else
   fi
 fi
 
+# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
+# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
+# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
+# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
+# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
+# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
+# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
+# a legitimate manifest, and that no expansion silently empties today.
+#
+# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
+# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
+# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
+# attributes no requirement, so the requirement-scoped selection would filter this check
+# out of exactly the change that introduces the bug. The invariant is whole-repo (same
+# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
+# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
+# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
+SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
+log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
+if [ ! -f "$SUBST_CHECK" ]; then
+  skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
+  summary "- subst-safety: **SKIPPED** (check missing)"
+  if [ "$rc" -eq 0 ]; then rc=3; fi
+else
+  subst_rc=0
+  bash "$SUBST_CHECK" || subst_rc=$?
+  case "$subst_rc" in
+    0) ok "Flux postBuild substitution safety passed"
+       summary "- subst-safety: **passed**" ;;
+    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
+       summary "- subst-safety: **SKIPPED** (nothing to audit)"
+       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
+    *) err "Flux postBuild substitution safety FAILED"
+       summary "- subst-safety: **FAILED**"; rc=1 ;;
+  esac
+fi
+
 case "$rc" in
   0) ok "L0 stage complete" ;;
   3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;

clusters/README.md

@@ -97,6 +97,27 @@ Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
 Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)
 ```
 
+### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
+
+The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
+controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
+and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
+reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
+before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
+**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
+`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
+have existed for months); it cost 12h on prod's cold start.
+
+So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
+platform-config`), and a rule:
+
+> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
+> **controller at runtime** rather than by a manifest in git.
+
+`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
+`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
+profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
+
 Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
 exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
 (`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own

clusters/base/apps-vms.yaml

@@ -4,8 +4,11 @@
 # NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
 # import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
 # health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
-# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
+# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
-# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
+# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
+# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
+# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
+# adoption concern).
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
@@ -19,6 +22,28 @@ spec:
   wait: false
   dependsOn:
     - name: platform-config
+    # LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
+    # storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
+    # decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
+    # (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
+    # live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
+    # with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
+    # to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
+    # conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
+    #
+    # HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
+    # derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
+    # status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
+    # it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
+    #
+    # This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
+    # deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
+    # dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
+    #
+    # The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
+    # edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
+    # cannot silently drop this tier's KubeVirt/StorageClass gate.
+    - name: platform-config-cdi
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/base/cilium.yaml

@@ -0,0 +1,60 @@
+---
+# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
+# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
+#
+# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
+#
+# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
+# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
+# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
+# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
+#
+#   * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
+#   * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
+#     tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
+#     `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
+#     the valve the 2026-09-01 substitution incident wanted and did not have.
+#   * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
+#     `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
+#
+# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
+# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
+# expressed as an explicit edge instead of an accident of co-tenancy.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: cilium
+  namespace: flux-system
+spec:
+  # A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
+  # kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
+  interval: 1h
+  retryInterval: 2m
+  # 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
+  # rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
+  timeout: 10m
+  prune: true
+  # TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
+  wait: true
+  # Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
+  # by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
+  # is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
+  # revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
+  # `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/cilium
+  # MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
+  # kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
+  # this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
+  # renders empty). That is the intended fail-closed behaviour — but it means the stanza is
+  # load-bearing for the CNI coming up at all. Do not remove it.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
+  # oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).

clusters/base/cluster-vars.yaml

@@ -17,9 +17,63 @@ data:
   CLUSTER_ENV: "base"                       # dev | prod  (overlays set this)
 
   # --- DNS / TLS ---
-  BASE_DOMAIN: "home.keithjackson.dev"      # platform hostname suffix (same both envs; split-horizon)
+  # THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
-  DNS_ZONE: "keithjackson.dev"              # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
+  # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
-                                            # ExternalDNS domain-filter must be the zone, not the subdomain
+  # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
+  #
+  # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
+  # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
+  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
+  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
+  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
+  # subdomain, so the two clusters can no longer describe the same name.
+  #
+  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
+  #   prod  -> prod.home.keithjackson.dev
+  #
+  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
+  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
+  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
+  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
+  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
+  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
+  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
+  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
+  BASE_DOMAIN: "home.keithjackson.dev"
+
+  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
+  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
+  # so it cannot be created, updated or deleted regardless of TXT ownership.
+  #
+  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
+  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
+  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
+  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
+  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
+  #
+  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
+  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
+  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
+  # — it can never delete another environment's records — and the fallback is to set prod's value
+  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
+  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
+
+  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
+  # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
+  # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
+  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
+  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
+  #
+  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
+  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
+  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
+  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
+  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
+  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
+  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
+
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
   GATEWAY_VIP: "192.168.1.53"                # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
@@ -47,10 +101,37 @@ data:
   # ship the literal string into the cluster.
 
   # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
-  # [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
+  #
-  # main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
+  # *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
-  # reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
+  # 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
-  K3S_API_HOST: "192.168.1.50"
+  # control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
+  # live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
+  # own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
+  # Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
+  # outage you would notice. A virtual address that follows the control plane is the fix, and it is
+  # what kube-vip was stood up to provide.
+  #
+  #   dev  -> 192.168.1.49   (= kube_vip_address, group_vars/all.yml; base default, below)
+  #   prod -> 192.168.2.9    (= kube_vip_address, group_vars/k3s_prod/vars.yml)
+  #
+  # THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
+  # unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
+  # move. Read it as "the API host Cilium dials", which is what it always meant.
+  #
+  # [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
+  # (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
+  # already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
+  # the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
+  # already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
+  # block at the top of group_vars/all.yml.
+  #
+  # Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
+  # in both envs, so it is NOT a knob.
+  #
+  # TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
+  # both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
+  # https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
+  K3S_API_HOST: "192.168.1.49"
 
   # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
   # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
@@ -74,6 +155,17 @@ data:
 
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
+  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
+  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
+  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
+  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
+  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
+  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
+  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
+  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
+  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
+  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
+  LONGHORN_DATA_PATH: "/var/lib/longhorn"
 
   # --- observability sizing (docs/10) ---
   MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long

clusters/base/kustomization.yaml

@@ -5,8 +5,24 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - flux-system.yaml     # GitRepository (reconciliation source)
+  # Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
+  # of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
+  # Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
+  # `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
+  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
+  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
+  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
+  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
+  # manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
+  # datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
+  # platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
+  # keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
+  # CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
+  # cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
+  # as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
+  - platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
   # Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
   # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.

clusters/base/platform-config-cdi.yaml

@@ -0,0 +1,135 @@
+---
+# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
+# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
+#
+# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
+#
+# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
+# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
+# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
+# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
+# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
+# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
+# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
+#
+# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
+# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
+# graph order itself, so a cold cluster needs no human in the loop.
+#
+# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
+# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
+# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
+# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
+# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
+# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
+# kind. One edge, no pre-apply.
+#
+# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
+# and everything else waits on: putting CDI's health on that critical path is the mistake the
+# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
+# StorageProfiles that were in the wrong pass.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: platform-config-cdi
+  namespace: flux-system
+spec:
+  interval: 10m
+  # 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
+  # very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
+  # now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
+  # about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
+  # cache is one retryInterval. Keep it short.
+  retryInterval: 2m
+  # 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
+  # image to pull and no controller to become Available, so a slow apply here means a real error, not
+  # a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
+  timeout: 2m
+  # TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
+  # component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
+  # It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
+  # second path also declares. See the transition note at the bottom of this file.
+  prune: true
+  # TRUE, but understand exactly what it buys and what it does NOT.
+  #
+  # CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
+  # provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
+  # `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
+  # it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
+  # is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
+  # If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
+  # `apps-vms` waits on it.
+  #
+  # WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
+  # provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
+  # storageprofile-controller computes that status from our spec override plus inference. So Ready
+  # here means "the spec override is accepted by the API server", not "CDI has folded it into
+  # status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
+  # below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
+  # a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
+  wait: true
+  dependsOn:
+    # platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
+    # causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
+    # ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
+    # flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform-config-cdi
+  # NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
+  # no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
+  # oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
+  # clusters/dev or clusters/prod for this tier.
+  #
+  # substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
+  # substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
+  #   1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
+  #      whose Kustomization declares substituteFrom. Omitting it would move
+  #      cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
+  #      could ship as a literal string, precisely the class of bug that check exists to stop.
+  #   2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
+  #      with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
+  #      empty. Without the stanza there is no substitution at all and the literal ships silently.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
+  # Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
+  # Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
+  # StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
+  # the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
+  # dual-ownership fight — the objects are pruned before they are re-applied under this tier's
+  # `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
+  #
+  # Why that is acceptable here — and it is NOT the platform/cilium case:
+  #   * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
+  #     nothing else in the repo declares a StorageProfile. The stale set is exactly two
+  #     cluster-scoped CDI CRs.
+  #   * The objects do not actually go missing: CDI's storageprofile-controller creates a
+  #     StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
+  #     inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
+  #     absent".
+  #   * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
+  #     existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
+  #     is disturbed and no data path is touched.
+  #   * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
+  #     instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
+  #     permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
+  #     dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
+  #     created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
+  #     could.
+  #
+  # PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
+  # happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
+  # config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
+  # healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
+  # --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
+  # StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
+  #   `flux get kustomization platform-config` must be Ready BEFORE merging.
+  # If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
+  # temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
+  # this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
+  # `prune: true` in a second commit.

clusters/base/platform.yaml

@@ -11,8 +11,53 @@ spec:
   interval: 10m
   retryInterval: 2m
   timeout: 5m
-  prune: true          # drift correction: delete resources removed from git (R-GITOPS-03)
+  # Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
+  # on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
+  # live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
+  # `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
+  # any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
+  #
+  # LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
+  # Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
+  # made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
+  # Cilium. Before moving ANY platform/* component in future, check whether it is the sole
+  # declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
+  prune: true
   wait: true           # block `apps` until platform is healthy
+  # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
+  #
+  # `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
+  # `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
+  # `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
+  # NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
+  # That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
+  #
+  # `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
+  # kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
+  # when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
+  # revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
+  # writes a newly-added `cluster-vars` knob — so this one edge closes the race.
+  #
+  # `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
+  # consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
+  # velero) chains transitively through it. One edge therefore protects the whole graph.
+  #
+  # TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
+  # ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
+  # Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
+  # the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
+  # entrypoint should stop the line, not let consumers render against a half-applied variable set.
+  # No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
+  # health-gates its own children (see clusters/dev/sync.yaml).
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+    # THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
+    # (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
+    # suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
+    # nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
+    # is healthy. It is also revision-gated, which is what makes the prune-safety argument above
+    # deterministic rather than a race.
+    - name: cilium
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/dev/kustomization.yaml

@@ -148,6 +148,33 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
+      # DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
+      # like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
+      # Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
+      # Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
+      # (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
+      # prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
+      # not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
+      # not a variable change: infra/PROD-PROVISIONING.md §10.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "home.keithjackson.dev"
+      # ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
+      # and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
+      # the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        value: "keithjackson.dev"
+      # DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
+      # registry under owner `default` (the external-dns chart default, which is what both clusters
+      # silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
+      # BY this string: any other value and it stops recognising every record it manages, then
+      # re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
+      # whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "default"
       # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
       # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
       # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -160,11 +187,25 @@ patches:
       - op: replace
         path: /data/GATEWAY_VIP
         value: "192.168.1.53"
-      # k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
+      # k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
-      # (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
+      # the address kube-vip floats, NOT the primary node .50.
+      # [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
+      # (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
+      #
+      # *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
+      # Every other value in this block is what dev is running right now. This one is not:
+      # the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
+      # DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
+      # AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
+      # in parity for exactly this reason). Do not push this to dev-live casually, and not
+      # while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
+      # but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
+      # already (k3s_tls_sans) and dev's own kubeconfig validates against it via
+      # tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
+      # same LAN as the nodes and ARP-announced by kube-vip on every server.
       - op: replace
         path: /data/K3S_API_HOST
-        value: "192.168.1.50"
+        value: "192.168.1.49"
       # Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
       # (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
       - op: replace
@@ -190,6 +231,16 @@ patches:
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
+      # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
+      # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
+      # else would orphan every one of them (they are not recoverable from the old path by
+      # rescheduling). Restated explicitly even though it equals the base default, exactly like the
+      # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
+      # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/var/lib/longhorn"
       # smaller retention / volumes to keep the dev workstation footprint light.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

clusters/prod/kustomization.yaml

@@ -5,11 +5,191 @@
 # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
 # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
 # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
+#
+# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
+# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
+# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
+# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
+# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
+# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
+# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
+# done here: prod is design-only and its root object is the prod bring-up owner's call.)
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - ../base
+  - sync.yaml          # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
 patches:
+  # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
+  # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
+  # The `sops-age` Secret is created OUT OF BAND on the prod cluster
+  # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
+  # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: platform
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-0
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-1
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-2
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: oauth2-proxy
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: headlamp
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: crafty
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: velero
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
+  #
+  # ############################################################################################
+  # ###                                                                                      ###
+  # ###   *** T E M P O R A R Y   B O O T S T R A P   S O U R C E  —  NOT PERMANENT ***       ###
+  # ###                                                                                      ###
+  # ###   The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside   ###
+  # ###   any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST   ###
+  # ###   be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this  ###
+  # ###   and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is  ###
+  # ###   a live single point of failure, not a settled design. See "CUTOVER" below.          ###
+  # ###                                                                                      ###
+  # ############################################################################################
+  #
+  # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
+  # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
+  # Two independent reasons, both consequences of the split-horizon design:
+  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
+  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
+  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
+  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
+  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
+  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
+  #      gitea.home.keithjackson.dev to itself and never reach a repository.
+  #
+  # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
+  #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
+  #   operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
+  #   reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
+  #   change was needed. This is the same "local bootstrap source" pattern dev itself used before
+  #   its Phase-6 cutover (docs/05 "Source-of-truth").
+  #
+  # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
+  # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
+  # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
+  # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
+  #
+  # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
+  # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
+  #   curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
+  #   git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git  -> refs/heads/main
+  # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
+  # THAT FIRST: a private repo here fails exactly that way.
+  #
+  # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
+  #   WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
+  #         through prod's own Gateway VIP with a publicly-trusted cert.
+  #   DO:   1. push/mirror this repo into the prod in-cluster Gitea;
+  #         2. change the url below to
+  #            https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+  #            and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
+  #            clusters/dev/kustomization.yaml, is what "done" looks like);
+  #         3. confirm `flux get sources git -n flux-system` reconciles the new revision;
+  #         4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
+  #            exactly as dev retained its bootstrap Gitea.
+  #   UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
+  #   powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
+  #   it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
+  #
+  # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
+  # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
+  # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
+  # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
+  # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
+  # is the revision you intend prod to run (`git ls-remote` above, against local `main`).
+  - patch: |-
+      apiVersion: source.toolkit.fluxcd.io/v1
+      kind: GitRepository
+      metadata:
+        name: flux-system
+        namespace: flux-system
+      spec:
+        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
+        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
+        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
+        ref:
+          branch: main
   - target:
       kind: ConfigMap
       name: cluster-vars
@@ -21,9 +201,80 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # ==========================================================================================
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
+      # ==========================================================================================
+      # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
+      # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
+      # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
+      # of which already happened:
+      #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
+      #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
+      #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
+      #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
+      #      have raced each other's `_acme-challenge` TXT records on that one zone.
+      # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
+      # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
+      #
+      #   dev   -> home.keithjackson.dev         (UNCHANGED — dev is live; phase 2 moves it)
+      #   prod  -> prod.home.keithjackson.dev    (this knob)
+      #
+      # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
+      # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
+      # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
+      # `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
+      # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
+      #
+      # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
+      # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
+      # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
+      # trusts a different Keycloak than the one prod actually runs.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "prod.home.keithjackson.dev"
+      # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
+      # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
+      # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
+      # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
+      # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
+      # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
+      # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
+      # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
+        # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
+        # record then skipped it with "no hosted zone matching record DNS Name", so prod would
+        # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
+        # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
+        # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
+        # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
+        value: "keithjackson.dev"
+      # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
+      # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
+      # its own and, under policy: sync, deleted the records behind them. This value MUST differ from
+      # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
+      # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "prod"
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
+      # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
+      # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
+      # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
+      # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
+      # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
+      #
+      # [parity] MUST equal node_baseline_longhorn_data_path in
+      # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
+      # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
+      # path on the OS filesystem instead of the data partition and /var fills up.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/data/longhorn"
       # longer retention / larger volumes on prod NVMe.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION
@@ -89,14 +340,32 @@ patches:
       - op: replace
         path: /data/CILIUM_LB_POOL_STOP
         value: "192.168.2.30"
-      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
       # k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
-      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
+      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
-      # Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
+      #
-      # platform/cilium/helmrelease.yaml. Not yet decided.
+      # *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
+      # read "Not yet decided" while shipping .10 — the founder's own address — and that is what
+      # the LIVE prod cluster is running: all three cilium-agents carry
+      # KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
+      # 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
+      # the datapath keeps running off Cilium's cache while no service or policy update lands, with
+      # nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
+      # control plane is, which is the whole reason kube-vip exists here.)
+      #
+      # It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
+      # this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
+      # equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
+      # that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
+      # `--server` join needs a host that is up before any VIP is floated. Two addresses, two
+      # variables, deliberately.
+      #
+      # .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
+      # LB-IPAM can never be handed the same IP — asserted by
+      # conformance/tests/l0-static/ha-control-plane-as-code.
       - op: replace
         path: /data/K3S_API_HOST
-        value: "192.168.2.10"
+        value: "192.168.2.9"
+      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
       # Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
       # prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
       # they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not

clusters/prod/sync.yaml

@@ -0,0 +1,32 @@
+---
+# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml).
+#
+# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars
+# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object
+# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs
+# its own sync.yaml -> ./clusters/prod"; this is it.
+#
+# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained
+# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent
+# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no
+# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does
+# not exist, and NOTHING downstream would ever reconcile.
+#
+# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches
+# Ready on apply success without health-gating its children (platform does its own wait/health
+# gating). Do not "harden" this to true.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: flux-system
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./clusters/prod

conformance/.gitignore

@@ -8,3 +8,22 @@ _artifacts/
 
 # Python bytecode from lib/ (status.py imports coverage.py); never committed.
 __pycache__/
+
+# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
+# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
+# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
+# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
+# fact, so it must survive in git.
+#
+# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
+# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
+# the `conformance/runs/` directory itself, so re-including the files here works (git
+# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
+# live under _artifacts/).
+#
+# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
+# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
+# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
+!runs/
+!runs/**
+!runs/**/*.junit.xml

conformance/README.md

@@ -22,6 +22,7 @@ behavioral / L3 fault-injection** (this harness).
 ./run.sh --lint                       # structural-validate test defs (no cluster)
 ./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
 ./run.sh --require-all                # phase-exit gate: every requirement must be covered
+./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
 ```
 
 **Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -105,13 +106,89 @@ lib/
   common.sh                  tagging convention + test discovery/selection helpers
   coverage.py                requirement-coverage generator (stdlib only)
   status.py                  requirement-STATUS generator (stdlib only; --selftest)
+  record_run.py              run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
   emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
+runs/
+  ledger.jsonl               TRACKED append-only proof-of-pass ledger (one row per
+                             testcase per run) — the only source status.py trusts
+  <UTC>-<short-sha>/         TRACKED per-run evidence: run.log + report.junit.xml
 tests/
   l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
 incluster/
   chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)
 ```
 
+## Proof of pass — the run ledger (`runs/`)
+
+`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
+proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
+root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
+only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
+asked for — **a committed, machine-readable run ledger**.
+
+```bash
+./run.sh --record                     # runs normally, and captures the evidence
+# -> conformance/runs/<UTC>-<short-sha>/run.log            (tee'd, carries the sentinels)
+# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml   (that run's JUnit)
+# -> conformance/runs/ledger.jsonl                         (one appended row per testcase)
+python3 lib/record_run.py --run-dir runs/<id> [--dry-run]  # or record an existing run dir
+python3 lib/record_run.py --selftest                       # deterministic, no cluster
+```
+
+**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
+that silently records itself is how a bad run becomes a permanent false proof. **Commit
+the run dir and the ledger** — an uncommitted ledger proves nothing.
+
+### Why a committed JUnit alone would be WORSE than nothing
+
+The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
+`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
+and `exit 0`, and their `check:` accepts either outcome —
+
+```yaml
+(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+ || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+```
+
+So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
+A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
+R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
+
+**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
+`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
+`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
+verdict token), so the table below is never hand-maintained:
+
+| Test | pass token (proof) | skip token (never proof) |
+|---|---|---|
+| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
+| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
+| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
+| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
+| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
+
+A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
+token present in that test's log lines, and **no** skip token anywhere in them. A skip
+token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
+`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
+non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
+skips as JUnit `<skipped/>`, never as a green pass.
+
+### Staleness — a proof is a claim about a tree, not a badge
+
+`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
+last commit touching that requirement's implementing manifests or its verifying test**.
+Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
+dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
+it never counts. Re-run with `--record` to refresh.
+
+A requirement reads `proven` only when **every** verifying test does; one skipped, failed
+or unrecorded test drags the whole requirement down. With an empty ledger `--status`
+behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
+
 ## How CI invokes it
 CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
 

conformance/lib/record_run.py

@@ -0,0 +1,949 @@
+#!/usr/bin/env python3
+"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
+
+WHY THIS EXISTS
+---------------
+`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
+fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
+`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
+`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
+that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
+and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
+tracked (see conformance/.gitignore for the negations that let it survive).
+
+THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
+------------------------------------------------------------------
+The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
+StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
+`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
+injection/dr-restore:
+
+    (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+     || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+
+So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
+(verified against the committed reports). Therefore:
+
+  *** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
+
+AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
+------------------------------------------------------
+Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
+it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
+the log of every run of that test. A section-blind grep therefore reads a skip token out
+of the source of every skip-gated test and can never record `proven` for any of them: the
+ledger fails closed so completely that it records nothing and proves nothing. Tokens are
+consequently read from SECTIONS (see classify_sections): a pass token only from
+`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
+pass — within genuine output, which is the only place either was ever evidence.
+
+That is the fake-green this project refuses, and it would be worse than no ledger at all
+because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
+PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
+files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
+unless the log carries that test's own pass token and carries NO skip token.
+
+HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
+--------------------------------------------------------
+Hard-coding a token table would rot the moment a test is edited. Instead every token is
+parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
+`check:` block is a declared sentinel. Classification is by the project's naming convention
+(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
+— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
+and this tool follows automatically; delete them and the test simply becomes unprovable
+rather than silently proven.
+
+  node-loss   pass NODE_LOSS_VOLUME_AVAILABLE_OK              skip NODE_LOSS_SKIPPED_OK
+  dr-restore  pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK     skip DR_RESTORE_SKIPPED_OK
+              (+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
+
+WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
+-----------------------------------------------------------
+The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
+dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
+any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
+pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
+can express "skipped" as "passed", and only they require a sentinel — see
+SENTINEL_REQUIRED_LAYERS.
+
+LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
+  ts_utc            run-dir timestamp, else the JUnit's mtime      (when)
+  git_sha           `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
+                    modified — a proof captured from a dirty tree is NOT honoured by
+                    status.py, because what ran is not what is committed  (what)
+  env               --env / CONF_ENV (cluster overlay the run targeted)
+  chainsaw_version  CHAINSAW_VERSION from versions.env (the pin IS the truth)
+  kube_version      `kubectl version` server gitVersion, else "unknown"
+  layer             L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
+  test              testcase name (== the test dir basename, asserted by convention)
+  requirements[]    the `conformance.home.k3s/R-*` labels on the test — the SAME
+                    authoritative source coverage.py uses, not the JUnit classname
+  junit_result      passed | failed | error | skipped
+  sentinel          the DECIDING token grepped from run.log, or null
+  proof             proven | skipped | failed | junit-skipped
+                    | unknown-no-sentinel-declared | unknown-sentinel-absent
+  run_id            the runs/<dir> name (provenance: which log backs this row)
+  note              one line saying WHY this row landed on that proof state
+
+`proven` is emitted on exactly one path (see decide()); every other path is non-green.
+
+RECORDING IS EXPLICIT, NEVER IMPLICIT
+-------------------------------------
+Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
+how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
+ledger row at all.
+
+EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
+  0  rows appended.
+  1  a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
+  3  the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
+     NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
+
+Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
+status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+_LIB = Path(__file__).resolve().parent
+if str(_LIB) not in sys.path:
+    sys.path.insert(0, str(_LIB))
+# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
+# tree, and the lint stage scans untracked files. Same guard as status.py.
+sys.dont_write_bytecode = True
+import coverage as cov  # noqa: E402  — reuse the registry/test parsers, don't fork them
+
+CONF_ROOT = _LIB.parent
+REPO_ROOT = CONF_ROOT.parent
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
+# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
+# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
+# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
+# by JUnit alone.
+SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
+
+# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
+# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
+# harmless for its own report, but here a mis-read layer decides whether a sentinel is
+# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
+_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
+                     ("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
+                     ("/meta/", "META"))
+
+
+def layer_of(path: Path) -> str:
+    rel = str(path)
+    for frag, layer in _LAYER_BY_SUBTREE:
+        if frag in rel:
+            return layer
+    return ""
+
+# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
+# Both quote styles are accepted; the repo uses single quotes throughout.
+_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
+
+# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
+# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
+# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
+# mis-classifying a SKIP token as a pass would manufacture one.
+_SKIP_TOKEN_RE = re.compile(r"_SKIP")
+
+# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
+# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
+#     l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
+# Field 2 is the test name; it is how output is attributed back to a testcase. The
+# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
+_LOG_HEADER_RE = re.compile(
+    r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
+    r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
+
+# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
+# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
+
+
+def _strip_noise(line: str) -> str:
+    return _ANSI_RE.sub("", line).replace("\b", "")
+
+
+# ===================================================================== output sections
+# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
+# -----------------------------------------------------------------------------
+# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
+# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
+# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
+# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
+# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
+# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
+# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
+# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
+# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
+#
+# So tokens are attributed to SECTIONS, never to raw lines:
+#     l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |     <- header, no section
+#         === COMMAND                                            <- marker (indent 8)
+#         /usr/bin/sh -c set -euo pipefail                       <- source, indent >= 8
+#           echo NODE_LOSS_SKIPPED_OK                            <- SOURCE, not output
+#     l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |    <- indent 4 => section ends
+#         === STDOUT                                             <- marker
+#         NODE_LOSS_VOLUME_AVAILABLE_OK                          <- the REAL emission
+#
+# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
+# (`=== RUN   chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
+# after the verb, so they match neither the indent nor the end-of-line anchor; they are
+# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
+# section that would swallow the whole log.
+_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
+_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
+
+# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
+# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
+# never echoes script source, so there is nothing to be confused by and the legacy
+# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
+# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
+# "" and are evidence for nothing.
+UNSECTIONED = "*"
+
+# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
+# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
+# never manufacture a proof.
+PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
+
+# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
+# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
+# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
+# one. Unknown section names therefore fail closed in both directions.
+SOURCE_SECTIONS = frozenset({"COMMAND"})
+
+
+def classify_sections(log_text: str) -> list[tuple[str, str]]:
+    """[(section, line)] for every line of a run log.
+
+    `section` is the enclosing `=== NAME` block, "" for a line inside no block, or
+    UNSECTIONED for every line when the log declares no sections at all. A block runs from
+    its marker until the first non-blank line indented LESS than the marker (blank lines do
+    not end it) or until the next marker; the marker line itself is content of nothing.
+    """
+    rows: list[tuple[str, str]] = []
+    cur, cur_indent, sectioned = "", 0, False
+    for raw in log_text.splitlines():
+        line = _strip_noise(raw)
+        m = _SECTION_MARKER_RE.match(line)
+        if m and m.group(2) not in _GO_TEST_MARKERS:
+            sectioned = True
+            cur, cur_indent = m.group(2), len(m.group(1))
+            rows.append(("", line))
+            continue
+        if cur:
+            if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
+                rows.append((cur, line))
+                continue
+            cur = ""
+        rows.append(("", line))
+    if not sectioned:
+        return [(UNSECTIONED, line) for _, line in rows]
+    return rows
+
+
+def scan_texts(lines) -> tuple[str, str]:
+    """(text a PASS token may be read from, text a SKIP token may be read from).
+
+    Accepts either [(section, line)] pairs from classify_sections() or a plain list of
+    strings, which is treated as an unsectioned log (legacy whole-log scanning).
+    """
+    pass_lines, skip_lines = [], []
+    for item in lines:
+        section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
+        if section in PASS_SECTIONS:
+            pass_lines.append(line)
+        if section and section not in SOURCE_SECTIONS:
+            skip_lines.append(line)
+    return "\n".join(pass_lines), "\n".join(skip_lines)
+
+_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
+
+
+# ========================================================================= sentinels
+def parse_sentinels(test_yaml: Path) -> dict:
+    """Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
+
+    Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
+    the LAST non-skip token in file order — the test's verdict token, i.e. the one its
+    final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
+    A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
+    unprovable rather than proven; see decide().
+    """
+    try:
+        text = test_yaml.read_text()
+    except (OSError, UnicodeDecodeError):
+        return {"pass": [], "skip": [], "terminal_pass": None}
+    toks = _SENTINEL_RE.findall(text)
+    passes, skips = [], []
+    for t in toks:
+        (skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
+    # De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
+    def uniq(xs: list[str]) -> list[str]:
+        return list(dict.fromkeys(xs))
+    passes, skips = uniq(passes), uniq(skips)
+    return {"pass": passes, "skip": skips,
+            "terminal_pass": passes[-1] if passes else None}
+
+
+def sentinel_map(tests_dir: Path) -> dict[str, dict]:
+    """{test-name: sentinels} for every test in the tree.
+
+    Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
+    name=...>`. This repo keeps that equal to the test's directory basename; the mapping
+    is by name, so it stays correct if that ever drifts.
+    """
+    out: dict[str, dict] = {}
+    for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
+        meta = cov.parse_test(f)
+        name = meta["name"] or f.parent.name
+        s = parse_sentinels(f)
+        s["dir"] = str(f.parent)
+        s["layer"] = layer_of(f) or meta["layer"]
+        s["requirements"] = meta["labels"]
+        out[name] = s
+    return out
+
+
+# =============================================================================== log
+def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
+                                                            bool]:
+    """Split a run log into per-testcase buckets of (section, line) pairs.
+
+    Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
+    script's sections underneath that header, so the test name is carried by a state
+    machine: a header naming a KNOWN test switches the current bucket, every following line
+    lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
+    chatter (and any other pipe-shaped line) from opening a bogus bucket.
+
+    Section labels come from classify_sections() and travel WITH the lines, so a bucket
+    still knows which of its lines were script source and which were real output.
+
+    Returns (buckets, attributed) where `attributed` is False when the log contained no
+    recognisable header at all — the signal for the degraded global scan in decide().
+    """
+    buckets: dict[str, list[tuple[str, str]]] = {}
+    cur: str | None = None
+    attributed = False
+    for section, line in classify_sections(log_text):
+        m = _LOG_HEADER_RE.match(line)
+        if m and m.group(1) in known:
+            cur = m.group(1)
+            attributed = True
+        if cur is not None:
+            buckets.setdefault(cur, []).append((section, line))
+    return buckets, attributed
+
+
+def unique_tokens(smap: dict[str, dict]) -> set[str]:
+    """Tokens claimed by EXACTLY ONE test in the tree.
+
+    Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
+    tests, so in an unattributable log it can never be evidence for either; but
+    `NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
+    """
+    owners: dict[str, set[str]] = {}
+    for name, s in smap.items():
+        for t in s["pass"] + s["skip"]:
+            owners.setdefault(t, set()).add(name)
+    return {t for t, o in owners.items() if len(o) == 1}
+
+
+# ============================================================================ decide
+def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
+           scope: str) -> tuple[str | None, str, str]:
+    """The ONE place a testcase becomes `proven` — or, far more often, does not.
+
+    Returns (sentinel, proof, note).
+
+    Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
+      1. JUnit failure/error            -> failed        (a red run is recorded as red)
+      2. JUnit <skipped/>               -> junit-skipped  (never green; ci/lib.sh rule)
+      3. ANY declared skip token seen   -> skipped        (BEATS a green JUnit and beats
+           IN REAL OUTPUT                                  the pass token: if any gated
+                                                           step self-skipped, the run did
+                                                           not prove the requirement)
+      4. no sentinel declared           -> proven only for L0/L1, whose skips are already
+                                           JUnit skips; unknown-no-sentinel-declared for
+                                           the Chainsaw layers
+      5. EVERY declared pass token seen -> proven
+      6. anything else                  -> unknown-sentinel-absent
+    Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
+    asserts `pass || skip`, so a missing pass token means a step did not reach its proven
+    branch, whatever the JUnit says.
+
+    Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
+    only in `=== STDOUT`, a skip token counts in any section that is not script source.
+    That narrows WHERE a token is read from and changes nothing about what it means — skip
+    still outranks pass, and it outranks it within genuine output, which is the only place
+    either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
+    strings (an unsectioned log; see scan_texts / UNSECTIONED).
+    """
+    if junit_result in ("failed", "error"):
+        return (None, "failed", f"JUnit recorded {junit_result}")
+    if junit_result == "skipped":
+        return (None, "junit-skipped",
+                "JUnit <skipped/> — the test did not execute; a skip is never green")
+
+    pass_text, skip_text = scan_texts(lines)
+    seen_skip = [t for t in sent["skip"] if t in skip_text]
+    if seen_skip:
+        return (seen_skip[0], "skipped",
+                f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
+                f"exited 0 without exercising the behaviour, so this is NOT proof")
+
+    if not sent["pass"] and not sent["skip"]:
+        if layer not in SENTINEL_EXEMPT_LAYERS:
+            return (None, "unknown-no-sentinel-declared",
+                    f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
+                    f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
+                    f"refusing to call it proven")
+        return (None, "proven",
+                f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
+                f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
+
+    missing = [t for t in sent["pass"] if t not in pass_text]
+    if not missing:
+        return (sent["terminal_pass"], "proven",
+                f"run log OUTPUT carries every declared pass token "
+                f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
+    return (None, "unknown-sentinel-absent",
+            f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
+            f"[{scope}] — refusing to infer proof from the JUnit alone")
+
+
+# ============================================================================= junit
+def junit_cases(path: Path) -> list[dict]:
+    """[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
+    root = ET.parse(path).getroot()
+    cases: list[dict] = []
+    for c in root.iter("testcase"):
+        if c.find("failure") is not None:
+            result = "failed"
+        elif c.find("error") is not None:
+            result = "error"
+        elif c.find("skipped") is not None:
+            result = "skipped"
+        else:
+            result = "passed"
+        props = {p.get("name"): (p.get("value") or "")
+                 for p in c.iter("property") if p.get("name")}
+        reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
+        layer = (props.get("layer") or "").upper()
+        if not layer:
+            layer = (c.get("classname") or "").split(":", 1)[0].upper()
+        cases.append({"name": c.get("name") or "?", "layer": layer,
+                      "requirements": reqs, "result": result})
+    return cases
+
+
+# =============================================================================== git
+def git_sha(repo: Path) -> str:
+    """HEAD, suffixed `-dirty` when TRACKED files differ from it.
+
+    Untracked files are deliberately ignored, matching status.py's `tracked_files()`
+    posture (untracked scratch is not repo state) — and because a `--record` run creates
+    untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
+    A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
+    the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
+    """
+    try:
+        sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
+                             capture_output=True, text=True, check=True).stdout.strip()
+    except (subprocess.CalledProcessError, OSError):
+        return "unknown"
+    dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--"],
+                           capture_output=True).returncode != 0
+    return f"{sha}-dirty" if dirty else sha
+
+
+def pinned_chainsaw_version(conf_root: Path) -> str:
+    """CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
+    f = conf_root / "versions.env"
+    if f.is_file():
+        m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
+        if m:
+            return m.group(1)
+    return "unknown"
+
+
+def server_kube_version() -> str:
+    """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
+    for args in (["kubectl", "version", "-o", "json"],
+                 ["kubectl", "version", "--output=json"]):
+        try:
+            p = subprocess.run(args, capture_output=True, text=True, timeout=15)
+        except (OSError, subprocess.TimeoutExpired):
+            continue
+        if p.returncode != 0 or not p.stdout.strip():
+            continue
+        try:
+            v = json.loads(p.stdout).get("serverVersion") or {}
+        except json.JSONDecodeError:
+            continue
+        if v.get("gitVersion"):
+            return v["gitVersion"]
+    return "unknown"
+
+
+def run_timestamp(run_dir: Path, junit: Path) -> str:
+    """UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
+    m = _RUN_ID_TS_RE.match(run_dir.name)
+    if m:
+        t = m.group(1)
+        return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
+    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
+
+
+# ============================================================================ record
+def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
+               kube_version: str, sha: str) -> list[dict]:
+    junit = run_dir / "report.junit.xml"
+    log = run_dir / "run.log"
+    smap = sentinel_map(tests_dir)
+    log_text = log.read_text(errors="replace")
+    buckets, attributed = attribute_log(log_text, set(smap))
+    uniq = unique_tokens(smap)
+    all_pairs = classify_sections(log_text)
+    sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
+    section_note = ("tokens read from real-output sections only"
+                    if sectioned else "log declares no === sections; whole-log scan")
+    ts = run_timestamp(run_dir, junit)
+
+    rows: list[dict] = []
+    for case in junit_cases(junit):
+        name = case["name"]
+        sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
+                               "layer": case["layer"], "requirements": []})
+        layer = sent.get("layer") or case["layer"] or "?"
+        # Requirements come from the test's LABELS (coverage.py's authoritative source);
+        # the JUnit property is the fallback for a testcase with no matching test file.
+        reqs = sent.get("requirements") or case["requirements"]
+
+        if name in buckets:
+            lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
+        elif not attributed:
+            # Degraded mode: the log had no recognisable Chainsaw header (a redirected or
+            # reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
+            # exactly one test, so a token another test emitted can never prove this one.
+            lines = all_pairs
+            sent = dict(sent,
+                        **{"pass": [t for t in sent["pass"] if t in uniq],
+                           "skip": [t for t in sent["skip"] if t in uniq]})
+            sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
+            scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
+                     + section_note)
+        else:
+            lines, scope = [], "no log lines attributed to this testcase"
+
+        sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
+        rows.append({
+            "ts_utc": ts,
+            "git_sha": sha,
+            "env": env,
+            "chainsaw_version": chainsaw_version,
+            "kube_version": kube_version,
+            "layer": layer,
+            "test": name,
+            "requirements": reqs,
+            "junit_result": case["result"],
+            "sentinel": sentinel,
+            "proof": proof,
+            "run_id": run_dir.name,
+            "note": note,
+        })
+    return rows
+
+
+def append_rows(ledger: Path, rows: list[dict]) -> None:
+    """Append-only, one compact JSON object per line. Never rewrites existing rows."""
+    ledger.parent.mkdir(parents=True, exist_ok=True)
+    with ledger.open("a", encoding="utf-8") as fh:
+        for r in rows:
+            fh.write(json.dumps(r, sort_keys=True) + "\n")
+
+
+# ========================================================================== selftest
+def _selftest() -> int:
+    """Deterministic, no-cluster self-test of every decision rule (house pattern:
+    `status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
+
+    The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
+    assertion standing between this ledger and a permanent false proof of R-STATE-03.
+    """
+    import tempfile
+    ok = True
+
+    def check(label: str, cond: bool) -> None:
+        nonlocal ok
+        if not cond:
+            sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
+            ok = False
+
+    # --- token derivation, against the dr-restore/node-loss shape ---------------------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
+            "  name: dr-restore\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-03: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-03\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo OFF_CLUSTER_TARGET_OK'\n"
+            "        check:\n"
+            "          (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
+            "    - script:\n        content: 'echo x'\n        check:\n"
+            "          (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
+        s = parse_sentinels(tdir / "chainsaw-test.yaml")
+        check("pass tokens derived from the test file",
+              s["pass"] == ["OFF_CLUSTER_TARGET_OK",
+                            "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
+        check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
+              s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
+        check("terminal pass token is the LAST non-skip token",
+              s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
+
+        smap = sentinel_map(root / "tests")
+        check("sentinel_map keys on metadata.name and carries layer + labels",
+              smap["dr-restore"]["layer"] == "L3"
+              and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
+
+    # --- decide(): the SKIP-is-never-proven contract ----------------------------------
+    l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
+          "skip": ["DR_RESTORE_SKIPPED_OK"],
+          "terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
+
+    # *** THE CASE THIS WHOLE FILE EXISTS FOR ***
+    sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
+    check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
+          proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
+                          "DR_RESTORE_SKIPPED_OK"], "t")
+    check("a skip token BEATS a pass token in the same log (worst wins)",
+          proof == "skipped")
+
+    _, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
+    check("green JUnit with no sentinel in the log is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "passed", l3, [], "t")
+    check("green JUnit with NO log lines attributed is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "failed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("a failing JUnit outranks a pass token in the log", proof == "failed")
+
+    _, proof, _ = decide("L3", "skipped", l3, [], "t")
+    check("JUnit <skipped/> records junit-skipped, never green",
+          proof == "junit-skipped")
+
+    multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
+    _, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
+    check("every declared pass token must be present, not just the terminal one",
+          proof == "unknown-sentinel-absent")
+    _, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
+    check("all pass tokens present == proven", proof == "proven")
+
+    none = {"pass": [], "skip": [], "terminal_pass": None}
+    _, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
+    check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("L1", "passed", none, [], "t")
+    check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
+          proof == "proven")
+    _, proof, _ = decide("L0", "skipped", none, [], "t")
+    check("an L0 JUnit skip is still never green", proof == "junit-skipped")
+    _, proof, _ = decide("META", "passed", none, [], "t")
+    check("META (a Chainsaw layer) needs a sentinel like L2/L3",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("", "passed", none, [], "t")
+    check("an unknown/blank layer fails CLOSED — sentinel required",
+          proof == "unknown-no-sentinel-declared")
+
+    # --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
+    # Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
+    # `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
+    # inside `=== COMMAND` on every run of a skip-gated test.
+    nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
+          "skip": ["NODE_LOSS_SKIPPED_OK"],
+          "terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+    _CMD = ("    l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |\n"
+            "        === COMMAND\n"
+            "        /usr/bin/sh -c set -euo pipefail\n"
+            "        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
+            "          echo NODE_LOSS_SKIPPED_OK\n"
+            "\n"
+            "          exit 0\n"
+            "        fi\n"
+            "        echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
+
+    ran = classify_sections(_CMD + "        === STDOUT\n"
+                            "        ready nodes: 3\n"
+                            "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
+    sent, proof, _ = decide("L3", "passed", nl, ran, "t")
+    check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
+          proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+    selfskipped = classify_sections(_CMD + "        === STDOUT\n"
+                                    "        SKIP: 'longhorn' StorageClass absent\n"
+                                    "        NODE_LOSS_SKIPPED_OK\n")
+    sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
+    check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
+          proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
+
+    neither = classify_sections(_CMD + "        === STDOUT\n        ready nodes: 3\n")
+    _, proof, _ = decide("L3", "passed", nl, neither, "t")
+    check("both tokens in COMMAND and neither in STDOUT is NOT proven",
+          proof == "unknown-sentinel-absent")
+
+    both_out = classify_sections("        === STDOUT\n"
+                                 "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                                 "        NODE_LOSS_SKIPPED_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, both_out, "t")
+    check("skip still BEATS pass when both are in genuine output", proof == "skipped")
+
+    stderr_skip = classify_sections("        === STDERR\n        NODE_LOSS_SKIPPED_OK\n"
+                                    "        === STDOUT\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
+    check("a skip token in a non-source section (STDERR) still costs the proof",
+          proof == "skipped")
+
+    stderr_pass = classify_sections("        === STDERR\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
+    check("a pass token OUTSIDE stdout cannot prove anything",
+          proof == "unknown-sentinel-absent")
+
+    # --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
+    gorows = classify_sections("=== RUN   chainsaw\n=== PAUSE chainsaw\n"
+                               "=== CONT  chainsaw/node-loss\n"
+                               "        === COMMAND\n        echo NODE_LOSS_SKIPPED_OK\n"
+                               "    l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
+                               "--- PASS: chainsaw/node-loss (131.09s)\n")
+    secs = [sec for sec, _ in gorows]
+    check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
+          secs[:3] == ["", "", ""])
+    check("a section covers its indented body and ends at a less-indented line",
+          secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
+    check("a log with no === markers stays wholly scannable (UNSECTIONED)",
+          {sec for sec, _ in classify_sections("plain line\n  another\n")} == {UNSECTIONED})
+
+    # --- the preserved run itself, as a fixture ----------------------------------------
+    # runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
+    # (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
+    fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
+    if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
+        rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
+        row = next((r for r in rows if r["test"] == "node-loss"), None)
+        check("preserved real L3 run records node-loss/R-STATE-02 as proven",
+              row is not None and row["proof"] == "proven"
+              and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
+              and row["requirements"] == ["R-STATE-02"])
+        check("preserved real L3 run attributes (no DEGRADED fallback needed)",
+              row is not None and "DEGRADED" not in row["note"])
+        pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
+        secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
+        check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
+              secs == {"COMMAND"})
+        secs = {sec for sec, line in pairs
+                if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+        check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
+
+    # --- log attribution --------------------------------------------------------------
+    log = ("[run.sh] Coverage report first\n"
+           "    | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
+           "        NODE_LOSS_SKIPPED_OK\n"
+           "    | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
+           "        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
+    b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
+    check("log lines are attributed to the testcase whose header preceded them",
+          attributed
+          and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
+          and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
+    check("a pass token does NOT leak across testcase boundaries",
+          not any("DR_RESTORED" in l for _, l in b["node-loss"]))
+    check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
+          all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
+
+    # Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
+    # regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
+    real_hdr = ("    l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
+                "\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
+                "        === STDOUT\n        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
+    check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
+          attributed_real and "node-loss" in b2
+          and ("STDOUT", "        NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
+    _, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
+    check("an unattributable log is reported as such (degraded mode)", not attributed2)
+
+    smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
+             "b": {"pass": ["DNS_OK"], "skip": []}}
+    check("only tokens owned by exactly one test may be used in degraded mode",
+          unique_tokens(smap2) == {"A_ONLY_OK"})
+
+    # --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "node-loss"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "kind: Test\nmetadata:\n  name: node-loss\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-02: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-02\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo hi'\n        check:\n"
+            "          (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
+            " || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
+        rd = root / "runs" / "20260901T120000Z-abc1234"
+        rd.mkdir(parents=True)
+        (rd / "report.junit.xml").write_text(
+            '<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
+            'name="node-loss"><properties>'
+            '<property name="requirements" value="R-STATE-02"/>'
+            '<property name="layer" value="L3"/></properties></testcase>'
+            "</testsuite></testsuites>")
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      SKIP: 'longhorn' StorageClass absent\n"
+            "      NODE_LOSS_SKIPPED_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
+              len(rows) == 1 and rows[0]["junit_result"] == "passed"
+              and rows[0]["proof"] == "skipped"
+              and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
+        check("end-to-end: row carries ts from the run-dir name and the R-ID label",
+              rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
+              and rows[0]["requirements"] == ["R-STATE-02"])
+
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
+              rows[0]["proof"] == "proven"
+              and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+        ledger = root / "ledger.jsonl"
+        append_rows(ledger, rows)
+        append_rows(ledger, rows)
+        check("ledger is append-only JSONL (one parsable object per line)",
+              len(ledger.read_text().strip().splitlines()) == 2
+              and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
+
+    if not ok:
+        return 1
+    sys.stderr.write("record_run.py selftest: PASS\n")
+    return 0
+
+
+# ============================================================================== main
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--run-dir", type=Path,
+                    help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"append-only ledger (default {DEFAULT_LEDGER})")
+    ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
+    ap.add_argument("--repo", type=Path, default=REPO_ROOT)
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
+    ap.add_argument("--chainsaw-version", default=None,
+                    help="override the versions.env pin (recorded verbatim)")
+    ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
+                    help="override the `kubectl version` probe (recorded verbatim)")
+    ap.add_argument("--dry-run", action="store_true",
+                    help="print the rows that WOULD be appended; write nothing")
+    ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
+    args = ap.parse_args()
+
+    if args.selftest:
+        return _selftest()
+    if args.run_dir is None:
+        sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
+        return 1
+
+    run_dir = args.run_dir.resolve()
+    junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
+    # BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
+    # recording from it alone is exactly the false proof this tool exists to prevent.
+    missing = [p.name for p in (junit, log) if not p.is_file()]
+    if missing:
+        sys.stderr.write(
+            f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
+            f"The JUnit alone cannot distinguish a self-skip from a real pass "
+            f"(see module docstring). SKIPPED (not green).\n")
+        return 3
+
+    try:
+        rows = build_rows(
+            run_dir, args.tests, args.env,
+            args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
+            args.kube_version or server_kube_version(),
+            git_sha(args.repo))
+    except ET.ParseError as e:
+        sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
+        return 1
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
+        return 1
+
+    if not rows:
+        sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
+                         f"SKIPPED (not green).\n")
+        return 3
+
+    for r in rows:
+        print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
+              f"({','.join(r['requirements']) or 'untagged'}) "
+              f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
+    counts: dict[str, int] = {}
+    for r in rows:
+        counts[r["proof"]] = counts.get(r["proof"], 0) + 1
+    print(f"[record] {len(rows)} testcase(s): "
+          + ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
+
+    if args.dry_run:
+        print("[record] --dry-run: ledger NOT written.")
+        return 0
+    try:
+        append_rows(args.ledger, rows)
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
+        return 1
+    print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
+    print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/lib/status.py

@@ -15,26 +15,42 @@ This generator derives, from the REPOSITORY ALONE, three orthogonal axes per req
                        from a Flux Kustomization entrypoint (i.e. actually reconciled)?
   2. VERIFICATION    — is there a requirement-tagged test, at which layer, and is it an
                        EXECUTABLE test or a TODO scaffold?
-  3. PROOF           — has that test actually passed?  *** DELIBERATELY NOT DERIVED. ***
+  3. PROOF           — has that test actually passed?  Derived ONLY from the committed
+                       run ledger, and ONLY as far as that ledger's evidence reaches.
 
-WHAT THIS TOOL REFUSES TO DERIVE (and why)
+PROOF — WHAT CHANGED, AND WHAT DID NOT
-------------------------------------------
+--------------------------------------
-PROOF-OF-PASS IS NOT DERIVABLE FROM REPO STATE. Nothing tracked in git records a test
+This report used to say `not-derivable` for every requirement, because nothing tracked in
-result: `conformance/_artifacts/` and `results/` are git-ignored, and the only prose
+git recorded a test result and the only prose record of a green run lived in
-record of a green run lives in DECISIONS-LOG.md — hand-maintained, unstructured, and
+DECISIONS-LOG.md — hand-maintained, unstructured, exactly the source the operator ruled
-exactly the kind of source the operator has ruled out. The project's own culture is that
+out. The fix this docstring asked for now exists: `conformance/runs/ledger.jsonl`, an
-"merged" is never "done" (agents/definition-of-done.md, R-TEST-03), so this report will
+append-only ledger written by `lib/record_run.py` from a run's JUnit **and its run log**
-NOT infer, guess, or default a requirement to green. Every requirement's proof state is
+(see conformance/.gitignore for the negations that keep both in git).
-reported as `not-derivable`, and the headline says so. If an untracked JUnit artifact
-happens to be lying in the workspace it is surfaced ONLY as a quarantined advisory with
-its mtime — never counted, never mixed into a status.
 
-The honest fix is a committed, machine-readable run ledger; until that exists, the most
+THE REFUSAL POSTURE IS UNCHANGED. Evidence replaced "not derivable" only where evidence
-this report can say is "wired and testable", never "proven".
+exists; nowhere is optimism substituted for it:
+
+  * A ledger row reaches `proven` only if `record_run.py` saw that test's own PASS
+    sentinel token in the run log. The L2/L3 tests SELF-SKIP GREEN — Chainsaw records
+    `passed` on `DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries no
+    stdout — so a skip token maps to `skipped` and can never become proof.
+  * A proof is honoured only while it still describes the current code: if any of the
+    requirement's implementing manifests or verifying tests changed in a commit that is
+    NOT an ancestor of the run's `git_sha`, the row degrades to `stale-proof`, which is
+    not green. A run recorded from a dirty tree (`<sha>-dirty`) is never honoured.
+  * A requirement with no ledger row, or whose verifying tests did not ALL report proven,
+    stays `not-derivable` / `unknown` — never rounded up.
+  * The untracked `_artifacts/report.junit.xml` is STILL only a quarantined advisory with
+    its mtime. It is not the ledger, it is not evidence, and it feeds no status.
+
+With an empty or absent ledger this report behaves exactly as it did before: every
+requirement `not-derivable`, headline PROVEN 0, "the most this report can say is wired and
+testable, never proven".
 
 READS (all read-only)
   requirements/registry.yaml     the authoritative R-* mirror  (via lib/coverage.py)
   tests/**/chainsaw-test.yaml    requirement tags + layer + executable annotation
+  runs/ledger.jsonl              COMMITTED proof-of-pass rows (via lib/record_run.py)
   clusters/<env>/               Flux entrypoint -> the reconciliation graph
   `git ls-files`                 TRACKED files only; untracked scratch is not repo state
   ../home-network-k3s/docs/00-vision-and-scope.md   OPTIONAL registry<->spec parity check
@@ -92,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io"
 #   conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there.
 #   ci/tests          — gate unit-test fixtures embed example R-IDs as DATA.
 #   registry.yaml     — the requirement mirror itself, not an implementation of anything.
-IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/")
+# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the
+# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as
+# implementing evidence makes `last_change` move every time proof is recorded, so a row's
+# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is
+# committed — recording the proof destroys it. The ledger is the proof record; it must not
+# participate in the freshness computation about itself. (Found live 2026-09-01 on the first
+# PROVEN row.)
+IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/")
 IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)
 
 
@@ -343,6 +366,169 @@ def spec_requirement_ids(spec_root: Path) -> set[str] | None:
     return set(RID_RE.findall(doc.read_text()))
 
 
+# ========================================================================== ledger
+# PROOF axis. Everything here reads `conformance/runs/ledger.jsonl` (written ONLY by an
+# explicit `lib/record_run.py` / `run.sh --record` invocation) and downgrades relentlessly.
+# Nothing in this section may ever turn absent, ambiguous or aged evidence into green.
+
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# Severity ranking, WORST FIRST. A requirement's proof is the MINIMUM over its verifying
+# tests, so one failed/skipped/missing test drags the whole requirement down and a single
+# proven test can never carry the others. `stale-proof` ranks below `proven` on purpose:
+# it is a real past pass that no longer describes the current code, and it is NOT green.
+PROOF_ORDER = {
+    "failed": 0,              # a ledger row recorded a JUnit failure/error
+    "unknown": 1,             # ran, but the log could not evidence pass vs self-skip
+    "not-derivable": 2,       # no ledger row for this test at all (the pre-ledger default)
+    "junit-skipped": 3,       # JUnit <skipped/> — did not execute
+    "skipped": 4,             # SELF-SKIPPED green: exited 0 without exercising the behaviour
+    "stale-proof": 5,         # proven at a sha the implementation/test has since moved past
+    "proven": 6,              # the ONLY green state
+}
+# record_run.py's two "we cannot tell" states collapse to one reported state here; the
+# per-test detail in status.json keeps the distinction for whoever has to fix it.
+_PROOF_ALIASES = {
+    "unknown-no-sentinel-declared": "unknown",
+    "unknown-sentinel-absent": "unknown",
+}
+
+
+def read_ledger(path: Path) -> list[dict]:
+    """Parse the append-only JSONL ledger. A malformed line is SKIPPED, never guessed at.
+
+    Absent or empty file -> [] , which reproduces this report's pre-ledger behaviour
+    exactly (every requirement `not-derivable`, headline PROVEN 0).
+    """
+    if not path.is_file():
+        return []
+    rows: list[dict] = []
+    for line in path.read_text(errors="replace").splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        try:
+            obj = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(obj, dict) and obj.get("test"):
+            rows.append(obj)
+    return rows
+
+
+def latest_by_test(rows: list[dict]) -> dict[str, dict]:
+    """Most recent ledger row per test name (ts_utc, then file order as the tiebreak).
+
+    Latest wins rather than best-ever: a test that has since gone back to self-skipping
+    must stop reading as proven.
+    """
+    latest: dict[str, dict] = {}
+    for i, r in enumerate(rows):
+        key = r["test"]
+        prev = latest.get(key)
+        if prev is None or (r.get("ts_utc", ""), i) >= (prev.get("ts_utc", ""),
+                                                       prev["_i"]):
+            latest[key] = {**r, "_i": i}
+    return latest
+
+
+class _GitOracle:
+    """Small cached wrapper for the two git questions the staleness rule asks."""
+
+    def __init__(self, repo: Path):
+        self.repo = repo
+        self._last: dict[tuple[str, ...], str | None] = {}
+        self._anc: dict[tuple[str, str], bool] = {}
+
+    def _git(self, *args: str) -> subprocess.CompletedProcess:
+        return subprocess.run(["git", "-C", str(self.repo), *args],
+                              capture_output=True, text=True)
+
+    def last_change(self, paths: tuple[str, ...]) -> str | None:
+        """Commit that last touched any of `paths`; None if git knows none of them."""
+        if not paths:
+            return None
+        if paths in self._last:
+            return self._last[paths]
+        p = self._git("log", "-1", "--format=%H", "--", *paths)
+        sha = p.stdout.strip() if p.returncode == 0 else ""
+        self._last[paths] = sha or None
+        return self._last[paths]
+
+    def is_ancestor(self, older: str, newer: str) -> bool:
+        """True iff `older` is an ancestor of (or equal to) `newer`.
+
+        False whenever either rev is unresolvable — an unknown or `-dirty` sha (a run
+        recorded from a modified working tree) can never satisfy the freshness test, which
+        is the intended fail-closed direction.
+        """
+        key = (older, newer)
+        if key in self._anc:
+            return self._anc[key]
+        ok = self._git("merge-base", "--is-ancestor", older, newer).returncode == 0
+        self._anc[key] = ok
+        return ok
+
+
+def derive_proof(git: _GitOracle, rid: str, verifiers: list[dict],
+                 impl_files: list[dict], latest: dict[str, dict]) -> dict:
+    """PROOF for one requirement, from the ledger, with the staleness rule applied.
+
+    STALENESS RULE — a proof is a claim about a specific tree, not a permanent badge:
+      relevant paths = every implementing artifact this report found for the requirement
+                       + every verifying test's directory
+      last_change    = `git log -1 -- <relevant paths>`
+      A row's `proven` is honoured only if `last_change` is an ANCESTOR of the row's
+      `git_sha`, i.e. the run happened at or after the newest change to the code it
+      claims to prove. Otherwise it degrades to `stale-proof` (not green). A `-dirty`
+      sha, a sha this clone does not have (rebased/dropped), or a path git has never
+      seen (an uncommitted test) all fail the ancestry test and degrade the same way.
+
+    The requirement's state is the MINIMUM over its verifying tests: every verifying test
+    must be proven and fresh before the requirement reads `proven`. A test with no ledger
+    row contributes `not-derivable`, so a partially-recorded run cannot manufacture green.
+    """
+    paths = sorted({f["path"] for f in impl_files}
+                   | {os.path.dirname(v["path"]) for v in verifiers if v.get("path")})
+    last_change = git.last_change(tuple(paths))
+
+    per_test: list[dict] = []
+    for v in verifiers:
+        row = latest.get(v["name"])
+        if row is None:
+            per_test.append({"test": v["name"], "state": "not-derivable",
+                             "reason": "no ledger row — this test has never been "
+                                       "recorded by run.sh --record / record_run.py"})
+            continue
+        base = _PROOF_ALIASES.get(row.get("proof", ""), row.get("proof", "unknown"))
+        if base not in PROOF_ORDER:
+            base = "unknown"
+        state, reason = base, row.get("note", "")
+        if base == "proven":
+            sha = row.get("git_sha", "")
+            if last_change and not git.is_ancestor(last_change, sha):
+                state = "stale-proof"
+                reason = (f"proven at {sha[:12]}, but the implementing manifests / test "
+                          f"changed in {last_change[:12]}, which that run predates")
+        per_test.append({"test": v["name"], "state": state, "reason": reason,
+                         "run_id": row.get("run_id"), "ts_utc": row.get("ts_utc"),
+                         "git_sha": row.get("git_sha"), "env": row.get("env"),
+                         "sentinel": row.get("sentinel"),
+                         "junit_result": row.get("junit_result")})
+
+    if not per_test:
+        state = "not-derivable"
+        reason = "no requirement-tagged test exists to be proven"
+    else:
+        worst = min(per_test, key=lambda d: PROOF_ORDER[d["state"]])
+        state, reason = worst["state"], worst["reason"]
+        if state == "not-derivable" and not reason:
+            reason = "no committed ledger row for this requirement's verifying test(s)"
+
+    return {"state": state, "reason": reason, "last_change_sha": last_change,
+            "evidence_paths": paths, "per_test": per_test}
+
+
 # ========================================================================== build
 IMPL_ORDER = {"none": 0, "off-cluster": 1, "unwired": 2, "reconciled-partial": 3,
               "reconciled": 4}
@@ -369,7 +555,13 @@ LABELS = {
 
 def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, list[dict]],
           applied: set[str], gitops_roots: set[str], spec_ids: set[str] | None,
-          env: str) -> dict:
+          env: str, ledger_rows: list[dict] | None = None) -> dict:
+    # PROOF axis inputs. `ledger_rows=None` / `[]` reproduces the pre-ledger behaviour:
+    # `latest` is empty, so every requirement derives `not-derivable` exactly as before.
+    ledger_rows = ledger_rows or []
+    latest = latest_by_test(ledger_rows)
+    git = _GitOracle(repo)
+
     classified = [classify_test(t) for t in tests]
     by_req_tests: dict[str, list[dict]] = {r["id"]: [] for r in registry}
     for t, c in zip(tests, classified):
@@ -439,11 +631,9 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
                 "blocked_by": sorted({t["blocked_by"] for t in ts if t["blocked_by"]}),
                 "tests": ts,
             },
-            # NEVER derived. See the module docstring.
+            # Derived ONLY from the committed ledger, and only as far as it reaches.
-            "proof": {"state": "not-derivable",
+            # With no ledger row this stays `not-derivable`, as it always was.
-                      "reason": "no committed, machine-readable record of a test run "
+            "proof": derive_proof(git, rid, ts, ev, latest),
-                                "exists in the repo (_artifacts/ and results/ are "
-                                "git-ignored); DECISIONS-LOG.md prose is not a signal"},
             "status": LABELS[(impl_state, verify_state)],
         }
 
@@ -492,8 +682,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
             and v["verification"]["state"] == "executable"),
         "safety_critical_total": count(lambda v: v["safety_critical"]),
         "safety_critical_without_executable_l3": len(safety_scaffold),
-        "proven": 0,
+        # PROVEN is now a real count — but only of requirements whose EVERY verifying
-        "proven_derivable": False,
+        # test has a committed ledger row that saw its pass sentinel, at a sha the
+        # implementation has not moved past. `stale-proof` and `skipped` are counted
+        # separately and are NOT green.
+        "proven": count(lambda v: v["proof"]["state"] == "proven"),
+        "proof_stale": count(lambda v: v["proof"]["state"] == "stale-proof"),
+        "proof_skipped": count(lambda v: v["proof"]["state"] == "skipped"),
+        "proof_failed": count(lambda v: v["proof"]["state"] == "failed"),
+        "proof_unknown": count(lambda v: v["proof"]["state"] == "unknown"),
+        "proof_not_derivable": count(
+            lambda v: v["proof"]["state"] in ("not-derivable", "junit-skipped")),
+        "ledger_rows": len(ledger_rows),
+        "proven_derivable": bool(ledger_rows),
         "coverage_style_covered": count(lambda v: v["verification"]["state"] != "none"),
     }
     summary["uncited_by_any_reconciled_manifest"] = count(
@@ -552,16 +753,39 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append(f"| Safety-critical requirements whose L3 test is a scaffold | "
              f"**{s['safety_critical_without_executable_l3']}** of "
              f"{s['safety_critical_total']} |")
-    o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
+    if s["proven_derivable"]:
-    o.append("")
+        o.append(f"| **PROVEN (every verifying test passed, evidenced)** | "
-    o.append("> **Proof is deliberately not derived.** Nothing tracked in git records a "
+                 f"**{s['proven']}** |")
-             "test result: `conformance/_artifacts/` and `results/` are git-ignored, and "
+        o.append(f"| Proof exists but is STALE (code changed after the run) | "
-             "the only record of a green run is DECISIONS-LOG.md prose — hand-maintained, "
+                 f"{s['proof_stale']} |")
-             "which is the very thing this report replaces. Per "
+        o.append(f"| Verifying test SELF-SKIPPED green (not proof) | {s['proof_skipped']} |")
-             "`agents/definition-of-done.md` a requirement is done only when its tagged "
+        o.append(f"| Verifying test FAILED in the recorded run | {s['proof_failed']} |")
-             "test **passes**, so the strongest claim this report can make is "
+        o.append(f"| Ran, but the log could not evidence pass vs self-skip | "
-             "*wired and testable*. Closing the gap needs a committed, machine-readable "
+                 f"{s['proof_unknown']} |")
-             "run ledger; until then, treat every row below as unproven.\n")
+        o.append(f"| No committed ledger row yet | {s['proof_not_derivable']} |")
+        o.append("")
+        o.append(f"> **Proof comes from `conformance/runs/ledger.jsonl`** "
+                 f"({s['ledger_rows']} row(s)), written only by an explicit "
+                 f"`run.sh --record` / `lib/record_run.py` invocation. A row reaches "
+                 f"`proven` ONLY when the run log carried that test's own pass sentinel "
+                 f"token — the L2/L3 tests self-skip GREEN (Chainsaw records `passed` on "
+                 f"`DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries "
+                 f"no stdout), so a skip token maps to `skipped` and can never become "
+                 f"proof. A proof is honoured only while the implementing manifests and "
+                 f"the test itself have not changed since the recorded `git_sha`; "
+                 f"otherwise it degrades to `stale-proof`, which is not green.\n")
+    else:
+        o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
+        o.append("")
+        o.append("> **No proof is derivable: `conformance/runs/ledger.jsonl` is absent or "
+                 "empty.** The ledger is the committed, machine-readable run record; until "
+                 "a run is captured with `run.sh --record`, nothing tracked in git records "
+                 "a test result (`conformance/_artifacts/` and `results/` are git-ignored, "
+                 "and DECISIONS-LOG.md prose is hand-maintained — the very thing this "
+                 "report replaces). Per `agents/definition-of-done.md` a requirement is "
+                 "done only when its tagged test **passes**, so the strongest claim this "
+                 "report can make is *wired and testable*; treat every row below as "
+                 "unproven.\n")
     o.append(f"> For contrast, `coverage.md` counts **{s['coverage_style_covered']}** "
              f"requirements as covered — that count includes the "
              f"{s['verify_scaffold_only']} whose only verifying test is a TODO scaffold.\n")
@@ -612,7 +836,15 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
             if ver["blocked_by"]:
                 vcell += f" · blocked-by: {'; '.join(ver['blocked_by'])}"
         sc = " ⚠safety" if v["safety_critical"] else ""
-        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | not-derivable |")
+        pr = v["proof"]
+        if pr["state"] == "not-derivable":
+            pcell = "not-derivable"
+        else:
+            runs = sorted({d["run_id"] for d in pr["per_test"] if d.get("run_id")})
+            pcell = f"**{pr['state']}**" if pr["state"] == "proven" else pr["state"]
+            if runs:
+                pcell += f" ({', '.join(runs)})"
+        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | {pcell} |")
 
     if rep["safety_critical_without_executable_l3"]:
         o.append("\n## Safety-critical requirements whose L3 fault test is a SCAFFOLD "
@@ -701,22 +933,76 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append("| executable vs scaffold | TODO markers in the test body outrank the "
              "`executable` annotation; a real body without the annotation still counts "
              "and is flagged | conservative — ambiguity never upgrades |")
-    o.append("| **test actually passed** | **NOT DERIVED** — nothing in git records a "
+    o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
-             "run result | refused |")
+             "every verifying test: JUnit result **plus** that test's own pass sentinel "
+             "token grepped from the run log | exact where a run was recorded, refused "
+             "everywhere else — a self-skip token or a missing/unrecognised sentinel "
+             "NEVER yields `proven` |")
+    o.append("| proof still current | the row's `git_sha` must be a descendant of the "
+             "last commit touching the requirement's implementing manifests or its test "
+             "| conservative — an unresolvable or `-dirty` sha degrades to `stale-proof` |")
     o.append("| cluster liveness (pods/CRs healthy) | **NOT DERIVED** — no cluster is "
              "contacted; `reconciled` means Flux *would* apply it, not that it is "
              "healthy | refused |")
     o.append("| phase/board status | **NOT DERIVED** — `agents/task-board.md` is stale "
              "and is not a signal | refused |")
 
+    o.append("\n## Run ledger — the committed proof-of-pass evidence\n")
+    if not s["proven_derivable"]:
+        o.append("- `conformance/runs/ledger.jsonl` holds no rows. Capture a run with "
+                 "`./run.sh --record` (recording is never implicit: a run that silently "
+                 "records itself is how a bad run becomes a permanent false proof), then "
+                 "commit `conformance/runs/<run-id>/` **with its `run.log`** and the "
+                 "ledger. The log is the load-bearing half — Chainsaw's JUnit carries no "
+                 "stdout, so without it a self-skip is indistinguishable from a pass.")
+    else:
+        o.append(f"- {s['ledger_rows']} row(s) in `conformance/runs/ledger.jsonl`. "
+                 f"Per-test evidence for every non-`not-derivable` requirement below; "
+                 f"full rows (env, chainsaw/kube versions, sentinel, note) are in "
+                 f"`status.json` under `by_requirement.<R-ID>.proof.per_test`.\n")
+        o.append("| Requirement | Proof | Test | Ledger state | Sentinel | Run |")
+        o.append("|---|---|---|---|---|---|")
+        for rid, v in rep["by_requirement"].items():
+            pr = v["proof"]
+            if pr["state"] == "not-derivable":
+                continue
+            for d in pr["per_test"]:
+                o.append(f"| {rid} | {pr['state']} | `{d['test']}` | {d['state']} | "
+                         f"`{d.get('sentinel') or '—'}` | {d.get('run_id') or '—'} |")
+        stale = [(rid, v) for rid, v in rep["by_requirement"].items()
+                 if v["proof"]["state"] == "stale-proof"]
+        if stale:
+            o.append("\n### Stale proofs — a real pass that no longer describes this tree\n")
+            o.append("These requirements DID pass, but their implementing manifests or "
+                     "their verifying test changed in a commit the recorded run predates. "
+                     "A stale proof is not green; re-run with `--record`.\n")
+            for rid, v in stale:
+                pr = v["proof"]
+                o.append(f"- **{rid}** — {pr['reason']}")
+        skipped = [(rid, v) for rid, v in rep["by_requirement"].items()
+                   if v["proof"]["state"] == "skipped"]
+        if skipped:
+            o.append("\n### SELF-SKIPPED green — recorded, and deliberately NOT proof\n")
+            o.append("Chainsaw reported `passed` for these, because the test exited 0 "
+                     "after emitting its skip token: a precondition (Longhorn "
+                     "StorageClass, Velero CRDs, >=3 Ready nodes) was absent, so the "
+                     "behaviour was never exercised. This is exactly the row a "
+                     "JUnit-only ledger would have recorded as green.\n")
+            for rid, v in skipped:
+                toks = ", ".join(sorted({d.get("sentinel") or "?"
+                                         for d in v["proof"]["per_test"]
+                                         if d["state"] == "skipped"}))
+                o.append(f"- **{rid}** — skip token(s): `{toks}`")
+
     o.append("\n## Advisory — untracked local run artifact (NOT counted)\n")
     if advisory:
         o.append(f"- `{advisory['path']}` (untracked, git-ignored), last written "
                  f"{advisory['mtime']}: {advisory['tests']} testcase(s), "
                  f"{advisory['failures']} failure(s), {advisory['skipped']} skipped.")
         o.append(f"- requirement IDs it names: {', '.join(advisory['requirements']) or '—'}")
-        o.append("- This is one developer's local workspace, not repo state. It is shown "
+        o.append("- This is one developer's local workspace, not repo state, and it is NOT "
-                 "for orientation only and contributes to NO status above.")
+                 "the ledger. It is shown for orientation only and contributes to NO "
+                 "status above — a JUnit alone cannot tell a self-skip from a pass.")
     else:
         o.append("- none present.")
     return "\n".join(o) + "\n"
@@ -841,6 +1127,122 @@ def _selftest() -> int:
               classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
               == "scaffold")
 
+    # ---------------------------------------------------------------- PROOF / ledger
+    # These are the assertions that stand between this report and a fake green.
+    check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
+          read_ledger(Path("/nonexistent/ledger.jsonl")) == [])
+
+    def row(**kw):
+        base = {"ts_utc": "2026-09-01T12:00:00Z", "git_sha": "deadbeef", "env": "dev",
+                "chainsaw_version": "v0.2.12", "kube_version": "v1.36.4", "layer": "L3",
+                "test": "dr-restore", "requirements": ["R-STATE-03"],
+                "junit_result": "passed", "sentinel": None, "proof": "skipped",
+                "run_id": "20260901T120000Z-deadbee", "note": ""}
+        base.update(kw)
+        return base
+
+    verifiers = [{"name": "dr-restore",
+                  "path": "conformance/tests/l3-fault-injection/dr-restore/"
+                          "chainsaw-test.yaml"}]
+
+    with tempfile.TemporaryDirectory() as td:
+        gr = Path(td)
+        env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
+               "GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
+
+        def git(*a):
+            return subprocess.run(["git", "-C", str(gr), *a], capture_output=True,
+                                  text=True, env=env)
+
+        git("init", "-q", "-b", "main")
+        (gr / "platform").mkdir()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03\n")
+        git("add", "-A"); git("commit", "-qm", "one")
+        sha_a = git("rev-parse", "HEAD").stdout.strip()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03 v2\n")
+        git("add", "-A"); git("commit", "-qm", "two")
+        sha_b = git("rev-parse", "HEAD").stdout.strip()
+
+        oracle = _GitOracle(gr)
+        impl = [{"path": "platform/velero.yaml"}]
+
+        # *** THE CASE THIS WHOLE FEATURE EXISTS FOR ***
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="skipped", junit_result="passed",
+                                              sentinel="DR_RESTORE_SKIPPED_OK",
+                                              git_sha=sha_b)]))
+        check("a SELF-SKIP row on a GREEN JUnit is 'skipped', NEVER 'proven'",
+              pr["state"] == "skipped")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a proven row at the CURRENT sha is honoured", pr["state"] == "proven")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_a,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("STALENESS: proven at a sha the implementation has moved past degrades "
+              "to stale-proof", pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=f"{sha_b}-dirty",
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a run recorded from a DIRTY tree is never honoured as proof",
+              pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha="0" * 40)]))
+        check("a git_sha this clone cannot resolve degrades, never upgrades",
+              pr["state"] == "stale-proof")
+
+        for bad, expect in (("failed", "failed"),
+                            ("junit-skipped", "junit-skipped"),
+                            ("unknown-sentinel-absent", "unknown"),
+                            ("unknown-no-sentinel-declared", "unknown"),
+                            ("something-new-from-the-future", "unknown")):
+            pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                              latest_by_test([row(proof=bad, git_sha=sha_b)]))
+            check(f"ledger state {bad!r} maps to {expect!r}, never to proven",
+                  pr["state"] == expect)
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl, {})
+        check("no ledger row at all stays not-derivable",
+              pr["state"] == "not-derivable")
+
+        # Two verifying tests, only one proven: the requirement must NOT read proven.
+        two = verifiers + [{"name": "data-survival",
+                            "path": "conformance/tests/l3-fault-injection/"
+                                    "data-survival/chainsaw-test.yaml"}]
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b)]))
+        check("a requirement is proven only when EVERY verifying test is",
+              pr["state"] == "not-derivable")
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b),
+                                          row(test="data-survival", proof="skipped",
+                                              git_sha=sha_b)]))
+        check("worst-of wins across verifying tests (skip drags proven down)",
+              pr["state"] == "skipped")
+
+        # Latest row wins, not best-ever: a test that regressed to self-skipping must
+        # stop reading as proven.
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(ts_utc="2026-09-01T10:00:00Z",
+                                              proof="proven", git_sha=sha_b),
+                                          row(ts_utc="2026-09-02T10:00:00Z",
+                                              proof="skipped", git_sha=sha_b)]))
+        check("the LATEST ledger row wins, not the best one ever recorded",
+              pr["state"] == "skipped")
+
+        # A malformed ledger line is skipped, never guessed at.
+        lf = gr / "ledger.jsonl"
+        lf.write_text('{"test":"a","proof":"proven"}\nnot json at all\n\n'
+                      '{"proof":"proven"}\n')
+        rows = read_ledger(lf)
+        check("malformed / test-less ledger lines are dropped, not guessed at",
+              len(rows) == 1 and rows[0]["test"] == "a")
+
     if not ok:
         return 1
     sys.stderr.write("status.py selftest: PASS\n")
@@ -865,6 +1267,10 @@ def main() -> int:
                     help="exit 1 on defects this tool owns: registry/spec drift, "
                          "executable-annotation conflicts, or a safety-critical "
                          "requirement whose only L3 test is a scaffold")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"committed proof-of-pass ledger written by lib/record_run.py "
+                         f"(default {DEFAULT_LEDGER}). Absent or empty => every "
+                         f"requirement stays `not-derivable`, as before the ledger existed")
     ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
     args = ap.parse_args()
 
@@ -895,7 +1301,12 @@ def main() -> int:
     applied, nodes = reconciled_files(repo, entry)
     gitops_roots = {n["resolved"].split("/", 1)[0] for n in nodes if n["exists"]}
     impl = scan_implementation(repo, tracked_files(repo))
-    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env)
+    ledger_rows = read_ledger(args.ledger)
+    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env,
+                ledger_rows)
+    rep["ledger"] = {"path": os.path.relpath(args.ledger, REPO_ROOT),
+                     "rows": len(ledger_rows),
+                     "present": args.ledger.is_file()}
     rep["flux_graph"] = nodes
     rep["gitops_roots"] = sorted(gitops_roots)
     rep["reconciled_file_count"] = len(applied)
@@ -915,7 +1326,11 @@ def main() -> int:
           f"(+{s['wired_or_partial_and_testable'] - s['wired_and_testable']} partial) | "
           f"scaffold-only {s['verify_scaffold_only']} | "
           f"safety-critical w/o executable L3 "
-          f"{s['safety_critical_without_executable_l3']} | PROVEN: not derivable")
+          f"{s['safety_critical_without_executable_l3']} | "
+          + (f"PROVEN: {s['proven']} (stale {s['proof_stale']}, self-skipped "
+             f"{s['proof_skipped']}, failed {s['proof_failed']}, unknown "
+             f"{s['proof_unknown']}) from {s['ledger_rows']} ledger row(s)"
+             if s["proven_derivable"] else "PROVEN: not derivable (empty run ledger)"))
     print(f"[status] wrote {args.out/'status.md'} and {args.out/'status.json'}")
 
     rc = 0

conformance/run.sh

@@ -14,12 +14,33 @@
 #   ./run.sh --status                 # DERIVED STATUS report only (no cluster): per
 #                                     #   requirement, is it implemented, is it actually
 #                                     #   reconciled by Flux, is its test executable or a
-#                                     #   scaffold. Proof-of-pass is deliberately NOT
+#                                     #   scaffold, and whether the committed run ledger
-#                                     #   derived — see lib/status.py.
+#                                     #   (runs/ledger.jsonl) actually proves it — see
+#                                     #   lib/status.py. No ledger row => not-derivable.
 #   ./run.sh --lint                   # structural-validate test defs (no cluster)
 #   ./run.sh --strict-coverage        # hygiene gate: fail on tag/annotation drift
 #   ./run.sh --requirements R-NET-01  # ALSO asserts those IDs are covered (merge gate)
 #   ./run.sh --require-all            # phase-exit: every requirement must be covered
+#   ./run.sh --record                 # RUN AND CAPTURE: tee the whole run to
+#                                     #   runs/<UTC>-<short-sha>/run.log, copy the JUnit
+#                                     #   beside it, and append one ledger row per
+#                                     #   testcase to runs/ledger.jsonl (lib/record_run.py)
+#
+# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
+# --------------------------------------------------------------------------
+# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
+# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
+# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
+# that — `conformance/runs/` (see conformance/.gitignore for the negations).
+#
+# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
+# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
+#
+# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
+# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
+# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
+# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
+# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
 #
 # Artifacts (always written under conformance/_artifacts/, gitignored):
 #   _artifacts/report.junit.xml        JUnit (Chainsaw output, or skipped-scaffold shim)
@@ -28,6 +49,11 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Run ledger (TRACKED — commit these; --record only):
+#   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
+#   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
+#   runs/ledger.jsonl                         append-only, one row per testcase per run
+#
 # Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
 # execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
 # green — the harness is proven reproducible while later phases fill in behavior.
@@ -47,27 +73,30 @@ STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
 PYTHON="${PYTHON:-python3}"
 
+# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
+# under `tee`. Captured BEFORE parsing so nothing is normalized away.
+ORIG_ARGS=("$@")
+
 LAYER="all"
 REQS=""
 MODE="run"          # run | coverage-only | status | lint
 STRICT_COVERAGE=0
 REQUIRE_ALL=0
 FORCE_NO_CLUSTER=0
+RECORD=0
 
 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
 # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
 # exemption is explicit and reviewable, not silent.
-#   R-ENV-02   (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in
+#
-#                                    coverage.md). The coarse path->requirement map keys
+# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted
-#                                    clusters/* AND ci/* to it and the R-ENV-02 doc token
+# because they were unscaffolded while the coarse path->requirement map keys clusters/*
-#                                    appears in platform/kustomization.yaml's header, so ANY
+# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real
-#                                    infra/overlay/CI edit (e.g. enabling a component, bumping
+# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the
-#                                    SPEC_PIN) would otherwise false-block on it.
+# substitution mechanism that IS the "authored once, promoted" contract, so they are
-#   R-GITOPS-01 (state in git/Flux) — same: owned by WS-GITOPS, unscaffolded, and the ci/*
+# covered rather than exempt and the completeness gate can hold them to it. That test
-#                                    path-map keys to it, so editing any ci/ file (SPEC_PIN
+# names its own limits in its header: it proves the STATIC property (no manifest renders
-#                                    adoption included) would false-block.
+# an undefined variable), not the live-cluster half of either requirement.
-# These are visible GAPs, not hidden — removing them from this list is the right move the
-# moment WS-TEST adds their verifying tests.
 #
 # FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
 # tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
@@ -83,9 +112,11 @@ FORCE_NO_CLUSTER=0
 # R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
 # but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
 # longer a gap and has been removed from this list; the coverage report counts it as covered.
-DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01"
+DOCUMENTED_GAPS=""
 
-usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
+# Print the header block (line 2 through the closing `####` rule, which is dropped) as
+# help. Bounded by the rule rather than a fixed line number so the header can grow.
+usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
 
 while [ $# -gt 0 ]; do
   case "$1" in
@@ -97,6 +128,7 @@ while [ $# -gt 0 ]; do
     --strict-coverage) STRICT_COVERAGE=1; shift ;;
     --require-all)    REQUIRE_ALL=1; shift ;;
     --no-cluster)     FORCE_NO_CLUSTER=1; shift ;;
+    --record)         RECORD=1; shift ;;
     -h|--help)        usage; exit 0 ;;
     *) echo "unknown arg: $1" >&2; usage; exit 2 ;;
   esac
@@ -106,9 +138,69 @@ mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
 
 log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
 
+###############################################################################
+# --record: capture this run as committed evidence, then hand it to the ledger.
+#
+# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
+# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
+# operator-visible output — no second, differently-behaving "recording mode" to drift.
+# CONF_RECORD_CHILD guards against recursion.
+#
+# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
+# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
+# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
+###############################################################################
+if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
+  case "${MODE}" in
+    run) ;;
+    # --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
+    # test, so a ledger row from one of them would be a row about nothing.
+    *) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
+  esac
+  RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
+  RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
+  RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
+  mkdir -p "${RUN_DIR}"
+  # Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
+  # ever capture a report THIS run produced. Copying a stale artifact would pair someone
+  # else's result with this run's log — a fabricated pairing, which is precisely the class
+  # of false proof the ledger exists to prevent.
+  rm -f "${JUNIT_OUT}"
+  # Strip --record from the argv handed to the child; everything else passes through.
+  CHILD_ARGS=()
+  for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
+    [ "$a" = "--record" ] || CHILD_ARGS+=("$a")
+  done
+  log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
+  set +e
+  CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
+    | tee "${RUN_DIR}/run.log"
+  RUN_RC=${PIPESTATUS[0]}
+  set -e
+  if [ -f "${JUNIT_OUT}" ]; then
+    cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
+  else
+    log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
+  fi
+  # Explicit, separate step: read the JUnit + the log and append one row per testcase.
+  # rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
+  # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
+  # to the real ledger; operators should never set it.
+  "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
+    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
+    --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
+  log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
+  log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
+  exit "${RUN_RC}"
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
-  local flags=(--allow-gap "${DOCUMENTED_GAPS}")
+  # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
+  # requirement is exempt, and handing the flag an empty string would be a silent no-op
+  # that reads like an exemption in the CLI log.
+  local flags=()
+  [ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}")
   [ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
   # A tagged subset run is also the merge gate: the touched requirements MUST be covered.
   [ -n "${REQS}" ] && flags+=(--require "${REQS}")

conformance/runs/20260901T235117Z-e9a4b66/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="131.090121" />
+  </testsuite>
+</testsuites>

conformance/runs/20260901T235117Z-e9a4b66/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 23:51:18 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1788306678898809444
+        victim node (currently hosts the attached volume): k3s-dev-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-dev-2 labeled
+        node/k3s-dev-2 cordoned
+        cordoned + labelled victim node: k3s-dev-2
+        pod "writer" deleted from chainsaw-fond-marlin namespace
+        evicted writer off the victim node; volume detached from k3s-dev-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
+        recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-dev-2 uncordoned
+        node/k3s-dev-2 unlabeled
+        restored node to service: k3s-dev-2
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-fond-marlin namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:53:29 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (131.09s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260902T001907Z-fdc3265/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="149.053701" />
+  </testsuite>
+</testsuites>

conformance/runs/20260902T001907Z-fdc3265/run.log

@@ -0,0 +1,671 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 00:19:07 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true http://192.168.1.9:9000 velero
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=velero  endpoint=http://192.168.1.9:9000
+        backup endpoint host: 192.168.1.9
+        backup endpoint resolves to: 192.168.1.9
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 81)
+        endpoint is not any Pod IP (checked 186)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=192.168.1.9 bucket=velero
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
+        namespace/chainsaw-cunning-hen-dr-src created
+        namespace/chainsaw-cunning-hen-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1788308352223263220
+        original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
+        backup.velero.io/chainsaw-cunning-hen-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=Completed
+        backup chainsaw-cunning-hen-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-cunning-hen-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-cunning-hen-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
+        recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
+        velero objects for chainsaw-cunning-hen cleaned up (ns velero)
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
+        released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:21:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (149.05s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -0,0 +1,2 @@
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
+{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}

conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml

@@ -0,0 +1,60 @@
+# =============================================================================
+# L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism:
+# `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each
+# Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes
+# the empty string while the Kustomization still reports Ready — which emptied Cilium's
+# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
+#
+# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
+# WHEN  each `spec.path` is resolved to the exact file set it renders (following
+#       kustomization.yaml `resources:` transitively, not globbing the repo),
+# THEN  every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml
+#       or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
+#       operator on an undefined name (the silent-empty class strict mode does NOT catch);
+#       no unescaped expansion is unparseable by envsubst (which fails the WHOLE
+#       Kustomization); every path rendering a knob actually substitutes cluster-vars;
+#       every overlay-patched knob has a base default; and the check is wired into the
+#       merge gate as a blocking, unconditionally-run stage.
+#
+# This is the prerequisite proof for enabling kustomize-controller's
+# StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library
+# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
+# Test wraps the SAME script.
+#
+# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
+# a path includes — a deliberate superset of the post-`kustomize build` stream. It does
+# not render, does not prove the feature gate is enabled on the live kustomize-controller,
+# and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice
+# only: what git says a Kustomization renders is fully determined, with no variable
+# silently emptied on the way in. The live-state half is L2, untested today.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: flux-postbuild-substitution
+  labels:
+    conformance.home.k3s/R-ENV-02: "true"
+    conformance.home.k3s/R-GITOPS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every variable a substituted Flux path renders is defined in cluster-vars or
+      escaped, so post-build substitution cannot silently empty a value and enabling
+      StrictPostBuildSubstitutions cannot fail a legitimate manifest.
+spec:
+  steps:
+    - name: audit-every-substituted-flux-path
+      description: >-
+        Resolve each postBuild.substituteFrom Kustomization to the files it renders and
+        assert every ${...} is defined or escaped, then assert the gate wires it blocking.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/flux-postbuild-substitution/check.sh

@@ -0,0 +1,369 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — flux-postbuild-substitution
+# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
+#                   — spec docs/02-environments-and-provisioning.md)
+#         R-GITOPS-01 ("All desired state in git, reconciled by Flux."
+#                   — spec docs/05-gitops-and-delivery.md)
+#
+# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN
+# --------------------------------------------------------
+# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml`
+# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders,
+# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an
+# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports
+# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put
+# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that
+# fails OPEN"). "Authored once, promoted" is therefore only true while every token a
+# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
+# the repo asserted that. This check does.
+#
+# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
+# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
+#   ${DEFINED}        -> substituted with the cluster-vars value.
+#   ${UNDEFINED}      -> EMPTY STRING, silently. `--strict` / the cluster-wide
+#                        StrictPostBuildSubstitutions feature gate turns this into a
+#                        loud Kustomization failure ("variable not set (strict mode)").
+#   ${UNDEFINED:-d}   -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does
+#                        NOT catch this (it exempts default-operator forms by design),
+#                        so a shell guard written `[ -z "${X:-}" ]` inside a rendered
+#                        manifest is pre-resolved to `[ -z "" ]` and its else-branch
+#                        becomes unreachable in the container. Silent, and permanent.
+#   $${VAR}, $${V:-d}, $${ARR[0]}  -> emitted as the LITERAL ${...} for a downstream
+#                        shell/cloud-init consumer. Escaping is total: it neutralises
+#                        even the indexed form.
+#   $VAR, $(cmd)      -> left alone by envsubst; they need no escaping.
+#   ${...}, ${a b}, ${ARR[0]}  (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
+#                        variable name" / "missing closing brace") which fails the
+#                        WHOLE Kustomization — every resource in the path, not just
+#                        the offending one.
+# The escape contract this enforces is the one already documented in
+# `clusters/base/apps-vms.yaml`; this check is that comment made executable.
+#
+# GIVEN the repository as authored,
+# WHEN  every Flux Kustomization under clusters/ that declares postBuild.substituteFrom
+#       is resolved to the exact file set its `spec.path` renders (by following
+#       kustomization.yaml `resources:` transitively — a path, not a glob),
+# THEN  (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`;
+#       (2) no UNESCAPED expansion carries a default/length/substitution operator on an
+#           undefined name (the silent-empty class strict mode does not catch);
+#       (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) —
+#           that fails the entire Kustomization, so it is fatal wherever it appears;
+#       (4) every Kustomization that renders a cluster-vars token actually declares
+#           `substituteFrom` a ConfigMap named cluster-vars (else the literal token
+#           ships into the cluster);
+#       (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
+#           the "add the knob with a base default BEFORE referencing it" rule that the
+#           incident's interim discipline depends on;
+#   AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
+#           ci/gate.sh sets the merge block when the l0 stage fails.
+# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
+# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
+# comment survives into the built apps/wave-1 output), so a braced expression in a
+# comment is a live hazard, not prose.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
+#   - It does NOT render the manifests. It scans the SOURCE files a path includes,
+#     which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize
+#     drops some comments). A finding in a comment may not reach envsubst TODAY; it is
+#     still reported, because whether a given comment survives the build is a kustomize
+#     implementation detail that changes with the pinned version, and the failure it
+#     causes is total (assertion 3). The end-to-end proof — `kustomize build <path> |
+#     flux envsubst --strict` for every substituted path — needs kustomize + flux,
+#     which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh
+#     l0), so it cannot run here without permanently SKIPPING. It was run by hand and
+#     recorded in DECISIONS-LOG; this check is the reproducible static half.
+#   - It does NOT prove the cluster's live state matches git, nor that the
+#     StrictPostBuildSubstitutions feature gate is actually enabled on
+#     kustomize-controller. Both are cluster facts (L2), untested here. Tagging
+#     R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
+#     is fully determined, with no variable silently emptied on the way in.
+#   - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
+#     that an overlay's LAN block is self-consistent). Presence, not correctness.
+#   - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
+#     several are deliberately reference-only and say so in cluster-vars.yaml
+#     (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so
+#     failing on them would be a false block on documented intent.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green) if the thing under test is not present at all.
+if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
+  echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit of Flux postBuild substitution safety. Pure standard library.
+
+No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are
+documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization
+`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one
+definition of "what a Flux path actually renders" in the harness.
+"""
+from __future__ import annotations
+
+import os
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+sys.path.insert(0, str(ROOT / "conformance" / "lib"))
+import status as st                     # noqa: E402 — reuse the walk, don't fork it
+
+FAILURES: list[str] = []
+ADVISORIES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+# ------------------------------------------------------------------ cluster-vars
+# Parse contract: `data:` block of a single ConfigMap, one `  KEY: "value"` per line
+# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and
+# continuation-comment lines are indented further and never match).
+_DATA_KEY_RE = re.compile(r'^  ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')')
+
+
+def cluster_vars_keys(path: Path) -> set[str]:
+    keys: set[str] = set()
+    in_data = False
+    for raw in path.read_text().splitlines():
+        if re.match(r"^data:\s*$", raw):
+            in_data = True
+            continue
+        if not in_data:
+            continue
+        if st._TOP_KEY_RE.match(raw):   # a new top-level key ends the data block
+            break
+        m = _DATA_KEY_RE.match(raw)
+        if m:
+            keys.add(m.group(1))
+    return keys
+
+
+VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml"
+KEYS = cluster_vars_keys(VARS_FILE)
+check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed")
+
+# ------------------------------------------- Flux Kustomizations with substituteFrom
+# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc
+# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND
+# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the
+# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped.
+_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M)
+_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)")
+_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
+
+
+def substitute_sources(doc: str) -> list[tuple[str, str]]:
+    """[(kind, name)] declared under postBuild.substituteFrom in this doc."""
+    m = _SUBFROM_RE.search(doc)
+    if not m:
+        return []
+    indent = len(m.group(1))
+    out: list[tuple[str, str]] = []
+    kind = ""
+    for raw in doc[m.end():].splitlines():
+        if raw.strip() == "" or raw.lstrip().startswith("#"):
+            continue
+        if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
+            break                                    # dedented out of the list
+        km = _SUBITEM_KIND_RE.match(raw)
+        if km:
+            kind = km.group(1)
+            continue
+        nm = _SUBITEM_NAME_RE.match(raw)
+        if nm and kind:
+            out.append((kind, nm.group(1)))
+            kind = ""
+    return out
+
+
+kustomizations: list[dict] = []
+for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
+    text = yfile.read_text()
+    if st.FLUX_API not in text:
+        continue
+    for doc in st._DOC_SPLIT_RE.split(text):
+        if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
+            continue
+        pm = st._PATH_RE.search(doc)
+        if not pm:
+            continue
+        nm = st._NAME_RE.search(doc)
+        srcs = substitute_sources(doc)
+        if not srcs:
+            continue                                  # no substitution => nothing to audit
+        kustomizations.append({
+            "file": str(yfile.relative_to(ROOT)),
+            "name": nm.group(1) if nm else "?",
+            "path": pm.group(1),
+            "sources": srcs,
+        })
+
+check(len(kustomizations) > 0,
+      "no Flux Kustomization with postBuild.substituteFrom found under clusters/ — "
+      "the parser or the repo layout changed; this check would silently audit nothing")
+
+# ------------------------------------------------------------------ token scanning
+# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree:
+# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only
+# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders).
+TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}")
+NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES
+# these, so an undefined name yields the default/empty AT RECONCILE TIME.
+OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$")
+LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
+
+
+def classify(dollars: int, inner: str) -> tuple[str, str] | None:
+    """Return (severity-tag, remediation) for a hazardous expansion, else None."""
+    if dollars >= 2:
+        return None                                   # escaped: emitted as a literal
+    if NAME_RE.match(inner):
+        if inner in KEYS:
+            return None                               # defined knob — the happy path
+        return ("UNDEFINED",
+                f"strict mode fails the Kustomization; today it renders EMPTY. Either add "
+                f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or "
+                f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer")
+    m = OP_RE.match(inner) or LEN_RE.match(inner)
+    if m:
+        base = m.group(1)
+        if base in KEYS:
+            return None                               # defined knob with an operator
+        return ("SILENT-EMPTY",
+                f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a "
+                f"cluster-var, so the rendered manifest carries the default, not the "
+                f"expression. StrictPostBuildSubstitutions does NOT catch this form. "
+                f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it")
+    return ("FATAL-PARSE",
+            "envsubst cannot parse this and fails the ENTIRE Kustomization (every resource "
+            "in the path). Escape it as $${...} or rewrite the prose without braces")
+
+
+def rendered_files(path_spec: str) -> tuple[list[str], bool]:
+    """Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively."""
+    rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "."
+    if not (ROOT / rel).is_dir():
+        return [], False
+    applied, _ = st.reconciled_files(ROOT, rel)
+    return sorted(applied), True
+
+
+findings: list[str] = []
+referenced: dict[str, set[str]] = {}
+inventory: list[str] = []
+
+for k in kustomizations:
+    files, exists = rendered_files(k["path"])
+    check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' "
+                  f"which is not a directory in this repo")
+    has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"]
+    n_def = n_esc = n_bad = 0
+    for rel in files:
+        f = ROOT / rel
+        try:
+            text = f.read_text()
+        except (OSError, UnicodeDecodeError):
+            continue
+        for lineno, line in enumerate(text.splitlines(), 1):
+            for m in TOKEN_RE.finditer(line):
+                dollars, inner = len(m.group(1)), m.group(2)
+                verdict = classify(dollars, inner)
+                if verdict is None:
+                    if dollars >= 2:
+                        n_esc += 1
+                    else:
+                        n_def += 1
+                        referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
+                                              set()).add(k["name"])
+                        # (4) a DEFINED knob is only substituted if this Kustomization
+                        #     actually pulls cluster-vars in; otherwise the literal ships.
+                        check(has_cluster_vars,
+                              f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but "
+                              f"Kustomization '{k['name']}' ({k['file']}) does not "
+                              f"substituteFrom ConfigMap/cluster-vars — the literal token "
+                              f"would ship into the cluster")
+                    continue
+                n_bad += 1
+                tag, fix = verdict
+                findings.append(f"  [{tag}] {rel}:{lineno}  {m.group(0)}\n"
+                                f"        rendered by Flux Kustomization '{k['name']}' "
+                                f"({k['file']}, path {k['path']})\n"
+                                f"        {fix}\n"
+                                f"        | {line.strip()[:120]}")
+    inventory.append(f"  {k['name']:<16} path={k['path']:<22} files={len(files):<3} "
+                     f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}")
+
+check(not findings,
+      f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
+
+# --------------------------------------------- (5) overlay knobs exist in the base
+# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
+# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
+# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or
+# manifest references it — that ordering is what closes the define-then-use race.
+_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M)
+for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
+    for key in set(_OP_PATH_RE.findall(overlay.read_text())):
+        check(key in KEYS,
+              f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE "
+              f"default in clusters/base/cluster-vars.yaml — a second overlay would render "
+              f"${{{key}}} empty (R-ENV-02: author once, supply per env)")
+
+# ------------------------------------------------------- (6) the check is load-bearing
+L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
+GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
+check("flux-postbuild-substitution/check.sh" in L0,
+      "ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
+      "conformance selection can filter this check out of a gate run, so l0.sh must invoke "
+      "it directly and unconditionally or the gate is not actually enforcing it")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+# ------------------------------------------------------------- advisory: dead knobs
+for key in sorted(KEYS - set(referenced)):
+    ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest "
+                      f"(dead knob, or documentation-only — advisory, not a failure)")
+
+# ------------------------------------------------------------------------- report
+print("Flux postBuild substitution inventory (per Kustomization):")
+for line in inventory:
+    print(line)
+print(f"\ncluster-vars knobs defined: {len(KEYS)}   referenced: {len(referenced)}")
+
+if ADVISORIES:
+    print("\nADVISORIES (not failures):")
+    for a in ADVISORIES:
+        print(f"  - {a}")
+
+if findings:
+    print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr)
+    for f in findings:
+        print(f, file=sys.stderr)
+
+print()
+if not FAILURES:
+    print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
+          f"(R-ENV-02, R-GITOPS-01)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -68,7 +68,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
@@ -76,8 +76,8 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -91,7 +91,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -91,7 +91,7 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true

infra/ansible/README.md

@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml   # 4. hand off to bootstrap/ (SOP
 > ```
 > (Both take the same target-group override as the rest — see **Targeting prod** below.)
 
+> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
+> ```
+> ansible-playbook playbooks/node-health.yml   -e node_health_hosts=k3s_prod   # read-only hardware health
+> ansible-playbook playbooks/memattr-survey.yml                                 # EFI memattr reboot survey
+> ```
+> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
+> until `-e memattr_survey_reboots_authorised=true` is passed. See
+> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
+
 ## Re-provisioning a node (tofu -replace / fresh OS)
 A node rebuild wipes **everything on the host filesystem** — including host-level config
 that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
@@ -70,11 +79,30 @@ Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
 afterward. The parity-critical values **must match** between this role's
 `templates/cilium-values.yaml.j2` and that HelmRelease:
 - `kubeProxyReplacement: true`
-- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
+- `k8sServiceHost: {{ k3s_apiserver_endpoint }}` — **the kube-vip CONTROL-PLANE VIP**
-- `k8sServicePort: 6443`
+  (`= kube_vip_address`: dev `192.168.1.49`, prod `192.168.2.9`), **not** `k3s_api_host`
+- `k8sServicePort: {{ k3s_api_port }}` (6443 in both environments)
 
-The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
+### Two API addresses, split 2026-09-06 — do not re-merge them
-to the same value this role uses (surfaced by `tofu output primary_node_ip`).
+`k3s_api_host` and `k3s_apiserver_endpoint` were one variable, and the conflation put every
+prod `cilium-agent` on `192.168.2.10` (`k3s-prod-1`'s own address) on a cluster whose point is
+HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no
+service or policy update ever lands, which is a silent failure, not an outage.
+
+| variable | value | consumed by | why it must be that address |
+|---|---|---|---|
+| `k3s_api_host` | the founder's own node IP (dev `192.168.1.50`, prod `192.168.2.10`) | `roles/k3s_server`: the secondary `--server https://…:6443` join URL, and one `k3s_tls_sans` entry | on a **cold** build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran `--cluster-init` and is up |
+| `k3s_apiserver_endpoint` | `{{ kube_vip_address }}` (dev `192.168.1.49`, prod `192.168.2.9`) | `roles/cilium` → `k8sServiceHost`; any steady-state client | it must survive losing any single server |
+
+Consequences already handled: `playbooks/site.yml` imports `kube-vip.yml` **between** `k3s.yml`
+and `cilium.yml` (the VIP has to be floating before the CNI dials it), and `roles/kube_vip`'s
+last task blocks until the VIP answers on 6443. `k3s_tls_sans` already carried both addresses,
+so no cert reissue and no k3s restart are involved.
+
+The Flux side of the same contract is `${K3S_API_HOST}` in `platform/cilium/helmrelease.yaml`,
+set per overlay in `clusters/<env>/kustomization.yaml`; `[parity]` means it must equal
+`k3s_apiserver_endpoint` for the SAME environment. Asserted by
+`conformance/tests/l0-static/ha-control-plane-as-code`.
 
 ## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
 `roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
@@ -109,6 +137,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
 `playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
 keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
 bring-up chain.
+`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
+for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
+role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
+can be mistyped; inventory membership cannot.
 
 ```bash
 # dev — completely unchanged, no extra vars
@@ -135,7 +167,9 @@ ansible-playbook playbooks/site.yml \
 What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
 `k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
 target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
-node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
+node-IP list), `k3s_api_host=192.168.2.10` (the founder, bootstrap-join only),
+`kube_vip_address=192.168.2.9` — and therefore `k3s_apiserver_endpoint=192.168.2.9`, which is what
+Cilium dials — and
 `cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
 `.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
 

infra/ansible/inventory/group_vars/all.yml

@@ -27,19 +27,85 @@ k3s_servers_group: k3s_servers
 # a kubeconfig is a credential.)
 cluster_kubeconfig_name: dev.kubeconfig
 
-# The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP.
+# --- TWO API ADDRESSES, AND THEY ARE NOT INTERCHANGEABLE (split 2026-09-06) ----------
+# This used to be ONE variable, k3s_api_host, consumed by BOTH the cold-bootstrap join and
+# Cilium's k8sServiceHost. That conflation is what pointed every cilium-agent on the LIVE
+# prod cluster at 192.168.2.10 — k3s-prod-1's own address — on a cluster whose entire
+# purpose is HA: verified 2026-09-06, all three agents carried
+# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held 192.168.2.9. Lose that one node
+# and no service/policy update reaches the CNI until it returns (the existing datapath
+# survives on Cilium's cache, so it fails SILENTLY). The two consumers have genuinely
+# different requirements, so they now have two variables.
+#
+#   k3s_api_host            THE FOUNDER'S OWN ADDRESS. Bootstrap-time only.
+#   k3s_apiserver_endpoint  THE HA ENDPOINT (the kube-vip VIP). Steady-state.
+#
+# DO NOT "simplify" these back into one. See each block below for why each half must be
+# the address it is.
+
+# THE FOUNDER'S OWN NODE IP — a BOOTSTRAP-TIME address, deliberately NOT the VIP.
 # NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node
-# IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san /
+# IP — so we read the real 192.168.1.x from k3s_node_ip instead.
-# secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux
+#
-# HelmRelease stay in PARITY (docs/02 contract).
+# EXACTLY TWO CONSUMERS, and both of them need the founder specifically:
+#   1. roles/k3s_server — the secondary `--server https://<host>:<port>` JOIN URL. On a COLD
+#      build the VIP DOES NOT EXIST YET at join time: kube-vip is a k3s auto-deploy manifest
+#      that only lands once a cluster exists, so a join aimed at the VIP is a join aimed at
+#      an unanswered address and the cold build is unjoinable. The founder is up (it just ran
+#      --cluster-init), which is the whole reason this variable is the founder.
+#   2. roles/k3s_server — one entry of k3s_tls_sans, so the founder's own address is a SAN
+#      alongside the VIP and every other server's IP.
+#
+# ANYTHING THAT RUNS AGAINST A CLUSTER THAT ALREADY EXISTS MUST NOT USE THIS. It is a
+# single point of failure by construction. Use k3s_apiserver_endpoint below.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}"
+
+# Shared by both halves: k3s serves the API on 6443 in every environment, so the port is
+# not a per-environment knob and is not duplicated per address.
 k3s_api_port: 6443
 
+# THE HA kube-apiserver ENDPOINT — a VIRTUAL address that reaches the cluster wherever the
+# control plane currently is, not a machine. OPERATOR DECISION 2026-09-06: in-cluster
+# components dial the VIP, as kube-vip was always intended to provide.
+#
+# DERIVED from kube_vip_address (below), never restated: an environment overrides the VIP in
+# ONE place (group_vars/k3s_prod/vars.yml sets kube_vip_address: 192.168.2.9) and this
+# follows automatically, so prod needs no override of its own and a fourth environment
+# cannot set one of the two and forget the other.
+#
+# NAMED FOR THE ROLE, NOT THE MECHANISM. Consumers want "the address that reaches the
+# apiserver, HA"; that it is currently floated by kube-vip is an implementation detail. If
+# the mechanism is ever something else (an external LB, a DNS name), this one line moves and
+# no consumer changes.
+#
+# CONSUMERS:
+#   - roles/cilium -> cilium-values.yaml.j2 -> k8sServiceHost.  [parity] with the Flux side:
+#     clusters/base/cluster-vars.yaml K3S_API_HOST (and each overlay's patch) MUST equal
+#     THIS value for that environment, or Flux and the bootstrap fight over the CNI's
+#     control-plane path. See platform/cilium/helmrelease.yaml's PARITY CONTRACT header.
+#
+# TLS IS ALREADY COVERED — verified, not assumed: k3s_tls_sans (roles/k3s_server/defaults)
+# has always included kube_vip_address, and on 2026-09-06 the prod apiserver was reached
+# through https://192.168.2.9:6443 with FULL CA verification and tls-server-name=192.168.2.9
+# (infra/ansible/.cache/prod.kubeconfig, no insecure-skip-tls-verify). So the serving cert
+# needs no change and no k3s restart for this switch.
+#
+# ORDERING CONSEQUENCE, and it is why playbooks/site.yml changed too: Cilium now needs the
+# VIP to be UP before it installs. site.yml therefore imports kube-vip.yml BETWEEN k3s.yml
+# and cilium.yml, and roles/kube_vip waits for the VIP to answer before returning. kube-vip
+# can precede the CNI because its DaemonSet is hostNetwork with
+# KUBERNETES_SERVICE_HOST=127.0.0.1 — it needs neither a CNI nor kube-proxy.
+k3s_apiserver_endpoint: "{{ kube_vip_address }}"
+
 # Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover
 # endpoint for the kube-apiserver, fronted by an operator pfSense port-forward at
-# 192.168.1.49:6443. Shared here so BOTH kube_vip AND the k3s_server tls-san list
+# 192.168.1.49:6443. Shared here so THREE things agree on the one address:
-# agree on the address (the serving cert must carry this IP or TLS via the VIP
+#   - roles/kube_vip          — what to ARP and lease-elect for;
-# fails to validate).
+#   - roles/k3s_server        — the tls-san entry (the serving cert must carry this IP or
+#                               TLS via the VIP fails to validate);
+#   - k3s_apiserver_endpoint  — above; what in-cluster components actually dial.
+# THIS is the one line an environment overrides (prod: 192.168.2.9, in
+# group_vars/k3s_prod/vars.yml); all three follow.
 kube_vip_address: "192.168.1.49"
 
 # Controller (dev container) reaches the kube-API DIRECTLY via the pfSense NAT

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -51,12 +51,18 @@ k3s_servers_group: k3s_prod
 # than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential.
 cluster_kubeconfig_name: prod.kubeconfig
 
-# --- k3s API host — safe to set now, the primary's real IP is already known -------
+# --- k3s API host — THE FOUNDER'S ADDRESS, bootstrap-time only --------------------
 # Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
 # dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
 # has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
 # address (see the KEY DIFFERENCE note there) — so point straight at it instead of
 # needing a k3s_node_ip split.
+#
+# WHAT THIS IS *NOT*, since 2026-09-06: it is not the endpoint anything steady-state
+# dials. That is k3s_apiserver_endpoint (group_vars/all.yml), which derives from
+# kube_vip_address below and therefore needs NO override here. k3s_api_host is consumed
+# only by the secondary `--server` cold-join URL and by one tls-san entry. Read the
+# TWO API ADDRESSES block at the top of group_vars/all.yml before touching either.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
 # by template reference, so it needs no separate override here — it picks up the real
@@ -75,8 +81,15 @@ k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # round (control-plane .49, ingress pool .53-.60).
 #
 # This is the address kube-vip floats for the API and that k3s tls-san's onto its
-# serving cert; secondaries use it as their --server join target. Dev's equivalent is
+# serving cert. Dev's equivalent is kube_vip_address: 192.168.1.49 in group_vars/all.yml.
-# kube_vip_address: 192.168.1.49 in group_vars/all.yml.
+#
+# CORRECTION 2026-09-06: an earlier version of this comment claimed "secondaries use it as
+# their --server join target". They do not, and they must not — on a COLD build the VIP is
+# not up when the secondaries join (kube-vip is a k3s auto-deploy manifest and needs a
+# cluster first), so the join uses k3s_api_host, the founder. What DOES use this address is
+# k3s_apiserver_endpoint (group_vars/all.yml, derived from this line), i.e. Cilium's
+# k8sServiceHost and any other steady-state client. Overriding kube_vip_address here moves
+# BOTH the VIP and prod's Cilium endpoint together, which is the point of deriving it.
 #
 # HISTORY: an earlier pass here wrongly set this to 192.168.2.20 — that is the INGRESS
 # VIP, not the control-plane VIP. Corrected. Keep the two distinct.
@@ -87,24 +100,155 @@ kube_vip_address: "192.168.2.9"
 # — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
 cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
 
-# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
+# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
-# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
+# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
-# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
+# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
-# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
+# backports kernel as a side effect of sharing the role), and this line switches it on for
-# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
+# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
-# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
+# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
-# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
+# rather than from anyone remembering.
 #
-# The overrides below are deliberately present-but-broken: leaving them UNSET would
+# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
-# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
+# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
-# hardware — a confusing failure that looks like a wiring bug. This value fails just
+# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
-# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
+# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
-# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
+# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
-# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
+# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
-# updates these two lines. All three nodes must end up with an identical layout
+# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
-# (Longhorn replica placement assumes symmetry) before any of this is used for real.
+# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
-node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
-node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# in §3.
+#
+# COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
+# guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
+# the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
+# back to false, roll back per the note the role writes to
+# /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
+# roles/node_baseline.
+#
+# The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
+# FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
+node_baseline_backports_kernel: true
+
+# --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
+# Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
+#
+#   DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
+#                     disk, claimed and mounted at the NEUTRAL mountpoint `/data`
+#                     (LABEL=k3s-data), then split between TWO tenants:
+#                       /data/longhorn   Longhorn's dataPath
+#                       /data/k3s        k3s --data-dir (etcd AND the containerd image store)
+#   ETCD           -> NO dedicated device. It lives inside /data/k3s on that partition.
+#
+# ================================ THE 2026-09-04 REVISION ================================
+# The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
+# /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
+# data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
+# deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
+# would therefore have filled /var during its FIRST image pulls — and with etcd on the same
+# partition, a full /var means etcd cannot write its WAL. That is cluster death, not
+# degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
+# mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
+# tenants. Nothing was installed on prod when this was decided, which is exactly why it was
+# decided then — it is cheap now and very expensive after bring-up.
+#
+# THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
+# were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
+# contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
+# MEASURED, not assumed — exit condition in §2.
+# =========================================================================================
+#
+# Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
+# not as a device path and a blank. That distinction is the point of this block:
+#
+# 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
+#    k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
+#    (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
+#    Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
+#    freezes a fact about today's hardware into inventory, where it goes stale silently the
+#    first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
+#    nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
+#    So inventory declares the INTENT ("the partition currently mounted at /srv") and the
+#    role discovers the device on the node, then proves it is safe to claim (a partition or
+#    disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
+#    unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
+#
+# 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
+#    describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
+#    layout etcd is inside /data/k3s on the claimed data partition, which is neither a
+#    dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
+#    Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
+#    and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
+#    on every run, that the filesystem backing the etcd data dir really is /data. If the
+#    mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
+#    WAL would land back on the OS filesystem, invisibly, until the disk filled. The
+#    `root-filesystem` value makes no such claim and gets no such assert — which is exactly
+#    why the two are distinct enum values rather than one with a comment.
+#
+# 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
+#    the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
+#    the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
+#    this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
+#    k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
+#    Longhorn volumes are untouched.
+node_baseline_longhorn_source: claim-mountpoint
+node_baseline_longhorn_claim_mountpoint: /srv
+node_baseline_etcd_source: shared-data-partition
+
+# --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
+# `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
+# CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
+# /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
+# as the destination would make "claimed" and "not claimed" indistinguishable.
+# `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
+# partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
+# else's named volume — so labelling ours `data` would make a stranger's `data` volume look
+# already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
+# `k3s-etcd` label convention already in this repo.
+node_baseline_data_mount: /data
+node_baseline_data_fs_label: k3s-data
+# Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
+# equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
+# side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
+# default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
+node_baseline_longhorn_data_path: /data/longhorn
+
+# --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
+# The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
+# `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
+# label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
+# would walk into the greenfield claim path — which fails closed at "the claim mountpoint
+# must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
+# is right but it is not convergence, so the transition is DECLARED here: the role unmounts
+# the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
+# the partition is empty (nothing orphaned) and is the device the retired label names.
+# Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
+# SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
+# then skip on the retired-label lookup and nothing else depends on them.
+node_baseline_data_retired_fs_label: longhorn
+node_baseline_data_retired_mount: /var/lib/longhorn
+
+# --- k3s --data-dir (§2) -----------------------------------------------------------
+# THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
+# claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
+# `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
+# which defers to this variable) creates the directory 0700 on the data partition and derives
+# etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
+# Ansible side cannot disagree with itself.
+#
+# Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
+# it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
+# dir would orphan that mount under a running cluster.
+k3s_data_dir: /data/k3s
+
+# Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
+# source uses a device path any more, but leaving these UNSET would let prod silently
+# inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
+# metal) the moment anyone flipped a source back to `device`. These values can never parse
+# as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
+# before mkfs is ever reached.
+node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
+node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
 
 # --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
 # roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
@@ -158,15 +302,58 @@ node_baseline_dns_servers:
   #   LAN clients       -> PiHole. Not a cluster concern.
   #   nodes             -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
   #
-  # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
+  # THE k3s API SERVER is a node process and carries `oidc-issuer-url=...` (built from
-  # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
+  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
-  # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
+  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
-  # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
-  # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
-  # with the dev Gateway VIP.
   #
-  # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
+  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
+  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
+  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
+  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
+  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
+  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
+  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  #
+  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
+  # external-dns has actually published the record. Until then the issuer does not resolve
+  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
+  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
+
+# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
+# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
+# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
+# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
+# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
+# postBuild.substituteFrom never sees it. Full rationale in
+# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
+#
+# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
+# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
+# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
+# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
+# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
+# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
+k3s_base_domain: "prod.home.keithjackson.dev"
+
+# --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
+# roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
+# and group_vars/all.yml carries no override, so dev keeps resolving exactly that).
+# Prod bootstraps its OWN overlay. Before this seam existed the role could bootstrap
+# clusters/dev ONLY — bootstrap/flux/bootstrap.sh's FLUX_ENV_PATH default was never
+# exported by the role — so `flux bootstrap ... --path=clusters/prod` (§8 step 4 of
+# infra/PROD-PROVISIONING.md) was simply not reachable without editing the role.
+#
+# Same seam as kube_vip_address / k3s_api_host / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel above: set once for the [k3s_prod] umbrella group, so a
+# FOURTH prod node inherits it from group membership with no per-host or per-run flag.
+#
+# WHAT clusters/prod STILL NEEDS before this is safe to run — do NOT invoke the
+# flux_bootstrap role until these are true (PROD-PROVISIONING.md §7):
+#   - the remaining TBD-PROD-* placeholders in clusters/prod/kustomization.yaml are real
+#     (S3_BACKUP_ENDPOINT); they are INVALID on purpose so an early apply fails loudly;
+#   - the GitRepository url there is the TEMPORARY workstation Gitea
+#     (http://192.168.2.132:3000/...) and that container is actually running and reachable
+#     from the prod nodes — it is outside every cluster and can simply be powered off;
+#   - the `sops-age` Secret is installed out of band on prod (bootstrap/sops).
+flux_bootstrap_env_path: "clusters/prod"

infra/ansible/inventory/hosts.ini

@@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 
 # =============================================================================
 # PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
-# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
+# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
-# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
+# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
-# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
+# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
-# nothing has been run against node 1, and the operator has decided the CLUSTER stays
+# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
-# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
+# check and the node-baseline PACKAGE floor has been run, and the operator has decided
-# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
+# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
-# later member-add dance.
+# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
+# single-server-etcd window, no later member-add dance.
 #
 # KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
 # hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
@@ -87,12 +88,23 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
 k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
 
 [k3s_prod_secondary]
-# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md
+# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
-# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so
+# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
-# no play can accidentally target a host that isn't there; uncomment each line only once
+# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
-# that node is actually on the bench and answering SSH.
+# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
-# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
+# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
-# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
+# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
+# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
+# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
+# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
+# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
+# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
+# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
+# three-way, not a node-1 outlier.
+# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
+# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
+k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
+k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
 
 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
 # pattern; named to match the group_vars directory so Ansible auto-loads it).

infra/ansible/playbooks/kube-vip.yml

@@ -3,6 +3,13 @@
 # Control-plane ONLY (svc_enable=false) — Cilium owns Service LoadBalancing.
 # Writes the manifest into each server's k3s auto-deploy dir; k3s applies it.
 #
+# PART OF THE ONE-COMMAND BRING-UP SINCE 2026-09-06, not a separate operator step:
+# playbooks/site.yml imports this play BETWEEN k3s.yml and cilium.yml, because Cilium's
+# k8sServiceHost is now this VIP (k3s_apiserver_endpoint, group_vars/all.yml) and the CNI
+# cannot be installed against an endpoint nothing announces yet. The role's final task waits
+# for the VIP to answer on 6443, so a failure to float it stops the run here with a named
+# cause instead of surfacing later as unreachable cilium-agents.
+#
 # TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
 #   ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
 # (the VIP itself comes from kube_vip_address, already scoped per environment in

infra/ansible/playbooks/memattr-survey.yml

@@ -0,0 +1,137 @@
+---
+# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
+# R-ENV-03, R-TEST-02).
+#
+# WHAT QUESTION THIS ANSWERS
+#   infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
+#   EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
+#   Linux 6.12 reserves everything from the table address to the top of RAM without
+#   validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
+#   playbook measures them:
+#
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
+#        failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
+#        fleet-wide were at other addresses. If bad outcomes map onto a narrow address
+#        range, the causal chain closes with nothing touched.
+#     2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+#   §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
+#   is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
+#   a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
+#   boots per arm; detecting a mere odds shift needs ~50.
+#
+# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
+# package installs, no hardware moves, no node_baseline run. Its job is to establish real
+# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
+# a Fast Boot flip, a 6.13 kernel) compares against it.
+#
+# HOW TO RUN IT
+#   Dry run first — every stage except the reboots, so the pipeline is proven before 45
+#   minutes of wall clock are committed:
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml'
+#
+#   The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
+#
+#     ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+#         ansible-playbook playbooks/memattr-survey.yml \
+#           -e memattr_survey_reboots_authorised=true'
+#
+#   Fewer rounds, or one node:
+#     -e memattr_survey_rounds=5        --limit k3s-prod-2
+#
+#   RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
+#   rounds are never lost or double-counted. Analysis alone, over existing artifacts:
+#     ansible-playbook playbooks/memattr-survey.yml --tags analyse
+#
+# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
+# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
+# not a member of memattr_survey_required_group — the dev cluster is live, and a survey
+# that reboots it 20 times because of a mistyped --limit is exactly the accident worth
+# engineering against.
+#
+# HOW TO READ THE OUTPUT
+#   infra/ansible/_artifacts/memattr-survey/  (gitignored, regenerated every run)
+#     <host>.rounds.jsonl  one JSON line per round, mirrored off the node after EVERY round
+#     <host>.boots.json    every boot the persistent journal holds, re-derived in one pass
+#                          at the end — the AUTHORITATIVE record
+#     summary.md           the analysis, also printed at the end of the run
+#   See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
+#   with NO memattr log line is a healthy third state and not a missing measurement.
+
+# =====================================================================================
+# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
+# before a single reboot. That is the right asymmetry: the cost of a false stop is a
+# re-run, the cost of a false start is rebooting a machine that was doing something.
+# =====================================================================================
+- name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  any_errors_fatal: true
+  tags: [survey]
+  tasks:
+    - name: Preflight and baseline probe
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: preflight
+
+# =====================================================================================
+# PLAY 2 — THE SURVEY.
+#
+# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
+# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
+# keep collecting.
+#
+# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
+# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
+# that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
+# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
+# blocks belongs to exactly one host, so the other two sit idle through it and the survey
+# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
+# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
+# appeared in the output. `free` lets each host walk its own task list at its own pace,
+# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
+#
+# Consequences worth knowing before editing: output INTERLEAVES between hosts (every
+# progress line is host-tagged for exactly this reason), and `run_once` is unreliable
+# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
+# directory create on every host instead. Concurrency is also capped by `forks`
+# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
+# than letting a bigger fleet silently serialise again.
+# =====================================================================================
+- name: memattr survey — reboot rounds and journal harvest
+  hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
+  gather_facts: false
+  become: true
+  strategy: free
+  tags: [survey]
+  tasks:
+    - name: Run the reboot rounds
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: rounds
+
+    - name: Harvest every retained boot from the persistent journal
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: harvest
+
+# =====================================================================================
+# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
+# even if every surveyed node died, because the artifacts are already on disk and the
+# analysis counts rounds from THOSE rather than from Ansible's host state. A node that
+# vanished shows up as a short arm with a wider interval, not as a missing report.
+# =====================================================================================
+- name: memattr survey — analyse
+  hosts: localhost
+  gather_facts: false
+  become: false
+  tags: [analyse]
+  tasks:
+    - name: Produce the analysis
+      ansible.builtin.include_role:
+        name: memattr_survey
+        tasks_from: analyse

infra/ansible/playbooks/site.yml

@@ -1,7 +1,10 @@
 ---
 # Full dev/test cluster bring-up, in order (docs/02 sequencing; run AFTER
 # `tofu apply` has created and cloud-init has baselined the VMs):
-#   k3s HA install -> Cilium CNI -> verify readiness -> Flux + SOPS hand-off.
+#   k3s HA install -> kube-vip control-plane VIP -> Cilium CNI -> verify readiness
+#   -> Flux + SOPS hand-off.
+# (kube-vip joined the chain 2026-09-06 — see the note at its import below; it is a
+# prerequisite of the CNI now, not a separate operator step.)
 #
 # TARGET (R-ENV-02): every imported play takes its host pattern from a var that
 # defaults to the dev group, so `ansible-playbook playbooks/site.yml` with no extra
@@ -20,6 +23,29 @@
 - name: Import - install k3s HA control plane
   ansible.builtin.import_playbook: k3s.yml
 
+# ADDED 2026-09-06, AND ITS POSITION IS LOAD-BEARING: kube-vip MUST come up between k3s and
+# Cilium. Cilium's k8sServiceHost is now the control-plane VIP (k3s_apiserver_endpoint =
+# kube_vip_address) rather than the founder's own node IP, so installing Cilium before the
+# VIP is announced would point every agent at an address nothing answers on — the exact
+# shape of the 2026-09-01 incident, where agents that could not reach the API server
+# deadlocked on a cold cluster.
+#
+# IT CANNOT COME EARLIER. kube-vip ships as a k3s auto-deploy manifest
+# (<data-dir>/server/manifests), so a cluster must exist before it can be applied — hence
+# after k3s.yml, not before.
+#
+# IT DOES NOT NEED THE CNI, which is what makes this ordering possible at all: the kube-vip
+# DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 (roles/kube_vip's template
+# explains why), so it reaches the node-local apiserver with no CNI and no kube-proxy, and
+# DaemonSet pods carry the default not-ready/network-unavailable tolerations that let them
+# schedule onto a still-NotReady node. roles/kube_vip blocks until the VIP actually answers
+# on 6443, so cilium.yml below cannot start against a VIP that is not up.
+#
+# Re-running this against the LIVE dev cluster is a no-op for this play: dev already runs
+# kube-vip with the same rendered manifest, and the wait finds the VIP already answering.
+- name: Import - float the control-plane HA VIP (kube-vip) BEFORE the CNI dials it
+  ansible.builtin.import_playbook: kube-vip.yml
+
 - name: Import - install Cilium CNI
   ansible.builtin.import_playbook: cilium.yml
 

infra/ansible/roles/cilium/defaults/main.yml

@@ -2,7 +2,17 @@
 # cilium role defaults (ADR-0002, docs/02 Cilium-ownership contract).
 # This role performs the INITIAL Cilium install; the Flux platform/cilium
 # HelmRelease adopts config afterward. The parity-critical values
-# (kubeProxyReplacement, k8sServiceHost/Port) are templated from k3s_api_host.
+# (kubeProxyReplacement, k8sServiceHost/Port) are templated from
+# k3s_apiserver_endpoint / k3s_api_port (group_vars/all.yml).
+#
+# k8sServiceHost IS THE HA VIP, NOT THE FOUNDER (changed 2026-09-06). It reads
+# k3s_apiserver_endpoint — which derives from kube_vip_address, so it is dev 192.168.1.49 /
+# prod 192.168.2.9 — and deliberately NOT k3s_api_host, the founder's own node IP. Those two
+# were one variable until 2026-09-06, and the conflation is what left all three prod
+# cilium-agents dialing k3s-prod-1 on an HA cluster. Do not re-merge them.
+#
+# THIS ROLE THEREFORE DEPENDS ON THE VIP EXISTING. playbooks/site.yml runs kube-vip.yml
+# before cilium.yml and roles/kube_vip blocks until the VIP answers on 6443.
 
 # Exact pin — MUST equal the platform/cilium HelmRelease chart.spec.version (1.20.1, an
 # exact pin, not a range). Renovate bumps both together.
@@ -15,9 +25,20 @@ cilium_namespace: "kube-system"
 # Cilium 1.20.1 targets Gateway API v1.6.1.
 gateway_api_version: "v1.6.1"
 gateway_api_crd_urls:
+  # ONE URL, deliberately. As of v1.6.0 the STANDARD channel bundle already contains
+  # TLSRoute (plus TCPRoute, UDPRoute, ListenerSet and BackendTLSPolicy), so the separate
+  # experimental TLSRoute manifest this list used to carry is redundant — and worse than
+  # redundant: v1.6 also ships a ValidatingAdmissionPolicy,
+  # `safe-upgrades.gateway.networking.k8s.io`, which REJECTS an experimental-channel CRD
+  # applied over a standard-channel one. Applying the old second URL therefore fails hard:
+  #   ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' ... denied request:
+  #   Installing experimental CRDs on top of standard channel CRDs is prohibited by default.
+  # Verified 2026-09-04 on the prod bootstrap: standard-install.yaml v1.6.1 alone produces the
+  # exact 10 gateway.networking.k8s.io CRDs the LIVE dev cluster is running, tlsroutes included
+  # and annotated channel=standard, bundle-version=v1.6.1 — so this is parity with dev, not a
+  # reduction. Dev needs no reapply; it already has them. If gateway_api_version is ever moved
+  # BACK below v1.6.0, the experimental TLSRoute URL has to come back with it.
   - "https://github.com/kubernetes-sigs/gateway-api/releases/download/{{ gateway_api_version }}/standard-install.yaml"
-  # TLSRoute (experimental) is required by Cilium's Gateway API support.
-  - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
 
 # Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
 # comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev

infra/ansible/roles/cilium/tasks/main.yml

@@ -89,7 +89,7 @@
   run_once: true
   changed_when: false
 
-- name: Render the Cilium values file (parity-critical settings from k3s_api_host)
+- name: Render the Cilium values file (parity-critical settings from k3s_apiserver_endpoint)
   ansible.builtin.template:
     src: cilium-values.yaml.j2
     dest: "{{ cilium_values_path }}"

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -4,9 +4,24 @@
 ---
 # [parity] k3s kube-proxy is disabled → Cilium replaces it (ADR-0001/0002).
 kubeProxyReplacement: true
-# [parity] = the k3s API endpoint (primary node IP). The Flux HelmRelease's
+# [parity] = k3s_apiserver_endpoint, THE HA CONTROL-PLANE VIP (kube_vip_address) — dev
-# k8sServiceHost "REPLACE-ME" must be set to this same value.
+# 192.168.1.49, prod 192.168.2.9. NOT k3s_api_host, which is the FOUNDER'S OWN node IP:
-k8sServiceHost: "{{ k3s_api_host }}"
+# pointing the CNI's kube-proxy replacement at one machine on a 3-server HA cluster means
+# losing that machine stops every service/policy update reaching the datapath, silently
+# (the existing datapath keeps working from Cilium's cache, so nothing looks broken). That
+# was the live state of prod until 2026-09-06 — all three cilium-agents carried
+# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held .9. See group_vars/all.yml.
+#
+# [parity] The Flux side is platform/cilium/helmrelease.yaml's k8sServiceHost:
+# "${K3S_API_HOST}", whose per-environment value comes from clusters/<env>/kustomization.yaml.
+# THAT VALUE MUST EQUAL k3s_apiserver_endpoint for the SAME environment. If it does not,
+# Flux and this bootstrap render different control-plane paths and fight over the CNI.
+#
+# THE VIP MUST BE UP BEFORE THIS ROLE RUNS. playbooks/site.yml imports kube-vip.yml between
+# k3s.yml and cilium.yml for exactly this reason, and roles/kube_vip blocks until the VIP
+# answers. Rendering this against a VIP that nothing announces yet reproduces the
+# 2026-09-01 shape: agents that cannot reach the API server on a cold cluster.
+k8sServiceHost: "{{ k3s_apiserver_endpoint }}"
 k8sServicePort: {{ k3s_api_port }}
 
 # Ingress via Gateway API — no Traefik (docs/09, ADR-0002).
@@ -42,5 +57,22 @@ cgroup:
     enabled: false
   hostRoot: /run/cilium/cgroupv2
 
+# Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
+# platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
+# 3-node cluster lets one bad render take down two of three agents simultaneously —
+# exactly the 2026-09-01 incident, where k8sServiceHost: "" made the agent's `config`
+# init container fall back to the in-cluster ClusterIP 10.43.0.1:443. Only the Cilium
+# agent can translate that ClusterIP, and the agent in question was the one being
+# replaced: `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked.
+# A CNI rollout has no healthy-peer fallback — the component being upgraded is the one
+# the upgrade needs to reach the API server. This bites hardest on a COLD BOOTSTRAP
+# (this role's job), where nothing healthy exists to recover from. With maxUnavailable: 1
+# the DaemonSet will not proceed past an unready pod, bounding damage to a single node.
+# Cost: Cilium upgrades serialise and take longer; at 3 nodes that is an acceptable trade.
+updateStrategy:
+  type: RollingUpdate
+  rollingUpdate:
+    maxUnavailable: 1
+
 operator:
   replicas: {{ cilium_operator_replicas }}

infra/ansible/roles/flux_bootstrap/defaults/main.yml

@@ -19,5 +19,22 @@ flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
 flux_bootstrap_sops_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/sops/install-age-key.sh"
 
 # Assumed interface of bootstrap/flux (GitOps Engineer, T-1.4):
-#   runs `flux bootstrap ... --path=clusters/dev` against this cluster.
+#   runs `flux install` + `kubectl apply -k <env path>` against this cluster.
 flux_bootstrap_flux_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/flux/bootstrap.sh"
+
+# --- WHICH ENVIRONMENT OVERLAY THIS BOOTSTRAPS (R-ENV-02) -------------------------
+# The one thing that makes this role environment-agnostic. bootstrap/flux/bootstrap.sh
+# reads FLUX_ENV_PATH (default clusters/dev) and ends with `kubectl apply -k <that>`;
+# tasks/main.yml exports this value into that variable, so the role NEVER names an
+# environment and the task graph is identical for dev and prod.
+#
+# THE DEFAULT IS DEV ON PURPOSE — dev is LIVE, and group_vars/all.yml carries no
+# override, so a dev run resolves exactly the value it always had (clusters/dev) and
+# behaves byte-for-byte as before.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same group_vars
+# seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel. No role fork, no per-run extra-var, no second
+# playbook: group membership decides. Before this existed the role could bootstrap
+# ONLY clusters/dev, so prod bring-up was impossible without editing the role.
+flux_bootstrap_env_path: "clusters/dev"

infra/ansible/roles/flux_bootstrap/tasks/main.yml

@@ -4,11 +4,17 @@
 # — it calls the entrypoints the Security + GitOps Engineers author there.
 #
 # Order matters: install the age decryption key FIRST (so Flux can decrypt on its
-# very first reconcile), THEN run `flux bootstrap` at clusters/dev.
+# very first reconcile), THEN bootstrap Flux at the environment's overlay.
 #
 # The two commands are variables (see defaults/main.yml). If the bootstrap workers
 # expose different entrypoints, override flux_bootstrap_sops_cmd / _flux_cmd — the
 # task graph does not change.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): this role names NO environment. Which overlay gets
+# bootstrapped comes from `flux_bootstrap_env_path` (defaults/main.yml, default
+# clusters/dev), exported below as FLUX_ENV_PATH — the variable bootstrap/flux/
+# bootstrap.sh already reads. Prod overrides it in group_vars/k3s_prod/vars.yml, the
+# same seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name.
 
 - name: Assert the bootstrap/ subtree exists (authored by the other worker)
   ansible.builtin.stat:
@@ -29,11 +35,14 @@
   run_once: true
   changed_when: true
 
-- name: Bootstrap Flux against clusters/dev (bootstrap/flux)
+- name: Bootstrap Flux against this environment's overlay (bootstrap/flux)
   ansible.builtin.command:
     cmd: "{{ flux_bootstrap_flux_cmd }}"
   environment:
     KUBECONFIG: "{{ flux_bootstrap_kubeconfig }}"
+    # The ONLY per-environment input. clusters/dev by default (dev unchanged);
+    # clusters/prod for [k3s_prod] hosts via group_vars.
+    FLUX_ENV_PATH: "{{ flux_bootstrap_env_path }}"
   delegate_to: localhost
   become: false
   run_once: true

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
 # the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
 k3s_disable: "traefik,servicelb,local-storage"
 
+# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
+# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
+# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
+# image store (agent/containerd) — which is by far the largest tenant.
+#
+# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
+# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
+# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
+# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
+# running cluster's entire state directory out from under it and orphan that etcd mount.
+#
+# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
+# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
+# node inherits it from group membership rather than from anyone remembering. It is set
+# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
+# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
+# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
+# this same variable). One line moves both.
+#
+# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
+# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
+# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
+# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
+# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
+#
+# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
+# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
+# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
+# Restart handler for a value that is not changing. The install flags are the correct and
+# sufficient place, and prod has no k3s installed yet.
+k3s_data_dir: "/var/lib/rancher/k3s"
+
 # Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
 k3s_server_mode: "secondary"
 
@@ -27,6 +59,17 @@ k3s_server_mode: "secondary"
 # serving cert on restart when this set changes. Deduplicated (the primary IP is
 # also k3s_api_host).
 #
+# THIS LIST COVERS BOTH API ADDRESSES, WHICH IS WHY THE 2026-09-06 SPLIT NEEDED NO CERT
+# CHANGE AND NO k3s RESTART. group_vars/all.yml now distinguishes k3s_api_host (the
+# founder, for the cold join) from k3s_apiserver_endpoint (= kube_vip_address, what Cilium
+# and every steady-state client dial). Both are already in this set, so repointing a
+# consumer from one to the other cannot produce a TLS failure. VERIFIED LIVE 2026-09-06,
+# not assumed: prod's apiserver answered through https://192.168.2.9:6443 with full CA
+# verification and tls-server-name=192.168.2.9. Do not remove either entry — dropping
+# kube_vip_address breaks every client of the VIP; dropping k3s_api_host breaks nothing
+# today only because it is also in the k3s_servers_group node-IP list below, and that
+# coincidence is not a contract.
+#
 # k3s_servers_group — NOT a literal group name — is what makes this role environment-
 # agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
 # group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
@@ -34,3 +77,33 @@ k3s_tls_sans: >-
   {{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
       + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
      | unique }}
+
+# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
+# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
+#
+# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
+# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
+# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
+# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
+#
+#   Flux side     clusters/base/cluster-vars.yaml   BASE_DOMAIN      (per-overlay)
+#   Ansible side  THIS variable                     k3s_base_domain  (per-inventory-group)
+#
+# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
+# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
+# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
+# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
+# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
+#
+# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
+# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
+# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
+# what it already is and the Restart handler does NOT fire on a re-run.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
+# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
+# from group membership rather than from anyone remembering.
+k3s_base_domain: "home.keithjackson.dev"
+
+# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
+k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -22,6 +22,7 @@
       --write-kubeconfig-mode=644
       {% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
       --node-ip={{ k3s_node_ip }}
+      --data-dir={{ k3s_data_dir }}
 
 # On an ALREADY-installed cluster the install task above is idempotent (creates:
 # k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
@@ -74,13 +75,19 @@
       # OIDC (Keycloak SSO for kubectl). The API server trusts ID tokens from the homelab realm;
       # the `kubernetes` PUBLIC client (PKCE) + kubelogin mint them, and RBAC binds the `groups`
       # claim (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins->cluster-admin,
-      # oidc:viewers->view). Nodes resolve keycloak.home.keithjackson.dev to the Cilium gateway VIP
+      # oidc:viewers->view). Nodes resolve keycloak.<k3s_base_domain> to the Cilium gateway VIP
-      # (192.168.1.53) and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
+      # and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
       # prefixes namespace federated identities away from cert / service-account subjects.
       # ADDITIVE + break-glass-safe: the cert-based admin kubeconfig keeps full access with or
       # without Keycloak, so a bad realm/OIDC state can never lock you out of the cluster.
+      #
+      # NOT a Flux ${BASE_DOMAIN} token: this is a k3s SERVER FLAG, not a manifest, so
+      # postBuild.substituteFrom never touches it. k3s_oidc_issuer_url derives from
+      # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
+      # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
+      # config.yaml is unchanged and the Restart handler does not fire.
       kube-apiserver-arg:
-        - "oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab"
+        - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
         - "oidc-client-id=kubernetes"
         - "oidc-username-claim=email"
         - "oidc-username-prefix=oidc:"
@@ -97,15 +104,20 @@
     INSTALL_K3S_VERSION: "{{ k3s_version }}"
   when: k3s_server_mode == 'primary'
 
+# PATH FOLLOWS --data-dir. k3s writes the node-token to <data-dir>/server/token, NOT to
+# the literal /var/lib/rancher/k3s. On dev k3s_data_dir IS /var/lib/rancher/k3s (the role
+# default, restated explicitly above) so this is byte-for-byte the old path; on prod
+# --data-dir is /data/k3s (group_vars/k3s_prod) and the hardcoded path did not exist at all
+# — this task would have timed out after 180s and the two secondaries could never join.
 - name: Wait for the k3s node-token to exist (primary)
   ansible.builtin.wait_for:
-    path: /var/lib/rancher/k3s/server/token
+    path: "{{ k3s_data_dir }}/server/token"
     timeout: 180
   when: k3s_server_mode == 'primary'
 
 - name: Read the k3s node-token (primary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   register: k3s_token_raw
   when: k3s_server_mode == 'primary'
 
@@ -125,9 +137,11 @@
 # "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
 # dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
 # delegates to the PROD primary using this same role, unchanged (R-ENV-02).
+# k3s_data_dir is a CLUSTER-WIDE value (group_vars for prod, role default for dev), so the
+# secondary's own copy of it is also the primary's data dir — correct for the delegated read.
 - name: Fetch the k3s node-token directly from the primary (secondary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   delegate_to: "{{ groups[k3s_primary_group][0] }}"
   register: k3s_primary_token_raw
   when:
@@ -141,6 +155,18 @@
          | default(k3s_primary_token_raw.content | b64decode | trim, true) }}
   when: k3s_server_mode == 'secondary'
 
+# THE JOIN TARGET IS k3s_api_host — THE FOUNDER — AND MUST NOT BECOME THE VIP.
+# group_vars/all.yml splits the two API addresses (k3s_api_host = the founder's own node IP,
+# k3s_apiserver_endpoint = the kube-vip HA VIP) and this is the canonical founder-only
+# consumer. On a COLD build the VIP does not exist at this point in the sequence: kube-vip
+# is a k3s auto-deploy manifest, so it cannot be applied until a cluster exists, and
+# playbooks/site.yml only imports kube-vip.yml AFTER k3s.yml. A `--server` pointed at the
+# VIP would therefore dial an address nothing announces and the cold build would be
+# unjoinable — with no diagnostic beyond a connection timeout.
+# On a WARM single-node rejoin (`--limit k3s-dev-N`) the founder is up and answering too, so
+# the founder works in both cases and the VIP works in only one. Cilium's k8sServiceHost is
+# the opposite trade (steady state, must survive losing the founder) and uses the endpoint
+# variable instead.
 - name: Install k3s secondary (join as server)
   ansible.builtin.command:
     cmd: >-

infra/ansible/roles/kube_vip/defaults/main.yml

@@ -24,6 +24,16 @@ kube_vip_port: 6443
 # only if detection is wrong.
 kube_vip_interface: ""
 
+# How long to wait for the VIP to actually ANSWER on kube_vip_port after the manifest is
+# written (seconds). This is a real gate, not cosmetics: since 2026-09-06 Cilium's
+# k8sServiceHost is this VIP (k3s_apiserver_endpoint), and playbooks/site.yml runs this role
+# immediately before cilium.yml — so if the VIP never comes up, the run must stop HERE with
+# a clear message rather than proceed to install a CNI pointed at an address nothing
+# answers. On a cold cluster the budget covers k3s's deploy controller applying the
+# manifest, the ghcr.io image pull and leader election; on an already-running cluster the
+# check returns immediately.
+kube_vip_wait_timeout: 300
+
 # Leader-election tuning (seconds) — a snappy failover without thrashing.
 kube_vip_lease_duration: 15
 kube_vip_renew_deadline: 10
@@ -31,4 +41,14 @@ kube_vip_retry_period: 2
 
 # k3s auto-deploy manifests dir: k3s applies anything here on start, so kube-vip
 # is reproducible and independent of Flux/Helm.
-kube_vip_manifest_path: "/var/lib/rancher/k3s/server/manifests/kube-vip.yaml"
+#
+# THE PATH FOLLOWS --data-dir. k3s reads auto-deploy manifests from
+# <data-dir>/server/manifests, so this must be derived from the cluster-wide
+# k3s_data_dir (roles/k3s_server turns the same variable into --data-dir), not from the
+# literal /var/lib/rancher/k3s. Dev leaves k3s_data_dir unset outside roles/k3s_server's
+# own defaults, so the fallback below reproduces dev's existing path exactly; prod sets
+# k3s_data_dir: /data/k3s in group_vars/k3s_prod, where the old hardcoded path was simply
+# not the directory k3s watches — the manifest would have been written and silently never
+# applied, and the control-plane VIP would never have come up.
+kube_vip_manifest_path: >-
+  {{ k3s_data_dir | default('/var/lib/rancher/k3s') }}/server/manifests/kube-vip.yaml

infra/ansible/roles/kube_vip/tasks/main.yml

@@ -1,17 +1,34 @@
 ---
 # Deploy kube-vip (control-plane HA VIP) via the k3s auto-deploy manifests dir.
-# k3s applies /var/lib/rancher/k3s/server/manifests/*.yaml on start, so this is
+# k3s applies <data-dir>/server/manifests/*.yaml on start, so this is
-# reproducible and independent of Flux. The same manifest is written on every
+# reproducible and independent of Flux (see kube_vip_manifest_path in defaults/main.yml —
+# the dir follows k3s's --data-dir, which prod moves to /data/k3s). The same manifest is written on every
 # server so the DaemonSet survives the loss of any single node.
 
-- name: Detect the LAN interface holding the node's 192.168.1.5x address
+# WHICH INTERFACE kube-vip ARPs THE VIP ON. Detected from this node's OWN k3s_node_ip
+# (inventory, per host) rather than a hardcoded subnet: the previous awk matched the
+# literal 192.168.1.5[0-2], i.e. DEV's node addresses, so on prod (192.168.2.10/.11/.12)
+# it returned nothing and the assert below aborted the play. k3s_node_ip is the same
+# address k3s advertises with --node-ip and is defined for dev and prod alike, so this
+# resolves eno1 on prod and the identical dev interface it always resolved before.
+# `ip addr show to <ip>/32` does the matching in iproute2 instead of a regex over dotted
+# quads (where `.` would match any character).
+- name: Detect the LAN interface holding this node's k3s_node_ip
   ansible.builtin.shell: >-
     set -o pipefail;
-    ip -o -4 addr show | awk '/192\.168\.1\.5[0-2]\//{print $2; exit}'
+    ip -o -4 addr show to {{ k3s_node_ip }}/32 | awk '{print $2; exit}'
   args:
     executable: /bin/bash
   register: kube_vip_iface_detect
   changed_when: false
+  # RUN THIS EVEN UNDER --check. It is `ip -o -4 addr show`: it reads and changes nothing.
+  # Without check_mode: false the shell module is SKIPPED in check mode, kube_vip_iface_detect
+  # has no stdout, and the assert two tasks down aborts the play with "Could not detect the
+  # interface ... set kube_vip_interface explicitly" — a misleading failure that says nothing
+  # about the node. That mattered less while this play was an out-of-band operator step; it
+  # matters now that playbooks/site.yml imports it, because `--check` on the bring-up before a
+  # cold prod build is exactly the rehearsal an operator should be able to run.
+  check_mode: false
 
 - name: Resolve the kube-vip VIP interface (override wins, else detected)
   ansible.builtin.set_fact:
@@ -25,7 +42,7 @@
       - kube_vip_vip_interface is defined
       - kube_vip_vip_interface | length > 0
     fail_msg: >-
-      Could not detect the 192.168.1.5x LAN interface for kube-vip on
+      Could not detect the interface holding {{ k3s_node_ip }} for kube-vip on
       {{ inventory_hostname }} — set kube_vip_interface explicitly.
 
 - name: Ensure the k3s auto-deploy manifests directory exists
@@ -43,3 +60,34 @@
     owner: root
     group: root
     mode: "0600"
+
+# PROVE THE VIP IS ACTUALLY UP BEFORE THIS PLAY RETURNS (added 2026-09-06).
+#
+# Templating the manifest is not the same as having a floating endpoint: k3s's deploy
+# controller still has to apply it, the image still has to pull, and leader election still
+# has to pick an owner. Nothing used to check, which was tolerable while the VIP was only an
+# operator convenience. It is not tolerable now — Cilium's k8sServiceHost IS this VIP
+# (k3s_apiserver_endpoint) and playbooks/site.yml runs cilium.yml straight after this play,
+# so an unannounced VIP would mean installing a CNI that cannot reach the API server. That
+# failure is the 2026-09-01 shape and it is very expensive to diagnose from the far side;
+# failing here instead names the actual cause.
+#
+# run_once: the VIP is ONE address for the whole environment, so one probe answers for the
+# environment. Probing from a server host (not the controller) is deliberate: the servers
+# are on the VIP's own LAN in both environments, whereas the dev controller reaches dev's
+# API only through a pfSense NAT — a controller-side probe would have been a property of
+# where the play was launched from. READ-ONLY: it opens a TCP connection and changes nothing.
+- name: Wait for the control-plane VIP to answer (the CNI's endpoint depends on it)
+  ansible.builtin.wait_for:
+    host: "{{ kube_vip_address }}"
+    port: "{{ kube_vip_port }}"
+    timeout: "{{ kube_vip_wait_timeout }}"
+  run_once: true
+  changed_when: false
+  vars:
+    ansible_become: false
+  # DELIBERATELY NOT check_mode: false, unlike the interface probe above — so a `--check`
+  # rehearsal SKIPS this and does not sit on a socket for kube_vip_wait_timeout. wait_for
+  # declares no check-mode support, so the skip is the module's own behaviour and this comment
+  # is here only so the skip does not read as an oversight. CONSEQUENCE, stated plainly: a
+  # `--check` run does NOT verify the VIP is up. Only a real run gates the CNI on it.

infra/ansible/roles/memattr_survey/README.md

@@ -0,0 +1,201 @@
+# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery
+
+**Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02
+**Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism
+is settled there and this role does **not** re-derive it.
+
+## The question
+
+§0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes
+Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it,
+and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from
+the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.
+
+Two things §0b leaves unmeasured, and this role measures:
+
+1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed
+   failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes
+   fleet-wide were at other addresses. If bad outcomes map onto a narrow address range,
+   the causal chain closes without touching hardware or firmware.
+2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each.
+
+## Why 20 rounds
+
+Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson
+interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is
+worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a
+mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds`
+is a variable; lowering it below 20 produces an arm that cannot answer the question it was
+built for, and the analysis will say so.
+
+## This is the BASELINE arm
+
+**Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware
+moves, no `node_baseline` run. The role installs nothing and configures nothing; its only
+write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's
+own allocation footprint — initrd size, module set, boot device — is *part of the causal
+chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that
+the kernel release and cmdline were constant on every node, for exactly this reason.
+
+## Running it
+
+```bash
+# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml'
+
+# The real thing. Reboots are OPT-IN; the interlock is deliberate.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml \
+      -e memattr_survey_reboots_authorised=true'
+
+# Re-analyse existing artifacts, no hardware touched.
+./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
+    ansible-playbook playbooks/memattr-survey.yml --tags analyse'
+```
+
+Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`,
+`-e memattr_survey_hosts=<group>`.
+
+20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are
+independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That
+parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count;
+preflight asserts the latter. See the playbook comment for why the default linear
+strategy silently serialises a looped `include_tasks` (it was measured, not assumed).
+
+## Safety — verified, not assumed
+
+The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That
+premise is re-checked on **every** run, because a document and a machine drift apart:
+
+| Check | Failure mode it prevents |
+|---|---|
+| Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. |
+| `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. |
+| `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. |
+| `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. |
+| `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. |
+
+Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the
+whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a
+false start is a rebooted production machine.
+
+## Fail-safe and resume
+
+* **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node
+  that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the
+  observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that
+  host only** and the others keep collecting. The abort is announced immediately and again
+  in the report — which counts rounds from the **artifacts**, not from Ansible's host
+  state, so even a node that goes fully unreachable (which `rescue` cannot catch) still
+  appears as a short arm with a wider interval, never as a silently missing report.
+* **Resumable.** Round numbers come from the on-node ledger, so re-running after an
+  interruption continues where it stopped and can neither lose nor duplicate a round. The
+  boot found on a resume is *not* re-appended: it is already round N of the interrupted
+  run, and double-counting it would bias the sample toward whichever outcome survived the
+  interruption.
+* **The abort is durable, not scrollback.** `<host>.aborted.json` records the round, the
+  failing task and the reason, so the report can say *why* an arm is short instead of
+  leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every
+  run and re-written only if that run aborts. Its *absence* is not proof of success — a
+  node can be lost abruptly enough that Ansible calls it unreachable before the rescue
+  runs — so completeness is still judged on round count, and the marker only supplies the
+  reason when there is one.
+* **Analysis survives everything.** It is a separate play on `localhost`, so it runs even
+  if every surveyed node died. With a node down, `--tags analyse` is the way in: a full
+  run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host.
+
+## Belt and braces on the data
+
+Two independent records, joined on `boot_id`:
+
+| | written by | authoritative for |
+|---|---|---|
+| `<host>.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** |
+| `<host>.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** |
+| `<host>.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** |
+
+The journal does not care whether the driver was alive: the boot happened, the kernel
+logged it, journald persisted it. So the harvest can only ever be a *superset* of the
+ledger, and a round the driver failed to append still shows up.
+
+## Reading the output
+
+Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as
+`roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never
+committed. `summary.md` is also printed at the end of the run.
+
+### The verdict is measured, never inferred from a log line
+
+This is the subtlest part of the design. There are **three** kernel readings, not two:
+
+| `kernel_read` | What the kernel logged | Memory |
+|---|---|---|
+| `valid` | **nothing at all** — it parsed a table it considered valid | intact |
+| `bail-version` | `Unexpected EFI Memory Attributes table version <garbage>` | intact — the *lucky* path, and it looks alarming |
+| `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** |
+
+Silence is therefore *evidence of health*, not a missing measurement — which is why the
+record carries an explicit `memattr_diagnostic_present: false` rather than an empty field.
+And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`,
+*later* than the `memblock_reserve()` that did the damage: it is a consequence, not the
+cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and
+`kernel_read` is recorded alongside it as an independent observation. §5a of the report
+cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.
+
+Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed
+clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything
+landing between them is reported as `indeterminate` and excluded from `n` — a new third
+thing deserves human eyes, not a coin flip into a bucket.
+
+### What is and is not pooled
+
+Boots the survey did **not cause** are shown in the per-round table for context and
+excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot
+each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so
+pooling it would double-count prior evidence into a run whose point is fresh trials).
+
+### The statistics
+
+`files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is.
+
+* **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the
+  Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly
+  instead of needing a special case. A bare fraction is what this exercise exists to stop
+  reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
+* **The address↔outcome question is tested three ways**, because the tempting version is
+  circular:
+  * **4b determinism** — was any single address seen with *both* outcomes? One such
+    address falsifies "the address determines the outcome" outright.
+  * **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed
+    up as a p-value: the bad set is defined after seeing the data.
+  * **4d the pre-registered prediction** — §0b, written before this run, names
+    `0xbeeb0018`. Fisher exact against this run's boots only. This is the one
+    non-circular test available.
+  * **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a
+    run produces no bad boot, `verdict` has no variance to explain, but `kernel_read`
+    does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in
+    §0b's mechanism. An address that reproduces the same *parse* boot after boot is the
+    address doing causal work, even when none of those parses was catastrophic.
+
+## Files
+
+| Path | What it is |
+|---|---|
+| `tasks/preflight.yml` | Safety gate, resume point, baseline probe |
+| `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock |
+| `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue |
+| `tasks/harvest.yml` | The authoritative journal pass |
+| `tasks/analyse.yml` | Controller-side analysis |
+| `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable |
+| `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report |
+
+## What this cannot tell you
+
+It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute
+the difference between nodes, because node 2 is simultaneously the only single-DIMM node
+and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled
+rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS
+settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this
+role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all
+three nodes, and that is not in dispute.

infra/ansible/roles/memattr_survey/defaults/main.yml

@@ -0,0 +1,92 @@
+---
+# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
+# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
+# settled and is NOT re-derived here.
+#
+# THE QUESTION THIS ROLE ANSWERS
+#   §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
+#   Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
+#     1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
+#        observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
+#        successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
+#        address range, the causal chain is closed with no hardware or firmware touched.
+#     2. Is the fault node-2-specific, or does any node reach the bad path given enough
+#        boots? Nodes 1 and 3 have each been observed exactly twice.
+#
+# WHY 20 ROUNDS AND NOT FIVE
+#   Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
+#   interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
+#   boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
+#   needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
+#   not a round number.
+#
+# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
+#   No BIOS writes, no kernel change, no package installs, no hardware moves. The role
+#   installs nothing and configures nothing; its only write to a node is its own append-
+#   only ledger under /var/lib. Anything that alters the boot chain's allocation
+#   footprint — and per §0b that includes the initrd, the module set and the boot device —
+#   changes the variable under test and voids the arm.
+
+# --- Sample size --------------------------------------------------------------------
+# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
+# machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
+memattr_survey_rounds: 20
+
+# --- Safety interlock ----------------------------------------------------------------
+# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
+# preflight, a baseline probe of the current boot, the journal harvest and the analysis.
+# That is a genuinely useful dry run and it is also how you prove the pipeline works
+# before committing 45 minutes of wall clock to it.
+memattr_survey_reboots_authorised: false
+
+# The role refuses to run against a host that is not in this group. The prod boxes carry
+# no k3s and nothing is running on them, which is the entire reason rebooting them is
+# sanctioned; the dev cluster is live and must never be caught by a stray --limit.
+memattr_survey_required_group: k3s_prod
+
+# Unit names whose presence or activity means something IS running on this node. Any hit
+# aborts the whole playbook before the first reboot (see tasks/preflight.yml).
+memattr_survey_forbidden_units:
+  - k3s
+  - k3s-agent
+  - kubelet
+  - containerd
+  - docker
+# Paths whose existence means k3s has been installed here, even if not currently running.
+memattr_survey_forbidden_paths:
+  - /etc/rancher/k3s
+  - /var/lib/rancher/k3s
+  - /usr/local/bin/k3s
+
+# --- Reboot behaviour -----------------------------------------------------------------
+# A node that does not answer within this is declared lost: the role stops rebooting THAT
+# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
+# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
+# slow boot.
+memattr_survey_reboot_timeout: 600
+memattr_survey_connect_timeout: 20
+# Let journald settle before reading the new boot's kernel log. The kernel log itself is
+# complete long before SSH is up; this is purely margin.
+memattr_survey_post_reboot_delay: 15
+
+# --- On-node ledger (belt) -------------------------------------------------------------
+# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
+# the controller dies. This is what makes the survey resumable: re-running skips rounds
+# already recorded here rather than starting over.
+memattr_survey_state_dir: /var/lib/memattr-survey
+memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
+
+# --- Controller artifacts (braces) ------------------------------------------------------
+# Same convention as roles/node_health and conformance/_artifacts/: generated output under
+# a `_artifacts/` directory that .gitignore excludes, never committed.
+#   <host>.rounds.jsonl  the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
+#                        mid-survey still leaves every completed round on the controller
+#   <host>.boots.json    the journal harvest — every retained boot re-derived in one
+#                        pass. THIS is the authoritative record; the ledger is the backup.
+#   summary.md           the analysis (also printed)
+memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
+memattr_survey_write_artifacts: true
+
+# Print each round's one-line result as it happens. A 45-minute unattended run with no
+# output is indistinguishable from a hung one.
+memattr_survey_show_progress: true

infra/ansible/roles/memattr_survey/files/memattr_analyse.py

@@ -0,0 +1,521 @@
+#!/usr/bin/env python3
+"""memattr_analyse.py — turn a memattr survey's artifacts into the answer.
+
+Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads
+  <host>.boots.json    the journal harvest — authoritative, one entry per retained boot
+  <host>.rounds.jsonl  the per-round ledger — authoritative for ROUND NUMBERS
+and joins them on boot_id.
+
+WHAT IT ANSWERS, in order:
+
+  1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated
+     loudly and up front, because every number below it is weaker than it looks.
+
+  2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing
+     this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and
+     completely different evidence. Clopper-Pearson is the exact (conservative) binomial
+     interval — no normal approximation, correct at 0/n and n/n where the approximation is
+     degenerate and where a clean arm is most likely to land.
+
+  3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three
+     ways, deliberately, because the tempting version of this test is circular:
+       (a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such
+           address falsifies "the address determines the outcome" outright, and no amount
+           of correlation elsewhere repairs it.
+       (b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported
+           as sorted min/max per outcome plus the count of good addresses falling inside
+           the bad range. This is descriptive; it is NOT dressed up as a p-value, because
+           the bad set was chosen after seeing the data.
+       (c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts
+           that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots
+           this run observed, which is the one non-circular test available.
+
+  4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory
+     ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell.
+
+Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import math
+import os
+import sys
+from collections import Counter, defaultdict
+
+# §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad
+# both times, while all five successes fleet-wide were at other addresses.
+PREREGISTERED_BAD_ADDRESS = "0xbeeb0018"
+
+
+# --- Exact binomial interval ----------------------------------------------------------
+def binom_tail_ge(n: int, k: int, p: float) -> float:
+    """P(X >= k) for X ~ Bin(n, p)."""
+    if k <= 0:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1))
+
+
+def binom_tail_le(n: int, k: int, p: float) -> float:
+    """P(X <= k) for X ~ Bin(n, p)."""
+    if k >= n:
+        return 1.0
+    return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1))
+
+
+def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
+    """Exact 95% (by default) binomial confidence interval for k successes in n trials.
+
+    Solved by bisection on the exact tail probabilities rather than via the Beta/F
+    quantile identity, so it needs nothing outside the standard library and the
+    boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing:
+      lower bound solves P(X >= k | p) = alpha/2,  and is 0 when k = 0
+      upper bound solves P(X <= k | p) = alpha/2,  and is 1 when k = n
+    """
+    if n == 0:
+        return (0.0, 1.0)
+
+    def bisect(fn, target, lo, hi):
+        for _ in range(200):
+            mid = (lo + hi) / 2
+            if fn(mid) < target:
+                lo = mid
+            else:
+                hi = mid
+        return (lo + hi) / 2
+
+    lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0)
+    upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0)
+    return (lower, upper)
+
+
+def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float:
+    """Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only.
+
+    Used ONLY for the pre-registered prediction (address == the §0b address vs outcome),
+    never for a bad set defined after the fact — that would be a garden of forking paths
+    with a p-value stapled on.
+    """
+    n = a + b + c + d
+    if n == 0:
+        return 1.0
+    row1, col1 = a + b, a + c
+
+    def prob(x):
+        return (
+            math.comb(row1, x)
+            * math.comb(n - row1, col1 - x)
+            / math.comb(n, col1)
+        )
+
+    observed = prob(a)
+    lo = max(0, col1 - (n - row1))
+    hi = min(row1, col1)
+    # 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating
+    # float noise in the equality case.
+    return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9)))
+
+
+# --- Loading ---------------------------------------------------------------------------
+def load_host(art_dir: str, host: str) -> dict:
+    """Harvest (authoritative outcomes) joined to ledger (authoritative round numbers)."""
+    with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle:
+        harvest = json.load(handle)
+
+    rounds_by_boot: dict[str, int] = {}
+    ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl")
+    ledger: list[dict] = []
+    if os.path.exists(ledger_path):
+        with open(ledger_path, encoding="utf-8") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                try:
+                    record = json.loads(line)
+                except json.JSONDecodeError:
+                    continue
+                ledger.append(record)
+                if record.get("boot_id") and isinstance(record.get("round"), int):
+                    rounds_by_boot[record["boot_id"]] = record["round"]
+
+    boots = []
+    for boot in harvest.get("boots", []):
+        entry = dict(boot)
+        entry["round"] = rounds_by_boot.get(boot.get("boot_id"))
+        boots.append(entry)
+    boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0)))
+
+    # Written by the per-round rescue when a node stops coming back. Its absence is not
+    # proof of success — a node can be lost so abruptly that Ansible calls it unreachable
+    # before the rescue runs — so completeness is still judged on round COUNT, and this
+    # only supplies the reason when one exists.
+    abort = None
+    abort_path = os.path.join(art_dir, f"{host}.aborted.json")
+    if os.path.exists(abort_path):
+        try:
+            with open(abort_path, encoding="utf-8") as handle:
+                abort = json.load(handle)
+        except (OSError, json.JSONDecodeError):
+            abort = {"reason": "abort marker present but unreadable"}
+
+    return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort}
+
+
+def discover_hosts(art_dir: str) -> list[str]:
+    return sorted(
+        os.path.basename(p)[: -len(".boots.json")]
+        for p in glob.glob(os.path.join(art_dir, "*.boots.json"))
+    )
+
+
+# --- Rendering ---------------------------------------------------------------------------
+def gib(kb) -> str:
+    return "?" if kb is None else f"{kb / 1048576:.2f}"
+
+
+def pct(x) -> str:
+    return "?" if x is None else f"{100 * x:.0f}%"
+
+
+def render(hosts: list[dict], expected_rounds: int) -> str:
+    out: list[str] = []
+    w = out.append
+
+    all_boots = [b for h in hosts for b in h["boots"]]
+    # Only boots this survey CAUSED count toward any statistic. Excluded, deliberately:
+    #   * pre-existing journal boots (round None) — observed under an unknown selection
+    #     process, and already counted once in §0b's table;
+    #   * round 0, the boot each node was found sitting on — it is a boot nobody chose,
+    #     but for node 2 it is literally §0b's "boot 0" row, so pooling it would double-
+    #     count prior evidence into a run whose entire point is fresh, independent trials.
+    # Both are still SHOWN in the per-round table, because context is not contamination.
+    surveyed = [b for b in all_boots if b.get("round") not in (None, 0)]
+
+    w("# EFI Memory Attributes Table — baseline reboot survey")
+    w("")
+    w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no")
+    w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this")
+    w("measures p(bad) per node and tests whether the outcome is predicted by the address")
+    w("the firmware published the table at.")
+    w("")
+
+    # --- 1. Completeness -----------------------------------------------------------------
+    w("## 1. Completeness — read this before any number below it")
+    w("")
+    w("| node | rounds recorded | target | journal boots harvested | short? |")
+    w("|---|---:|---:|---:|---|")
+    short_arms = []
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        is_short = n < expected_rounds
+        if is_short:
+            short_arms.append((h["host"], n))
+        w(
+            f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | "
+            f"{'**YES — arm incomplete**' if is_short else 'no'} |"
+        )
+    w("")
+    if short_arms:
+        w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms))
+        w("> Every interval below is correspondingly wider than planned.")
+        w("")
+        for h in hosts:
+            if h.get("abort"):
+                a = h["abort"]
+                w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of "
+                  f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}")
+                w(f"> (failing task: {a.get('failed_task', 'unknown')})")
+                w("")
+
+    # --- 2. Per-round table ---------------------------------------------------------------
+    w("## 2. Per-round results")
+    w("")
+    for h in hosts:
+        installed = h["harvest"].get("smbios_installed_kb")
+        w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)")
+        w("")
+        w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |")
+        w("|---:|---|---|---|---:|---:|---:|---|")
+        for b in h["boots"]:
+            rnd = b.get("round")
+            label = "pre" if rnd is None else str(rnd)
+            usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb")
+            w(
+                f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | "
+                f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | "
+                f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | "
+                f"{gib(usable)} | {pct(b.get('usable_fraction'))} | "
+                f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |"
+            )
+        w("")
+        w("`round` = survey round. `pre` = a boot already in the journal before the survey;")
+        w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED")
+        w("from every statistic below. `kernel read`: `valid` = table")
+        w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage")
+        w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =")
+        w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from")
+        w("the presence of a log line.")
+        w("")
+
+    # --- 3. p(bad) -------------------------------------------------------------------------
+    w("## 3. p(bad) per node — 95% Clopper-Pearson")
+    w("")
+    w("| node | bad | n | p(bad) | 95% CI | reading |")
+    w("|---|---:|---:|---:|---|---|")
+    for h in hosts:
+        rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
+        n = len(rounds)
+        bad = sum(1 for b in rounds if b.get("verdict") == "bad")
+        lo, hi = clopper_pearson(bad, n)
+        if n == 0:
+            reading = "no data"
+        elif bad == 0:
+            reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}"
+        elif bad == n:
+            reading = f"bad every time in {n}; p(bad) at least {lo:.2f}"
+        else:
+            reading = f"p(bad) between {lo:.2f} and {hi:.2f}"
+        w(
+            f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | "
+            f"{lo:.2f} – {hi:.2f} | {reading} |"
+        )
+    w("")
+    indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")]
+    if indet:
+        w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are")
+        w("> reported as `indeterminate` rather than forced into a bucket. They are")
+        w("> excluded from n above. Inspect them by hand.")
+        w("")
+
+    # --- 4. Address vs outcome -----------------------------------------------------------
+    w("## 4. Is the outcome predicted by where the table landed?")
+    w("")
+    by_addr: dict[str, Counter] = defaultdict(Counter)
+    addr_hosts: dict[str, set] = defaultdict(set)
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        by_addr[addr][b.get("verdict")] += 1
+        addr_hosts[addr].add(b.get("host") or "?")
+
+    w("### 4a. Every address observed in the survey")
+    w("")
+    w("| MEMATTR | good | bad | other | nodes | verdict for this address |")
+    w("|---|---:|---:|---:|---|---|")
+    ambiguous = []
+    for addr in sorted(by_addr, key=lambda a: (a == "none", a)):
+        counts = by_addr[addr]
+        good, bad = counts.get("good", 0), counts.get("bad", 0)
+        other = sum(v for k, v in counts.items() if k not in ("good", "bad"))
+        if good and bad:
+            note = "**BOTH — address does not determine outcome**"
+            ambiguous.append(addr)
+        elif bad:
+            note = "always bad"
+        elif good:
+            note = "always good"
+        else:
+            note = "—"
+        w(
+            f"| `{addr}` | {good} | {bad} | {other} | "
+            f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |"
+        )
+    w("")
+
+    w("### 4b. Determinism")
+    w("")
+    if not by_addr:
+        w("No surveyed boots — nothing to test.")
+    elif ambiguous:
+        w(f"**FALSIFIED for {len(ambiguous)} address(es):** "
+          + ", ".join(f"`{a}`" for a in ambiguous)
+          + " was seen with BOTH outcomes. The address alone cannot determine the outcome.")
+    else:
+        w("**Not falsified.** No address in this survey was ever seen with both outcomes —")
+        w("consistent with the address determining the outcome, though with this many")
+        w("distinct addresses most were seen only once, which makes the test weak.")
+        singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1)
+        w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)")
+    w("")
+
+    w("### 4c. Separation of the address ranges")
+    w("")
+    bad_addrs = sorted({b["memattr_address"] for b in surveyed
+                        if b.get("verdict") == "bad" and b.get("memattr_address")})
+    good_addrs = sorted({b["memattr_address"] for b in surveyed
+                         if b.get("verdict") == "good" and b.get("memattr_address")})
+    if bad_addrs and good_addrs:
+        blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16)
+        inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi]
+        w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`")
+        w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`")
+        w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**"
+          + (" — the ranges interleave, so 'a narrow bad range' does not hold"
+             if inside else " — the ranges are disjoint"))
+        w("")
+        w("Descriptive only. The bad set was defined after seeing the data, so this is a")
+        w("picture, not a test.")
+    elif not bad_addrs:
+        w("No bad boot was observed in this survey, so there is no bad range to separate.")
+    else:
+        w("No good boot was observed in this survey.")
+    w("")
+
+    w("### 4d. The pre-registered prediction")
+    w("")
+    w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the")
+    w("bad address (both prior failures were there; all five prior successes elsewhere).")
+    w("Tested against this survey's boots only:")
+    w("")
+    a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
+    d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
+    w(f"| | bad | good |")
+    w(f"|---|---:|---:|")
+    w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |")
+    w(f"| elsewhere | {c} | {d} |")
+    w("")
+    if a + b_ == 0:
+        w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the")
+        w("prediction had no opportunity to be right or wrong. That is itself informative:")
+        w("the address moves every boot, so a specific address is not a reproducible handle.")
+    else:
+        p = fisher_exact_2x2(a, b_, c, d)
+        w(f"Fisher exact (two-sided) p = **{p:.4g}**"
+          + ("  — the prediction holds." if p < 0.05 and a and not b_
+             else "  — not significant at 0.05." if p >= 0.05
+             else "  — significant, but read the cell counts before believing it."))
+    w("")
+
+    w("### 4e. Does the address predict how the KERNEL READ the table?")
+    w("")
+    w("The strongest test this data can support when no bad boot occurred. `verdict` has")
+    w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both")
+    w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:")
+    w("the kernel reads whatever bytes happen to be at the address the firmware published.")
+    w("If the address determines the reading, the address is doing causal work, one step")
+    w("short of the catastrophic path.")
+    w("")
+    reads_by_addr: dict[str, set] = defaultdict(set)
+    n_by_addr: Counter = Counter()
+    for b in surveyed:
+        addr = b.get("memattr_address") or "none"
+        reads_by_addr[addr].add(b.get("kernel_read"))
+        n_by_addr[addr] += 1
+    repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1}
+    inconsistent = {a: r for a, r in repeated.items() if len(r) > 1}
+    distinct_reads = {r for rs in reads_by_addr.values() for r in rs}
+    w(f"- distinct addresses: **{len(reads_by_addr)}**, of which "
+      f"**{len(repeated)}** were observed more than once")
+    w(f"- distinct kernel readings across the survey: "
+      + ", ".join(f"`{r}`" for r in sorted(distinct_reads)))
+    w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**")
+    w("")
+    if len(distinct_reads) < 2:
+        w("Only one reading occurred in the whole survey, so there is nothing to predict.")
+    elif inconsistent:
+        w("**The address does NOT determine the reading:** "
+          + "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items()))
+    elif not repeated:
+        w("No address recurred, so the test could not be run.")
+    else:
+        w(f"**The address determines the reading, on every address observed more than once**")
+        w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both")
+        w("readings present in the sample). This is direct evidence for the causal chain")
+        w("§0b describes: the same address reproduces the same parse, boot after boot.")
+        w("It does NOT establish which addresses are catastrophic — no bad boot occurred —")
+        w("only that WHERE the table lands is what decides what the kernel finds there.")
+    w("")
+    w("| MEMATTR | boots | kernel reading(s) |")
+    w("|---|---:|---|")
+    for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)):
+        marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else ""
+        w(f"| `{addr}` | {n_by_addr[addr]} | "
+          f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |")
+    w("")
+
+    # --- 5. Corroboration -------------------------------------------------------------------
+    w("## 5. Corroborating evidence")
+    w("")
+    w("### 5a. Kernel's own reading vs the measured outcome")
+    w("")
+    cross: dict[tuple, int] = Counter()
+    for b in surveyed:
+        cross[(b.get("kernel_read"), b.get("verdict"))] += 1
+    w("| kernel read | verdict | boots |")
+    w("|---|---|---:|")
+    for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))):
+        w(f"| {read} | {verdict} | {count} |")
+    w("")
+    disagreements = [
+        (read, verdict, count)
+        for (read, verdict), count in cross.items()
+        if (read == "failed-map") != (verdict == "bad")
+    ]
+    if disagreements:
+        w("> **The log line and the measurement DISAGREE** in "
+          f"{sum(c for _, _, c in disagreements)} boot(s): "
+          + "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements)
+          + ". That is a finding in its own right — §0b's model says `failed-map` and only")
+        w("> `failed-map` costs memory.")
+    else:
+        w("`failed-map` and only `failed-map` cost memory, in every boot observed —")
+        w("exactly as §0b's mechanism predicts, and independent of the address question.")
+    w("")
+
+    w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)")
+    w("")
+    w("| verdict | NODE_DATA end address | boots |")
+    w("|---|---|---:|")
+    nd: Counter = Counter()
+    for b in surveyed:
+        line = b.get("node_data_line") or ""
+        end = line.split("-")[-1].rstrip("]") if "-" in line else "?"
+        nd[(b.get("verdict"), end)] += 1
+    for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
+        w(f"| {verdict} | `{end}` | {count} |")
+    w("")
+
+    # --- 6. Invariants ------------------------------------------------------------------
+    w("## 6. Nothing under test changed")
+    w("")
+    w("| node | kernels seen | distinct cmdlines | SMBIOS installed |")
+    w("|---|---|---:|---|")
+    for h in hosts:
+        kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")})
+        cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")}
+        w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | "
+          f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |")
+    w("")
+    w("More than one kernel or more than one cmdline on a node would void that node's arm.")
+    w("")
+    return "\n".join(out) + "\n"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--artifact-dir", required=True)
+    parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted")
+    parser.add_argument("--out", default=None, help="also write the report here")
+    args = parser.parse_args()
+
+    names = discover_hosts(args.artifact_dir)
+    if not names:
+        print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr)
+        return 1
+    hosts = [load_host(args.artifact_dir, name) for name in names]
+    report = render(hosts, args.rounds)
+    if args.out:
+        with open(args.out, "w", encoding="utf-8") as handle:
+            handle.write(report)
+    sys.stdout.write(report)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

infra/ansible/roles/memattr_survey/files/memattr_probe.py

@@ -0,0 +1,489 @@
+#!/usr/bin/env python3
+"""memattr_probe.py — read one boot's EFI Memory Attributes Table outcome off a node.
+
+Part of roles/memattr_survey (infra/ansible). See that role's README.md for the
+question this answers; see infra/PROD-PROVISIONING.md §0b for the settled mechanism.
+
+WHY A SCRIPT AND NOT A HANDFUL OF `command:` TASKS
+  The record for one boot needs six sources cross-referenced (kernel log config-table
+  line, kernel log memattr diagnostic, kernel log `Memory:` line, kernel log
+  `NODE_DATA` line, /proc/meminfo, SMBIOS) and one derived verdict. Doing that in
+  Jinja across six registered results is unreadable and would put the classification
+  logic in a template instead of in one testable place. It also has to be exactly the
+  same logic for a live boot and for a boot harvested out of the journal, which a
+  single --all-boots mode gives for free.
+
+MODES
+  (default)      one JSON record for the CURRENT boot, on stdout
+  --all-boots    one JSON object with a `boots` list: EVERY boot the persistent
+                 journal still holds, re-derived from scratch. This is the
+                 AUTHORITATIVE record — it survives the driver script dying mid-run.
+  --status       {"completed_rounds": N, "records": M} from the on-node ledger, so the
+                 playbook can resume without re-deriving anything.
+
+  --round N      stamp the record with the survey round number
+  --append PATH  ALSO append the record to PATH as one JSON line (the on-node ledger),
+                 created 0644 with parents. Appended with O_APPEND in one write() so a
+                 concurrent reader never sees a half line.
+
+VERDICT IS MEASURED, NOT INFERRED FROM A LOG LINE
+  A node whose kernel reads a VALID table logs no memattr diagnostic at all and is
+  perfectly healthy; a node whose kernel bails out on a garbage version > 2 logs a
+  scary warning and is ALSO perfectly healthy. Only the third state — "Failed to map"
+  — costs memory, and even that message is emitted by a LATER function than the one
+  that did the damage. So the verdict comes from usable RAM versus SMBIOS installed
+  RAM, and `kernel_read` is recorded ALONGSIDE it as an independent observation. If
+  the two ever disagree, that disagreement is itself the finding.
+
+EXIT STATUS is 0 whenever a record could be produced, even a mostly-null one: a field
+this script could not read comes back as null with a reason in `notes`, never as an
+empty string and never as a hard failure. A survey that aborts on one unreadable field
+loses the other 19 rounds.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+
+SCHEMA = "memattr-survey/1"
+
+# --- Verdict thresholds -------------------------------------------------------------
+# Observed on this fleet: a good boot yields usable/installed of 0.95-0.98 (kernel code,
+# reserved firmware regions and struct page overhead account for the rest); the bad path
+# yields 0.15-0.17 (everything above the table address is reserved). Two orders of
+# separation, so the thresholds are nowhere near either cluster. Anything landing in the
+# gap is deliberately NOT called: it is a new third thing and wants human eyes.
+VERDICT_GOOD_MIN_FRACTION = 0.85
+VERDICT_BAD_MAX_FRACTION = 0.60
+
+# `Memory: 2568544K/16546088K available (... 13958980K reserved ...)`
+RE_MEMORY_LINE = re.compile(
+    r"^Memory:\s+(?P<available>\d+)K/(?P<present>\d+)K available\b.*?"
+    r"(?:(?P<reserved>\d+)K reserved)?"
+)
+# `efi: ACPI=0x... SMBIOS=0x... MEMATTR=0xbeeb0018 MOKvar=0x... INITRD=0x...`
+RE_MEMATTR_ADDR = re.compile(r"\bMEMATTR=(?P<addr>0x[0-9a-fA-F]+)")
+# `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749`
+RE_TABLE_VERSION = re.compile(r"table version\s+(?P<version>-?\d+)")
+# `NODE_DATA(0) allocated [mem 0xbee85680-0xbeeaffff]` — per §0b the cleanest A/B tell:
+# on the bad path pgdat is forced immediately BELOW the table address instead of landing
+# at the top of RAM. Captured because it is free and it is independent evidence.
+RE_NODE_DATA = re.compile(r"^NODE_DATA\(\d+\) allocated \[mem (?P<range>[0-9a-fx]+-[0-9a-fx]+)\]")
+RE_CMDLINE = re.compile(r"^Command line:\s*(?P<cmdline>.*)$")
+# dmidecode type 17 `Size: 16 GB` / `Size: 8192 MB` / `Size: No Module Installed`
+RE_DMI_SIZE = re.compile(r"^\s*Size:\s+(?P<n>\d+)\s+(?P<unit>[kKmMgGtT]B)\s*$")
+
+DMI_UNIT_KB = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def run(argv: list[str], timeout: int = 60) -> tuple[int, str, str]:
+    """Run a command, never raise. Returns (rc, stdout, stderr); rc 127 if absent."""
+    try:
+        proc = subprocess.run(
+            argv, capture_output=True, text=True, timeout=timeout, check=False
+        )
+    except FileNotFoundError:
+        return 127, "", f"{argv[0]}: not found"
+    except subprocess.TimeoutExpired:
+        return 124, "", f"{' '.join(argv)}: timed out after {timeout}s"
+    except OSError as exc:  # pragma: no cover - defensive
+        return 126, "", f"{' '.join(argv)}: {exc}"
+    return proc.returncode, proc.stdout, proc.stderr
+
+
+def read_text(path: str) -> str | None:
+    try:
+        with open(path, "r", encoding="utf-8", errors="replace") as handle:
+            return handle.read()
+    except OSError:
+        return None
+
+
+def kernel_log(boot: str) -> tuple[list[str], str | None]:
+    """Kernel ring buffer for `boot` ('0' = current, '-1' = previous, or a boot ID).
+
+    `-o cat` is deliberate: it yields the message VERBATIM with no host/facility
+    decoration, which is what the survey wants to record, and it is stable across
+    systemd versions. `-k` restricts to _TRANSPORT=kernel.
+    """
+    rc, out, err = run(["journalctl", "-k", "-b", boot, "-o", "cat", "--no-pager"], timeout=180)
+    if rc != 0:
+        return [], f"journalctl -k -b {boot} failed (rc={rc}): {err.strip()[:200]}"
+    return out.splitlines(), None
+
+
+def list_boots() -> tuple[list[dict], str | None]:
+    """Every boot the journal still holds, oldest first.
+
+    Prefers `--list-boots -o json` (systemd >= 250, so always on Debian 13) and falls
+    back to parsing the table, because a survey that cannot enumerate boots has lost
+    its authoritative record and must say so rather than silently reporting one boot.
+    """
+    rc, out, err = run(["journalctl", "--list-boots", "-o", "json", "--no-pager"], timeout=120)
+    if rc == 0:
+        try:
+            parsed = json.loads(out)
+        except json.JSONDecodeError as exc:
+            return [], f"--list-boots -o json unparseable: {exc}"
+        boots = [
+            {
+                "index": int(entry.get("index", 0)),
+                "boot_id": entry.get("boot_id"),
+                "first_entry_usec": entry.get("first_entry"),
+                "last_entry_usec": entry.get("last_entry"),
+            }
+            for entry in parsed
+        ]
+        return sorted(boots, key=lambda b: b["index"]), None
+
+    rc, out, err = run(["journalctl", "--list-boots", "--no-pager"], timeout=120)
+    if rc != 0:
+        return [], f"journalctl --list-boots failed (rc={rc}): {err.strip()[:200]}"
+    boots = []
+    for line in out.splitlines():
+        fields = line.split()
+        if len(fields) < 2 or not re.fullmatch(r"-?\d+", fields[0]):
+            continue  # header row
+        boots.append(
+            {
+                "index": int(fields[0]),
+                "boot_id": fields[1],
+                "first_entry_usec": None,
+                "last_entry_usec": None,
+            }
+        )
+    return sorted(boots, key=lambda b: b["index"]), None
+
+
+def journal_storage() -> dict:
+    """Is the journal PERSISTENT? The harvest is only authoritative if it is.
+
+    /var/log/journal existing is what makes journald write to disk; /run/log/journal is
+    the volatile fallback and would mean every reboot erases the record.
+    """
+    return {
+        "var_log_journal": os.path.isdir("/var/log/journal"),
+        "run_log_journal": os.path.isdir("/run/log/journal"),
+        "persistent": os.path.isdir("/var/log/journal"),
+    }
+
+
+def smbios_installed_kb() -> tuple[int | None, str | None]:
+    """Installed RAM per SMBIOS type-17 records — the GROUND TRUTH for the verdict.
+
+    Deliberately not /proc/meminfo (that is the number under test) and not dmesg's
+    `Memory: x/yK` denominator (which is already post-e820 and post-reservation).
+    Hardware does not change between boots, so one read applies to harvested boots too.
+    """
+    rc, out, err = run(["dmidecode", "-t", "17"], timeout=60)
+    if rc != 0:
+        return None, f"dmidecode -t 17 failed (rc={rc}): {err.strip()[:200]}"
+    total = 0
+    seen = 0
+    for line in out.splitlines():
+        match = RE_DMI_SIZE.match(line)
+        if not match:
+            continue  # "No Module Installed" and every non-Size line
+        total += int(match.group("n")) * DMI_UNIT_KB[match.group("unit").lower()]
+        seen += 1
+    if seen == 0:
+        return None, "dmidecode -t 17 reported no populated memory devices"
+    return total, None
+
+
+def meminfo_total_kb() -> tuple[int | None, str | None]:
+    text = read_text("/proc/meminfo")
+    if text is None:
+        return None, "/proc/meminfo unreadable"
+    for line in text.splitlines():
+        if line.startswith("MemTotal:"):
+            return int(line.split()[1]), None
+    return None, "/proc/meminfo has no MemTotal"
+
+
+def classify_kernel_read(memattr_addr, diagnostics: list[str]) -> tuple[str, int | None]:
+    """What did the kernel make of the table? THREE outcomes, not two (§0b).
+
+    Returns (classification, table_version_if_reported).
+
+      no-table      firmware published no MEMATTR pointer at all (the dev nodes)
+      valid         a MEMATTR pointer AND no diagnostic => the kernel parsed a table it
+                    considered valid. This is SILENCE AS EVIDENCE, and it is a healthy
+                    state, not a missing measurement.
+      bail-version  garbage version > 2 => kernel warned and bailed. Memory intact.
+                    The LUCKY path, and it looks alarming in the log.
+      failed-map    the catastrophic path. The message comes from
+                    efi_memattr_apply_permissions() and is therefore PROOF the bogus
+                    memblock_reserve() already happened upstream of it.
+      bogus-size    a kernel >= 6.13 rejecting the table. Should never appear on 6.12;
+                    if it does, the kernel under test changed and the arm is void.
+    """
+    version = None
+    for line in diagnostics:
+        match = RE_TABLE_VERSION.search(line)
+        if match:
+            version = int(match.group("version"))
+    joined = " | ".join(diagnostics)
+    if "Failed to map" in joined:
+        return "failed-map", version
+    if "Corrupted EFI Memory Attributes Table" in joined:
+        return "bogus-size", version
+    if "Unexpected EFI Memory Attributes table version" in joined:
+        return "bail-version", version
+    if diagnostics:
+        return "diagnostic-unrecognised", version
+    if memattr_addr is None:
+        return "no-table", version
+    return "valid", version
+
+
+def derive_verdict(usable_kb, installed_kb) -> tuple[str, float | None]:
+    if usable_kb is None or not installed_kb:
+        return "unknown", None
+    fraction = usable_kb / installed_kb
+    if fraction >= VERDICT_GOOD_MIN_FRACTION:
+        return "good", fraction
+    if fraction <= VERDICT_BAD_MAX_FRACTION:
+        return "bad", fraction
+    return "indeterminate", fraction
+
+
+def build_record(
+    *,
+    boot_selector: str,
+    boot_meta: dict | None,
+    installed_kb,
+    installed_note,
+    round_number,
+    live: bool,
+) -> dict:
+    notes: list[str] = []
+    if installed_note:
+        notes.append(installed_note)
+
+    lines, log_note = kernel_log(boot_selector)
+    if log_note:
+        notes.append(log_note)
+
+    config_table_line = None
+    memattr_addr = None
+    diagnostics: list[str] = []
+    memory_line = None
+    node_data_line = None
+    cmdline = None
+
+    for line in lines:
+        stripped = line.strip()
+        if memattr_addr is None:
+            match = RE_MEMATTR_ADDR.search(stripped)
+            if match and stripped.startswith("efi:"):
+                memattr_addr = match.group("addr").lower()
+                config_table_line = stripped
+        # Every memattr-related diagnostic, verbatim. Matched on the pr_fmt prefix the
+        # kernel actually uses (`efi: memattr: `) AND on the message text, so a prefix
+        # change upstream cannot silently drop the line.
+        if stripped.startswith("efi: memattr:") or "EFI Memory Attributes" in stripped:
+            if stripped != config_table_line:
+                diagnostics.append(stripped)
+        if memory_line is None and stripped.startswith("Memory:"):
+            if RE_MEMORY_LINE.match(stripped):
+                memory_line = stripped
+        if node_data_line is None:
+            match = RE_NODE_DATA.match(stripped)
+            if match:
+                node_data_line = stripped
+        if cmdline is None:
+            match = RE_CMDLINE.match(stripped)
+            if match:
+                cmdline = match.group("cmdline").strip()
+
+    available_kb = present_kb = reserved_kb = None
+    if memory_line:
+        match = RE_MEMORY_LINE.match(memory_line)
+        available_kb = int(match.group("available"))
+        present_kb = int(match.group("present"))
+        reserved = match.group("reserved")
+        reserved_kb = int(reserved) if reserved else None
+    else:
+        notes.append("no `Memory: .../... available` line found in this boot's kernel log")
+
+    kernel_read, table_version = classify_kernel_read(memattr_addr, diagnostics)
+
+    # /proc/meminfo describes the CURRENT boot only. For a harvested past boot the
+    # equivalent measurement is the kernel's own `Memory: available` figure, which
+    # tracks MemTotal to within ~7% on both paths. Which one was used is recorded, so
+    # nobody has to guess later.
+    mem_total_kb, meminfo_note = (None, None)
+    if live:
+        mem_total_kb, meminfo_note = meminfo_total_kb()
+        if meminfo_note:
+            notes.append(meminfo_note)
+    usable_kb = mem_total_kb if mem_total_kb is not None else available_kb
+    basis = "MemTotal" if mem_total_kb is not None else "dmesg Memory: available"
+    verdict, fraction = derive_verdict(usable_kb, installed_kb)
+
+    record = {
+        "schema": SCHEMA,
+        "round": round_number,
+        "host": os.uname().nodename,
+        "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+        "recorded_at_epoch": int(time.time()),
+        "live_boot": live,
+        "boot_index": (boot_meta or {}).get("index"),
+        "boot_id": (boot_meta or {}).get("boot_id"),
+        "boot_first_entry_usec": (boot_meta or {}).get("first_entry_usec"),
+        "kernel_release": os.uname().release,
+        "kernel_cmdline": cmdline,
+        # --- the independent variable -------------------------------------------------
+        "efi_config_table_line": config_table_line,
+        "memattr_address": memattr_addr,
+        "memattr_diagnostics": diagnostics,
+        "memattr_diagnostic_count": len(diagnostics),
+        # An explicit marker, never an empty field: "the kernel said nothing" is a
+        # measurement (it read a VALID table), not a gap in the data.
+        "memattr_diagnostic_present": bool(diagnostics),
+        "memattr_table_version": table_version,
+        "kernel_read": kernel_read,
+        # --- the dependent variable ---------------------------------------------------
+        "memory_line": memory_line,
+        "memory_available_kb": available_kb,
+        "memory_present_kb": present_kb,
+        "memory_reserved_kb": reserved_kb,
+        "mem_total_kb": mem_total_kb,
+        "smbios_installed_kb": installed_kb,
+        "usable_kb": usable_kb,
+        "usable_fraction": round(fraction, 5) if fraction is not None else None,
+        "verdict": verdict,
+        "verdict_basis": f"{basis} vs SMBIOS installed",
+        # --- corroborating evidence ---------------------------------------------------
+        "node_data_line": node_data_line,
+        "notes": notes,
+    }
+    return record
+
+
+def append_record(path: str, record: dict) -> str | None:
+    """One O_APPEND write of one line, so a reader never sees a torn record."""
+    try:
+        parent = os.path.dirname(path) or "."
+        os.makedirs(parent, mode=0o755, exist_ok=True)
+        payload = (json.dumps(record, sort_keys=True) + "\n").encode()
+        fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
+        try:
+            os.write(fd, payload)
+            os.fsync(fd)  # the point of the ledger is surviving the next reboot
+        finally:
+            os.close(fd)
+    except OSError as exc:
+        return f"could not append to {path}: {exc}"
+    return None
+
+
+def ledger_status(path: str) -> dict:
+    """Resume support: how far did a previous run get?
+
+    Round numbers come from the ledger itself rather than a line count, so the
+    baseline round 0 record and any hand-added line cannot shift the numbering.
+    """
+    rounds: list[int] = []
+    records = 0
+    malformed = 0
+    if os.path.exists(path):
+        with open(path, "r", encoding="utf-8", errors="replace") as handle:
+            for line in handle:
+                line = line.strip()
+                if not line:
+                    continue
+                records += 1
+                try:
+                    parsed = json.loads(line)
+                except json.JSONDecodeError:
+                    malformed += 1
+                    continue
+                value = parsed.get("round")
+                if isinstance(value, int):
+                    rounds.append(value)
+    return {
+        "schema": SCHEMA,
+        "ledger": path,
+        "records": records,
+        "malformed_records": malformed,
+        "completed_rounds": max(rounds) if rounds else 0,
+        "rounds_recorded": sorted(rounds),
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--round", type=int, default=None, help="survey round number to stamp")
+    parser.add_argument("--append", default=None, help="on-node ledger to append to")
+    parser.add_argument("--all-boots", action="store_true", help="harvest every retained boot")
+    parser.add_argument("--status", default=None, metavar="LEDGER", help="report resume state")
+    args = parser.parse_args()
+
+    if args.status:
+        print(json.dumps(ledger_status(args.status), indent=2, sort_keys=True))
+        return 0
+
+    installed_kb, installed_note = smbios_installed_kb()
+    boots, boots_note = list_boots()
+    storage = journal_storage()
+
+    if args.all_boots:
+        records = []
+        for boot in boots:
+            # Address boots by BOOT ID, not by relative index: indices shift under the
+            # harvester's feet if journald rotates mid-harvest, and the ID is what the
+            # per-round ledger recorded, so the two records join cleanly.
+            selector = boot["boot_id"] or str(boot["index"])
+            records.append(
+                build_record(
+                    boot_selector=selector,
+                    boot_meta=boot,
+                    installed_kb=installed_kb,
+                    installed_note=installed_note,
+                    round_number=None,
+                    live=(boot["index"] == 0),
+                )
+            )
+        payload = {
+            "schema": SCHEMA,
+            "host": os.uname().nodename,
+            "harvested_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "journal_storage": storage,
+            "smbios_installed_kb": installed_kb,
+            "boots_listed": len(boots),
+            "notes": [n for n in [boots_note, installed_note] if n],
+            "boots": records,
+        }
+        print(json.dumps(payload, indent=2, sort_keys=True))
+        return 0
+
+    current = next((b for b in boots if b["index"] == 0), None)
+    record = build_record(
+        boot_selector="0",
+        boot_meta=current,
+        installed_kb=installed_kb,
+        installed_note=installed_note,
+        round_number=args.round,
+        live=True,
+    )
+    record["journal_storage"] = storage
+    if boots_note:
+        record["notes"].append(boots_note)
+    if args.append:
+        note = append_record(args.append, record)
+        if note:
+            record["notes"].append(note)
+    print(json.dumps(record, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

infra/ansible/roles/memattr_survey/tasks/analyse.yml

@@ -0,0 +1,42 @@
+---
+# Controller-side analysis. Runs on localhost over the artifact directory, so it is
+# independent of whether any surveyed node is still alive — and so it can be re-run on
+# its own (`--tags analyse`) without touching hardware.
+#
+# The statistics live in files/memattr_analyse.py, not in Jinja: Clopper-Pearson and
+# Fisher belong somewhere testable, and a confidence interval assembled out of template
+# filters is a confidence interval nobody will ever check.
+
+- name: Locate the survey artifacts
+  ansible.builtin.find:
+    paths: "{{ memattr_survey_artifact_dir }}"
+    patterns: "*.boots.json"
+  register: memattr_survey_artifacts
+
+- name: Report that there is nothing to analyse
+  ansible.builtin.debug:
+    msg: >-
+      No *.boots.json under {{ memattr_survey_artifact_dir }} — no survey has produced a
+      harvest yet. Run the playbook with -e memattr_survey_reboots_authorised=true.
+  when: memattr_survey_artifacts.matched | int == 0
+
+- name: Analyse the survey
+  ansible.builtin.command:
+    argv:
+      - /usr/bin/env
+      - python3
+      - "{{ role_path }}/files/memattr_analyse.py"
+      - "--artifact-dir"
+      - "{{ memattr_survey_artifact_dir }}"
+      - "--rounds"
+      - "{{ memattr_survey_rounds }}"
+      - "--out"
+      - "{{ memattr_survey_artifact_dir }}/summary.md"
+  register: memattr_survey_analysis
+  changed_when: false
+  when: memattr_survey_artifacts.matched | int > 0
+
+- name: Show the analysis
+  ansible.builtin.debug:
+    msg: "{{ memattr_survey_analysis.stdout.split('\n') }}"
+  when: memattr_survey_analysis is not skipped

infra/ansible/roles/memattr_survey/tasks/harvest.yml

@@ -0,0 +1,66 @@
+---
+# The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from
+# scratch in one pass by the same code that produced the per-round ledger.
+#
+# WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and
+# the append, that round exists on the machine but not in the ledger. The journal does not
+# care whether the driver was alive — the boot happened, the kernel logged it, journald
+# persisted it. So the harvest can only ever be a superset of the ledger, and the analysis
+# joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and
+# any boot the ledger missed still shows up.
+#
+# This runs even for a node that aborted mid-survey, which is exactly when it matters most.
+
+- name: Harvest every retained boot from the persistent journal
+  ansible.builtin.script:
+    cmd: memattr_probe.py --all-boots
+    executable: /usr/bin/python3
+  register: memattr_survey_harvest_raw
+  changed_when: false   # strictly a read of the journal
+
+- name: Parse the harvest
+  ansible.builtin.set_fact:
+    memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}"
+
+- name: Ensure the artifact directory exists (on the controller)
+  ansible.builtin.file:
+    path: "{{ memattr_survey_artifact_dir }}"
+    state: directory
+    mode: "0755"
+  delegate_to: localhost
+  become: false
+  # NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and
+  # run_once is unreliable there. `state: directory` is idempotent, so three hosts each
+  # asserting the directory exists is cheaper than a conditional that might not fire.
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Write the harvested boot record
+  ansible.builtin.copy:
+    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json"
+    mode: "0644"
+    content: "{{ memattr_survey_harvest | to_nice_json }}"
+  delegate_to: localhost
+  become: false
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Mirror the final ledger to the controller
+  ansible.builtin.fetch:
+    src: "{{ memattr_survey_state_file }}"
+    dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
+    flat: true
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Harvest summary
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots
+      from the journal
+      ({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad,
+      {{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good,
+      {{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other)
+      {%- if memattr_survey_aborted | default(false) | bool %}
+      — NOTE: this node ABORTED ({{ memattr_survey_abort_reason }})
+      {%- endif %}

infra/ansible/roles/memattr_survey/tasks/main.yml

@@ -0,0 +1,23 @@
+---
+# memattr_survey — see defaults/main.yml for the question, README.md for how to run and
+# read it, and infra/PROD-PROVISIONING.md §0b for the settled mechanism this measures.
+#
+# ORDER OF OPERATIONS
+#   1. preflight.yml — refuse to touch a node that is not prod, not idle, or has no
+#      persistent journal; read the resume point; record the boot we found it on.
+#   2. round.yml x N — reboot, measure, append, mirror. Per-node fail-safe.
+#   3. harvest.yml   — re-derive every retained boot from the journal (authoritative).
+#
+# The playbook splits these across plays so that preflight is any_errors_fatal (one unsafe
+# node stops everything) while the rounds are not (one dead node stops only itself), and
+# so that the controller-side analysis runs even if every node dies. This file is the
+# whole-role entry point kept for completeness; the playbook calls the stages directly.
+
+- name: Preflight and baseline
+  ansible.builtin.include_tasks: preflight.yml
+
+- name: Run the survey rounds
+  ansible.builtin.include_tasks: rounds.yml
+
+- name: Harvest the authoritative record
+  ansible.builtin.include_tasks: harvest.yml

infra/ansible/roles/memattr_survey/tasks/preflight.yml

@@ -0,0 +1,149 @@
+---
+# Safety gate. Runs with any_errors_fatal at the play level (playbooks/memattr-survey.yml),
+# so a violation on ANY node stops the entire playbook before a single reboot — the
+# correct posture when the failure mode is "rebooted a machine that was doing something".
+#
+# VERIFY, DO NOT ASSUME. The premise that makes these reboots safe is that prod carries no
+# k3s and runs nothing. That premise is checked here on every run rather than trusted from
+# a document, because the document and the machine drift independently.
+
+- name: Assert the target host is in the prod group
+  # The dev cluster is LIVE. This is the one check that a mistyped --limit or a copied
+  # command line cannot slip past: membership is read from the inventory, not the pattern.
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_required_group in (group_names | default([]))
+    fail_msg: >-
+      REFUSING TO SURVEY {{ inventory_hostname }}: it is not a member of
+      '{{ memattr_survey_required_group }}' (groups: {{ group_names | default([]) | join(', ') }}).
+      This role reboots its targets. Point it at prod, or override
+      memattr_survey_required_group deliberately.
+    quiet: true
+
+- name: Check whether any cluster or container runtime unit is active
+  # `is-active` returns non-zero for inactive/unknown units, which is the expected answer
+  # here, so the rc is data and not a failure.
+  ansible.builtin.command:
+    argv: "{{ ['systemctl', 'is-active'] + memattr_survey_forbidden_units }}"
+  register: memattr_survey_units
+  changed_when: false
+  failed_when: false
+
+- name: Check whether k3s has been installed on this node
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  loop: "{{ memattr_survey_forbidden_paths }}"
+  register: memattr_survey_paths
+  changed_when: false
+
+- name: Assert nothing is running on this node
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_units.stdout_lines | reject('equalto', 'inactive') | reject('equalto', 'unknown') | list | length == 0
+      - memattr_survey_paths.results | selectattr('stat.exists') | list | length == 0
+    fail_msg: >-
+      REFUSING TO REBOOT {{ inventory_hostname }} — it is not idle.
+      units {{ memattr_survey_forbidden_units | zip(memattr_survey_units.stdout_lines) | map('join', '=') | join(' ') }};
+      present paths: {{ memattr_survey_paths.results | selectattr('stat.exists') | map(attribute='item') | list | join(', ') | default('none', true) }}.
+      The survey is only sanctioned because these boxes carry no k3s and run nothing.
+    quiet: true
+
+- name: Clear any abort marker from a previous run
+  # A marker left by a run that has since been resumed successfully would make this run's
+  # report describe the LAST failure rather than this one. The marker means "this run
+  # aborted", so it is cleared at the start of every run and re-written only if it does.
+  ansible.builtin.file:
+    path: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
+    state: absent
+  delegate_to: localhost
+  become: false
+  changed_when: false
+  when: memattr_survey_write_artifacts | bool
+
+- name: Assert forks allows every node to be surveyed concurrently
+  # The survey play runs `strategy: free` so the nodes reboot in parallel, but forks is a
+  # hard cap on that: with fewer forks than hosts the survey quietly serialises and takes
+  # a multiple of the wall clock it should. Cheap to satisfy (`-f <n>`), so assert rather
+  # than discover it three hours later.
+  ansible.builtin.assert:
+    that:
+      - (lookup('ansible.builtin.config', 'DEFAULT_FORKS') | int) >= (ansible_play_hosts_all | length)
+    fail_msg: >-
+      forks={{ lookup('ansible.builtin.config', 'DEFAULT_FORKS') }} but this survey targets
+      {{ ansible_play_hosts_all | length }} hosts. The rounds would serialise. Re-run with
+      -f {{ ansible_play_hosts_all | length }} (or higher).
+    quiet: true
+  run_once: true
+
+- name: Read the node's resume state and baseline boot
+  # One call does double duty: it reports how far a previous run got (so this one resumes)
+  # and, with --round 0, records the boot we found the node on. Round 0 is a free data
+  # point — it is a boot nobody chose, so it is not selected for by the survey.
+  ansible.builtin.script:
+    cmd: >-
+      memattr_probe.py --status {{ memattr_survey_state_file | quote }}
+    executable: /usr/bin/python3
+  register: memattr_survey_status_raw
+  changed_when: false
+
+- name: Record the resume point
+  ansible.builtin.set_fact:
+    memattr_survey_status: "{{ memattr_survey_status_raw.stdout | from_json }}"
+    memattr_survey_aborted: false
+    memattr_survey_abort_reason: ""
+
+- name: Assert the probe schema is the one this role understands
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_status.schema == 'memattr-survey/1'
+    fail_msg: >-
+      Probe returned schema {{ memattr_survey_status.schema | default('<none>') }};
+      this role understands memattr-survey/1. Role and probe are out of step.
+    quiet: true
+
+- name: Probe the boot the node is currently on
+  # On a FIRST run this is stamped round 0 and appended: the boot the node was already
+  # sitting on is a free, unselected data point. On a RESUME it is read but NOT appended,
+  # because that boot is already round N of the interrupted run and double-counting it
+  # would bias the sample toward whatever outcome happened to survive the interruption.
+  ansible.builtin.script:
+    cmd: >-
+      memattr_probe.py
+      {% if memattr_survey_status.completed_rounds | int == 0 %}
+      --round 0 --append {{ memattr_survey_state_file | quote }}
+      {% endif %}
+    executable: /usr/bin/python3
+  register: memattr_survey_baseline_raw
+  # The append is the experiment's own ledger, not a change to the node under test: this
+  # role reports `changed` only for things that could affect a boot, and a file written
+  # under /var/lib long after the kernel parsed the table cannot.
+  changed_when: false
+
+- name: Parse the baseline record
+  ansible.builtin.set_fact:
+    memattr_survey_baseline: "{{ memattr_survey_baseline_raw.stdout | from_json }}"
+
+- name: Assert the journal is PERSISTENT
+  # The end-of-run harvest is the authoritative record precisely because it re-derives
+  # every boot from the journal. On a volatile journal (/run/log/journal only) each
+  # reboot erases the evidence and the harvest would silently return one boot — a survey
+  # that looks complete and holds nothing. Verified, not assumed.
+  ansible.builtin.assert:
+    that:
+      - memattr_survey_baseline.journal_storage.persistent | bool
+    fail_msg: >-
+      {{ inventory_hostname }} has NO persistent journal (/var/log/journal missing).
+      Every reboot would erase the record this survey depends on. Refusing to run.
+    quiet: true
+
+- name: Report the starting state
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: idle, prod, persistent journal.
+      kernel {{ memattr_survey_baseline.kernel_release | default('?') }},
+      installed {{ (memattr_survey_baseline.smbios_installed_kb | default(0) | int / 1048576) | round(1) }} GiB.
+      baseline boot MEMATTR={{ memattr_survey_baseline.memattr_address | default('none') }}
+      kernel_read={{ memattr_survey_baseline.kernel_read | default('?') }}
+      verdict={{ memattr_survey_baseline.verdict | default('?') | upper }}.
+      resuming from round {{ memattr_survey_status.completed_rounds }}
+      of {{ memattr_survey_rounds }}.

infra/ansible/roles/memattr_survey/tasks/round.yml

@@ -0,0 +1,98 @@
+---
+# ONE round on ONE node: reboot, wait for it back, measure the new boot, record it twice.
+# Included in a loop from tasks/main.yml with loop_var `memattr_survey_round`.
+#
+# FAIL-SAFE, PER NODE. The block/rescue below is what keeps one dead node from ending the
+# survey: on any failure this host sets memattr_survey_aborted and every later round is
+# skipped FOR THIS HOST ONLY, while the other nodes keep going (Ansible runs the loop for
+# all hosts in lockstep, so the reboots are genuinely concurrent). The abort is reported
+# loudly here AND again in the final analysis, which counts rounds from the artifacts
+# rather than from Ansible's host state — so a node that vanishes so hard that it is
+# marked unreachable (which rescue cannot catch) still shows up as a short arm.
+
+- name: "Round {{ memattr_survey_round }} — survey step"
+  block:
+    - name: "Round {{ memattr_survey_round }} — reboot and wait for the node to return"
+      ansible.builtin.reboot:
+        reboot_timeout: "{{ memattr_survey_reboot_timeout | int }}"
+        connect_timeout: "{{ memattr_survey_connect_timeout | int }}"
+        post_reboot_delay: "{{ memattr_survey_post_reboot_delay | int }}"
+        msg: "memattr survey round {{ memattr_survey_round }} of {{ memattr_survey_rounds }}"
+
+    - name: "Round {{ memattr_survey_round }} — measure the new boot and append to the ledger"
+      ansible.builtin.script:
+        cmd: >-
+          memattr_probe.py --round {{ memattr_survey_round }}
+          --append {{ memattr_survey_state_file | quote }}
+        executable: /usr/bin/python3
+      register: memattr_survey_round_raw
+      changed_when: false   # see the note in preflight.yml — the ledger is not the node
+
+    - name: "Round {{ memattr_survey_round }} — parse the record"
+      ansible.builtin.set_fact:
+        memattr_survey_record: "{{ memattr_survey_round_raw.stdout | from_json }}"
+
+    - name: "Round {{ memattr_survey_round }} — result"
+      ansible.builtin.debug:
+        msg: >-
+          [{{ inventory_hostname }} r{{ memattr_survey_round }}/{{ memattr_survey_rounds }}]
+          MEMATTR={{ memattr_survey_record.memattr_address | default('none') }}
+          read={{ memattr_survey_record.kernel_read }}
+          {%- if memattr_survey_record.memattr_table_version is not none %}
+          (version {{ memattr_survey_record.memattr_table_version }})
+          {%- endif %}
+          MemTotal={{ (memattr_survey_record.mem_total_kb | default(0) | int / 1048576) | round(2) }}G
+          of {{ (memattr_survey_record.smbios_installed_kb | default(0) | int / 1048576) | round(0) | int }}G
+          => {{ memattr_survey_record.verdict | upper }}
+          boot={{ memattr_survey_record.boot_id | default('?') | truncate(9, true, '') }}
+      when: memattr_survey_show_progress | bool
+
+    # BRACES. The on-node ledger already survives a reboot; re-fetching it every round
+    # means the controller also holds every completed round if the DRIVER dies, without
+    # waiting for the end-of-run harvest. One tiny file per host, per-host destination
+    # paths, so three concurrent fetches cannot collide.
+    - name: "Round {{ memattr_survey_round }} — mirror the ledger to the controller"
+      ansible.builtin.fetch:
+        src: "{{ memattr_survey_state_file }}"
+        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
+        flat: true
+      changed_when: false
+      when: memattr_survey_write_artifacts | bool
+
+  rescue:
+    - name: "Round {{ memattr_survey_round }} — ABORT this node, continue the others"
+      ansible.builtin.set_fact:
+        memattr_survey_aborted: true
+        memattr_survey_abort_reason: >-
+          round {{ memattr_survey_round }} failed:
+          {{ ansible_failed_result.msg | default('unknown error') | trim }}
+
+    # The abort must survive the run, not live only in terminal scrollback. Without
+    # this the report can see that an arm is SHORT but not WHY, and "0 rounds" reads
+    # like a targeting mistake instead of a node that failed to come back.
+    - name: "Round {{ memattr_survey_round }} — record the abort as an artifact"
+      ansible.builtin.copy:
+        dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
+        mode: "0644"
+        content: "{{ memattr_survey_abort_record | to_nice_json }}"
+      vars:
+        memattr_survey_abort_record:
+          schema: "memattr-survey/1"
+          host: "{{ inventory_hostname }}"
+          aborted_at_round: "{{ memattr_survey_round }}"
+          target_rounds: "{{ memattr_survey_rounds }}"
+          reason: "{{ memattr_survey_abort_reason | trim }}"
+          failed_task: "{{ ansible_failed_task.name | default('unknown') }}"
+      delegate_to: localhost
+      become: false
+      changed_when: false
+      when: memattr_survey_write_artifacts | bool
+
+    - name: "Round {{ memattr_survey_round }} — announce the abort"
+      ansible.builtin.debug:
+        msg: >-
+          *** SURVEY ABORTED ON {{ inventory_hostname }} at round
+          {{ memattr_survey_round }}/{{ memattr_survey_rounds }}:
+          {{ memattr_survey_abort_reason }}
+          *** The other nodes continue. This node's arm is SHORT — the analysis will
+          say so, and its p(bad) interval will be correspondingly wider.

infra/ansible/roles/memattr_survey/tasks/rounds.yml

@@ -0,0 +1,40 @@
+---
+# The loop itself, kept separate from round.yml so the loop bounds (which encode the
+# RESUME logic) are readable in one place.
+#
+# RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a
+# re-run after an interruption continues where it stopped and can never lose or duplicate
+# an earlier round. Already at or past the target count => nothing to do, and the run
+# collapses to a harvest.
+#
+# NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which
+# makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest,
+# analysis) still runs, so a dry run exercises the entire pipeline without touching a
+# single machine.
+
+- name: Announce the plan
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}:
+      {%- if not (memattr_survey_reboots_authorised | bool) %}
+      DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm).
+      {%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %}
+      already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do.
+      {%- else %}
+      rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }}
+      to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots).
+      {%- endif %}
+
+- name: Survey round
+  ansible.builtin.include_tasks: round.yml
+  loop: >-
+    {{ range(memattr_survey_status.completed_rounds | int + 1,
+             memattr_survey_rounds | int + 1) | list }}
+  loop_control:
+    loop_var: memattr_survey_round
+    label: "{{ inventory_hostname }} round {{ memattr_survey_round }}"
+  # Evaluated per iteration, so the abort flag set by a rescue in one round skips every
+  # round after it on THIS host and no others.
+  when:
+    - memattr_survey_reboots_authorised | bool
+    - not (memattr_survey_aborted | bool)

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -18,22 +18,193 @@ node_baseline_packages:
   # node from first baseline — package installation is the role's job, never a manual step
   # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
   - smartmontools
+# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
+# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
+# baseline — not a one-off `apt install` someone has to remember on the next node.
+#
+# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
+# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
+# from its contents, so an unlucky boot reserves everything above the table and the node
+# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
+# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
+# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
+# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
+# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
+# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+#
+# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
+# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
+# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
+# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
+# encode the environment into the ROLE, would silently miss a fourth prod node named
+# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
+# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
+# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
+#
+# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
+# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
+# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
+# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
+#
+# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
+# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
+# delete this block. Nothing else in the repo depends on it.
+node_baseline_backports_kernel: false
+# The suite, mirror and components the source line is built from. Split out so a mirror
+# change is a variable, not an edit to a heredoc in tasks/main.yml.
+node_baseline_backports_suite: trixie-backports
+node_baseline_backports_mirror: "http://deb.debian.org/debian"
+# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
+# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
+# surface beyond what the node already trusts is a separate decision nobody has taken.
+node_baseline_backports_components:
+  - main
+  - non-free-firmware
+# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
+# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
+# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
+# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
+# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
+# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
+# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
+node_baseline_backports_kernel_packages:
+  - linux-image-amd64
+# Provenance/rollback note written on the node. An operator standing in front of a prod box
+# with no repo checkout must be able to answer "why is this machine not on the distro
+# kernel, and how do I undo it?" from the machine itself.
+node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
+
 node_baseline_kernel_modules:
   - overlay
   - br_netfilter
   - iscsi_tcp
   - dm_crypt
   - nfs
+
+# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
+# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
+# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
+#
+#   node_baseline_longhorn_source:
+#     device           dedicated block device at node_baseline_longhorn_device (DEV: the
+#                      tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
+#     claim-mountpoint re-purpose the partition the node ALREADY has mounted at
+#                      node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
+#                      path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
+#                      and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
+#                      is correct and a hardcoded one would eventually mkfs the wrong
+#                      partition on a future node. Destructive; heavily guarded — read
+#                      tasks/disks.yml before changing anything here.
+#
+#   node_baseline_etcd_source:
+#     device                dedicated block device at node_baseline_etcd_device (DEV:
+#                           /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
+#                           isolated from Longhorn rebuild I/O after dev was burned by HDD
+#                           contention).
+#     root-filesystem       etcd lives on the OS filesystem, with NO dedicated device and
+#                           NO claimed data partition under it. Kept as a valid, supported
+#                           shape; NOT what prod uses any more (see below).
+#     shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
+#                           CLAIMED data partition (PROD). No dedicated device, but also NOT
+#                           the OS filesystem: the whole point of the 2026-09-04 layout
+#                           revision is that /var (11.8 GiB) can hold neither the container
+#                           image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
+#
+# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
+# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
+# var, which still fails the `disks` assert.
+node_baseline_longhorn_source: device
 node_baseline_longhorn_device: /dev/vdb
-node_baseline_longhorn_mount: /var/lib/longhorn
+# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
-node_baseline_longhorn_fs_label: longhorn
+# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
+# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
+# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
+# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
+#
+# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
+# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
+# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
+# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
+# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
+node_baseline_data_mount: /var/lib/longhorn
+node_baseline_data_fs_label: longhorn
+# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
+# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
+# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
+# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
+# same fact, and the two are checked against each other by nothing but this comment.
+node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
+# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
+# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
+# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
+# install flags, this role reads the same value to create the directory and to place etcd,
+# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
+# value (higher precedence than any role default) is what actually resolves here, even
+# though roles/k3s_server is not in the node-baseline play; the default() keeps
+# node-baseline.yml runnable standalone.
+node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
+# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
+# default so a dev host cannot enter the claim path even by accident.
+node_baseline_longhorn_claim_mountpoint: ""
+# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
+# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
+# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
+# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
+# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
+# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
+# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
+node_baseline_data_retired_fs_label: ""
+node_baseline_data_retired_mount: ""
+# Sanity floor for the claim path: a data partition smaller than this is not "a small
+# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
+node_baseline_longhorn_min_size_gb: 100
 # Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
 # so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
 # See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
-# device renames (matches the Longhorn LABEL= approach).
+# device renames (matches the data partition's LABEL= approach).
+node_baseline_etcd_source: device
 node_baseline_etcd_device: /dev/vdc
-node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
+# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
+# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
+# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
+# to /data/k3s/server/db, inside the claimed partition.
+node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
 node_baseline_etcd_fs_label: k3s-etcd
+# Mounts whose backing devices the claim path must NEVER touch. Used to build the
+# protected-device set from the node's own gathered facts (plus live swap), so a node
+# partitioned differently from the fleet is judged on its own layout rather than against
+# a hardcoded list of partition names.
+node_baseline_protected_mounts:
+  - /
+  - /boot
+  - /boot/efi
+  - /usr
+  - /var
+  - /home
+  - /tmp
+
+# --- WIRELESS (tasks/main.yml, tag `wireless`) ---------------------------------------
+# A CLUSTER NODE GETS EXACTLY ONE PATH ONTO THE NETWORK. ON by default — unlike the
+# backports kernel above, which is a prod-only deviation, this is a cluster-wide policy and
+# it is a provable NO-OP where it does not apply: the tasks act only on interfaces the node
+# reports as real 802.11 devices (a `phy80211` link in sysfs), and dev's Proxmox VMs have
+# none, so on dev nothing is written, nothing is unloaded and nothing changes. Defaulting it
+# true is what makes a FOURTH node with a wifi card inherit the policy from being a cluster
+# node, rather than from someone remembering to set a flag.
+#
+# WHY IT EXISTS: k3s-prod-3 came up on both eno1 (192.168.2.12, the inventory address) and
+# wlp1s0 (192.168.2.195, DHCP on the same /24). The wifi lease's dhcpcd run rewrote
+# /etc/resolv.conf three minutes after this role wrote it, and a second address on the LAN
+# where kube-vip floats 192.168.2.9 and Cilium L2-announces .20-.30 is a problem in its own
+# right. See the `wireless` block in tasks/main.yml for the full reasoning, including why the
+# DRIVER MODULE is the layer that gets blocked rather than the link, a service, or rfkill.
+#
+# SET IT FALSE for a host that must genuinely keep wifi (there is none today). That does NOT
+# undo an already-applied block — delete the file below on the node as well.
+node_baseline_disable_wireless: true
+# Where the persistent block lives. 99- so it is read last and overrides any earlier
+# permissive rule; the name says what it does, on a node with no repo checkout to hand.
+node_baseline_wireless_modprobe_file: /etc/modprobe.d/99-k3s-no-wireless.conf
 
 # --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
 # These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml

infra/ansible/roles/node_baseline/tasks/disks.yml

@@ -0,0 +1,642 @@
+---
+# =====================================================================================
+# node_baseline — DATA DISKS.  Every task here carries the `disks` tag (applied once by
+# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
+# Split into its own file rather than living in main.yml because it is the one part of
+# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
+#
+# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
+#
+#   dev  (group_vars/all.yml, role defaults)
+#     node_baseline_longhorn_source: device          -> mkfs /dev/vdb, mount LABEL=longhorn
+#                                                       at /var/lib/longhorn (== dataPath)
+#     node_baseline_etcd_source:     device          -> mkfs /dev/vdc, mount LABEL=k3s-etcd
+#     k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
+#     THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
+#     exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
+#
+#   prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
+#     node_baseline_longhorn_source: claim-mountpoint  (claim mountpoint /srv)
+#     node_baseline_data_mount:      /data            (LABEL=k3s-data)
+#     node_baseline_etcd_source:     shared-data-partition
+#     k3s_data_dir:                  /data/k3s
+#     ONE physical disk per node, already fully partitioned by the Debian installer. The
+#     ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
+#     between TWO tenants:  /data/longhorn (Longhorn replicas)  +  /data/k3s (the whole k3s
+#     data dir: etcd AND the containerd image store).
+#
+# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
+# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
+# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
+# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
+# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
+# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
+# partition is the fix; the partition therefore serves two tenants, and neither its
+# mountpoint nor its filesystem label may name only one of them.
+#
+# WHY /data AND LABEL k3s-data.
+#   * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
+#     Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
+#     at, and the idempotency contract is that /srv is gone from /proc/mounts and
+#     /etc/fstab once the claim has happened. Re-using it as the destination would make
+#     "claimed" and "not yet claimed" indistinguishable.
+#   * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
+#     accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
+#     `data` as an example of somebody else's named volume. Labelling ours `data` would make
+#     a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
+#     cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
+#     convention in this repo.
+#
+# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
+# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
+# empty string, and the second one must never silently succeed. `root-filesystem` and
+# `shared-data-partition` are explicit, greppable statements of intent that the assert
+# below accepts; an unset or placeholder device is still a hard failure. The two differ in
+# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
+# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
+# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
+# than printing a claim it never checked (see the etcd section).
+#
+# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
+# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
+# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
+# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
+# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
+# point at the wrong partition on some future node and mkfs it. So the role asks the NODE
+# which block device is mounted at the claim mountpoint, and then proves that answer is
+# safe (see the assert wall below) before touching it. The inventory declares INTENT
+# ("the partition currently mounted at /srv"); the node supplies the FACT.
+# =====================================================================================
+
+# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
+# its data-disk layout. Two failure shapes are caught:
+#   * an undeclared/typo'd source enum, and
+#   * source: device with a device path that is a PLACEHOLDER rather than a real device.
+# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
+# vars now that prod uses neither: if anyone ever flips a source back to `device` without
+# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
+# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
+# stays in the token list because that was the pre-decision placeholder and may still be
+# carried by other environments/branches.
+- name: Assert this environment's data-disk layout has actually been declared
+  ansible.builtin.assert:
+    that:
+      - node_baseline_longhorn_source in ['device', 'claim-mountpoint']
+      - node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
+      # `shared-data-partition` is only coherent when there IS a claimed data partition to
+      # share, and only when the k3s data dir actually sits INSIDE it. Catch the
+      # contradiction here, in inventory terms, rather than three sections later when etcd
+      # has silently landed back on the OS filesystem.
+      - >-
+        node_baseline_etcd_source != 'shared-data-partition'
+        or node_baseline_longhorn_source == 'claim-mountpoint'
+      - >-
+        node_baseline_etcd_source != 'shared-data-partition'
+        or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
+      # The transition knobs are a matched PAIR: a retired label with no retired mountpoint
+      # (or vice versa) cannot describe a state to migrate from.
+      - >-
+        (node_baseline_data_retired_fs_label | length > 0)
+        == (node_baseline_data_retired_mount | length > 0)
+      - >-
+        node_baseline_longhorn_source != 'device'
+        or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
+      - >-
+        node_baseline_etcd_source != 'device'
+        or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
+      - >-
+        node_baseline_longhorn_source != 'claim-mountpoint'
+        or (node_baseline_longhorn_claim_mountpoint | length) > 0
+    fail_msg: >-
+      This host's data-disk layout is not declared.
+      longhorn_source={{ node_baseline_longhorn_source }}
+      (device={{ node_baseline_longhorn_device }},
+      claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
+      etcd_source={{ node_baseline_etcd_source }}
+      (device={{ node_baseline_etcd_device }}).
+      Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
+      `claim-mountpoint` (re-purpose the partition already mounted at
+      node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
+      `device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
+      it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
+      data partition — prod's shape since 2026-09-04, which additionally requires
+      longhorn_source `claim-mountpoint` and a k3s data dir under
+      node_baseline_data_mount={{ node_baseline_data_mount }};
+      k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
+      operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
+      Declare it in the environment's group_vars and record the reasoning in
+      §2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
+      `--tags packages`, which skips every disk task including this assert.
+    quiet: true
+
+# =====================================================================================
+# THE DATA PARTITION  (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
+# =====================================================================================
+
+# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
+# rather than a reformat. Once the claim has happened the partition carries the ext4
+# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
+# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
+# mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
+# "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
+# path means converged, rc != 0 means the claim has not been made. Everything destructive
+# below is gated on rc != 0.
+- name: "Data partition: look for a filesystem already carrying the data-partition label"
+  ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
+  register: node_baseline_lh_labelled
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+# --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
+# WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
+# label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
+# takes over — which then fails closed at "the claim mountpoint must actually be a
+# mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
+# behaviour but it is not convergence, and the prod nodes are in exactly that state today:
+# claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
+# So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
+#
+# WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
+# partition with the new label. It never touches the DATA and never reformats: e2label is a
+# metadata write, exactly as in the claim path below. The common mount task then mounts
+# LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
+# `blkid -L <new label>` succeeds and none of this is reachable.
+#
+# THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
+# volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
+# unmounting it would orphan every one of them:
+#   1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
+#      group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
+#   2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
+#      the same structural gate that keeps dev out of the claim path keeps it out of this one.
+#   3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
+#      replica directories on it fails closed and nothing is unmounted.
+# Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
+- name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
+  ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
+  register: node_baseline_lh_retired_labelled
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when:
+    - node_baseline_longhorn_source == 'claim-mountpoint'
+    - node_baseline_data_retired_fs_label | length > 0
+    - node_baseline_lh_labelled.rc != 0
+
+- name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
+  ansible.builtin.set_fact:
+    # A node claimed under the RETIRED label. Mutually exclusive with the claim below.
+    node_baseline_lh_transition_needed: >-
+      {{ node_baseline_longhorn_source == 'claim-mountpoint'
+         and node_baseline_data_retired_fs_label | length > 0
+         and node_baseline_lh_labelled.rc != 0
+         and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
+    # True only for a claim-mountpoint environment that has NOT been claimed under EITHER
+    # label. A `device` environment (dev) can never enter the claim path at all — that is
+    # the structural guarantee that no generalisation of prod's behaviour can reach the
+    # LIVE dev nodes and their /dev/vdb Longhorn replicas.
+    node_baseline_lh_claim_needed: >-
+      {{ node_baseline_longhorn_source == 'claim-mountpoint'
+         and node_baseline_lh_labelled.rc != 0
+         and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
+
+- name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_data_retired_mount }}
+  register: node_baseline_lh_retired_findmnt
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_transition_needed | bool
+
+# Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
+# not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
+# an unmount of a populated Longhorn store would orphan every replica on the node.
+- name: "Data partition (transition): read the contents of the retired mountpoint"
+  ansible.builtin.find:
+    paths: "{{ node_baseline_data_retired_mount }}"
+    file_type: any
+    hidden: true
+    recurse: false
+    excludes: ['lost+found']
+  register: node_baseline_lh_retired_contents
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): prove this node is safe to re-label"
+  ansible.builtin.assert:
+    that:
+      # 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
+      #    retired label names. The only state this role knows how to transition is the one
+      #    the previous version of it produced; anything else is an operator decision.
+      - (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
+      - >-
+        (node_baseline_lh_retired_findmnt.stdout | default('') | trim)
+        == (node_baseline_lh_retired_labelled.stdout | default('') | trim)
+      # 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
+      #    not the node we think it is, or a tenant has already written data — in both cases
+      #    unmounting and re-labelling under it is destructive and the role must stop.
+      - (node_baseline_lh_retired_contents.files | length) == 0
+    fail_msg: >-
+      REFUSING to transition this node's data partition from
+      LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
+      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
+      device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
+      device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
+      (findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
+      non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
+      {{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
+      If the partition holds DATA, this transition would orphan it — Longhorn replicas are
+      not recoverable from an unmounted, re-labelled filesystem. Migrate the data
+      deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
+      Do NOT relax this assert.
+    quiet: true
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): report exactly what is about to change"
+  ansible.builtin.debug:
+    msg: >-
+      Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
+      from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
+      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
+      unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
+      No filesystem is created and no data is moved.
+  when: node_baseline_lh_transition_needed | bool
+
+# e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
+# breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
+- name: "Data partition (transition): unmount the retired mountpoint"
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_retired_mount }}"
+    state: unmounted
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_retired_mount }}"
+    state: absent_from_fstab
+    backup: true
+  when: node_baseline_lh_transition_needed | bool
+
+- name: "Data partition (transition): re-stamp the partition with the new filesystem label"
+  ansible.builtin.command: >-
+    e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
+    {{ node_baseline_data_fs_label }}
+  changed_when: true
+  when: node_baseline_lh_transition_needed | bool
+
+# --- The claim (prod, first run only) -------------------------------------------------
+# Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
+# mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
+# returning the device behind `/` — which is precisely the mistake that would hand the
+# root partition to e2label.
+- name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
+  register: node_baseline_lh_claim_findmnt
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
+  ansible.builtin.assert:
+    that:
+      - node_baseline_lh_claim_findmnt.rc == 0
+      - (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
+    fail_msg: >-
+      node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
+      yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
+      this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
+      stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
+      The role will NOT go looking for "a big partition" to claim instead. Either the
+      node is partitioned differently from the fleet — in which case fix the node or give
+      this host its own declaration — or the claim already happened and the label was
+      removed by hand.
+    quiet: true
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): inspect the candidate partition"
+  ansible.builtin.command: >-
+    lsblk --json --nodeps --bytes
+    --output PATH,TYPE,FSTYPE,LABEL,SIZE
+    {{ node_baseline_lh_claim_findmnt.stdout | trim }}
+  register: node_baseline_lh_claim_lsblk
+  changed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+# The devices this role must never, under any circumstances, hand to e2label. Built from
+# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
+# NODE's account of what is load-bearing, not a hardcoded list of partition names.
+- name: "Longhorn (claim): list the devices currently backing active swap"
+  ansible.builtin.command: swapon --noheadings --show=NAME
+  register: node_baseline_lh_swapdevs
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_lh_claim_needed | bool
+
+# Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
+# ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
+- name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
+  ansible.builtin.find:
+    paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    file_type: any
+    hidden: true
+    recurse: false
+    excludes: ['lost+found']
+  register: node_baseline_lh_claim_contents
+  when: node_baseline_lh_claim_needed | bool
+
+# THE GUARD WALL. Five independent conditions, every one of which must hold before the
+# role modifies a filesystem it did not create. Any single failure stops the play with
+# the evidence in the message — nothing is unmounted, relabelled or mounted first.
+- name: "Longhorn (claim): prove the candidate partition is safe to claim"
+  ansible.builtin.assert:
+    that:
+      # 1. It is a real block device — a partition or a whole disk. Catches a claim
+      #    mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
+      #    which would have produced a non-/dev SOURCE or an lsblk row of another type.
+      - node_baseline_lh_candidate.type in ['part', 'disk']
+      # 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
+      #    so a differently-partitioned node is caught by its own facts.
+      - node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
+      # 3. It already holds ext4. The claim path RELABELS rather than reformats (see
+      #    below), which is only meaningful for the filesystem type we are about to
+      #    declare in fstab. Anything else is an operator decision, not a role's.
+      - node_baseline_lh_candidate.fstype == 'ext4'
+      # 4. It carries no filesystem label, or already carries ours. A partition labelled
+      #    `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
+      #    it would make it unfindable by its own fstab entry.
+      - (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
+      # 5. It is big enough to be the partition we meant. A data partition of a few GB is
+      #    not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
+      - (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
+      # 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
+      #    ext4 5% root reserve; this is where that premise is verified per node instead
+      #    of trusted. If any node has real data on it, the play stops here — the role
+      #    never wipes data it cannot account for.
+      - (node_baseline_lh_claim_contents.files | length) == 0
+    fail_msg: >-
+      REFUSING to claim {{ node_baseline_lh_candidate.path }}
+      (mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
+      type={{ node_baseline_lh_candidate.type }},
+      fstype={{ node_baseline_lh_candidate.fstype }},
+      label='{{ node_baseline_lh_candidate.label | default('', true) }}',
+      size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
+      (minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
+      non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
+      {{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
+      devices this node needs and that must never be claimed:
+      {{ node_baseline_lh_protected_devices }}.
+      This is the destructive step of the role and it fails CLOSED. If the partition
+      holds data, back it up or pick a different one; do NOT relax this assert.
+    quiet: true
+  vars:
+    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
+    # The node's OWN account of what it cannot afford to lose: the devices behind the
+    # mounts in node_baseline_protected_mounts (from gathered facts, so a node
+    # partitioned differently is judged on its own layout) plus every active swap device.
+    # Built here rather than hardcoded so no partition-name list can go stale.
+    node_baseline_lh_protected_devices: >-
+      {{ (ansible_facts.mounts | default([])
+          | selectattr('mount', 'in', node_baseline_protected_mounts)
+          | map(attribute='device') | list)
+         + (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): report exactly what is about to be claimed"
+  ansible.builtin.debug:
+    msg: >-
+      Claiming {{ node_baseline_lh_candidate.path }}
+      ({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
+      currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
+      as the data partition: unmount -> drop its fstab entry ->
+      relabel LABEL={{ node_baseline_data_fs_label }} ->
+      mount at {{ node_baseline_data_mount }}.
+  vars:
+    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
+  when: node_baseline_lh_claim_needed | bool
+
+# --- Why the role must unmount and relabel, not just point fstab at the device ---------
+# The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
+# /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
+# path does not. The installer-created /srv filesystem has NO label (fstab pins it by
+# UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
+# the unmount first — and the fstab entry must go in the same breath, or the next boot
+# re-mounts the same partition at /srv underneath Longhorn.
+#
+# RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
+# mkfs would buy nothing and would destroy anything the emptiness check got wrong.
+# e2label is a metadata write: if some assert above was too permissive, the worst case is
+# a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
+# cannot do this job: it declines to touch a device that already has a filesystem, so it
+# would silently no-op and then the LABEL= mount would fail with no useful message.)
+- name: "Longhorn (claim): unmount the claim mountpoint"
+  ansible.posix.mount:
+    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    state: unmounted
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
+  ansible.posix.mount:
+    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
+    state: absent_from_fstab
+    backup: true
+  when: node_baseline_lh_claim_needed | bool
+
+- name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
+  ansible.builtin.command: >-
+    e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
+    {{ node_baseline_data_fs_label }}
+  changed_when: true
+  when: node_baseline_lh_claim_needed | bool
+
+# --- The dedicated-device path (dev) --------------------------------------------------
+# Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
+# never reformats a device that already carries a filesystem.
+- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_longhorn_device }}"
+    opts: "-L {{ node_baseline_data_fs_label }}"
+  when: node_baseline_longhorn_source == 'device'
+
+# --- Common to ALL paths (device, claim, transition) ----------------------------------
+# Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
+# mount task. Idempotent: on a converged node this reports ok, which is what makes a
+# re-run a no-op rather than a reformat. It is also what completes the transition above:
+# the retired mountpoint is gone from fstab and the partition now answers to the new label,
+# so this task mounts it at the new path.
+- name: Mount the data partition
+  ansible.posix.mount:
+    path: "{{ node_baseline_data_mount }}"
+    src: "LABEL={{ node_baseline_data_fs_label }}"
+    fstype: ext4
+    opts: "defaults,nofail,x-systemd.device-timeout=5s"
+    state: mounted
+
+# --- The per-tenant subdirectories ---------------------------------------------------
+# WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
+# tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
+# subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
+# /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
+# either there would at best be a no-op and at worst re-mode the running dev cluster's data
+# dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
+# mounted — i.e. only the directories it is actually responsible for. That test is derived
+# from the paths themselves, so no environment has to remember to switch it off.
+#
+# MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
+# the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
+# etcd database, the node token and the containerd image store, and k3s would tighten it to
+# 0700 itself on first start — doing it here means it is never briefly world-readable.
+- name: Create the per-tenant data directories on the data partition
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
+    - { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
+  loop_control:
+    label: "{{ item.path }} ({{ item.tenant }})"
+  when: item.path.startswith(node_baseline_data_mount ~ '/')
+
+# =====================================================================================
+# ETCD DATA DIR
+# =====================================================================================
+
+# --- The dedicated-device path (dev) --------------------------------------------------
+# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
+# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
+# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
+# a disk that already carries a filesystem, so re-running against an already-migrated
+# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
+# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
+# role would mount over a populated directory.
+- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
+  community.general.filesystem:
+    fstype: ext4
+    dev: "{{ node_baseline_etcd_device }}"
+    opts: "-L {{ node_baseline_etcd_fs_label }}"
+  when: node_baseline_etcd_source == 'device'
+
+- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
+  ansible.posix.mount:
+    path: "{{ node_baseline_etcd_mount }}"
+    src: "LABEL={{ node_baseline_etcd_fs_label }}"
+    fstype: ext4
+    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
+    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
+    opts: "defaults,noatime"
+    dump: "0"
+    passno: "2"
+    state: mounted
+  when: node_baseline_etcd_source == 'device'
+
+# --- The no-dedicated-device paths (prod) ---------------------------------------------
+# Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
+# populated — so etcd gets no device of its own. That is an operator decision
+# (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
+# UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
+# on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
+# latency rather than collapsing it. Accepted to be MEASURED, not assumed.
+#
+# WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
+# filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
+# 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
+# 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
+# could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
+# partition and etcd moved with it: `shared-data-partition`. Both enum values are still
+# supported, and the role now PROVES which one is true instead of printing a claim.
+#
+# There is nothing to CREATE here for either shape — the data-partition section above
+# already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
+# role's job is to verify the assumption rather than act on it.
+- name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
+  ansible.builtin.command: >-
+    findmnt --noheadings --first-only --output SOURCE
+    --mountpoint {{ node_baseline_etcd_mount }}
+  register: node_baseline_etcd_existing_mount
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+- name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
+  ansible.builtin.assert:
+    that:
+      - node_baseline_etcd_existing_mount.rc != 0
+    fail_msg: >-
+      node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
+      NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
+      ALREADY a separate mount, backed by
+      {{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
+      contradiction the role will not resolve on its own: either this node was previously
+      built with a dedicated etcd device (in which case unmount it deliberately, after
+      accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
+      the node in a state where the declared layout and the real one disagree.
+    quiet: true
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+# Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
+# WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
+# existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
+# that does exist — the same filesystem the data dir will be created on. TARGET is captured
+# separately from the human-readable line because the assert below judges on it.
+- name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
+  ansible.builtin.shell: |
+    set -eu
+    p={{ node_baseline_etcd_mount | quote }}
+    while [ ! -d "$p" ]; do p=$(dirname "$p"); done
+    findmnt --noheadings --first-only --output TARGET --target "$p"
+    findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
+  register: node_baseline_etcd_backing_fs
+  changed_when: false
+  check_mode: false
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
+
+# THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
+# `shared-data-partition` is a distinct enum value rather than a comment. Declaring
+# k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
+# above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
+# of container images would land back on the OS filesystem — the exact failure this change
+# exists to prevent, and completely invisible until the disk filled. So the role checks that
+# the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
+# run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
+# which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
+# whole record for that shape.
+- name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
+  ansible.builtin.assert:
+    that:
+      - >-
+        (node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
+        == node_baseline_data_mount
+    fail_msg: >-
+      node_baseline_etcd_source is `shared-data-partition`, which asserts that
+      {{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
+      {{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
+      '{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
+      (full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
+      The data partition is therefore NOT mounted where this layout requires, and k3s would
+      put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
+      filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
+      image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
+      this node until the mount is correct.
+    quiet: true
+  when: node_baseline_etcd_source == 'shared-data-partition'
+
+- name: "etcd (no dedicated device): show which filesystem etcd will live on"
+  ansible.builtin.debug:
+    msg: >-
+      etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
+      (source={{ node_baseline_etcd_source }}); it lands on
+      {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
+      (SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
+      measured risk — §2.
+  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1,6 +1,6 @@
 ---
-# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
+# TAGS — every task here carries one of `packages`, `kernel`, `system`, `wireless`, `disks` or
-# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
+# `dns`. An untagged run is unchanged (all six run, in this order). The split exists because the PACKAGE floor
 # and the DATA-DISK work have different readiness: packages are safe on any node from
 # first contact, while the disk tasks need block devices that a new environment may not
 # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -12,6 +12,16 @@
 # mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
 # at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
 #   ansible-playbook playbooks/node-baseline.yml --tags dns
+# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
+# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
+# it must be runnable without touching a disk decision that is still open:
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
+# `wireless` (a cluster node gets exactly ONE path onto the network) is separated for a third
+# reason: it must be runnable BEFORE `dns` and without touching packages, the kernel or a disk
+# decision, because a node with a second DHCP-managed interface reverts the `dns` write. It is
+# placed before the disk work for the same reason, and the pair is the whole remedy for the
+# k3s-prod-3 incident:
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags wireless,dns
 
 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
   ansible.builtin.apt:
@@ -21,6 +31,237 @@
     cache_valid_time: 3600
   tags: [packages]
 
+# =====================================================================================
+# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
+# =====================================================================================
+# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
+# FUTURE prod node gets it from provisioning instead of from someone remembering.
+#
+# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
+# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
+# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
+# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
+# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
+# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
+# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
+# (first released in v6.13) validates the size and rejects the table instead. It is absent
+# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
+# stable feed will ever pick it up. There is NO kernel cmdline workaround —
+# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
+# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
+# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
+#
+# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
+# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
+# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
+# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
+# change for no benefit. See defaults/main.yml for why this is a variable rather than a
+# `k3s-prod-*` hostname test.
+#
+# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
+# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
+# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
+# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
+# before touching the next, never node 2 first — and a role cannot enforce that. The last
+# task here reports the pending reboot loudly instead of letting a green play imply the node
+# is already fixed.
+- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
+  when: node_baseline_backports_kernel | bool
+  tags: [kernel]
+  block:
+
+    # The suite name and the running release must agree, or apt resolves no candidate and
+    # `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
+    # a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
+    # dist-upgraded to forky while group_vars still says trixie-backports).
+    - name: Assert the configured backports suite matches this node's Debian release
+      ansible.builtin.assert:
+        that:
+          - ansible_facts['distribution'] == 'Debian'
+          - node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
+        fail_msg: >-
+          node_baseline_backports_kernel is enabled but node_baseline_backports_suite
+          ({{ node_baseline_backports_suite }}) is not the backports suite for this node's
+          release ({{ ansible_facts['distribution'] }}
+          {{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
+          candidate and this play would report success while leaving the node on the
+          vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
+        quiet: true
+
+    # deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
+    # prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
+    # rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
+    # idempotent in a way `apt_repository` — which rewrites and reformats — is not.
+    #
+    # THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
+    # backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
+    # apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
+    # 500 for a package that was explicitly installed from there so it keeps getting updates.
+    # Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
+    # HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
+    # be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
+    - name: Enable the Debian backports apt source (deb822)
+      ansible.builtin.copy:
+        dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
+          # A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
+          # machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
+          # Table its firmware publishes (upstream 8fbe4c49c0cc). See
+          # infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
+          # on this node.
+          #
+          # Backports is NotAutomatic, so nothing is taken from here unless asked for by name
+          # with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
+          # ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
+          # Pin-Priority would switch that off.
+          Types: deb
+          URIs: {{ node_baseline_backports_mirror }}
+          Suites: {{ node_baseline_backports_suite }}
+          Components: {{ node_baseline_backports_components | join(' ') }}
+          Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
+      register: node_baseline_backports_source
+
+    - name: Refresh the apt cache after enabling backports
+      ansible.builtin.apt:
+        update_cache: true
+      when: node_baseline_backports_source is changed
+
+    # `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
+    # nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
+    # nothing at all — the single most likely way to write this task and have it silently fix
+    # nothing forever. `latest` + `default_release` resolves the candidate inside backports
+    # and upgrades the metapackage to it. Once the node is current a re-run is a genuine
+    # no-op: no change, no churn, no spurious reboot flag. Scope is exactly
+    # node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
+    # is dragged out of the main archive.
+    - name: Install the fixed kernel from backports (>= 6.13)
+      ansible.builtin.apt:
+        name: "{{ node_baseline_backports_kernel_packages }}"
+        state: latest
+        default_release: "{{ node_baseline_backports_suite }}"
+        update_cache: true
+        cache_valid_time: 3600
+      register: node_baseline_backports_kernel_apt
+
+    # WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
+    # metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
+    # answer anybody can roll back from six months later.
+    - name: Read back the installed backports kernel metapackage version
+      ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
+      register: node_baseline_backports_kernel_version
+      changed_when: false
+      check_mode: false
+
+    # `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
+    # neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
+    # list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
+    # GRUB's "Advanced options" without uninstalling anything.
+    - name: List every kernel image installed on this node, newest first
+      ansible.builtin.shell:
+        cmd: set -o pipefail; linux-version list | linux-version sort --reverse
+        executable: /bin/bash
+      register: node_baseline_installed_kernels
+      changed_when: false
+      check_mode: false
+
+    # The node must be able to explain itself to an operator standing in front of it with no
+    # repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
+    # role is a real no-op rather than a permanent daily "changed".
+    - name: Record the deviation, the exact version, and the rollback path on the node
+      ansible.builtin.copy:
+        dest: "{{ node_baseline_backports_kernel_stamp }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
+          #
+          # THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
+          #
+          # WHY
+          #   This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
+          #   Linux < 6.13 reserves memory from it without validating its size, and an unlucky
+          #   boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
+          #   k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
+          #   is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
+          #   v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
+          #   Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
+          #
+          # PROOF THE FIX IS ACTIVE (not merely a lucky boot)
+          #   BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
+          #   >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
+          #     zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
+          #   (roles/node_health does the version check for you; see the health report's
+          #   firmware.efi_memattr finding.)
+          #
+          #   FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
+          #   and each prints a DIFFERENT line — all three are rejections:
+          #     tbl->version > 2         'Unexpected EFI Memory Attributes table version %d'
+          #                              (NOTE: printed with %d, so garbage may read NEGATIVE)
+          #     bad desc_size            'Unexpected EFI Memory Attributes descriptor size ...'
+          #     num_entries > 64K        '[Firmware Bug]: Corrupted EFI Memory Attributes
+          #                               Table detected! (version == ..., num_entries == ...)'
+          #   Which one appears depends on WHICH field this boot's garbage corrupted, so do
+          #   NOT treat the absence of the FW_BUG line as the fix being absent — all three
+          #   prod nodes printed the *version* line on 2026-09-04.
+          #   The one line that means the fix is NOT active:
+          #     'Failed to map EFI Memory Attributes table @ 0x...'  — the kernel trusted the
+          #     corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
+          #     escalate, do not shrug.
+          #
+          # INSTALLED
+          #   source   /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
+          #   package  {{ node_baseline_backports_kernel_packages | join(' ') }}
+          #   version  {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
+          #   kernels installed on this node, newest first:
+          {% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
+          #     {{ kv }}
+          {% endfor %}
+          #
+          # SECURITY COST
+          #   Packages from *-backports are provided AS IS. They are NOT covered by the Debian
+          #   Security Team's guarantees for the main archive: no security-tracker commitment,
+          #   no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
+          #
+          # ROLLBACK — no repo checkout needed
+          #   1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
+          #      and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
+          #      GNU/Linux". That answers "is the new kernel the problem?" without
+          #      uninstalling anything. Note it reinstates the memattr bug for that boot.
+          #   2. PERMANENT. Set node_baseline_backports_kernel: false in
+          #      inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
+          #      puts the backports kernel straight back — then, on this node:
+          #        apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
+          #        apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
+          #        rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
+          #        apt-get update && update-grub && reboot
+          #
+          # EXIT CONDITION
+          #   If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
+          #   rebase), this deviation can be removed entirely: flip the variable to false,
+          #   roll back as above, and delete the `kernel` block from roles/node_baseline.
+
+    # Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
+    # it means the fixed kernel is on disk. See the block header for why the role will not
+    # reboot on its own.
+    - name: Report whether this node is running the kernel it now has installed
+      ansible.builtin.debug:
+        msg: >-
+          {{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
+              ~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
+              prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
+              node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
+              with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
+             if newest != ansible_facts['kernel']
+             else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
+                   ~ '); no reboot pending for the memattr fix.') }}
+      vars:
+        newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
+
 - name: Persist kernel modules for k3s/Cilium/Longhorn
   ansible.builtin.copy:
     dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
@@ -79,70 +320,330 @@
     state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
   tags: [system]
 
-# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
+# =====================================================================================
-# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
+# WIRELESS — tag `wireless`: a cluster node gets exactly ONE path onto the network
-# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
+# =====================================================================================
-# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
+# WHAT WENT WRONG (k3s-prod-3, reported by the operator 2026-09-04). The node was up on BOTH
-# which do not exist on bare metal either — a confusing failure that looks like a wiring
+# eno1 (192.168.2.12 — the inventory address) and wlp1s0 (192.168.2.195, DHCP, the SAME /24).
-# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
+# The wifi lease carried a dhcpcd run with it, and dhcpcd's `20-resolv.conf` hook REWROTE
-# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
+# /etc/resolv.conf, undoing the resolvers this role had just written: the file read
-- name: Assert this environment's data-disk devices have actually been decided
+# "# Generated by dhcpcd from wlp1s0.dhcp, wlp1s0.dhcp6, wlp1s0.ra" with nameserver
+# 192.168.2.1 plus two Google IPv6 servers, timestamped THREE MINUTES AFTER the role's own
+# write (01:32 -> 01:35). The role had reported `changed` and green. That is exactly the
+# "successful run that fixed nothing" the `contended` branch of the DNS block below exists to
+# prevent, and the detection blind spot that let it through is closed there.
+#
+# WHY THE INTERFACE IS REMOVED RATHER THAN JUST REPORTED. Closing the detection hole alone
+# would have turned a silent revert into a loud failure — an improvement, but it leaves the
+# node with a second, DHCP-managed path onto the LAN, and on a k3s server that is three
+# distinct problems beyond DNS:
+#   * a SECOND DEFAULT ROUTE, whose metric decides which source address the node uses;
+#   * a SECOND ADDRESS ON THE SAME L2 — the segment where kube-vip floats 192.168.2.9 and
+#     Cilium L2-announces 192.168.2.20-.30 (§4). One host answering ARP on one subnet from
+#     two NICs is the exact shape neither mechanism can reason about;
+#   * an UNMANAGED address: nothing in this repo knows about .195, and the DHCP server is
+#     free to hand out a different one after any lease expiry.
+# None of that is a resolver bug, and none of it is fixed by writing resolv.conf harder.
+#
+# WHY THE DRIVER LAYER — the four alternatives, and why each is weaker. What has to hold is:
+# survives a REBOOT, survives a LEASE RENEWAL, and cannot be undone by a routine `ifup`.
+# Measured on the nodes themselves — all three are HP EliteDesk 800 G5 minis, eno1 is
+# `e1000e`, the wifi NIC is an Intel AX200 on `iwlwifi`, and the OS is Debian 13 trixie with
+# ifupdown + `networking.service`, NO NetworkManager, NO systemd-networkd, NO systemd-resolved:
+#   `ip link set <wifi> down` — NOT persistent. Any `ifup`, any hotplug event, any reboot
+#       undoes it. It is what had already been done by hand on node 3, and the stale dhcpcd
+#       /etc/resolv.conf outlived it — the link was down and the wrong resolvers were still
+#       in the file, across a reboot.
+#   mask the service that manages it — THERE IS NO SUCH SERVICE HERE. dhcpcd on these nodes
+#       is `dhcpcd-base`: no dhcpcd.service exists at all (`systemctl is-enabled dhcpcd` =
+#       not-found), it is spawned per-interface by ifupdown. The only unit that can bring
+#       wireless up is `networking.service` — which ALSO owns eno1's static address, so
+#       masking it would cut the one SSH path to a machine whose operator is not on site.
+#   `rfkill` soft-block — weaker on every axis. The netdev still EXISTS, so ifupdown and
+#       dhcpcd can still be pointed at it; persistence depends on systemd-rfkill restoring
+#       saved state (the kernel's own `persistent` flag reads 0 on these devices, and the
+#       `rfkill` CLI is not even installed); and one `rfkill unblock wlan` undoes it.
+#   the BIOS `M.2 WLAN/BT` toggle — genuinely the deepest layer, and it reads `Enable` on all
+#       three nodes (hp-bioscfg survey, §1b). NOT taken here: it needs a REBOOT to take
+#       effect, and a prod reboot is a serialized, supervised operator step (§0b "SECOND
+#       FAULT" — k3s-prod-2 hung on a warm reboot and needed the power button). It also
+#       disables Bluetooth on the same M.2 card. Left as an operator recommendation in §4.
+#   THE DRIVER MODULE — chosen. `blacklist` plus `install <module> /bin/false` in
+#       /etc/modprobe.d means kmod refuses to load the wifi driver on this boot and every
+#       future one, so THE NETDEV NEVER EXISTS. No interface means no DHCP lease, no
+#       resolv.conf hook, no second route, no second address — and every re-activation path
+#       above (udev hotplug, a hand-run `ifup`, wpa_supplicant, a hand-run dhcpcd) needs a
+#       device that is not there. `blacklist` alone would still allow a dependency- or
+#       hand-triggered `modprobe`; the `install ... /bin/false` line is what makes it refuse
+#       outright. It is ONE declarative file, readable on the node, and undone by deleting
+#       it. Paired with the runtime unload below, it needs NO reboot to take effect.
+#
+# SAFE AND IDEMPOTENT ON ALL THREE PROD NODES *AND* ON THE LIVE DEV CLUSTER. Nothing here is
+# hostname- or hardware-specific: the block acts only on interfaces the node itself reports as
+# real 802.11 devices (a `phy80211` link in sysfs) and on the driver modules behind them.
+#   * dev (Proxmox VMs, virtio NICs) has no such interface, so the discovery comes back empty,
+#     every task below is skipped, and NOT EVEN THE modprobe.d FILE IS WRITTEN. The block
+#     cannot touch dev's networking, which is why it is safe to have on by default.
+#   * k3s-prod-1 (wlp1s0) and k3s-prod-2 (wlp2s0) already have the link DOWN, so the link-down
+#     task is skipped by its own `when` and only the modprobe.d file and the module unload
+#     remain — and both are no-ops on a re-run.
+#   * k3s-prod-3 (wlp1s0) is the reported node.
+# A re-run changes nothing on any of them: the file is content-identical, `ip link set down`
+# is gated on the link not already being down, and the unload is gated on the module still
+# being loaded.
+#
+# AND IT REFUSES TO CUT ITS OWN PATH IN. The assert below stops the play if the address
+# Ansible is connected to is bound to a wireless interface, or if the default route leaves
+# over one, or if the wireless driver module is also the driver of the default-route
+# interface. That is the one case where doing this would lock everybody out of a machine
+# nobody can walk up to, so it is checked per node, on every run, from that node's own facts.
+- name: Survey this node's 802.11 devices, their drivers and any live DHCP client
+  ansible.builtin.shell:
+    executable: /bin/sh
+    cmd: |
+      set -u
+      # Flat KEY|value lines so the Ansible side needs one single-group regex per fact and
+      # no parsing cleverness. A wireless interface is one with a phy80211 link in sysfs —
+      # that is the kernel's own answer to "is this 802.11", not a name pattern like wlp*.
+      wifi=''
+      for d in /sys/class/net/*; do
+        [ -e "$d/phy80211" ] || continue
+        i=$(basename "$d")
+        wifi="$wifi $i"
+        m=$(basename "$(readlink -f "$d/device/driver" 2>/dev/null || echo unknown)")
+        state=$(cat "$d/operstate" 2>/dev/null || echo unknown)
+        printf 'IFACE|%s\n' "$i"
+        printf 'DETAIL|%s driver=%s operstate=%s\n' "$i" "$m" "$state"
+        # LIVE means "not already down" — the link-down task keys off this, so a node whose
+        # radio is already down reports no change.
+        [ "$state" = down ] || printf 'IFACE_LIVE|%s\n' "$i"
+        ip -o addr show dev "$i" 2>/dev/null | awk '{print "IFACE_ADDR|" $4}'
+        [ "$m" = unknown ] || printf 'MODULE|%s\n' "$m"
+      done
+      for m in $(printf '%s\n' $wifi | while read -r i; do
+            [ -n "$i" ] && basename "$(readlink -f "/sys/class/net/$i/device/driver" 2>/dev/null || echo unknown)"
+          done | sort -u); do
+        [ "$m" = unknown ] && continue
+        [ -d "/sys/module/$m" ] || continue
+        printf 'MODULE_LOADED|%s\n' "$m"
+        # Holders must be unloaded FIRST: `modprobe -r iwlwifi` fails while iwlmvm still
+        # holds it. Read from sysfs rather than assumed, so a different wifi driver stack
+        # (ath*, rtw*, mt76*) works without editing this role.
+        for h in $(ls "/sys/module/$m/holders" 2>/dev/null); do printf 'MODULE_HOLDER|%s\n' "$h"; done
+      done
+      dr=$(ip -o route show default 2>/dev/null | awk '{for (i = 1; i < NF; i++) if ($i == "dev") print $(i + 1)}' | head -1)
+      printf 'DEFROUTE_IFACE|%s\n' "${dr:-none}"
+      if [ -n "${dr:-}" ] && [ -e "/sys/class/net/$dr" ]; then
+        printf 'DEFROUTE_MODULE|%s\n' "$(basename "$(readlink -f "/sys/class/net/$dr/device/driver" 2>/dev/null || echo unknown)")"
+      fi
+      # Any DHCP client process, with the interfaces it was started for. Used only to stop a
+      # client that holds a WIRELESS interface and nothing else.
+      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
+        printf 'DHCPCLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
+      done
+      exit 0
+  register: node_baseline_wifi_scan
+  changed_when: false
+  check_mode: false
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Interpret the wireless survey
+  ansible.builtin.set_fact:
+    node_baseline_wifi_ifaces: "{{ scan | regex_findall('(?m)^IFACE\\|(.+)$') }}"
+    node_baseline_wifi_live: "{{ scan | regex_findall('(?m)^IFACE_LIVE\\|(.+)$') }}"
+    node_baseline_wifi_addrs: "{{ scan | regex_findall('(?m)^IFACE_ADDR\\|(\\S+)') | map('regex_replace', '/.*$', '') | list }}"
+    node_baseline_wifi_modules: "{{ scan | regex_findall('(?m)^MODULE\\|(.+)$') | unique }}"
+    node_baseline_wifi_modules_loaded: "{{ scan | regex_findall('(?m)^MODULE_LOADED\\|(.+)$') | unique }}"
+    node_baseline_wifi_module_holders: "{{ scan | regex_findall('(?m)^MODULE_HOLDER\\|(.+)$') | unique }}"
+    node_baseline_wifi_detail: "{{ scan | regex_findall('(?m)^DETAIL\\|(.+)$') }}"
+    node_baseline_wifi_dhcp_clients: "{{ scan | regex_findall('(?m)^DHCPCLIENT\\|(.+)$') }}"
+    node_baseline_defroute_iface: "{{ (scan | regex_findall('(?m)^DEFROUTE_IFACE\\|(.+)$') | first) | default('none', true) }}"
+    node_baseline_defroute_module: "{{ (scan | regex_findall('(?m)^DEFROUTE_MODULE\\|(.+)$') | first) | default('unknown', true) }}"
+  vars:
+    scan: "{{ node_baseline_wifi_scan.stdout | default('') }}"
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Report this node's wireless hardware
+  ansible.builtin.debug:
+    msg: >-
+      wireless devices = {{ node_baseline_wifi_detail | default([]) | join('; ') | default('(none)', true) }}
+      | driver modules = {{ node_baseline_wifi_modules | default([]) | join(', ') | default('(none)', true) }}
+      (loaded: {{ node_baseline_wifi_modules_loaded | default([]) | join(', ') | default('(none)', true) }},
+      held by: {{ node_baseline_wifi_module_holders | default([]) | join(', ') | default('(none)', true) }})
+      | default route via {{ node_baseline_defroute_iface | default('?') }}
+      (driver {{ node_baseline_defroute_module | default('?') }})
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+# The one way this block could do real harm is by removing the interface it is being run
+# over. Checked from the node's own facts, per node, every run — not assumed from the fact
+# that prod "should" be wired.
+- name: Refuse to disable wireless if the path to this node runs OVER the wireless interface
   ansible.builtin.assert:
     that:
-      - "'BLOCKED' not in node_baseline_longhorn_device"
+      - node_baseline_defroute_iface not in node_baseline_wifi_ifaces
-      - "'BLOCKED' not in node_baseline_etcd_device"
+      - node_baseline_defroute_module not in node_baseline_wifi_modules
+      - (ansible_host | default(inventory_hostname)) not in node_baseline_wifi_addrs
     fail_msg: >-
-      node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
+      REFUSING to disable wireless on this node: the connection to it appears to BE the
-      placeholders for this host's environment
+      wireless path. ansible_host={{ ansible_host | default(inventory_hostname) }}, wireless
-      (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
+      interfaces={{ node_baseline_wifi_ifaces | join(', ') }} carrying
-      The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
+      {{ node_baseline_wifi_addrs | join(', ') | default('no addresses', true) }}, default route
-      and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
+      via {{ node_baseline_defroute_iface }} (driver {{ node_baseline_defroute_module }}).
-      the SMART read that decision waits on can be taken) re-run this play with
+      Disabling the driver here would drop this node off the network with no remote way back.
-      `--tags packages`, which skips every disk task including this assert.
+      Give the node a wired address, point the inventory at it, and re-run.
     quiet: true
-  tags: [disks]
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_ifaces | length > 0
+  tags: [wireless]
 
-- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
+# THE PERSISTENT HALF. This file is the whole reboot story: kmod consults /etc/modprobe.d on
-  community.general.filesystem:
+# every load attempt, so from here on the driver cannot be loaded by udev at boot, as a
-    fstype: ext4
+# dependency, or by hand. Written ONLY when the node actually has a wireless driver, so a
-    dev: "{{ node_baseline_longhorn_device }}"
+# node without wifi hardware (every dev VM) gets no file and no change at all.
-    opts: "-L {{ node_baseline_longhorn_fs_label }}"
+# `update-initramfs` is deliberately NOT run: no wifi module is loaded from the initramfs on
-  tags: [disks]
+# these nodes (root is a local SATA/NVMe device), the driver is autoloaded by udev after
+# switch-root, and re-generating an initramfs on a prod node is a bigger risk than the one it
+# would cover.
+- name: Block the wireless driver module(s) from ever loading again
+  ansible.builtin.copy:
+    dest: "{{ node_baseline_wireless_modprobe_file }}"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # Managed by the Ansible node_baseline role (tags: wireless) — do not edit by hand.
+      #
+      # WHY THIS NODE HAS NO WIRELESS. A k3s server gets exactly ONE path onto the network.
+      # A second, DHCP-managed interface on the same LAN gives the node a second default
+      # route, a second address nothing in the cluster's config knows about, and a DHCP
+      # client whose resolv.conf hook silently overwrites the resolvers Ansible manages
+      # (that is what happened on k3s-prod-3, 2026-09-04). The link is also brought down and
+      # the module unloaded at the same time, so no reboot is needed for this to take effect.
+      #
+      # `blacklist` stops udev's alias-driven autoload; the `install ... /bin/false` line is
+      # what also refuses a dependency-triggered or hand-run `modprobe`. Both are needed.
+      #
+      # TO RE-ENABLE WIRELESS ON THIS NODE: delete this file and `modprobe` the module(s)
+      # below (or reboot). To stop Ansible putting it back, set
+      # node_baseline_disable_wireless: false for this host first — otherwise the next
+      # node-baseline run recreates it. Full rationale: roles/node_baseline/tasks/main.yml,
+      # the `wireless` block, and infra/PROD-PROVISIONING.md §4.
+      {% for module in node_baseline_wifi_modules %}
+      blacklist {{ module }}
+      install {{ module }} /bin/false
+      {% endfor %}
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_modules | length > 0
+  tags: [wireless]
 
-- name: Mount the Longhorn data disk
+# THE RUNTIME HALF — everything below makes the change true NOW, so the node does not have to
-  ansible.posix.mount:
+# be rebooted to stop carrying a second address (this role never reboots: §0b "SECOND FAULT").
-    path: "{{ node_baseline_longhorn_mount }}"
+#
-    src: "LABEL={{ node_baseline_longhorn_fs_label }}"
+# Stop a DHCP client that is holding a wireless interface first, so it releases its lease
-    fstype: ext4
+# instead of being left holding a device that is about to disappear. HEAVILY GUARDED: it runs
-    opts: "defaults,nofail,x-systemd.device-timeout=5s"
+# only for clients whose command line names a wireless interface and names NO non-wireless
-    state: mounted
+# interface, because `dhcpcd -x` against a shared master process would also drop a wired
-  tags: [disks]
+# lease. On these prod nodes eno1 is a static ifupdown address with no DHCP client at all, so
+# this task has nothing to do; it exists so the ordering is correct on a node where wifi is
+# still actively leasing.
+- name: Stop the dhcpcd instance that holds a wireless interface (and nothing else)
+  ansible.builtin.command:
+    argv: [dhcpcd, --exit, "{{ item }}"]
+  loop: "{{ node_baseline_wifi_ifaces }}"
+  register: node_baseline_wifi_dhcpcd_exit
+  changed_when: rc == 0
+  failed_when: false
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_dhcp_wifi_only | length > 0
+    - node_baseline_wifi_dhcp_wifi_only | select('search', 'dhcpcd') | list | length > 0
+  vars:
+    # A client is "wireless-only" when its argv mentions a wireless interface and mentions no
+    # other interface this node has. Empty interface sets are turned into a pattern that
+    # cannot match, so an odd node fails towards doing nothing rather than towards guessing.
+    node_baseline_non_wifi_ifaces: "{{ ansible_facts['interfaces'] | default([]) | difference(node_baseline_wifi_ifaces) | difference(['lo']) }}"
+    node_baseline_wifi_dhcp_wifi_only: >-
+      {{ node_baseline_wifi_dhcp_clients
+         | select('search', '\b(' ~ ((node_baseline_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
+         | reject('search', '\b(' ~ ((node_baseline_non_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
+         | list }}
+  tags: [wireless]
 
-# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
+- name: Take each wireless link down now
-# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
+  ansible.builtin.command:
-# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
+    argv: [ip, link, set, dev, "{{ item }}", down]
-# a disk that already carries a filesystem, so re-running against an already-migrated
+  loop: "{{ node_baseline_wifi_live }}"
-# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
+  changed_when: true
-# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
+  when: node_baseline_disable_wireless | bool
-# role would mount over a populated directory.
+  tags: [wireless]
-- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
-  community.general.filesystem:
-    fstype: ext4
-    dev: "{{ node_baseline_etcd_device }}"
-    opts: "-L {{ node_baseline_etcd_fs_label }}"
-  tags: [disks]
 
-- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
+# Unload the driver so the netdev is gone immediately — holders (iwlmvm on these nodes) first,
-  ansible.posix.mount:
+# then the driver itself, because `modprobe -r` refuses a module that is still held.
-    path: "{{ node_baseline_etcd_mount }}"
+# NOT fatal on failure: the persistent block above is already in place, so a module that
-    src: "LABEL={{ node_baseline_etcd_fs_label }}"
+# cannot be unloaded right now is gone at the next boot regardless. The report below says so
-    fstype: ext4
+# out loud rather than letting a green play imply the radio is already off.
-    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
+- name: Unload the wireless driver module(s) now (no reboot needed)
-    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
+  community.general.modprobe:
-    opts: "defaults,noatime"
+    name: "{{ item }}"
-    dump: "0"
+    state: absent
-    passno: "2"
+  loop: "{{ node_baseline_wifi_module_holders + node_baseline_wifi_modules_loaded }}"
-    state: mounted
+  register: node_baseline_wifi_unload
+  failed_when: false
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Report the wireless state this node is left in
+  ansible.builtin.debug:
+    msg: >-
+      {{ 'No 802.11 device on this node — nothing to disable (no modprobe.d file written).'
+         if (node_baseline_wifi_ifaces | length) == 0
+         else ((('WIRELESS DISABLED: ' ~ (node_baseline_wifi_modules | join(', ')) ~ ' blocked in '
+                 ~ node_baseline_wireless_modprobe_file ~ ' (survives reboot) and unloaded now; '
+                 ~ (node_baseline_wifi_ifaces | join(', ')) ~ ' is down and the netdev is gone.')
+                if (failed_unloads | length) == 0
+                else ('WIRELESS BLOCKED FOR THE NEXT BOOT BUT STILL LOADED NOW: could not unload '
+                      ~ (failed_unloads | join(', ')) ~ '. ' ~ node_baseline_wireless_modprobe_file
+                      ~ ' is in place, so the driver will not load again after a reboot, but the '
+                      ~ 'radio is live until then — check `ip link` and whether something still '
+                      ~ 'holds the device.'))) }}
+  vars:
+    failed_unloads: >-
+      {{ (node_baseline_wifi_unload.results | default([]))
+         | selectattr('failed', 'defined') | selectattr('failed') | map(attribute='item') | list }}
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+# =====================================================================================
+# DATA DISKS — tag `disks`, in tasks/disks.yml
+# =====================================================================================
+# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
+# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
+# should be readable end-to-end on its own; and both environments' storage shapes — dev's
+# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
+# the OS filesystem — are declared there as an explicit enum rather than inferred from
+# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
+#
+# The `disks` tag is applied ONCE here, in BOTH places it is needed:
+#   * `tags:` on the include itself, so `--tags packages` never even reaches the include
+#     and a placeholder device path is not so much as templated; and
+#   * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
+#     optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
+#     its own tags to its children the way `import_tasks` does, so without `apply` the
+#     included tasks are untagged and `--tags disks` selects the include and then runs
+#     NOTHING inside it. That failure is silent and reports success, which is the worst
+#     shape a guard can have.
+# A FULL run against a node whose layout is undeclared stops on the assert at the top of
+# that file instead of failing obscurely at mkfs time.
+- name: Provision the data disks (Longhorn store, etcd data dir)
+  ansible.builtin.include_tasks:
+    file: disks.yml
+    apply:
+      tags: [disks]
   tags: [disks]
 
 - name: Disable swap now
@@ -180,6 +681,11 @@
 # node and the matching seam is written. Adding a third mechanism means adding a branch
 # here, not rewriting the role.
 #
+# AND DETECTION IS ONLY AS GOOD AS THE SET OF OWNERS IT KNOWS ABOUT. It originally looked for
+# resolvconf/openresolv and NetworkManager, missed dhcpcd, and therefore reported a green,
+# `changed` run on k3s-prod-3 whose resolvers a wifi lease reverted three minutes later. dhcpcd
+# is now part of that set — see the detection task below and the `contended` failure.
+#
 # All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
 # documented at the top of this file) still installs nothing but packages.
 
@@ -199,6 +705,55 @@
   check_mode: false
   tags: [dns]
 
+# THE dhcpcd BLIND SPOT, CLOSED (k3s-prod-3, 2026-09-04). dhcpcd's `20-resolv.conf` hook
+# rewrites /etc/resolv.conf on every lease, renewal and carrier change, and it is a THIRD
+# way to own that file — neither resolvconf nor NetworkManager. On node 3 it silently reverted
+# this role's write three minutes later, and because nothing below looked for it the role had
+# already reported `changed` and green: the precise failure the `contended` branch exists to
+# make impossible.
+#
+# WHY THIS LOOKS FOR A LIVE CLIENT AND NOT FOR AN INSTALLED PACKAGE. `dhcpcd-base` is
+# installed on all three prod nodes and always will be — it is the DHCP client ifupdown
+# reaches for — and there is NO dhcpcd.service on them at all (`systemctl is-active dhcpcd` =
+# not-found; it is spawned per-interface by ifupdown). Keying off the package, or off a unit
+# that does not exist, would put every prod node permanently in `contended` and leave the DNS
+# block unable to configure a node it is perfectly safe to configure. What actually matters is
+# whether a DHCP client CAN still write the file, so all three real signals are checked:
+#   CLIENT| a dhcpcd/dhclient process running right now (with the argv that started it);
+#   UNIT|   a dhcpcd/dhclient unit that IS active or IS enabled, on a distro that has one;
+#   STANZA| an UNCOMMENTED `iface <x> inet[6] dhcp` in /etc/network/interfaces{,.d} — the
+#           configuration that makes ifupdown spawn a client at the next boot or `ifup`, even
+#           when none is running at this instant.
+# Any one of them means "another manager is in play" and the play stops below. None of them
+# means no client can run, and the file is genuinely ours to own.
+- name: Determine whether a DHCP client (dhcpcd/dhclient) can rewrite /etc/resolv.conf
+  ansible.builtin.shell:
+    executable: /bin/sh
+    cmd: |
+      set -u
+      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
+        printf 'CLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
+      done
+      for u in dhcpcd dhcpcd5 dhclient isc-dhcp-client; do
+        a=$(systemctl is-active "$u" 2>/dev/null || true)
+        e=$(systemctl is-enabled "$u" 2>/dev/null || true)
+        # Compared exactly, never with a substring match: `is-active` answers "inactive",
+        # which contains the word "active".
+        if [ "$a" = active ] || [ "$e" = enabled ] || [ "$e" = enabled-runtime ]; then
+          printf 'UNIT|%s active=%s enabled=%s\n' "$u" "$a" "$e"
+        fi
+      done
+      # `^[[:space:]]*iface` cannot match a commented-out stanza, which is how the wireless
+      # stanza is left on these nodes.
+      grep -rhE '^[[:space:]]*iface[[:space:]]+[^[:space:]]+[[:space:]]+inet6?[[:space:]]+dhcp' \
+        /etc/network/interfaces /etc/network/interfaces.d/ 2>/dev/null \
+        | sed -e 's/^[[:space:]]*/STANZA|/'
+      exit 0
+  register: node_baseline_dhcp_client_scan
+  changed_when: false
+  check_mode: false
+  tags: [dns]
+
 # follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
 # points, which is the strongest single signal of who owns the file.
 - name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
@@ -229,9 +784,10 @@
 #       resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
 #       then the ONLY thing that changes real behaviour; writing resolv.conf would be
 #       overwritten by resolved (or, worse, land in /run and vanish at reboot).
-#   contended — nobody's stub, but resolvconf or NetworkManager is present/running and
+#   contended — nobody's stub, but resolvconf, NetworkManager or a DHCP client (dhcpcd /
-#       would regenerate the file underneath us. We do NOT write. The role fails loudly
+#       dhclient) is present/running/configured and would regenerate the file underneath us.
-#       below rather than leaving an operator believing a change stuck when it did not.
+#       We do NOT write. The role fails loudly below rather than leaving an operator
+#       believing a change stuck when it did not.
 #   resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
 #       Safe to own outright.
 # resolved running but NOT owning resolv.conf correctly falls through to the file branch:
@@ -241,13 +797,21 @@
     node_baseline_dns_mechanism: >-
       {{
         'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
-        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
+        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present
+                              or node_baseline_dhcp_client_live)
               else 'resolv-conf-file')
       }}
+    # Kept as facts (not just task vars) so the failure message, the report and the
+    # reclaim note below all quote the SAME evidence this decision was made on.
+    node_baseline_dhcp_client_evidence: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) | regex_findall('(?m)^(?:CLIENT|UNIT|STANZA)\\|(.+)$') }}"
+    # A file dhcpcd WROTE, versus a dhcpcd that can still write it — two different facts, and
+    # only the second one is contention. See the reclaim note below.
+    node_baseline_resolv_dhcpcd_written: "{{ 'Generated by dhcpcd' in (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
   vars:
     node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
     node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
     node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
+    node_baseline_dhcp_client_live: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) is search('(?m)^(?:CLIENT|UNIT|STANZA)\\|') }}"
     node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
     node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
     node_baseline_resolved_owns_resolv: >-
@@ -264,7 +828,9 @@
       NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
       /etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
       target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
-      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
+      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }},
+      DHCP client evidence={{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }},
+      resolv.conf written by dhcpcd={{ node_baseline_resolv_dhcpcd_written | default(false) }})
   tags: [dns]
 
 # Same shape, and the same reason, as the data-disk assert above: this environment's
@@ -297,16 +863,45 @@
 - name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
   ansible.builtin.fail:
     msg: >-
-      /etc/resolv.conf on this node is not the systemd-resolved stub, but
+      /etc/resolv.conf on this node is not the systemd-resolved stub, but something else would
-      resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
+      regenerate the file underneath this role: resolvconf/openresolv
-      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
+      (/etc/resolvconf/resolv.conf.d present:
-      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
+      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}), NetworkManager
-      would regenerate the file underneath this role. Writing it here would look like it
+      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}), or a DHCP client
-      worked and silently revert. Add the matching branch to roles/node_baseline (the
+      that can still write it — dhcpcd/dhclient evidence:
-      `dns`-tagged block) for whichever manager this node really uses — do not force it.
+      {{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }}.
+      Writing it here would look like it worked and silently revert — which is exactly what
+      happened on k3s-prod-3 on 2026-09-04, where a wifi lease's dhcpcd hook reverted this
+      role's resolvers three minutes after a green run. If the evidence above is a DHCP client
+      on a WIRELESS interface, the fix is the `wireless` block earlier in this role (a cluster
+      node gets one network path) — run it, then re-run this. If it is a DHCP client on the
+      node's real interface, or another manager entirely, add the matching branch to
+      roles/node_baseline for the manager this node really uses — do not force it.
   when: node_baseline_dns_mechanism == 'contended'
   tags: [dns]
 
+# THE CORPSE CASE, and why it is NOT a hole in the fail-closed rule above. After the
+# `wireless` block disables the radio, k3s-prod-3 is left with a /etc/resolv.conf that dhcpcd
+# WROTE (its "# Generated by dhcpcd from wlp1s0.dhcp..." header, the gateway as sole IPv4
+# resolver, two Google IPv6 servers) and NO dhcpcd that can ever run again. `contended` means
+# "another manager is live and will undo this write"; a file left behind by a client that no
+# longer exists is not a manager, it is the damage — and refusing to write here would leave the
+# node permanently mis-resolved, which is the very outcome the operator reported. So the file
+# branch below reclaims it, and says so.
+- name: Note that a dhcpcd-written /etc/resolv.conf is being reclaimed
+  ansible.builtin.debug:
+    msg: >-
+      /etc/resolv.conf on this node was written by dhcpcd, but no DHCP client can write it any
+      more (no process, no active/enabled unit, no `inet dhcp` stanza). Taking ownership and
+      replacing it with the declared upstreams
+      ({{ node_baseline_dns_servers | join(', ') }}). If a DHCP client ever comes back on this
+      node, the `contended` check above will stop the play instead of letting it revert
+      silently.
+  when:
+    - node_baseline_dns_mechanism == 'resolv-conf-file'
+    - node_baseline_resolv_dhcpcd_written | default(false)
+  tags: [dns]
+
 # --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
 # A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
 # never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses

infra/ansible/roles/node_health/README.md

@@ -32,7 +32,11 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 - **Works on bare metal and VMs.** Everything absent in a guest (SMART on virtio, DMI,
   EDAC, `coretemp`, CPU throttle counters) skips cleanly instead of erroring. Verified
   against both `k3s-prod-1` (HP EliteDesk, bare metal) and `k3s-dev-1` (Proxmox VM).
-- **Never `become`s unless it must.** Only SMART needs root.
+- **Never `become`s unless it must.** Three things need root: SMART, the kernel ring
+  buffer (`dmesg`, because Debian sets `kernel.dmesg_restrict=1`) and the SMBIOS DIMM
+  inventory (`dmidecode -t 17`). Everything else is read unprivileged in one pass, and
+  the privileged second pass re-runs the *same* collector with `--privileged` so the
+  parsing has one home. Without escalation those checks SKIP with the exact reason.
 
 ## What it checks
 
@@ -44,15 +48,99 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
 | `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
+| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
 | `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
+| `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
 
 Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
 in task logic.
 
+
+## The EFI memattr check
+
+`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
+
+HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
+does not validate it before reserving memory from its contents. **Two different kernel
+functions matter, and the check used to conflate them:**
+
+- `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both
+  fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so
+  **catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through
+  `phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything
+  above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and
+  costs nothing measurable.
+- `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints
+  "Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or
+  merely large.
+
+So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not*
+prove memory was lost: that depends on the garbage **value** at that address on that boot,
+not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03)
+booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on
+measured memory, never on the log line alone:**
+
+| kernel log | measured shortfall | verdict |
+|---|---|---|
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
+| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic |
+| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working |
+| no memattr line, log reaches back to boot | (not consulted) | PASS |
+| no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
+| `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
+
+The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt
+table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt
+table is not passing a health check even when it kept its RAM.
+
+### The FW_BUG line is proof, but its ABSENCE proves nothing
+
+`efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG
+line:
+
+| gate | message | reachable when |
+|---|---|---|
+| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
+| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not |
+| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not |
+
+**All three are rejections.** Which one a boot prints is decided by *which field* that boot's
+garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very
+often prints the *first* message, and looking only for the FW_BUG line will make a working
+fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed
+the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version
+alongside the log line, and why the check must never demand the FW_BUG string as its
+evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in
+the kernel image itself, not a line in `dmesg`.
+
+**The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged
+*negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that
+until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected
+a corrupt table was reported as one that "accepted it without complaint" — a false PASS on
+the exact bug this check exists to find.
+
+**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
+`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
+workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
+rebooting/CMOS/BIOS/USB changes only move where the table lands.
+
+### The two memory checks cannot contradict each other
+
+`memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine
+lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus
+the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in
+`tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads
+it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of
+its own, so there is exactly one verdict on lost memory per host and both checks quote it.
+A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a
+future memory loss with a different cause is still caught.
+
 ## Output
 
 1. A human summary printed in the play output.

infra/ansible/roles/node_health/defaults/main.yml

@@ -45,6 +45,33 @@ node_health_thresholds:
   # Memory (percentage of MemTotal still available)
   mem_available_pct_warn: 15
   mem_available_pct_fail: 5
+  # --- MEMORY THE MACHINE HAS THAT THE OS WILL NEVER USE ----------------------------
+  # Two independent measurements of the same question, both fed by memory.physical_total:
+  #
+  #   unmanaged  /proc/zoneinfo present pages MINUS managed pages — RAM the kernel's own
+  #              memory map says exists, that never reached the allocator. Needs no root
+  #              and no tools, works identically on metal and in a VM.
+  #   shortfall  MemTotal against the DIMM capacity SMBIOS (dmidecode -t 17) says is
+  #              installed. Needs root, and only bare metal reports it. Independent of the
+  #              kernel's memory map entirely, so it also catches RAM the firmware never
+  #              handed the kernel in the first place.
+  #
+  # MEASURED BASELINES for the unmanaged figure, so these are tuned against numbers and
+  # not taste: 2.0% k3s-prod-1 (32 GiB metal), 2.1% (32 GiB Fedora workstation), 2.2%
+  # k3s-dev-1 (24 GiB VM) — versus 83.4% on k3s-prod-2 in the EFI-memattr failure state.
+  # The healthy figure is the kernel image, memmap and firmware reservations; it is a
+  # near-constant few hundred MB, so a small machine sits higher in PERCENT terms and 10%
+  # keeps clearance for that. There is no plausible tuning of a threshold in this band
+  # that changes any of those five verdicts.
+  #
+  # The DIMM shortfall runs a little higher than the unmanaged figure on the same node
+  # (2.7% on k3s-prod-1) because installed capacity also covers e820-reserved regions and
+  # the 64 MiB of graphics stolen memory — legitimately missing RAM that was never the
+  # kernel's to manage. Same bands: the signal being caught is 83%, not 3%.
+  mem_unmanaged_pct_warn: 10
+  mem_unmanaged_pct_fail: 25
+  mem_dimm_shortfall_pct_warn: 10
+  mem_dimm_shortfall_pct_fail: 25
   # Temperatures (deg C, any sensor)
   temp_warn_c: 70
   temp_fail_c: 85

infra/ansible/roles/node_health/files/node_health_collect.py

@@ -97,10 +97,22 @@ def read_kv(path, sep=":"):
     return out
 
 
+def which(tool):
+    """Locate a binary, including the sbin dirs a NON-ROOT PATH usually omits.
+
+    `dmidecode` lives in /usr/sbin and is simply invisible to `shutil.which()` under the
+    unprivileged `k3s` user's PATH. Reporting "not installed" for a tool that IS installed
+    but unreachable would be a false reason on a SKIP, which is the one thing this
+    collector may never do."""
+    return shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+
+
 def run(argv, timeout=20):
     """Run a read-only query command. Returns (rc, stdout, stderr); rc None if absent."""
-    if shutil.which(argv[0]) is None:
+    exe = which(argv[0])
+    if exe is None:
         return None, "", "%s not installed" % argv[0]
+    argv = [exe] + list(argv[1:])
     try:
         proc = subprocess.run(
             argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout
@@ -284,9 +296,300 @@ def collect_memory():
         edac["ce_total"] = ce_total
         edac["ue_total"] = ue_total
     mem["edac"] = edac
+    # Physically present vs actually usable — the cause-independent memory-loss signal.
+    mem["zones"] = collect_zoneinfo(mem["mem_total_kb"])
+    # Firmware's own inventory of what is physically installed. Root-only; attempted
+    # here anyway so the unprivileged pass records the exact reason it could not read.
+    mem["devices"] = collect_memory_devices()
     return mem
 
 
+# --- Memory the kernel has but never hands out --------------------------------------
+# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
+# no DMI. Per zone it reports:
+#   spanned  pages in the zone's address range, holes included
+#   present  spanned MINUS the holes — i.e. RAM the kernel's e820/EFI memory map says
+#            physically exists here
+#   managed  present MINUS everything memblock reserved before the buddy allocator was
+#            handed the rest (kernel image, memmap, initrd, crashkernel, firmware
+#            reservations). sum(managed) IS MemTotal, exactly, on every node measured.
+# So `present - managed` is, by construction, "RAM this machine has that the OS will
+# never allocate", whatever the reason. On a healthy node it is the couple of hundred
+# MB the kernel legitimately keeps: measured 2.0% (k3s-prod-1), 2.2% (k3s-dev-1) and
+# 2.7% (a 24 GiB VM). On k3s-prod-2 in the EFI-memattr failure state it is 83.4% —
+# `Normal` zone present 3340288 / managed 0. That is a three-order-of-magnitude
+# separation, which is why this needs no clever tuning to be trustworthy.
+#
+# It is deliberately NOT a dmesg-wording check: it would still catch the same symptom if
+# a future firmware, kernel or hypervisor lost the memory for an entirely different
+# reason, and it needs neither root nor a parseable kernel log.
+def collect_zoneinfo(mem_total_kb):
+    raw = read_text("/proc/zoneinfo", optional=False)
+    page_kb = os.sysconf("SC_PAGE_SIZE") // 1024
+    out = {
+        "available": raw is not None,
+        "reason": None if raw is not None else "/proc/zoneinfo could not be read",
+        "page_size_kb": page_kb,
+        "zones": [],
+        "present_kb": None,
+        "managed_kb": None,
+        "unmanaged_kb": None,
+        "unmanaged_pct": None,
+    }
+    if raw is None:
+        return out
+    node = zone = None
+    cur = {}
+    present_pages = managed_pages = 0
+    for line in raw.splitlines():
+        fields = line.split()
+        if line.startswith("Node ") and "zone" in line:
+            # "Node 0, zone   Normal"
+            if cur:
+                out["zones"].append(cur)
+            node, zone = fields[1].rstrip(","), fields[-1]
+            cur = {"node": node, "zone": zone, "spanned": None, "present": None, "managed": None}
+        elif cur and len(fields) == 2 and fields[0] in ("spanned", "present", "managed"):
+            try:
+                cur[fields[0]] = int(fields[1])
+            except ValueError:
+                cur[fields[0]] = None
+    if cur:
+        out["zones"].append(cur)
+    # An empty parse is a PARSE failure, not "this node has no memory" — say so and stay
+    # null rather than emitting a confident 0.
+    if not out["zones"]:
+        out["available"] = False
+        out["reason"] = "/proc/zoneinfo was read but no zone could be parsed from it"
+        return out
+    for z in out["zones"]:
+        present_pages += z["present"] or 0
+        managed_pages += z["managed"] or 0
+    out["present_kb"] = present_pages * page_kb
+    out["managed_kb"] = managed_pages * page_kb
+    out["unmanaged_kb"] = out["present_kb"] - out["managed_kb"]
+    if out["present_kb"] > 0:
+        out["unmanaged_pct"] = round(out["unmanaged_kb"] * 100.0 / out["present_kb"], 1)
+    # Cross-check for the report: sum(managed) should equal MemTotal. If it does not, the
+    # two readings disagree and the discrepancy itself is worth showing.
+    out["matches_mem_total"] = (
+        None if (mem_total_kb is None or out["managed_kb"] is None)
+        else abs(out["managed_kb"] - mem_total_kb) <= page_kb
+    )
+    return out
+
+
+# --- Installed DIMMs (SMBIOS type 17 — ROOT ONLY) -----------------------------------
+# The independent answer to "how much RAM is physically in this box", from the firmware's
+# own inventory rather than from anything the kernel decided. Needs root: dmidecode reads
+# /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem. Unprivileged it fails, and that
+# failure is recorded as the SKIP reason — never as zero DIMMs.
+#
+# On a VM there is usually either no type-17 structure at all or a single synthetic entry
+# that merely restates the configured guest RAM; both are handled by reporting what was
+# found and letting the check SKIP when nothing populated was reported.
+DMI_SIZE_UNITS = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
+
+
+def collect_memory_devices():
+    out = {
+        "available": False,
+        "reason": None,
+        "devices": [],
+        "slots": 0,
+        "populated_slots": 0,
+        "installed_kb": None,
+    }
+    if which("dmidecode") is None:
+        out["reason"] = ("dmidecode is not installed on this node; it ships with "
+                         "roles/node_baseline (node_baseline_packages). "
+                         "This role never installs packages.")
+        return out
+    rc, stdout, stderr = run(["dmidecode", "-t", "17"], timeout=30)
+    if rc != 0:
+        out["reason"] = "dmidecode -t 17 failed (rc=%s): %s" % (
+            rc, (stderr or stdout).strip().splitlines()[-1] if (stderr or stdout).strip() else "no output")
+        return out
+    # Blocks are separated by blank lines; each "Memory Device" block is one SLOT, which
+    # may or may not hold a module ("Size: No Module Installed").
+    total_kb = 0
+    saw_size = False
+    for block in stdout.split("\n\n"):
+        if "Memory Device" not in block:
+            continue
+        fields = {}
+        for line in block.splitlines():
+            key, sep, value = line.partition(":")
+            if sep:
+                fields[key.strip()] = value.strip()
+        if "Size" not in fields:
+            continue
+        out["slots"] += 1
+        size_kb = None
+        parts = fields["Size"].split()
+        if len(parts) == 2 and parts[1].lower() in DMI_SIZE_UNITS:
+            try:
+                size_kb = int(parts[0]) * DMI_SIZE_UNITS[parts[1].lower()]
+            except ValueError:
+                size_kb = None
+        if size_kb:
+            saw_size = True
+            total_kb += size_kb
+            out["populated_slots"] += 1
+        out["devices"].append(
+            {
+                "locator": fields.get("Locator"),
+                "bank_locator": fields.get("Bank Locator"),
+                # Verbatim, so an unparsed spelling is visible rather than silently null.
+                "size_raw": fields["Size"],
+                "size_kb": size_kb,
+                "type": fields.get("Type"),
+                "speed": fields.get("Speed"),
+                "configured_speed": fields.get("Configured Memory Speed"),
+                "manufacturer": fields.get("Manufacturer"),
+                "part_number": fields.get("Part Number"),
+                "rank": fields.get("Rank"),
+                # Serial deliberately NOT collected: the artifact already carries drive
+                # serials and a DIMM serial adds identifying data no check reads.
+            }
+        )
+    out["available"] = saw_size
+    if not saw_size:
+        out["reason"] = ("dmidecode ran but reported no populated memory device "
+                         "(%d slot structure(s) found) — typical of a VM with no SMBIOS "
+                         "type 17 inventory" % out["slots"])
+    else:
+        out["installed_kb"] = total_kb
+    return out
+
+
+# --- Kernel ring buffer: the EFI Memory Attributes Table ----------------------------
+# WHY THIS IS HERE. See infra/PROD-PROVISIONING.md Section 0b for the authoritative
+# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
+# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
+# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
+# the product). The kernel prints one of two lines:
+#
+#   "Failed to map EFI Memory Attributes table @ 0x..."   the garbage version read <= 2, so
+#       the kernel TRUSTED the table and reserved memory from it. This is emitted later, by
+#       efi_memattr_apply_permissions()'s memremap(), and it fails for ANY sufficiently
+#       wrong tbl_size. So it proves the table was corrupt and trusted — it does NOT prove
+#       memory was lost: only a NEGATIVE tbl_size (bit 31 set, sign-extended and clamped to
+#       PHYS_ADDR_MAX) reserves everything above the table, while a garbage-but-positive
+#       tbl_size costs nothing measurable. Whether memory actually went missing is decided
+#       by MEASUREMENT (collect_zoneinfo below), not by this line: k3s-prod-2 lost 13.3 GiB
+#       twice at 0xbeeb0018 and then booted with all 15.1 GiB at the same address, same line.
+#   "Unexpected EFI Memory Attributes table version <N>"  the garbage version read > 2, so
+#       the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
+#       firmware is still publishing a corrupt table and the outcome is decided afresh by
+#       what the garbage reads on every boot. That is luck, not immunity.
+#
+# This collector reports the LINES and the addresses; it never rules on memory. Classifying
+# is roles/node_health/tasks/assess.yml, which rules on the measured shortfall.
+#
+# It is boot-to-boot NON-DETERMINISTIC, which is precisely why it is detected on every
+# health run instead of diagnosed once.
+#
+# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
+# every node here). The unprivileged pass therefore records WHY it could not read, and
+# roles/node_health re-runs this collector with `--privileged` when escalation works.
+BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Command line:")
+EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
+MEMATTR_FAILED_RE = re.compile(
+    r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
+# NOTE the `-?`. The kernel prints this version with %d — SIGNED — so a garbage field with
+# bit 31 set is logged NEGATIVE (k3s-prod-2, 2026-09-04: "version -1569980416"). Without the
+# minus sign this regex missed those lines entirely and the state fell through to "clean",
+# i.e. a node whose kernel had just REJECTED a corrupt table was reported as one whose
+# kernel "accepted it without complaint" — a false PASS on the exact bug this check exists
+# to find. Anchor future edits on that: the field is signed, and every diagnostic that
+# quotes it must survive a negative value.
+MEMATTR_VERSION_RE = re.compile(
+    r"Unexpected EFI Memory Attributes table version\s+(-?\d+)", re.I)
+MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
+# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
+# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
+# captured separately, above, and quoting it as evidence would bury the one line that
+# matters in 200 characters of unrelated table addresses.
+MEMATTR_LINE_RE = re.compile(r"memattr:|EFI Memory Attributes", re.I)
+# "Memory: 2568544K/16546088K available (... 13958980K reserved, ...)" — the same line
+# Section 0b tabulates per boot. Collected as corroborating evidence, not as a verdict.
+MEMORY_LINE_RE = re.compile(r"Memory:\s+(\d+)K/(\d+)K available")
+MEMORY_RESERVED_RE = re.compile(r"(\d+)K reserved")
+
+
+def collect_klog():
+    out = {
+        "available": False,
+        "reason": None,
+        "boot_covered": None,
+        "memattr_state": None,
+        "memattr_published": None,
+        "memattr_table_address": None,
+        "memattr_version": None,
+        "memattr_lines": [],
+        "memory_line": None,
+    }
+    rc, stdout, stderr = run(["dmesg"], timeout=30)
+    if rc is None:
+        out["reason"] = "dmesg is not installed on this node"
+        return out
+    if rc != 0:
+        detail = (stderr or stdout).strip().splitlines()
+        out["reason"] = "dmesg failed (rc=%d, uid=%d): %s%s" % (
+            rc, os.geteuid(), detail[-1] if detail else "no output",
+            " — kernel.dmesg_restrict=1, so reading the ring buffer needs root/CAP_SYSLOG"
+            if read_int("/proc/sys/kernel/dmesg_restrict") == 1 else "")
+        return out
+
+    out["available"] = True
+    # Does the buffer still reach back to early boot? The ring buffer WRAPS, and every
+    # memattr line is printed in the first fraction of a second. If the boot header is
+    # gone then so may be a memattr line that was printed — absence would prove nothing,
+    # and the check must SKIP rather than PASS. This is the difference between
+    # "unaffected" and "could not tell", and the role is not allowed to blur it.
+    out["boot_covered"] = bool(BOOT_ANCHOR_RE.search(stdout))
+
+    published = EFI_CONFIG_RE.search(stdout)
+    if published:
+        out["memattr_published"] = True
+        out["memattr_table_address"] = published.group(1)
+    elif re.search(r"^\[[^]]*\]\s*efi:\s", stdout, re.M):
+        # UEFI booted and the config-table line is present, but it lists no MEMATTR=.
+        out["memattr_published"] = False
+    # else: no efi: line at all (BIOS/CSM boot, or the line has scrolled away) -> None.
+
+    for line in stdout.splitlines():
+        if MEMATTR_LINE_RE.search(line):
+            out["memattr_lines"].append(line.strip())
+    joined = "\n".join(out["memattr_lines"])
+    if MEMATTR_CORRUPT_RE.search(joined):
+        # Kernel >= 6.13 (commit 8fbe4c49c0cc): the table is validated and rejected.
+        out["memattr_state"] = "corrupt_rejected"
+    elif MEMATTR_FAILED_RE.search(joined):
+        out["memattr_state"] = "failed_to_map"
+        addr = MEMATTR_FAILED_RE.search(joined).group(1)
+        if addr:
+            out["memattr_table_address"] = addr
+    elif MEMATTR_VERSION_RE.search(joined):
+        out["memattr_state"] = "bad_version"
+        out["memattr_version"] = int(MEMATTR_VERSION_RE.search(joined).group(1))
+    else:
+        out["memattr_state"] = "clean"
+
+    mem_line = MEMORY_LINE_RE.search(stdout)
+    if mem_line:
+        full = [l for l in stdout.splitlines() if mem_line.group(0) in l]
+        reserved = MEMORY_RESERVED_RE.search(full[0]) if full else None
+        out["memory_line"] = {
+            "available_kb": int(mem_line.group(1)),
+            "total_kb": int(mem_line.group(2)),
+            "reserved_kb": int(reserved.group(1)) if reserved else None,
+            "raw": full[0].strip() if full else None,
+        }
+    return out
+
+
 # --- Thermals -----------------------------------------------------------------------
 def collect_thermal():
     zones = []
@@ -708,15 +1011,34 @@ def collect_os():
 def collect_tools():
     tools = {}
     for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
-        tools[tool] = shutil.which(tool) or shutil.which(tool, path="/usr/sbin:/sbin:/usr/bin:/bin")
+        tools[tool] = which(tool)
     return tools
 
 
 def main():
+    # --privileged: re-collect ONLY the fields that genuinely need root, for the second
+    # pass roles/node_health makes when (and only when) escalation is proven to work
+    # (tasks/collect_privileged.yml). Parsing therefore lives in exactly one place —
+    # re-implementing the kernel-log grep in Jinja would be a second source of truth for
+    # the same fact, and the two would drift.
+    if "--privileged" in sys.argv[1:]:
+        doc = {
+            "schema": SCHEMA,
+            "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
+            "collector_uid": os.geteuid(),
+            "klog": collect_klog(),
+            "memory": {"devices": collect_memory_devices()},
+        }
+        doc["unreadable"] = UNREADABLE
+        json.dump(doc, sys.stdout, indent=2, sort_keys=True)
+        sys.stdout.write("\n")
+        return
+
     doc = {
         "schema": SCHEMA,
         "collected_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
         "collector_uid": os.geteuid(),
+        "klog": collect_klog(),
         "os": collect_os(),
         "dmi": collect_dmi(),
         "cpu": collect_cpu(),

infra/ansible/roles/node_health/tasks/assess.yml

@@ -387,6 +387,118 @@
                ~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
                ~ avail_pct | string ~ '%)') }}
 
+# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
+# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
+# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
+# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
+# from the presence of a kernel-log line — and they duly contradicted each other on the same
+# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
+# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
+# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
+# measurement was right; the inference from the log line was wrong — the memattr block below
+# has the refined mechanism.
+#
+# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
+# place and both checks read this fact: neither derives a shortfall of its own, and the
+# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
+# disagree about whether memory was lost — there is only one verdict on that question, and
+# it is this one. Thresholds and their measured baselines: defaults/main.yml.
+- name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
+  ansible.builtin.set_fact:
+    node_health_mem_shortfall: "{{ {
+      'zones_pct': z_pct, 'zones_status': z_status,
+      'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
+      'status': worst_status } }}"
+  vars:
+    ansible_become: false
+    th: "{{ node_health_thresholds }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+
+    # 1. /proc/zoneinfo: pages the kernel says are physically PRESENT, minus the pages it
+    #    actually handed to the allocator (sum(managed) IS MemTotal). World-readable, no
+    #    tools, identical on metal and in a VM — so this half is available almost always.
+    z_pct: "{{ z.unmanaged_pct if (z.available | bool) else none }}"
+    z_status: >-
+      {{ 'SKIP' if z_pct is not number
+         else ('FAIL' if z_pct | float >= th.mem_unmanaged_pct_fail
+               else ('WARN' if z_pct | float >= th.mem_unmanaged_pct_warn else 'PASS')) }}
+
+    # 2. SMBIOS type 17 vs MemTotal. Root-only and bare-metal-only, so it SKIPS on the dev
+    #    VMs and wherever escalation is unavailable — with dmidecode's own reason attached,
+    #    never as a zero DIMM count. Independent of the kernel's memory map entirely.
+    inst_kb: "{{ dev.installed_kb if (dev.available | bool) else none }}"
+    d_pct: >-
+      {{ (((inst_kb | int) - (m.mem_total_kb | int)) * 100.0 / (inst_kb | int)) | round(1)
+         if (inst_kb is number and m.mem_total_kb is number and (inst_kb | int) > 0)
+         else none }}
+    d_status: >-
+      {{ 'SKIP' if d_pct is not number
+         else ('FAIL' if d_pct | float >= th.mem_dimm_shortfall_pct_fail
+               else ('WARN' if d_pct | float >= th.mem_dimm_shortfall_pct_warn else 'PASS')) }}
+
+    # Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
+    # SKIPs must never add up to a PASS.
+    worst_status: >-
+      {{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
+         | default('SKIP', true) }}
+
+# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
+# firmware.efi_memattr names the one cause we have actually hit here; this one asks the
+# blunter question — "is this machine using the RAM it has?" — and answers it from two
+# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
+# losing memory for some other reason still shows up.
+- name: Assess — usable memory vs the memory this machine physically has
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'memory.physical_total',
+      'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
+      'status': phys_status,
+      'detail': phys_detail }] }}"
+  vars:
+    ansible_become: false
+    ms: "{{ node_health_mem_shortfall }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    dev: "{{ node_health_facts.memory.devices }}"
+    m: "{{ node_health_facts.memory }}"
+    # Straight from the shared measurement — this check re-derives nothing.
+    z_pct: "{{ ms.zones_pct }}"
+    inst_kb: "{{ ms.installed_kb }}"
+    d_pct: "{{ ms.dimm_pct }}"
+    phys_status: "{{ ms.status }}"
+    # What the kernel log says, IF it was read — quoted here only as attribution for the
+    # measured shortfall. The verdict above is the shared measurement and never depends on
+    # the log; firmware.efi_memattr reads that same verdict, so the two always agree.
+    phys_cause: >-
+      {{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
+           firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
+           if (node_health_facts.klog.memattr_state | default('')) == 'failed_to_map'
+         else ('. The kernel log was NOT read here, so this shortfall is unattributed —
+                see check firmware.efi_memattr')
+           if not (node_health_facts.klog.available | bool)
+         else '' }}
+    phys_detail: >-
+      {{ (('present ' ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB, managed '
+           ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB — '
+           ~ z_pct | string ~ '% of the RAM the kernel memory map reports as physically PRESENT
+             was never handed to the allocator (/proc/zoneinfo)')
+          if z_pct is number
+          else ('present-vs-managed pages NOT read — ' ~ (z.reason | default('no reason recorded', true))))
+         ~ '. '
+         ~ (('SMBIOS reports ' ~ ((inst_kb | int) / 1048576) | round(1) | string ~ ' GiB installed across '
+             ~ dev.populated_slots | string ~ ' of ' ~ dev.slots | string ~ ' slot(s) ('
+             ~ (dev.devices | selectattr('size_kb', 'number') | map(attribute='locator') | join(', '))
+             ~ '); MemTotal is ' ~ ((m.mem_total_kb | int) / 1048576) | round(1) | string
+             ~ ' GiB — a ' ~ d_pct | string ~ '% shortfall')
+            if d_pct is number
+            else ('installed DIMM capacity NOT read — ' ~ (dev.reason | default('no reason recorded', true))))
+         ~ (('. A few percent is normal and expected — the kernel image, memmap, firmware
+              reservations and (on this platform) 64 MiB of graphics stolen memory.
+              A shortfall of this size is memory the machine HAS and cannot use')
+            if phys_status in ['WARN', 'FAIL'] else '')
+         ~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
+
 - name: Assess — swap (kubelet requires it off)
   ansible.builtin.set_fact:
     node_health_checks: "{{ node_health_checks + [{
@@ -674,3 +786,245 @@
                ~ (d.bios_version | default('?')) ~ ' (' ~ (d.bios_date | default('?')) ~ ')'
                ~ (' — serial numbers not read (root-only DMI fields)'
                   if d.product_serial is none else '')) }}
+
+# --- EFI Memory Attributes Table ------------------------------------------------------
+# THE AUTHORITATIVE WRITE-UP IS infra/PROD-PROVISIONING.md §0b. Do not re-derive it here;
+# what follows is only what this check needs to be read correctly.
+#
+# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
+# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
+# are involved, and conflating them is what made this check wrong until 2026-09-03:
+#
+#   efi_memattr_init()              tbl_size = sizeof(*tbl) + num_entries * desc_size, then
+#                                   memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
+#                                   is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
+#                                   NEGATIVE: a garbage product with bit 31 set sign-extends
+#                                   through phys_addr_t and memblock_cap_size() clamps it to
+#                                   PHYS_ADDR_MAX — everything above the table is reserved.
+#                                   A garbage-but-POSITIVE tbl_size reserves a modest, wrong
+#                                   amount and costs nothing measurable.
+#   efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
+#                                   THIS is what prints "Failed to map". It fails for ANY
+#                                   sufficiently wrong tbl_size — negative OR merely large.
+#
+# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
+# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
+# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
+# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
+# is the counter-example that falsified the old inference.
+#
+# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
+#
+#   "Failed to map ... @ 0x..."   the bug FIRED: corrupt table, trusted by the kernel.
+#       FAIL when the shared measurement (node_health_mem_shortfall) says memory is
+#       catastrophically missing — memory is lost NOW.
+#       WARN otherwise — this boot drew a survivable garbage value; the next may not.
+#   "Unexpected ... version <N>"  version read > 2, so the kernel bailed out before
+#       reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
+#       safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
+#       gates and no draw can reach the fatal arithmetic any more. Same line, opposite
+#       meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
+#   FW_BUG "Corrupted EFI Memory Attributes Table detected!"  the >= 6.13 THIRD gate
+#       (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
+#       must not be treated as such: it is only reachable on a boot whose garbage leaves the
+#       version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
+#       line instead. All three prod nodes did exactly that on 2026-09-04.
+#
+# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
+# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
+# measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
+# firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
+# the draw still happens but every outcome is a rejection, so the same lines mean the
+# validation is working.
+#
+# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
+# so it cannot be diagnosed once. It has to be measured on every health run.
+#
+# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
+# node_health_mem_shortfall, computed once further up this file; this check does not measure
+# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
+# one verdict on any host and both checks quote it.
+#
+# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
+# ONLY if the kernel log was actually read AND still reaches back to early boot — every
+# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
+# wraps. So this check SKIPs, with the reason, when dmesg could not be read at all, and
+# SKIPs again when it was read but carries no boot anchor. Neither is allowed to become a
+# PASS. memory.physical_total measures the SYMPTOM from /proc/zoneinfo and needs no kernel
+# log at all, so a node whose log is unreadable is still not unmeasured.
+- name: Assess — EFI Memory Attributes table (firmware bug + unvalidated kernel parse)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'firmware.efi_memattr',
+      'title': 'EFI Memory Attributes table — corrupt table reserved as RAM by the kernel',
+      'status': memattr_status,
+      'detail': memattr_detail }] }}"
+  vars:
+    ansible_become: false
+    k: "{{ node_health_facts.klog }}"
+    z: "{{ node_health_facts.memory.zones }}"
+    # THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
+    # near memory.physical_total. This check contributes the kernel-log facts only.
+    ms: "{{ node_health_mem_shortfall }}"
+    ma_state: "{{ k.memattr_state | default(none) }}"
+    # DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
+    # carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
+    # THREE checks before it computes tbl_size at all:
+    #     tbl->version > 2                       -> "Unexpected ... table version %d"
+    #     desc_size implausible                  -> "Unexpected ... descriptor size %u ..."
+    #     num_entries > SZ_64K                   -> FW_BUG "Corrupted EFI Memory Attributes
+    #                                               Table detected! ..."
+    # ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
+    # given boot produces is decided by WHICH field that boot's garbage lands in — it is not
+    # a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
+    # version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
+    # memblock_reserve(). That is the whole difference, and it is why the verdict for
+    # `bad_version` below is opposite on the two kernels: same log line, completely different
+    # meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
+    # something the `version` test can parse.
+    memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
+    memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
+    # `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
+    # COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
+    # this check and memory.physical_total can no longer disagree about lost memory.
+    # `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
+    # boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
+    # WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
+    # forever would make the check unable to ever show the fix landing, which is exactly what
+    # it is for.
+    memattr_status: >-
+      {{ 'SKIP' if not (k.available | bool)
+         else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
+               else ('WARN' if ma_state == 'failed_to_map'
+                     else ('PASS' if ma_state == 'corrupt_rejected'
+                           else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
+                                 if ma_state == 'bad_version'
+                                 else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
+    # One remediation, quoted by both findings. NOTE what it must NOT say: there is no
+    # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
+    # could take effect. Suggesting one would be a fix that quietly does nothing.
+    memattr_fix: >-
+      FIX: boot a kernel >= 6.13 (Debian 13: linux-image-amd64 from trixie-backports) —
+      upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly
+      bogus", first released in v6.13 and absent from v6.12 and from the current
+      linux-6.12.y stable branch. This node runs {{ node_health_facts.os.kernel | default('an unknown kernel') }}.
+      There is NO kernel cmdline workaround: efi_memattr_init() runs before any efi=
+      bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
+      unplugging USB only change where the table lands and may APPEAR to fix it — they are
+      a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
+    # The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
+    # to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
+    # kernel log has no part in; the verdict word comes from ms.status, so it always agrees
+    # with memory.physical_total. When the measurement is missing it says so, and says why.
+    memattr_figures: >-
+      {{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+          ~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
+          ~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
+            allocator (/proc/zoneinfo)')
+         if ms.zones_pct is number
+         else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
+                reports as installed (/proc/zoneinfo NOT read — '
+                ~ (z.reason | default('no reason recorded', true)) ~ ')')
+               if ms.dimm_pct is number else '') }}
+    memattr_corroboration: >-
+      {{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
+          ~ ms.status ~ '): ' ~ memattr_figures
+          ~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
+             else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
+                   else ' — the normal few percent (kernel image, memmap, firmware
+                        reservations, graphics stolen memory). NO measurable memory was lost
+                        on this boot.')))
+         if (memattr_figures | trim | length) > 0
+         else (' Whether memory was lost on this boot could NOT be measured here: neither
+                present-vs-managed pages nor installed DIMM capacity was read — see check
+                memory.physical_total for the reasons.') }}
+    # The kernel log lines themselves, verbatim. A verdict the operator cannot check
+    # against the evidence is not much better than a wrong one.
+    memattr_evidence: >-
+      {{ ((' Kernel log: ' ~ (k.memattr_lines | join(' | ')))
+          if (k.memattr_lines | default([]) | length) > 0 else '')
+         ~ ((' At boot the kernel reported '
+             ~ ((k.memory_line.available_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
+             ~ ((k.memory_line.total_kb | int) / 1048576) | round(1) | string ~ ' GiB available, '
+             ~ ((k.memory_line.reserved_kb | int) / 1048576) | round(1) | string ~ ' GiB reserved'
+             ~ ' — the per-boot figure PROD-PROVISIONING.md §0b tabulates.')
+            if (k.memory_line is mapping and (k.memory_line.reserved_kb | default(none)) is number)
+            else '') }}
+    memattr_detail: >-
+      {%- if not (k.available | bool) -%}
+      the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
+      Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
+      lives only in dmesg.{{ memattr_corroboration }}
+      {%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
+      THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
+      line. The kernel failed to map the EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }}, which proves the
+      table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
+      from its contents; the measurement below shows that reservation was the catastrophic
+      kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
+      through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
+      table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'failed_to_map' -%}
+      THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
+      loss either. The kernel failed to map the EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }}: the table was
+      corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
+      WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
+      the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
+      with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
+      tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
+      measurable and still fails the later memremap() that printed this line. This boot drew a survivable
+      value; the draw is repeated on EVERY boot, at the same address — this platform has
+      been seen to lose 13.3 GiB and to keep its full memory at one identical table address
+      (infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
+      {{ memattr_fix }}
+      {%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
+      the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
+      kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
+      VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
+      boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
+      outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
+      efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
+      WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
+      Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
+      (num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
+      version field <= 2. Both messages are rejections, and which one a boot prints is the
+      lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
+      path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
+      PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
+      {{ memattr_corroboration }}
+      {%- elif ma_state == 'bad_version' -%}
+      EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
+      CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
+      version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
+      kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
+      EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
+      tbl_size the garbage produces — nothing measurable if that value is positive,
+      everything above the table if it is negative.{{ memattr_evidence }}
+      {{ memattr_corroboration }} {{ memattr_fix }}
+      {%- elif ma_state == 'corrupt_rejected' -%}
+      the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
+      validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
+      cannot cost this node memory on this or any future boot; no
+      action.{{ memattr_evidence }}{{ memattr_corroboration }}
+      {%- elif not (k.boot_covered | bool) -%}
+      NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
+      back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
+      the ring buffer, which wraps). Every memattr diagnostic is printed within the first
+      fraction of a second, so its absence here proves NOTHING — this is an UNKNOWN, not a
+      pass. Read /var/log/kern.log for this boot, or re-run after a
+      reboot.{{ memattr_corroboration }}
+      {%- elif k.memattr_published is sameas true -%}
+      the firmware publishes an EFI Memory Attributes table at
+      {{ k.memattr_table_address | default('an unreported address') }} and the kernel
+      accepted it without complaint, in a log that still reaches back to early boot. Not
+      affected on this boot.
+      {%- elif k.memattr_published is sameas false -%}
+      this firmware publishes NO EFI Memory Attributes table (the UEFI config-table line
+      lists no MEMATTR=), so the bug cannot arise on this node. Normal for a Proxmox/OVMF
+      guest.
+      {%- else -%}
+      no UEFI config-table line in the kernel log at all — this node did not boot via UEFI,
+      so there is no EFI Memory Attributes table to be corrupt. Log reaches back to early
+      boot, so that absence is measured, not assumed.
+      {%- endif -%}

infra/ansible/roles/node_health/tasks/collect_privileged.yml

@@ -0,0 +1,62 @@
+---
+# The SECOND pass of the collector, run with escalation, for the two facts that are
+# genuinely root-gated:
+#
+#   * the KERNEL RING BUFFER (`dmesg`) — Debian sets kernel.dmesg_restrict=1, so an
+#     unprivileged read gets EPERM. This is where the EFI Memory Attributes Table
+#     diagnostic lives (infra/PROD-PROVISIONING.md §0b), and that line is the difference
+#     between a node that has lost 13.3 GiB and one that has not.
+#   * SMBIOS type 17 (`dmidecode -t 17`) — the firmware's own inventory of installed
+#     DIMMs, read from /sys/firmware/dmi/tables/DMI (mode 0400) or /dev/mem.
+#
+# WHY RE-RUN THE COLLECTOR RATHER THAN SHELL OUT HERE. Both facts need real parsing (a
+# kernel-log grep with three alternative messages, a boot-coverage anchor, a blank-line-
+# delimited SMBIOS block parse). Re-implementing that in Jinja would create a SECOND
+# source of truth for the same fact, and the two would drift. `--privileged` makes the
+# collector emit exactly those subtrees and nothing else, so the parsing stays in one
+# place and is identical whichever pass produced it.
+#
+# REACHED ONLY when probe_privilege.yml proved escalation works AND the unprivileged pass
+# actually failed to read one of them (main.yml checks both). On a node where
+# dmesg_restrict=0 the first pass already succeeded and this never runs — the role's rule
+# is that everything readable WITHOUT root is read without root.
+#
+# If escalation is unavailable the checks that consume these SKIP with the exact reason
+# the unprivileged attempt recorded. They never silently become a PASS: "no memattr line"
+# and "could not read the kernel log" are different answers and the role says which.
+
+- name: Re-collect the root-only facts with escalation
+  ansible.builtin.script:
+    cmd: node_health_collect.py --privileged
+    executable: /usr/bin/python3
+  become: true
+  vars:
+    ansible_become: true
+  register: node_health_privileged_raw
+  changed_when: false   # strictly read-only; it can never change the node
+
+- name: Assert the privileged payload is the schema this role understands
+  ansible.builtin.assert:
+    that:
+      - (node_health_privileged_raw.stdout | from_json).schema == 'node-health/1'
+    fail_msg: >-
+      Privileged collector pass returned an unexpected schema; role and collector are
+      out of step.
+    quiet: true
+  vars:
+    ansible_become: false
+
+# Merge, field by field, over the unprivileged facts. Explicit rather than a recursive
+# `combine` because `unreadable` is a LIST and must be APPENDED — a recursive combine
+# would REPLACE it and throw away the reasons the first pass recorded.
+- name: Merge the privileged facts over the unprivileged ones
+  ansible.builtin.set_fact:
+    node_health_facts: >-
+      {{ node_health_facts | combine({
+           'klog': priv.klog,
+           'memory': node_health_facts.memory | combine({'devices': priv.memory.devices}),
+           'unreadable': (node_health_facts.unreadable | default([])) + (priv.unreadable | default([])),
+         }) }}
+  vars:
+    ansible_become: false
+    priv: "{{ node_health_privileged_raw.stdout | from_json }}"

infra/ansible/roles/node_health/tasks/main.yml

@@ -4,9 +4,11 @@
 # ORDER OF OPERATIONS
 #   1. Collect everything readable WITHOUT root (one script, one round trip).
 #   2. Probe whether privilege escalation actually works on this host.
-#   3. Only if it does AND smartmontools is installed: read SMART.
+#   3. Only if it does, and only for what the first pass could NOT read: re-run the same
-#   4. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#      collector with --privileged (kernel ring buffer, SMBIOS DIMM inventory).
-#   5. Render the human summary + write the JSON artifact.
+#   4. Only if it does AND smartmontools is installed: read SMART.
+#   5. Turn facts into a list of explicit PASS/WARN/FAIL/SKIP checks.
+#   6. Render the human summary + write the JSON artifact.
 #
 # BECOME HANDLING — the subtle bit, read before editing.
 #   `ansible_become` set as an inventory/group var (which is exactly what
@@ -43,6 +45,16 @@
   ansible.builtin.include_tasks: probe_privilege.yml
   when: node_health_try_privileged | bool
 
+# Only when the unprivileged pass actually failed to read one of them. On a node with
+# kernel.dmesg_restrict=0 the first pass already has the kernel log and this is skipped
+# entirely — "everything readable WITHOUT root is read without root" (defaults/main.yml).
+- name: Re-read the root-only facts with escalation (kernel log, SMBIOS DIMMs)
+  ansible.builtin.include_tasks: collect_privileged.yml
+  when:
+    - node_health_root_available | bool
+    - (not (node_health_facts.klog.available | bool))
+      or (not (node_health_facts.memory.devices.available | bool))
+
 - name: Read SMART data (privileged)
   ansible.builtin.include_tasks: collect_smart.yml
   when:

infra/ansible/roles/node_health/templates/summary.txt.j2

@@ -12,6 +12,17 @@
 {% macro gib(n) -%}
 {{ (n | int / 1073741824) | round(1) }} GiB
 {%- endmacro %}
+{# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not
+   report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of
+   its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised
+   "object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it
+   as `n/a` (the same token this template already uses for unreported readings) rather than
+   defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The
+   collector already guards this — tasks/collect_smart.yml uses selectattr('thresh',
+   'defined') and `a.thresh | default(0)` — so only the template was exposed. #}
+{% macro thr(a) -%}
+{{ a.thresh | default('n/a') }}
+{%- endmacro %}
 ================================================================================
  NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
  collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
@@ -45,6 +56,15 @@
 {% endif %}
  Memory       {{ (node_health_facts.memory.mem_total_kb | int / 1048576) | round(1) }} GiB total, {{ (node_health_facts.memory.mem_available_kb | int / 1048576) | round(1) }} GiB available, swap {{ (node_health_facts.memory.swap_total_kb | int / 1048576) | round(2) }} GiB
  ECC/EDAC     {{ 'controllers: ' ~ (node_health_facts.memory.edac.controllers | map(attribute='name') | join(', ')) ~ ' CE=' ~ node_health_facts.memory.edac.ce_total | string ~ ' UE=' ~ node_health_facts.memory.edac.ue_total | string if node_health_facts.memory.edac.controllers_present else 'not exposed (no EDAC controller bound — ECC status UNKNOWN on this platform)' }}
+{% set mz = node_health_facts.memory.zones %}
+{% set md = node_health_facts.memory.devices %}
+ Present RAM  {{ ((mz.present_kb | int / 1048576) | round(1) | string ~ ' GiB present / ' ~ (mz.managed_kb | int / 1048576) | round(1) | string ~ ' GiB managed — ' ~ mz.unmanaged_pct | string ~ '% never handed to the allocator') if mz.available and mz.unmanaged_pct is not none else ('NOT READ — ' ~ (mz.reason | default('no reason recorded', true))) }}
+ DIMMs        {{ ((md.installed_kb | int / 1048576) | round(1) | string ~ ' GiB installed in ' ~ md.populated_slots | string ~ ' of ' ~ md.slots | string ~ ' slot(s)') if md.available else ('NOT READ — ' ~ (md.reason | default('no reason recorded', true)) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14)) }}
+{% for dimm in md.devices | default([]) if dimm.size_kb %}
+              {{ '%-8s' | format(dimm.locator | default('?', true)) }} {{ '%-8s' | format(dimm.size_raw) }} {{ dimm.type | default('?', true) }} {{ dimm.speed | default('', true) }} {{ dimm.manufacturer | default('', true) }} {{ dimm.part_number | default('', true) }}
+{% endfor %}
+{% set klog = node_health_facts.klog %}
+ EFI memattr  {{ (klog.memattr_lines | join(' | ')) if klog.memattr_lines else (('kernel log NOT READ — ' ~ (klog.reason | default('no reason recorded', true))) if not klog.available else ('no memattr line in the kernel log' ~ ('' if klog.boot_covered else ' — BUT the ring buffer no longer reaches early boot, so that is an UNKNOWN, not a pass'))) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
  Load         {{ node_health_facts.cpu.loadavg | default([]) | join(' ') }}
 
 --------------------------------------------------------------------------------
@@ -64,19 +84,19 @@
               lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }}  temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
               pending(197)={{ sm.pending_sectors | default('?') }}  offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }}  reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }}  crc(199)={{ sm.crc_errors | default('?') }}
 {% if sm.failing_now or sm.failed_past %}
-              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %}
+              THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %}
 
 {% elif sm.thresholds_present %}
-              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %}
+              no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %}
 
 {% endif %}
               vendor context (NOT used for the verdict — raw units are vendor-defined):
               reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }}  grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }}  erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }}  program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }}  unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
 {% if sm.realloc_attr %}
-              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged)
+              attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged)
 {% endif %}
 {% for a in sm.attributes | default([]) %}
-              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
+              [{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %}  <<< when_failed={{ a.when_failed }}{% endif %}
 
 {% endfor %}
 {% else %}

platform-config-cdi/kustomization.yaml

@@ -0,0 +1,41 @@
+---
+# platform-config-cdi — the SECOND-ORDER config wave: custom resources whose CRD is created at
+# RUNTIME by a controller that a `platform-config` custom resource switches on.
+#
+# ── THE BUG THIS DIRECTORY EXISTS TO FIX (observed on PROD, 2026-09-05) ───────────────────────────
+# The repo already splits CONTROLLERS (`platform`) from CONFIGS (`platform-config`) because a CR
+# cannot be dry-run before its CRD is registered. That split assumes a CRD is registered by the
+# controller INSTALL. CDI breaks the assumption: `cdi-operator.yaml` registers exactly ONE CRD
+# (`cdis.cdi.kubevirt.io`). Every OTHER CDI CRD — `storageprofiles`, `datavolumes`, `datasources`,
+# `dataimportcrons`, … — is created by cdi-operator only AFTER it has reconciled the `CDI` CR.
+#
+# So `cdi-cr.yaml` is a FIRST-order CR (its CRD comes from the install) and belongs in
+# platform-config, while `cdi-storageprofiles.yaml` is a SECOND-order CR (its CRD comes from
+# reconciling the first-order CR). Having both in ONE Flux Kustomization is a true DEADLOCK, not a
+# race: kustomize-controller server-side dry-runs EVERY object before applying ANY of them, so the
+# unknown `StorageProfile` kind aborts the whole pass → the `CDI` CR is never applied → the
+# StorageProfile CRD is never created → platform-config can NEVER converge, and every tier that
+# dependsOn it stays blocked forever. On prod, cdi-operator sat Available for 12h with no CR:
+#
+#   StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile"
+#   in version "cdi.kubevirt.io/v1beta1"
+#
+# Invisible on dev (whose CDI CRDs have existed for months) and fatal on any COLD cluster.
+#
+# ── THE RULE, so this is not rediscovered a third time ────────────────────────────────────────────
+# A CR belongs HERE (not in platform-config) when the CRD it needs is created by a CONTROLLER AT
+# RUNTIME rather than by a manifest in git. Grep test: if `grep -rn "kind: CustomResourceDefinition"`
+# over ./platform (following remote bases) does not produce the CRD, and no chart installs it, then
+# some operator creates it after being switched on — and this is the tier for it.
+#
+# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
+# manifests stay beside the component they configure). Build/verify locally with:
+#   kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config-cdi
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  # CDI StorageProfiles — snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
+  # `longhorn-vm` virt-default class (clears CDIDefaultStorageClassDegraded). `StorageProfile`'s CRD
+  # is created by cdi-operator only once the `CDI` CR (applied by platform-config) is reconciled,
+  # which is exactly why these cannot live in the same pass as that CR.
+  - ../platform/kubevirt/cdi-storageprofiles.yaml

platform-config/kustomization.yaml

@@ -25,6 +25,14 @@ resources:
   - ../platform/kyverno/policies/restrict-host-namespaces.yaml
   - ../platform/kyverno/policies/require-egress-class-label.yaml
   - ../platform/kyverno/policies/require-auth-mode-label.yaml
+  # COLD-START FIX (2026-09-06, prod): a MUTATE policy that must travel WITH the two label
+  # policies above. CDI generates its own short-lived worker pods (importer-*, uploader-*,
+  # cloner-*) inside app namespaces; nothing in git can label them, so both policies DENIED them
+  # and `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled with no disk. This policy adds
+  # the two labels to pods created by ServiceAccount cdi/cdi-sa, so the requirement is SATISFIED
+  # rather than bypassed. Applying it here (not later) matters: if it lands after the validating
+  # policies, the next import is blocked in the gap.
+  - ../platform/kyverno/policies/label-cdi-worker-pods.yaml
   - ../platform/kyverno/admission/secret-provenance.yaml
   # cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
   - cilium/l2-ip-pool.yaml
@@ -41,9 +49,23 @@ resources:
   # ConfigMap kept beside them for locality.
   - ../platform/kubevirt/kubevirt-cr.yaml
   - ../platform/kubevirt/cdi-cr.yaml
-  # CDI StorageProfiles: snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
+  # *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
-  # `longhorn-vm` virt-default class (consume cdi.kubevirt.io; clears CDIDefaultStorageClassDegraded).
+  # The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
-  - ../platform/kubevirt/cdi-storageprofiles.yaml
+  # (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
+  #
+  # `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
+  # `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
+  # only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
+  # in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
+  # `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
+  # created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
+  # A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
+  #   StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
+  #   "cdi.kubevirt.io/v1beta1"
+  # Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
+  #
+  # THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
+  # is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
   - ../platform/kubevirt/servicemonitor.yaml
   - ../platform/kubevirt/alerts.yaml
   - ../platform/kubevirt/dashboard.yaml

platform/cilium/helmrelease.yaml

@@ -10,7 +10,8 @@
 #
 # SOURCE OF TRUTH (do not diverge without updating both):
 #   infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
-#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version, k3s_api_host/port)
+#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version)
+#   infra/ansible/inventory/group_vars/all.yml     (k3s_apiserver_endpoint, k3s_api_port)
 #
 # Every value tagged [parity] below is asserted identical to the Ansible render. If you change one
 # here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.)
@@ -45,11 +46,35 @@ spec:
     # [parity] MUST equal Ansible: kubeProxyReplacement: true
     kubeProxyReplacement: true
     # [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint
-    # (k3s_api_host/k3s_api_port). This is the primary control-plane node IP.
+    # (**k3s_apiserver_endpoint** / k3s_api_port, infra/ansible/inventory/group_vars/all.yml).
     # Setting these wrong breaks Cilium's connection to the API server on adopt.
-    # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.50 | prod set in clusters/prod.
+    #
-    # The `platform` Flux Kustomization substitutes it; the Ansible side is already parameterized
+    # THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS — operator decision 2026-09-06.
-    # from inventory (k3s_api_host), so the parity contract holds per-environment, not per-literal.
+    # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.49 | prod 192.168.2.9, each equal to
+    # that environment's kube_vip_address. The knob's NAME still says "host"; read it as "the API
+    # host Cilium dials", and see clusters/base/cluster-vars.yaml for why it was not renamed.
+    #
+    # WHY, in one line: this value used to be the PRIMARY NODE's IP, so on a 3-server HA cluster the
+    # CNI's control-plane path depended on one machine. Verified on live prod 2026-09-06 — every
+    # cilium-agent carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1) while kube-vip held .9.
+    # Losing that node does not break the datapath (Cilium keeps serving from cache); it silently
+    # stops every service and policy update from ever landing, which is worse to detect.
+    #
+    # [parity] IS NOW AGAINST k3s_apiserver_endpoint AND NOT k3s_api_host. Those were one Ansible
+    # variable until 2026-09-06 and the conflation is the defect. k3s_api_host still exists and
+    # still holds the FOUNDER's own address, because roles/k3s_server's secondary
+    # `--server https://<host>:6443` join must target a host that is already up — on a COLD build
+    # no VIP exists yet. So: founder for the join, VIP for the CNI. If you change this line, change
+    # cilium-values.yaml.j2 in the same PR, and point it at the SAME variable.
+    #
+    # The `platform` Flux Kustomization substitutes it; the Ansible side is parameterized from
+    # inventory (k3s_apiserver_endpoint derives from kube_vip_address), so the parity contract holds
+    # per-environment, not per-literal.
+    #
+    # NEEDS NO CERT WORK: k3s_tls_sans has always carried kube_vip_address alongside every node IP,
+    # so both addresses are already SANs on the serving cert (verified live 2026-09-06 — prod
+    # answers through https://192.168.2.9:6443 under full CA verification, no insecure skip). This
+    # switch reissues nothing and restarts no apiserver.
     k8sServiceHost: "${K3S_API_HOST}"
     k8sServicePort: 6443
     # --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 ---
@@ -89,6 +114,27 @@ spec:
       autoMount:
         enabled: false
       hostRoot: /run/cilium/cgroupv2
+    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
+    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
+    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
+    # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
+    # Helm render take down two of three agents simultaneously. That is exactly what happened
+    # when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config`
+    # init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium
+    # agent itself can translate — and the agent translating it was the one being replaced.
+    # `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once.
+    # A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the
+    # upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet
+    # will not proceed past an unready pod, so a bad render costs one node, not the cluster.
+    # This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is
+    # no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH).
+    # Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and
+    # it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster
+    # grows enough that serialised CNI upgrades become an operational problem.
+    updateStrategy:
+      type: RollingUpdate
+      rollingUpdate:
+        maxUnavailable: 1
     # NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets
     # replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field
     # (Flux/prod may differ). Only the [parity] values above must match the bootstrap.

platform/cilium/values.yaml

@@ -6,16 +6,20 @@
 #
 # SOURCE OF TRUTH (all [parity] fields MUST equal these):
 #   infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
-#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version: 1.20.1, k3s_api_host/port)
+#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version: 1.20.1)
+#   infra/ansible/inventory/group_vars/all.yml     (k3s_apiserver_endpoint = the kube-vip VIP,
+#                                                   k3s_api_port) — NOT k3s_api_host, which is the
+#                                                   founder's own address and bootstrap-only
 #
 # chart version (helmrelease.yaml chart.spec.version): 1.20.1   # [parity] = cilium_version
 
 # [parity] k3s kube-proxy disabled → Cilium replaces it (ADR-0001/0002)
 kubeProxyReplacement: true
-# [parity] = the k3s API endpoint (primary control-plane node IP) + port.
+# [parity] = the k3s API endpoint + port. THE CONTROL-PLANE VIP (kube-vip), not the primary node —
+# changed 2026-09-06; the Ansible half is k3s_apiserver_endpoint, NOT k3s_api_host.
 # Mirrors the HelmRelease's ${K3S_API_HOST} cluster-vars token (R-ENV-02). This file is NOT a
 # kustomize resource, so nothing substitutes it — the token is shown with its dev value beside it.
-k8sServiceHost: "${K3S_API_HOST}"   # dev: 192.168.1.50
+k8sServiceHost: "${K3S_API_HOST}"   # the CONTROL-PLANE VIP — dev: 192.168.1.49 | prod: 192.168.2.9
 k8sServicePort: 6443
 # [parity] ingress via Gateway API, no Traefik (docs/09)
 gatewayAPI:
@@ -42,6 +46,16 @@ cgroup:
   autoMount:
     enabled: false
   hostRoot: /run/cilium/cgroupv2
+# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
+# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
+# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
+# ClusterIP 10.43.0.1:443, translatable only by the very agent being replaced). A CNI upgrade
+# has no healthy-peer fallback, least of all on a cold prod bootstrap; maxUnavailable: 1 stops
+# the rollout at the first unready pod. Cost: serialised, slower upgrades (fine at 3 nodes).
+updateStrategy:
+  type: RollingUpdate
+  rollingUpdate:
+    maxUnavailable: 1
 # NON-parity (Flux/prod may differ; dev sets 1 to save RAM) — omitted from the HelmRelease:
 #   operator.replicas
 #

platform/coredns/coredns-custom.yaml

@@ -29,9 +29,9 @@ metadata:
 data:
   # Filename `home.server` -> imported as a standalone server block for the whole zone.
   home.server: |
-    home.keithjackson.dev:53 {
+    ${BASE_DOMAIN}:53 {
         errors
-        # Answer EVERY name in the zone (apex + all `<svc>.home.keithjackson.dev`) with the Gateway
+        # Answer EVERY name in the zone (apex + all `<svc>.${BASE_DOMAIN}`) with the Gateway
         # VIP. The template plugin's default match is `.*`, so this covers current and FUTURE app
         # hostnames without editing this file per service. `{{ .Name }}` echoes the queried FQDN.
         template IN A {

platform/crafty/deployment.yaml

@@ -125,7 +125,7 @@ spec:
             - --http-address=0.0.0.0:4180
             - --reverse-proxy=true
             - --provider=oidc
-            - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+            - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
             # Gate the co-located Crafty UI. Crafty serves HTTPS with a SELF-SIGNED cert, so skip verify
             # on this in-pod (127.0.0.1) hop; the browser-facing TLS is the trusted wildcard at the Gateway.
             - --upstream=https://127.0.0.1:8443
@@ -138,9 +138,9 @@ spec:
             - --allowed-group=platform-admins
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
-            - --redirect-url=https://crafty.home.keithjackson.dev/oauth2/callback
+            - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.home.keithjackson.dev
+            - --cookie-domain=.${BASE_DOMAIN}
-            - --whitelist-domain=.home.keithjackson.dev
+            - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true
             - --set-xauthrequest=true