feat(prod): kernel fix + disk layout — prod nodes ready for k3s

Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled changes, all applied and verified on all three nodes. === 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have === A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on. roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's Debian release, writes a deb822 trixie-backports source, installs, reads back the exact version, writes a provenance/rollback note to the node. Scoped by node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen over a hostname test so a fourth prod node inherits it from group membership. Verified: --check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest` not `present` (linux-image-amd64 was already installed from main, so `present` would have silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic + ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF automatic security upgrades for the very kernel being installed). Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable. THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it 2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same address, handled correctly; not a lucky draw. §0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was corrected: >=6.13 has three rejection gates and only the third prints that line; all three nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing /boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither exists in 6.12. TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown: - MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node 2's real value was -1569980416, which did not match, so state fell through to `clean` and the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the exact bug the check exists to find. Now (-?\d+). - bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever. Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above. === 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded === Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe). Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory. Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on the first re-image and the stale value's failure mode is mkfs on the WRONG partition. Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr /var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat. /srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used" was ext4's default 5% root reserve). === 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd === Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB. Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled /var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster death, not degradation, arriving during bring-up. Invisible when the disk decision was made, because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var. Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and "unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make a stranger's `data` volume look already-claimed and weaken an assert. The transition off the existing LABEL=longhorn mount is DECLARED, not inferred (node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness assert fails closed on dev's 18 volumes even if both were defeated. k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back on the OS filesystem invisibly until the disk filled. LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn, prod /data/longhorn. DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn volumes on that path. All 12 substituting Flux paths rendered through `kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing the same harness detects prod changing on exactly one line — an all-identical result is worthless if the harness cannot see change. Live check after: dev's default-data-path setting still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready. Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks` selected the include and ran nothing inside it while reporting success. OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node 3's decided resolvers will silently revert. Gate: MERGE GATE: PASS. L0 substitution check 106 assertions. Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 00:39:10 -05:00
1 parent 8e913fdd7b
commit 05baae90ec
@@ -74,6 +74,17 @@ data:
  # --- storage (ADR-0003) ---
  LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
  LONGHORN_DATA_PATH: "/var/lib/longhorn"
  # --- observability sizing (docs/10) ---
  MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long
@@ -190,6 +190,16 @@ patches:
      - op: replace
        path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
        value: "3"
      # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
      # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
      # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
      # else would orphan every one of them (they are not recoverable from the old path by
      # rescheduling). Restated explicitly even though it equals the base default, exactly like the
      # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
      # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
      - op: replace
        path: /data/LONGHORN_DATA_PATH
        value: "/var/lib/longhorn"
      # smaller retention / volumes to keep the dev workstation footprint light.
      - op: replace
        path: /data/MONITORING_PROMETHEUS_RETENTION
@@ -157,6 +157,20 @@ patches:
      - op: replace
        path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
        value: "3"
      # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
      # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
      # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
      # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
      # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
      # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
      #
      # [parity] MUST equal node_baseline_longhorn_data_path in
      # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
      # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
      # path on the OS filesystem instead of the data partition and /var fills up.
      - op: replace
        path: /data/LONGHORN_DATA_PATH
        value: "/data/longhorn"
      # longer retention / larger volumes on prod NVMe.
      - op: replace
        path: /data/MONITORING_PROMETHEUS_RETENTION
@@ -559,7 +559,7 @@ nothing ever transmits `new_password` to firmware. It is write-only into a buffe
   password in cleartext on the command line — it would land in process listings and shell history
   unless wrapped. Store it in the Ansible vault **and** offline.
-## 2. Disk layout — **OPEN DECISION, blocks the build**
+## 2. Disk layout — **DECIDED (operator, 2026-09-03) · REVISED (operator, 2026-09-04)**
 `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
 further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
@@ -576,6 +576,218 @@ sda  477G  C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
 └─sda5 433G  /srv          <- the only large free space
 ```
 ### THE DECISION (operator, 2026-09-03; layout REVISED 2026-09-04) — made, applied, and measured-not-assumed
 | | |
 |---|---|
 | **Data partition** | the **existing `/srv` partition** on each node's single physical disk (~432 GB nodes 1–2, ~422 GB node 3), re-purposed, re-labelled **`k3s-data`** and mounted at the neutral **`/data`** |
 | **Longhorn store** | **`/data/longhorn`** — a *subdirectory* of that partition |
 | **k3s `--data-dir`** | **`/data/k3s`** — the *sibling* subdirectory. Holds **embedded etcd AND the containerd image store** |
 | **etcd** | **no dedicated device** — it lives at `/data/k3s/server/db`, i.e. on the data partition, **not** on the OS filesystem |
 | **Shared-device I/O contention** | **ACCEPTED, knowingly** — unchanged by the 2026-09-04 revision |
 #### THE 2026-09-04 REVISION, AND THE CAPACITY EVIDENCE THAT FORCED IT
 The 2026-09-03 ruling gave this partition to Longhorn **only** and left k3s at its default
 `/var/lib/rancher/k3s`. That was wrong, for a reason that is arithmetic rather than judgement.
 **`/var` on these nodes is a separate partition of 11.8 GiB** (11.6 on node 3) — see the
 partition map above; it is `/dev/sda3` / `/dev/nvme0n1p3`, not the 30 GiB `/`. k3s's data dir
 puts **two** things on it: etcd's database and WAL (`server/db`) and, far larger, the
 **containerd image store** (`agent/containerd`).
 **Measured on the fully-deployed dev cluster** — the same platform and app set prod will run:
 | | |
 |---|---|
 | `agent/containerd` per node | **21–25 GB** |
 | total `/var` per node | **58–70 GB** |
 | prod `/var` capacity | **11.8 GiB** |
 So prod would have **filled `/var` during its first image pulls** — not eventually, not under
 load, but on the way up. And because etcd was on that same partition, a full `/var` means
 **etcd cannot write its WAL**. That is cluster death, not degradation: a wedged etcd on all
 three servers is an unavailable control plane, and it would have happened during bring-up.
 **The fix is to move k3s's whole data dir onto the 424 GB partition**, which is why that
 partition's mountpoint and filesystem label are no longer Longhorn-specific — it now serves
 **two tenants**. `/var` retains only the OS's own usage (503 MB, 5% used, measured 2026-09-04).
 **Why now.** Nothing was installed on prod when this was decided — verified 2026-09-04: no
 `/usr/local/bin/k3s`, no `/etc/rancher/k3s`, no `/var/lib/rancher` on any node. Moving a data
 dir before first install is a one-line inventory change; after bring-up it means draining and
 re-imaging three servers.
 **Why `/data` and `LABEL=k3s-data`.**
 - **`/data`** — short, neutral, tenant-agnostic, and *not* under `/var`. Deliberately **not
  `/srv`**: `/srv` is the *claim mountpoint* the role discovers the device at, and the
  idempotency contract is that `/srv` is gone from `/proc/mounts` and `/etc/fstab` once the
  claim has happened. Re-using it as the destination would make "claimed" and "not yet
  claimed" indistinguishable.
 - **`LABEL=k3s-data`** — deliberately **not** the bare `data`. Guard #4 of the role's assert
  wall accepts a partition whose label is empty *or already ours*, and names `data` as an
  example of somebody else's named volume; labelling ours `data` would make a stranger's
  `data` volume look already-claimed and weaken that guard. `k3s-data` is cluster-scoped
  rather than tenant-scoped, and matches the `k3s-etcd` label convention already in the repo.
 **The shared-device I/O risk is UNCHANGED and still accepted.** Longhorn and etcd were already
 going to share this spindle under the 2026-09-03 ruling — the revision changes *which*
 filesystem etcd sits on, not *which device*. The acceptance rationale and its exit condition
 below stand verbatim.
 **How prod's data dir is scoped away from dev.** `k3s_data_dir` is a **single** variable read
 by **two** roles: `roles/k3s_server` turns it into `--data-dir` in the install flags, and
 `roles/node_baseline` (via `node_baseline_k3s_data_dir`, which defers to it) creates the
 directory `0700` on the data partition and derives etcd's location from it
 (`node_baseline_etcd_mount` = `<data-dir>/server/db`). The role default is **k3s's own
 default**, `/var/lib/rancher/k3s`, so **dev is unchanged** — and dev also has a *dedicated
 etcd device* mounted at `server/db` under that path, which moving dev's data dir would orphan.
 Prod overrides it in one line in `inventory/group_vars/k3s_prod/vars.yml`, the same seam
 `kube_vip_address` / `k3s_api_host` / `node_baseline_backports_kernel` use, so a fourth prod
 node inherits it from group membership rather than from anyone remembering.
 **Longhorn's `defaultDataPath` follows the R-ENV-02 knob pattern, not a base edit.**
 `platform/longhorn/helmrelease.yaml` referenced a hardcoded `/var/lib/longhorn`; it now
 references `${LONGHORN_DATA_PATH}`, a `cluster-vars` knob with a base default and an explicit
 value in **both** overlays:
 | | |
 |---|---|
 | `clusters/base/cluster-vars.yaml` | `/var/lib/longhorn` (base default = the known-good dev value, same convention as `GATEWAY_VIP`) |
 | `clusters/dev` | `/var/lib/longhorn` — **must never change**; the live dev cluster has 18 volumes whose replicas are on disk under that path |
 | `clusters/prod` | `/data/longhorn` |
 `kustomize-controller` runs with `--feature-gates=StrictPostBuildSubstitutions=true`, so an
 undefined variable fails the Kustomization outright — hence base-first, then both overlays.
 `[parity]`: the prod value MUST equal `node_baseline_longhorn_data_path` in
 `group_vars/k3s_prod/vars.yml`. Ansible creates and mounts the directory; Helm tells Longhorn
 to use it. If the two disagree, Longhorn writes replicas onto the OS filesystem and `/var`
 fills up — the exact failure this revision exists to prevent.
 **Verified 2026-09-04 that dev's rendered output is byte-identical.** All 12 Flux
 Kustomizations that declare `postBuild.substituteFrom cluster-vars` were rendered
 (`kubectl kustomize <spec.path> | flux envsubst --strict`, substituting the dev overlay's
 *effective* cluster-vars) both before and after the change: 12/12 identical per-file SHA-256,
 and one aggregate SHA-256 over every rendered byte, `b34c5d3d…89e7d`, unchanged. The only
 dev-side difference anywhere is the presence of the new knob in the ConfigMap itself, carrying
 dev's existing literal. The same harness shows prod's render changing on exactly one line
 (`defaultDataPath: /var/lib/longhorn` → `/data/longhorn`), which is what proves the comparison
 could have detected a regression.
 **Per-node device paths — they are NOT the same, and that is why nothing hardcodes them.**
 Verified 2026-09-03 with `findmnt --mountpoint /srv` on each node:
 | Node | Boot media | `/srv` partition | Size | Contents before the claim |
 |---|---|---|---|---|
 | `k3s-prod-1` | SATA SSD `C400-MTFDDAC512M` | `/dev/sda5` | 432.5 GiB ext4 | `lost+found` only (`du -sh` 20K; `df` 2.1 M of 425 G) |
 | `k3s-prod-2` | **NVMe** `PM9A1 Samsung` | **`/dev/nvme0n1p5`** | 432.5 GiB ext4 | `lost+found` only (same) |
 | `k3s-prod-3` | SATA SSD `WDC WDS500G2B0A` | `/dev/sda5` | 422.2 GiB ext4 | `lost+found` only (same) |
 The `df` "5.1% used" that made `/srv` look occupied is **ext4's default 5% root-reserved
 blocks**, not data — confirmed per node by `ls -A /srv` (only `lost+found`) and
 `du -sh --exclude=lost+found /srv` (20 K). Nothing was destroyed.
 **How the role resolves the device — discovery, not per-host inventory.** A single
 `k3s_prod` group value cannot be right when node 2 is NVMe, and per-host `host_vars` would
 freeze a *fact about today's hardware* into inventory, where it goes stale the first time a
 node is re-imaged or a disk is swapped — and the stale value's failure mode is `mkfs` on the
 wrong partition. So `group_vars/k3s_prod/vars.yml` declares **intent**
 (`node_baseline_longhorn_source: claim-mountpoint`,
 `node_baseline_longhorn_claim_mountpoint: /srv`) and
 `roles/node_baseline/tasks/disks.yml` asks the **node** which block device is mounted there,
 then proves the answer is safe before writing anything. Six independent asserts, all
 fail-closed: it is a partition or whole disk (not tmpfs/NFS/bind), it is **not** backing
 `/`, `/boot`, `/boot/efi`, `/usr`, `/var`, `/home`, `/tmp` or active swap (that set is built
 from the node's *own* gathered facts plus `swapon`, never a hardcoded partition list), it is
 already ext4, it carries no other filesystem label, it is ≥ 100 GiB, and it is **empty**.
 **Unmount → relabel, not reformat.** `/srv` is already formatted and already mounted, and
 `community.general.filesystem` will not touch a device that has a filesystem — it would
 silently no-op and then the `LABEL=longhorn` mount would fail. The role therefore unmounts
 `/srv`, removes its `/etc/fstab` entry (installer-written, pinned by UUID — leaving it would
 re-mount the partition at `/srv` underneath Longhorn on the next boot), and stamps the label
 with `e2label`. Relabelling rather than reformatting is deliberate: the partition is already
 ext4 and already verified empty, so `mkfs` buys nothing, and if some assert were ever too
 permissive the worst case is a renamed filesystem instead of a destroyed one.
 **Idempotency guard.** A second run is a no-op, not a reformat. After the claim the
 partition carries the ext4 label `k3s-data` and `/srv` is gone from `/proc/mounts` *and*
 `/etc/fstab` — so "what is mounted at `/srv`" would find nothing on run 2. The role pivots
 on the durable on-disk evidence instead: `blkid -L k3s-data` returning a device means the
 node is already claimed, and **every** destructive task is gated on that returning nothing.
 **The 2026-09-04 transition, handled idempotently.** All three nodes had *already* been
 claimed under the previous naming (`LABEL=longhorn` at `/var/lib/longhorn`). Because the
 idempotency pivot asks "does the **current** label exist?", changing the label makes an
 already-claimed node answer *no* — and it would then walk into the greenfield claim path and
 fail closed at "the claim mountpoint must actually be a mountpoint", `/srv` having been
 consumed by that first claim. Fail-closed is correct, but it is not convergence, so the
 transition is **declared** rather than inferred:
 `node_baseline_data_retired_fs_label: longhorn` + `node_baseline_data_retired_mount:
 /var/lib/longhorn` in `group_vars/k3s_prod/vars.yml`. The role then unmounts the retired
 mountpoint, drops its `fstab` entry and re-stamps the label with `e2label` — a **metadata
 write, never a reformat**, exactly as in the claim path — having first proved that the retired
 mountpoint is still a mount, that it is backed by the very device the retired label names, and
 that it is **empty** so nothing is orphaned. The common `LABEL=`-based mount task then brings
 it up at `/data`, and the two per-tenant subdirectories are created (`0755` Longhorn, `0700`
 k3s, since the latter holds the etcd DB, the node token and the image store).
 The transition block is **one-time and self-retiring**: on every subsequent run
 `blkid -L k3s-data` succeeds and none of it is reachable. The two retired-* lines are safe to
 delete from `group_vars` once the fleet is converged (it is, as of 2026-09-04).
 **Three independent reasons the LIVE dev cluster cannot enter that transition** — it has 18
 Longhorn volumes on `/var/lib/longhorn` and unmounting it would orphan every replica:
 (1) both retired-* vars default to `""` and are set only in the prod-scoped `group_vars`;
 (2) the path is gated on `longhorn_source == 'claim-mountpoint'`, and dev is `device` — the
 same structural gate that keeps dev out of the claim path; (3) the emptiness assert, which
 fails closed on a partition with replica directories even if (1) and (2) were both defeated.
 Dev's mountpoint, label and k3s data dir are also simply **not changing**.
 **Run evidence (2026-09-04).** Full `node-baseline.yml` against `k3s_prod`: run 1 converged
 (5–6 changed per node), run 2 **`changed=0` on all three nodes** with every transition and
 claim task skipping. Per node afterwards: `blkid -L k3s-data` → `/dev/sda5` (nodes 1, 3) /
 `/dev/nvme0n1p5` (node 2); `blkid -L longhorn` → gone; `/data` mounted with 403 G / 394 G
 available; `/var/lib/longhorn` and `/srv` no longer mountpoints; `fstab` carrying exactly
 `LABEL=k3s-data /data ext4 defaults,nofail,x-systemd.device-timeout=5s 0 0`; and `/var` down
 to **5% used, ~11 GiB free**.
 **Dev can never reach this path.** Both role defaults stay `source: device`, so the LIVE
 dev cluster keeps its dedicated `/dev/vdb` (Longhorn) and `/dev/vdc` (etcd) and the claim
 tasks are structurally unreachable there — they are gated on `claim-mountpoint`, which only
 the prod-scoped `group_vars` sets.
 ### The accepted risk, and its exit condition
 Dev was burned by exactly this shape: a Longhorn rebuild starved etcd WAL fsync
 (`docs/plans/ssd-etcd-isolation.md`), which is why dev has a third, dedicated etcd device.
 **Prod accepts the same shared-device contention** because the failure mode is materially
 cheaper here: dev's contention was on a **spinning HDD** with seek-bound queueing; every
 prod node is **SSD or NVMe** end-to-end, where concurrent rebuild I/O costs latency rather
 than collapsing it. The three nodes also have no second device to give etcd — one physical
 disk each, already fully partitioned, no free slot verified populated.
 **Accepted to be MEASURED, not assumed.** Exit condition, in order of preference:
 - **Measure it before it matters.** Under a deliberate Longhorn replica rebuild, watch
  `etcd_disk_wal_fsync_duration_seconds` p99 and `etcd_disk_backend_commit_duration_seconds`
  p99, plus `etcd_server_leader_changes_seen_total`. If fsync p99 stays inside etcd's
  practical budget (~10 ms wal fsync / ~25 ms backend commit) with no leader churn, the
  acceptance holds and this becomes a recorded measurement rather than an assumption.
 - **If it does not hold:** add an M.2 NVMe per node (chassis slot to be verified
  physically) and flip `node_baseline_etcd_source` back to `device` — the role still
  supports that path unchanged, and it is the dev shape.
 **~~Known, separate consequence — `/var` is only 12 GiB.~~ RESOLVED 2026-09-04 by moving
 k3s's data dir.** This was recorded here on 2026-09-03 as a capacity item "to watch or
 repartition". Re-measuring it against dev turned it from a watch item into a blocker: the
 container image store alone is **21–25 GB per node** on the fully-deployed dev cluster and
 `/var` totals **58–70 GB**, against prod's **11.8 GiB**. So prod would have filled `/var`
 during its first image pulls, and with etcd on the same partition a full `/var` means etcd
 cannot write its WAL. The fix was neither "watch" nor "repartition" but `--data-dir` — see
 "THE 2026-09-04 REVISION" above. `/var` now carries only the OS's own usage (503 MB, 5%).
 Consequences to settle before imaging nodes 2 and 3:
 - [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
@@ -609,24 +821,43 @@ Consequences to settle before imaging nodes 2 and 3:
      a member corrupts — this drive has already logged 32 of them.
      (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
      `DECISIONS-LOG.md`. It is not part of this verdict.)
- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
+- [x] **Layout DECIDED and APPLIED 2026-09-03** — see "THE DECISION" above. `roles/node_baseline`
-      (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
+      no longer expects a block-device path for either store; it takes a declared **source
-      Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
+      enum** per environment (`roles/node_baseline/tasks/disks.yml`):
-      prod must override them in `group_vars`. On a single-disk node there is no second device at
+      - **Longhorn** — `node_baseline_longhorn_source: claim-mountpoint` +
-      all, so one of:
+        `node_baseline_longhorn_claim_mountpoint: /srv`. The `/dev/vdb` default (a virtio
-      - **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
+        name that does not exist on bare metal) is unused on prod; the real partition is
-        SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
+        discovered on the node and guarded, because it is `/dev/sda5` on nodes 1 and 3 and
-        spindle.
+        `/dev/nvme0n1p5` on node 2.
-      - **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
+      - **etcd** — `node_baseline_etcd_source: shared-data-partition` (was `root-filesystem`
-        Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
+        until 2026-09-04). "No dedicated device" is a first-class, explicitly-configured state
-      - **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
+        that the `disks` assert **accepts**, not a blank variable that the tasks happen to
-        physical failure domain.
+        skip — and the *name* of that state now describes reality, because etcd is no longer
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
+        on the OS filesystem. The distinction is load-bearing, not cosmetic:
- [ ] Record the chosen device paths here and set them in
+        `shared-data-partition` is **asserted**, per node on every run, by checking that the
-      `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
+        filesystem backing the etcd data dir really is the data partition's mount. Declaring
-      `/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
+        `k3s_data_dir: /data/k3s` does not by itself achieve that — had the mount silently
-      `node-baseline` run against a node still carrying those placeholders now stops on an explicit
+        failed, `/data` would be a plain directory on `/` and etcd plus 21–25 GB of images
-      assert naming this section, instead of failing obscurely at `mkfs` time.
+        would land back on the OS filesystem, invisibly, until the disk filled. Leaving the
        value at `root-filesystem` would have been a lie inventory told about the cluster, and
        `root-filesystem` makes no such claim and gets no such assert. The role also still
        asserts nothing is already mounted at the etcd data dir (so a half-migrated node
        cannot be inherited silently) and reports which filesystem etcd actually lands on
        (measured 2026-09-04: `/data`, 403 G / 394 G available, vs `/var`'s 10.7 GiB).
      - The two rejected options stand rejected: **add a second drive** (no verified free slot;
        the exit condition above reopens it if measurement fails) and **repartition** (same
        single failure domain, plus a destructive re-image of three working nodes for no
        isolation the SSDs do not already give).
 - [x] **All three nodes get the identical layout.** Same declaration, same mountpoint, same
      resulting `/data` + `/data/longhorn` + `/data/k3s`. Not the same *device path* and not
      the same *size* (432.5 / 432.5 / 422.2 GiB) — Longhorn tolerates unequal node capacity;
      it is replica *placement* that assumes a symmetric layout, and the layout is symmetric.
      Confirmed per node 2026-09-04 by `findmnt`/`lsblk`/`blkid` after the transition.
 - [x] **Chosen paths recorded here and set in `group_vars`.** The BLOCKED placeholders in
      `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` are replaced — by the source
      enums above, plus deliberately-unusable `/dev/NOT-A-DEVICE-…` values for the two device
      variables so that flipping a source back to `device` without supplying a real path
      still fails on the role's assert instead of silently inheriting dev's `/dev/vdb`.
 - [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
      is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
      tasks this decision blocks — see §5's drive-health box for the exact command.
@@ -639,6 +870,138 @@ Consequences to settle before imaging nodes 2 and 3:
 - [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
      `roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
      so the same role covers bare metal. Nothing to port.
 - [x] **KERNEL: a standing, deliberate deviation from the distro baseline — `linux-image-amd64`
      from `trixie-backports` (DONE on all three nodes, 2026-09-04).**
      **What.** `roles/node_baseline` gained a `kernel`-tagged block that enables
      `trixie-backports` as a deb822 apt source and installs the `linux-image-amd64`
      **metapackage** from it. Applied 2026-09-04; all three nodes landed on
      **`linux-image-amd64` 7.1.8-1~bpo13+1 → kernel `7.1.8+deb13-amd64`**, rebooted one at a
      time, and are running it.
      ```bash
      ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
      ```
      **Why.** §0b. HP's R21 firmware publishes a corrupt EFI Memory Attributes Table; Linux
      < 6.13 reserves memory from it without validating the size, and `k3s-prod-2` has booted
      with 2.6 GiB of its 16 GB, twice. The fix is upstream commit **`8fbe4c49c0cc`**
      *"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, first
      released in **v6.13**, hardened by `5d0faa8e8369`). It is absent from v6.12 and from
      `linux-6.12.y`, carries neither `Cc: stable` nor `Fixes:`, and there is **no kernel
      cmdline workaround**. A k3s **server** that intermittently comes up with 2.6 GiB will
      fail under etcd plus its workload share on any reboot, so this is a **prerequisite to
      installing k3s**, not a follow-up.
      **How it is scoped.** `node_baseline_backports_kernel` defaults to **false** in
      `roles/node_baseline/defaults/main.yml` and is set **true** exactly once, in
      `inventory/group_vars/k3s_prod/vars.yml` — the same seam `kube_vip_address` /
      `k3s_api_host` / `cluster_kubeconfig_name` use. Dev's Proxmox VMs publish no MAT and are
      genuinely unaffected; they must not be dragged onto a backports kernel as a side effect
      of sharing the role, and a fourth prod node must inherit the deviation from **group
      membership**, not from a `k3s-prod-*` hostname test. Verified: a `--tags kernel` run
      against dev skips all 8 tasks on all 3 hosts.
      **Security cost, accepted.** `*-backports` packages are provided **as-is** and are
      **not** covered by the Debian Security Team's guarantees for the main archive — no
      security-tracker commitment, no DSAs. Recorded here as the price of the nodes keeping
      their RAM.
      **No apt pin, deliberately.** The backports `Release` file carries `NotAutomatic: yes`
      + `ButAutomaticUpgrades: yes`, which apt already reads as *"priority 100 so nothing is
      taken from backports by accident, 500 for a package explicitly installed from there so
      it keeps getting updates."* Writing an explicit `Pin-Priority: 100` in
      `preferences.d` would look like belt-and-braces and would in fact **switch off the
      automatic security upgrades for the very kernel this deviation installs.** Enabling the
      source is the whole configuration.
      **Metapackage, not a pinned ABI.** A frozen `linux-image-6.19.14+deb13-amd64` would stop
      receiving even backports' own updates the moment the ABI rolls. The metapackage tracks
      whatever backports ships (>= 6.13 by construction). The **exact** version each node
      landed on is recorded on the node itself at `/etc/node-baseline-backports-kernel`,
      written by the same role, along with the rollback path below.
      **Rollback.**
      1. *Non-destructive, try first.* The stock kernels are still installed
         (`6.12.107+deb13-amd64` and `6.12.94+deb13-amd64` on all three) and selectable from
         GRUB → *Advanced options for Debian GNU/Linux*. That answers "is the new kernel the
         problem?" without uninstalling anything — it also reinstates the memattr bug for that
         boot.
      2. *Permanent.* Set `node_baseline_backports_kernel: false` in
         `group_vars/k3s_prod/vars.yml` **first** (otherwise the next Ansible run puts it
         back), then on each node:
         ```bash
         apt-get install --reinstall linux-image-amd64/trixie
         apt-get purge linux-image-7.1.8+deb13-amd64
         rm /etc/apt/sources.list.d/trixie-backports.sources
         apt-get update && update-grub && reboot
         ```
      **The role does NOT reboot**, and that is deliberate — see §0b "SECOND FAULT". Reboots
      are a serialized, supervised operator step: one node at a time, verify each returns
      before touching the next, never node 2 first. On 2026-09-04 all three returned in
      **39–41 s** (node 1 39 s, node 3 41 s, node 2 39 s); node 2 did **not** reproduce its
      2026-09-03 hang, but one clean warm reboot is not a fix and that defect stays open.
      **Exit condition — how this deviation ends.** If `8fbe4c49c0cc` is ever picked up into
      `linux-6.12.y` (and therefore into Debian's 6.12 rebase, which reaches us by plain
      `apt upgrade`), this can be removed entirely: set `node_baseline_backports_kernel:
      false`, roll back per the note on each node, and delete the `kernel` block from
      `roles/node_baseline`. §0b records why that pickup will not happen on its own and what
      request would cause it.
 - [x] **Verification of the fix, and a correction to how it is verified (2026-09-04).**
      §0b says *"A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table
      detected!`"*. That is true but **conditional, and it did not happen on any of the three
      nodes.** `efi_memattr_init()` on >= 6.13 has **three** gates and only the third prints
      that line:
      | gate | message | reachable when |
      |---|---|---|
      | `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
      | implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, `desc_size` does not |
      | `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* `desc_size` read plausibly, entry count does not |
      **All three are rejections**, and which one a boot prints is decided by *which field*
      that boot's garbage corrupts. On 7.1.8 all three nodes hit the **first** gate:
      | node | `MEMATTR=` | kernel line on 7.1.8 | MemTotal | reserved at boot |
      |---|---|---|---|---|
      | 1 | `0xbee84018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1030` | 31.1 GiB | 0.9 GiB |
      | 2 | `0xbeeb0018` | `efi: memattr: Unexpected EFI Memory Attributes table version -1569980416` | 15.4 GiB | 0.6 GiB |
      | 3 | `0xbeecc018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` | 31.1 GiB | 0.9 GiB |
      Node 2 is the result that matters: **`0xbeeb0018` is the exact address that produced
      2.6 GiB twice**, and on 7.1.8 it reserved 634,904 K instead of 13,913,856 K.
      **Boot-independent proof the fix is compiled in**, since the log line is a lottery: the
      running kernel image itself contains the string. Decompressing `/boot/vmlinuz-$(uname
      -r)` on all three nodes yields, verbatim —
      ```
      efi: memattr: [Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)
      efi: memattr: Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)
      ```
      — neither of which exists in 6.12. That is the positive proof; the `dmesg` line is not
      available on demand.
      **Two bugs in `roles/node_health` were fixed as a consequence**, because without them
      the health check could not show the fix landing:
      - `MEMATTR_VERSION_RE` in `files/node_health_collect.py` matched `(\d+)` only. The
        kernel prints that field with `%d` — **signed** — so node 2's `-1569980416` did not
        match, the state fell through to `clean`, and the node was reported **PASS "the kernel
        accepted it without complaint"**: a false pass on the exact bug the check exists to
        find. Now `(-?\d+)`.
      - `firmware.efi_memattr` treated `bad_version` as WARN unconditionally, so a node on a
        fixed kernel would have stayed WARN forever. The verdict now keys on the running
        kernel version (`memattr_kernel_fixed`, `>= 6.13`): the same log line is WARN on
        < 6.13 (a lucky boot) and PASS on >= 6.13 (the validation working). The pre-existing
        `corrupt_rejected` PASS branch is unchanged and remains reachable.
      **`playbooks/node-health.yml -e node_health_hosts=k3s_prod`, after the reboots:
      `firmware.efi_memattr` = PASS on all three, `memory.physical_total` = PASS on all
      three.** Dev re-run unchanged.
 ## 4. Network (per node)
@@ -87,24 +87,155 @@ kube_vip_address: "192.168.2.9"
 # — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
 cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
-# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
+# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
-# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
+# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
-# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
+# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
-# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
+# backports kernel as a side effect of sharing the role), and this line switches it on for
-# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
+# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
-# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
+# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
-# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
+# rather than from anyone remembering.
 #
-# The overrides below are deliberately present-but-broken: leaving them UNSET would
+# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
-# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
+# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
-# hardware — a confusing failure that looks like a wiring bug. This value fails just
+# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
-# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
+# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
-# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
+# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
-# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
+# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
-# updates these two lines. All three nodes must end up with an identical layout
+# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
-# (Longhorn replica placement assumes symmetry) before any of this is used for real.
+# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
-node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
-node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
+# in §3.
 #
 # COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
 # guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
 # the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
 # back to false, roll back per the note the role writes to
 # /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
 # roles/node_baseline.
 #
 # The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
 # FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
 node_baseline_backports_kernel: true
 # --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
 # Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
 #
 #   DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
 #                     disk, claimed and mounted at the NEUTRAL mountpoint `/data`
 #                     (LABEL=k3s-data), then split between TWO tenants:
 #                       /data/longhorn   Longhorn's dataPath
 #                       /data/k3s        k3s --data-dir (etcd AND the containerd image store)
 #   ETCD           -> NO dedicated device. It lives inside /data/k3s on that partition.
 #
 # ================================ THE 2026-09-04 REVISION ================================
 # The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
 # /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
 # data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
 # deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
 # would therefore have filled /var during its FIRST image pulls — and with etcd on the same
 # partition, a full /var means etcd cannot write its WAL. That is cluster death, not
 # degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
 # mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
 # tenants. Nothing was installed on prod when this was decided, which is exactly why it was
 # decided then — it is cheap now and very expensive after bring-up.
 #
 # THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
 # were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
 # contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
 # MEASURED, not assumed — exit condition in §2.
 # =========================================================================================
 #
 # Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
 # not as a device path and a blank. That distinction is the point of this block:
 #
 # 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
 #    k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
 #    (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
 #    Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
 #    freezes a fact about today's hardware into inventory, where it goes stale silently the
 #    first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
 #    nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
 #    So inventory declares the INTENT ("the partition currently mounted at /srv") and the
 #    role discovers the device on the node, then proves it is safe to claim (a partition or
 #    disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
 #    unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
 #
 # 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
 #    describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
 #    layout etcd is inside /data/k3s on the claimed data partition, which is neither a
 #    dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
 #    Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
 #    and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
 #    on every run, that the filesystem backing the etcd data dir really is /data. If the
 #    mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
 #    WAL would land back on the OS filesystem, invisibly, until the disk filled. The
 #    `root-filesystem` value makes no such claim and gets no such assert — which is exactly
 #    why the two are distinct enum values rather than one with a comment.
 #
 # 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
 #    the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
 #    the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
 #    this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
 #    k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
 #    Longhorn volumes are untouched.
 node_baseline_longhorn_source: claim-mountpoint
 node_baseline_longhorn_claim_mountpoint: /srv
 node_baseline_etcd_source: shared-data-partition
 # --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
 # `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
 # CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
 # /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
 # as the destination would make "claimed" and "not claimed" indistinguishable.
 # `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
 # partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
 # else's named volume — so labelling ours `data` would make a stranger's `data` volume look
 # already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
 # `k3s-etcd` label convention already in this repo.
 node_baseline_data_mount: /data
 node_baseline_data_fs_label: k3s-data
 # Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
 # equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
 # side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
 # default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
 node_baseline_longhorn_data_path: /data/longhorn
 # --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
 # The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
 # `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
 # label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
 # would walk into the greenfield claim path — which fails closed at "the claim mountpoint
 # must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
 # is right but it is not convergence, so the transition is DECLARED here: the role unmounts
 # the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
 # the partition is empty (nothing orphaned) and is the device the retired label names.
 # Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
 # SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
 # then skip on the retired-label lookup and nothing else depends on them.
 node_baseline_data_retired_fs_label: longhorn
 node_baseline_data_retired_mount: /var/lib/longhorn
 # --- k3s --data-dir (§2) -----------------------------------------------------------
 # THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
 # claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
 # `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
 # which defers to this variable) creates the directory 0700 on the data partition and derives
 # etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
 # Ansible side cannot disagree with itself.
 #
 # Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
 # it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
 # dir would orphan that mount under a running cluster.
 k3s_data_dir: /data/k3s
 # Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
 # source uses a device path any more, but leaving these UNSET would let prod silently
 # inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
 # metal) the moment anyone flipped a source back to `device`. These values can never parse
 # as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
 # before mkfs is ever reached.
 node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
 node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
 # --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
 # roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
 # the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
 k3s_disable: "traefik,servicelb,local-storage"
 # --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
 # EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
 # WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
 # image store (agent/containerd) — which is by far the largest tenant.
 #
 # THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
 # LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
 # /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
 # server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
 # running cluster's entire state directory out from under it and orphan that etcd mount.
 #
 # PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
 # kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
 # node inherits it from group membership rather than from anyone remembering. It is set
 # there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
 # turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
 # data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
 # this same variable). One line moves both.
 #
 # WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
 # separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
 # alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
 # FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
 # not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
 #
 # NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
 # INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
 # config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
 # Restart handler for a value that is not changing. The install flags are the correct and
 # sufficient place, and prod has no k3s installed yet.
 k3s_data_dir: "/var/lib/rancher/k3s"
 # Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
 k3s_server_mode: "secondary"
@@ -22,6 +22,7 @@
      --write-kubeconfig-mode=644
      {% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
      --node-ip={{ k3s_node_ip }}
      --data-dir={{ k3s_data_dir }}
 # On an ALREADY-installed cluster the install task above is idempotent (creates:
 # k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
@@ -18,22 +18,170 @@ node_baseline_packages:
  # node from first baseline — package installation is the role's job, never a manual step
  # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
  - smartmontools
 # --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
 # OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
 # baseline — not a one-off `apt install` someone has to remember on the next node.
 #
 # WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
 # Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
 # from its contents, so an unlucky boot reserves everything above the table and the node
 # comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
 # 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
 # first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
 # no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
 # kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
 # >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
 #
 # WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
 # publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
 # backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
 # overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
 # encode the environment into the ROLE, would silently miss a fourth prod node named
 # differently, and could not be flipped for a one-off test host. So it is a plain boolean,
 # default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
 # kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
 #
 # SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
 # covered by the Debian Security Team's guarantees for the main archive — no security-tracker
 # commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
 # is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
 #
 # EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
 # 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
 # delete this block. Nothing else in the repo depends on it.
 node_baseline_backports_kernel: false
 # The suite, mirror and components the source line is built from. Split out so a mirror
 # change is a variable, not an edit to a heredoc in tasks/main.yml.
 node_baseline_backports_suite: trixie-backports
 node_baseline_backports_mirror: "http://deb.debian.org/debian"
 # Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
 # contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
 # surface beyond what the node already trusts is a separate decision nobody has taken.
 node_baseline_backports_components:
  - main
  - non-free-firmware
 # The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
 # would stop receiving even backports' own updates the moment the ABI rolls, leaving the
 # node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
 # The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
 # far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
 # means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
 # is recorded on the node itself — see node_baseline_backports_kernel_stamp.
 node_baseline_backports_kernel_packages:
  - linux-image-amd64
 # Provenance/rollback note written on the node. An operator standing in front of a prod box
 # with no repo checkout must be able to answer "why is this machine not on the distro
 # kernel, and how do I undo it?" from the machine itself.
 node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
 node_baseline_kernel_modules:
  - overlay
  - br_netfilter
  - iscsi_tcp
  - dm_crypt
  - nfs
 # --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
 # The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
 # a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
 #
 #   node_baseline_longhorn_source:
 #     device           dedicated block device at node_baseline_longhorn_device (DEV: the
 #                      tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
 #     claim-mountpoint re-purpose the partition the node ALREADY has mounted at
 #                      node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
 #                      path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
 #                      and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
 #                      is correct and a hardcoded one would eventually mkfs the wrong
 #                      partition on a future node. Destructive; heavily guarded — read
 #                      tasks/disks.yml before changing anything here.
 #
 #   node_baseline_etcd_source:
 #     device                dedicated block device at node_baseline_etcd_device (DEV:
 #                           /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
 #                           isolated from Longhorn rebuild I/O after dev was burned by HDD
 #                           contention).
 #     root-filesystem       etcd lives on the OS filesystem, with NO dedicated device and
 #                           NO claimed data partition under it. Kept as a valid, supported
 #                           shape; NOT what prod uses any more (see below).
 #     shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
 #                           CLAIMED data partition (PROD). No dedicated device, but also NOT
 #                           the OS filesystem: the whole point of the 2026-09-04 layout
 #                           revision is that /var (11.8 GiB) can hold neither the container
 #                           image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
 #
 # Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
 # recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
 # var, which still fails the `disks` assert.
 node_baseline_longhorn_source: device
 node_baseline_longhorn_device: /dev/vdb
-node_baseline_longhorn_mount: /var/lib/longhorn
+# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
-node_baseline_longhorn_fs_label: longhorn
+# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
 # tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
 # store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
 # both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
 #
 # THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
 # Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
 # /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
 # at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
 # partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
 node_baseline_data_mount: /var/lib/longhorn
 node_baseline_data_fs_label: longhorn
 # Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
 # root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
 # sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
 # in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
 # same fact, and the two are checked against each other by nothing but this comment.
 node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
 # k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
 # one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
 # NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
 # install flags, this role reads the same value to create the directory and to place etcd,
 # and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
 # value (higher precedence than any role default) is what actually resolves here, even
 # though roles/k3s_server is not in the node-baseline play; the default() keeps
 # node-baseline.yml runnable standalone.
 node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
 # Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
 # default so a dev host cannot enter the claim path even by accident.
 node_baseline_longhorn_claim_mountpoint: ""
 # --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
 # A node claimed by an EARLIER version of this role carries the OLD label at the OLD
 # mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
 # "not claimed yet" and walk into the greenfield claim path — which then fails closed at
 # "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
 # two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
 # are NOT changing, so dev has nothing to transition and cannot enter the transition path.
 node_baseline_data_retired_fs_label: ""
 node_baseline_data_retired_mount: ""
 # Sanity floor for the claim path: a data partition smaller than this is not "a small
 # store", it is the wrong partition. Prod's /srv is 415-425 GiB.
 node_baseline_longhorn_min_size_gb: 100
 # Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
 # so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
 # See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
-# device renames (matches the Longhorn LABEL= approach).
+# device renames (matches the data partition's LABEL= approach).
 node_baseline_etcd_source: device
 node_baseline_etcd_device: /dev/vdc
-node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
+# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
 # the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
 # /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
 # to /data/k3s/server/db, inside the claimed partition.
 node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
 node_baseline_etcd_fs_label: k3s-etcd
 # Mounts whose backing devices the claim path must NEVER touch. Used to build the
 # protected-device set from the node's own gathered facts (plus live swap), so a node
 # partitioned differently from the fleet is judged on its own layout rather than against
 # a hardcoded list of partition names.
 node_baseline_protected_mounts:
  - /
  - /boot
  - /boot/efi
  - /usr
  - /var
  - /home
  - /tmp
 # --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
 # These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
@@ -0,0 +1,642 @@
 ---
 # =====================================================================================
 # node_baseline — DATA DISKS.  Every task here carries the `disks` tag (applied once by
 # the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
 # Split into its own file rather than living in main.yml because it is the one part of
 # the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
 #
 # TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
 #
 #   dev  (group_vars/all.yml, role defaults)
 #     node_baseline_longhorn_source: device          -> mkfs /dev/vdb, mount LABEL=longhorn
 #                                                       at /var/lib/longhorn (== dataPath)
 #     node_baseline_etcd_source:     device          -> mkfs /dev/vdc, mount LABEL=k3s-etcd
 #     k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
 #     THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
 #     exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
 #
 #   prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
 #     node_baseline_longhorn_source: claim-mountpoint  (claim mountpoint /srv)
 #     node_baseline_data_mount:      /data            (LABEL=k3s-data)
 #     node_baseline_etcd_source:     shared-data-partition
 #     k3s_data_dir:                  /data/k3s
 #     ONE physical disk per node, already fully partitioned by the Debian installer. The
 #     ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
 #     between TWO tenants:  /data/longhorn (Longhorn replicas)  +  /data/k3s (the whole k3s
 #     data dir: etcd AND the containerd image store).
 #
 # WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
 # §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
 # BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
 # cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
 # have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
 # WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
 # partition is the fix; the partition therefore serves two tenants, and neither its
 # mountpoint nor its filesystem label may name only one of them.
 #
 # WHY /data AND LABEL k3s-data.
 #   * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
 #     Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
 #     at, and the idempotency contract is that /srv is gone from /proc/mounts and
 #     /etc/fstab once the claim has happened. Re-using it as the destination would make
 #     "claimed" and "not yet claimed" indistinguishable.
 #   * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
 #     accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
 #     `data` as an example of somebody else's named volume. Labelling ours `data` would make
 #     a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
 #     cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
 #     convention in this repo.
 #
 # WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
 # "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
 # empty string, and the second one must never silently succeed. `root-filesystem` and
 # `shared-data-partition` are explicit, greppable statements of intent that the assert
 # below accepts; an unset or placeholder device is still a hard failure. The two differ in
 # a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
 # `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
 # until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
 # than printing a claim it never checked (see the etcd section).
 #
 # WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
 # nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
 # correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
 # that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
 # arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
 # point at the wrong partition on some future node and mkfs it. So the role asks the NODE
 # which block device is mounted at the claim mountpoint, and then proves that answer is
 # safe (see the assert wall below) before touching it. The inventory declares INTENT
 # ("the partition currently mounted at /srv"); the node supplies the FACT.
 # =====================================================================================
 # Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
 # its data-disk layout. Two failure shapes are caught:
 #   * an undeclared/typo'd source enum, and
 #   * source: device with a device path that is a PLACEHOLDER rather than a real device.
 # group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
 # vars now that prod uses neither: if anyone ever flips a source back to `device` without
 # also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
 # not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
 # stays in the token list because that was the pre-decision placeholder and may still be
 # carried by other environments/branches.
 - name: Assert this environment's data-disk layout has actually been declared
  ansible.builtin.assert:
    that:
      - node_baseline_longhorn_source in ['device', 'claim-mountpoint']
      - node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
      # `shared-data-partition` is only coherent when there IS a claimed data partition to
      # share, and only when the k3s data dir actually sits INSIDE it. Catch the
      # contradiction here, in inventory terms, rather than three sections later when etcd
      # has silently landed back on the OS filesystem.
      - >-
        node_baseline_etcd_source != 'shared-data-partition'
        or node_baseline_longhorn_source == 'claim-mountpoint'
      - >-
        node_baseline_etcd_source != 'shared-data-partition'
        or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
      # The transition knobs are a matched PAIR: a retired label with no retired mountpoint
      # (or vice versa) cannot describe a state to migrate from.
      - >-
        (node_baseline_data_retired_fs_label | length > 0)
        == (node_baseline_data_retired_mount | length > 0)
      - >-
        node_baseline_longhorn_source != 'device'
        or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
      - >-
        node_baseline_etcd_source != 'device'
        or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
      - >-
        node_baseline_longhorn_source != 'claim-mountpoint'
        or (node_baseline_longhorn_claim_mountpoint | length) > 0
    fail_msg: >-
      This host's data-disk layout is not declared.
      longhorn_source={{ node_baseline_longhorn_source }}
      (device={{ node_baseline_longhorn_device }},
      claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
      etcd_source={{ node_baseline_etcd_source }}
      (device={{ node_baseline_etcd_device }}).
      Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
      `claim-mountpoint` (re-purpose the partition already mounted at
      node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
      `device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
      it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
      data partition — prod's shape since 2026-09-04, which additionally requires
      longhorn_source `claim-mountpoint` and a k3s data dir under
      node_baseline_data_mount={{ node_baseline_data_mount }};
      k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
      operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
      Declare it in the environment's group_vars and record the reasoning in
      §2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
      `--tags packages`, which skips every disk task including this assert.
    quiet: true
 # =====================================================================================
 # THE DATA PARTITION  (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
 # =====================================================================================
 # THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
 # rather than a reformat. Once the claim has happened the partition carries the ext4
 # label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
 # mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
 # mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
 # "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
 # path means converged, rc != 0 means the claim has not been made. Everything destructive
 # below is gated on rc != 0.
 - name: "Data partition: look for a filesystem already carrying the data-partition label"
  ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
  register: node_baseline_lh_labelled
  changed_when: false
  failed_when: false
  check_mode: false
 # --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
 # WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
 # label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
 # takes over — which then fails closed at "the claim mountpoint must actually be a
 # mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
 # behaviour but it is not convergence, and the prod nodes are in exactly that state today:
 # claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
 # So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
 #
 # WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
 # partition with the new label. It never touches the DATA and never reformats: e2label is a
 # metadata write, exactly as in the claim path below. The common mount task then mounts
 # LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
 # `blkid -L <new label>` succeeds and none of this is reachable.
 #
 # THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
 # volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
 # unmounting it would orphan every one of them:
 #   1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
 #      group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
 #   2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
 #      the same structural gate that keeps dev out of the claim path keeps it out of this one.
 #   3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
 #      replica directories on it fails closed and nothing is unmounted.
 # Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
 - name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
  ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
  register: node_baseline_lh_retired_labelled
  changed_when: false
  failed_when: false
  check_mode: false
  when:
    - node_baseline_longhorn_source == 'claim-mountpoint'
    - node_baseline_data_retired_fs_label | length > 0
    - node_baseline_lh_labelled.rc != 0
 - name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
  ansible.builtin.set_fact:
    # A node claimed under the RETIRED label. Mutually exclusive with the claim below.
    node_baseline_lh_transition_needed: >-
      {{ node_baseline_longhorn_source == 'claim-mountpoint'
         and node_baseline_data_retired_fs_label | length > 0
         and node_baseline_lh_labelled.rc != 0
         and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
    # True only for a claim-mountpoint environment that has NOT been claimed under EITHER
    # label. A `device` environment (dev) can never enter the claim path at all — that is
    # the structural guarantee that no generalisation of prod's behaviour can reach the
    # LIVE dev nodes and their /dev/vdb Longhorn replicas.
    node_baseline_lh_claim_needed: >-
      {{ node_baseline_longhorn_source == 'claim-mountpoint'
         and node_baseline_lh_labelled.rc != 0
         and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
 - name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
  ansible.builtin.command: >-
    findmnt --noheadings --first-only --output SOURCE
    --mountpoint {{ node_baseline_data_retired_mount }}
  register: node_baseline_lh_retired_findmnt
  changed_when: false
  failed_when: false
  check_mode: false
  when: node_baseline_lh_transition_needed | bool
 # Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
 # not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
 # an unmount of a populated Longhorn store would orphan every replica on the node.
 - name: "Data partition (transition): read the contents of the retired mountpoint"
  ansible.builtin.find:
    paths: "{{ node_baseline_data_retired_mount }}"
    file_type: any
    hidden: true
    recurse: false
    excludes: ['lost+found']
  register: node_baseline_lh_retired_contents
  when: node_baseline_lh_transition_needed | bool
 - name: "Data partition (transition): prove this node is safe to re-label"
  ansible.builtin.assert:
    that:
      # 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
      #    retired label names. The only state this role knows how to transition is the one
      #    the previous version of it produced; anything else is an operator decision.
      - (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
      - >-
        (node_baseline_lh_retired_findmnt.stdout | default('') | trim)
        == (node_baseline_lh_retired_labelled.stdout | default('') | trim)
      # 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
      #    not the node we think it is, or a tenant has already written data — in both cases
      #    unmounting and re-labelling under it is destructive and the role must stop.
      - (node_baseline_lh_retired_contents.files | length) == 0
    fail_msg: >-
      REFUSING to transition this node's data partition from
      LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
      device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
      device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
      (findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
      non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
      {{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
      If the partition holds DATA, this transition would orphan it — Longhorn replicas are
      not recoverable from an unmounted, re-labelled filesystem. Migrate the data
      deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
      Do NOT relax this assert.
    quiet: true
  when: node_baseline_lh_transition_needed | bool
 - name: "Data partition (transition): report exactly what is about to change"
  ansible.builtin.debug:
    msg: >-
      Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
      from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
      to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
      unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
      No filesystem is created and no data is moved.
  when: node_baseline_lh_transition_needed | bool
 # e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
 # breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
 - name: "Data partition (transition): unmount the retired mountpoint"
  ansible.posix.mount:
    path: "{{ node_baseline_data_retired_mount }}"
    state: unmounted
  when: node_baseline_lh_transition_needed | bool
 - name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
  ansible.posix.mount:
    path: "{{ node_baseline_data_retired_mount }}"
    state: absent_from_fstab
    backup: true
  when: node_baseline_lh_transition_needed | bool
 - name: "Data partition (transition): re-stamp the partition with the new filesystem label"
  ansible.builtin.command: >-
    e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
    {{ node_baseline_data_fs_label }}
  changed_when: true
  when: node_baseline_lh_transition_needed | bool
 # --- The claim (prod, first run only) -------------------------------------------------
 # Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
 # mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
 # returning the device behind `/` — which is precisely the mistake that would hand the
 # root partition to e2label.
 - name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
  ansible.builtin.command: >-
    findmnt --noheadings --first-only --output SOURCE
    --mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
  register: node_baseline_lh_claim_findmnt
  changed_when: false
  failed_when: false
  check_mode: false
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
  ansible.builtin.assert:
    that:
      - node_baseline_lh_claim_findmnt.rc == 0
      - (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
    fail_msg: >-
      node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
      yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
      this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
      stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
      The role will NOT go looking for "a big partition" to claim instead. Either the
      node is partitioned differently from the fleet — in which case fix the node or give
      this host its own declaration — or the claim already happened and the label was
      removed by hand.
    quiet: true
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): inspect the candidate partition"
  ansible.builtin.command: >-
    lsblk --json --nodeps --bytes
    --output PATH,TYPE,FSTYPE,LABEL,SIZE
    {{ node_baseline_lh_claim_findmnt.stdout | trim }}
  register: node_baseline_lh_claim_lsblk
  changed_when: false
  check_mode: false
  when: node_baseline_lh_claim_needed | bool
 # The devices this role must never, under any circumstances, hand to e2label. Built from
 # gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
 # NODE's account of what is load-bearing, not a hardcoded list of partition names.
 - name: "Longhorn (claim): list the devices currently backing active swap"
  ansible.builtin.command: swapon --noheadings --show=NAME
  register: node_baseline_lh_swapdevs
  changed_when: false
  failed_when: false
  check_mode: false
  when: node_baseline_lh_claim_needed | bool
 # Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
 # ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
 - name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
  ansible.builtin.find:
    paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
    file_type: any
    hidden: true
    recurse: false
    excludes: ['lost+found']
  register: node_baseline_lh_claim_contents
  when: node_baseline_lh_claim_needed | bool
 # THE GUARD WALL. Five independent conditions, every one of which must hold before the
 # role modifies a filesystem it did not create. Any single failure stops the play with
 # the evidence in the message — nothing is unmounted, relabelled or mounted first.
 - name: "Longhorn (claim): prove the candidate partition is safe to claim"
  ansible.builtin.assert:
    that:
      # 1. It is a real block device — a partition or a whole disk. Catches a claim
      #    mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
      #    which would have produced a non-/dev SOURCE or an lsblk row of another type.
      - node_baseline_lh_candidate.type in ['part', 'disk']
      # 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
      #    so a differently-partitioned node is caught by its own facts.
      - node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
      # 3. It already holds ext4. The claim path RELABELS rather than reformats (see
      #    below), which is only meaningful for the filesystem type we are about to
      #    declare in fstab. Anything else is an operator decision, not a role's.
      - node_baseline_lh_candidate.fstype == 'ext4'
      # 4. It carries no filesystem label, or already carries ours. A partition labelled
      #    `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
      #    it would make it unfindable by its own fstab entry.
      - (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
      # 5. It is big enough to be the partition we meant. A data partition of a few GB is
      #    not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
      - (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
      # 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
      #    ext4 5% root reserve; this is where that premise is verified per node instead
      #    of trusted. If any node has real data on it, the play stops here — the role
      #    never wipes data it cannot account for.
      - (node_baseline_lh_claim_contents.files | length) == 0
    fail_msg: >-
      REFUSING to claim {{ node_baseline_lh_candidate.path }}
      (mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
      type={{ node_baseline_lh_candidate.type }},
      fstype={{ node_baseline_lh_candidate.fstype }},
      label='{{ node_baseline_lh_candidate.label | default('', true) }}',
      size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
      (minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
      non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
      {{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
      devices this node needs and that must never be claimed:
      {{ node_baseline_lh_protected_devices }}.
      This is the destructive step of the role and it fails CLOSED. If the partition
      holds data, back it up or pick a different one; do NOT relax this assert.
    quiet: true
  vars:
    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
    # The node's OWN account of what it cannot afford to lose: the devices behind the
    # mounts in node_baseline_protected_mounts (from gathered facts, so a node
    # partitioned differently is judged on its own layout) plus every active swap device.
    # Built here rather than hardcoded so no partition-name list can go stale.
    node_baseline_lh_protected_devices: >-
      {{ (ansible_facts.mounts | default([])
          | selectattr('mount', 'in', node_baseline_protected_mounts)
          | map(attribute='device') | list)
         + (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): report exactly what is about to be claimed"
  ansible.builtin.debug:
    msg: >-
      Claiming {{ node_baseline_lh_candidate.path }}
      ({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
      currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
      as the data partition: unmount -> drop its fstab entry ->
      relabel LABEL={{ node_baseline_data_fs_label }} ->
      mount at {{ node_baseline_data_mount }}.
  vars:
    node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
  when: node_baseline_lh_claim_needed | bool
 # --- Why the role must unmount and relabel, not just point fstab at the device ---------
 # The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
 # /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
 # path does not. The installer-created /srv filesystem has NO label (fstab pins it by
 # UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
 # the unmount first — and the fstab entry must go in the same breath, or the next boot
 # re-mounts the same partition at /srv underneath Longhorn.
 #
 # RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
 # mkfs would buy nothing and would destroy anything the emptiness check got wrong.
 # e2label is a metadata write: if some assert above was too permissive, the worst case is
 # a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
 # cannot do this job: it declines to touch a device that already has a filesystem, so it
 # would silently no-op and then the LABEL= mount would fail with no useful message.)
 - name: "Longhorn (claim): unmount the claim mountpoint"
  ansible.posix.mount:
    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
    state: unmounted
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
  ansible.posix.mount:
    path: "{{ node_baseline_longhorn_claim_mountpoint }}"
    state: absent_from_fstab
    backup: true
  when: node_baseline_lh_claim_needed | bool
 - name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
  ansible.builtin.command: >-
    e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
    {{ node_baseline_data_fs_label }}
  changed_when: true
  when: node_baseline_lh_claim_needed | bool
 # --- The dedicated-device path (dev) --------------------------------------------------
 # Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
 # never reformats a device that already carries a filesystem.
 - name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
  community.general.filesystem:
    fstype: ext4
    dev: "{{ node_baseline_longhorn_device }}"
    opts: "-L {{ node_baseline_data_fs_label }}"
  when: node_baseline_longhorn_source == 'device'
 # --- Common to ALL paths (device, claim, transition) ----------------------------------
 # Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
 # mount task. Idempotent: on a converged node this reports ok, which is what makes a
 # re-run a no-op rather than a reformat. It is also what completes the transition above:
 # the retired mountpoint is gone from fstab and the partition now answers to the new label,
 # so this task mounts it at the new path.
 - name: Mount the data partition
  ansible.posix.mount:
    path: "{{ node_baseline_data_mount }}"
    src: "LABEL={{ node_baseline_data_fs_label }}"
    fstype: ext4
    opts: "defaults,nofail,x-systemd.device-timeout=5s"
    state: mounted
 # --- The per-tenant subdirectories ---------------------------------------------------
 # WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
 # tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
 # subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
 # /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
 # either there would at best be a no-op and at worst re-mode the running dev cluster's data
 # dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
 # mounted — i.e. only the directories it is actually responsible for. That test is derived
 # from the paths themselves, so no environment has to remember to switch it off.
 #
 # MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
 # the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
 # etcd database, the node token and the containerd image store, and k3s would tighten it to
 # 0700 itself on first start — doing it here means it is never briefly world-readable.
 - name: Create the per-tenant data directories on the data partition
  ansible.builtin.file:
    path: "{{ item.path }}"
    state: directory
    owner: root
    group: root
    mode: "{{ item.mode }}"
  loop:
    - { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
    - { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
  loop_control:
    label: "{{ item.path }} ({{ item.tenant }})"
  when: item.path.startswith(node_baseline_data_mount ~ '/')
 # =====================================================================================
 # ETCD DATA DIR
 # =====================================================================================
 # --- The dedicated-device path (dev) --------------------------------------------------
 # Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
 # mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
 # etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
 # a disk that already carries a filesystem, so re-running against an already-migrated
 # node leaves the etcd data intact. On EXISTING nodes the one-time data migration
 # (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
 # role would mount over a populated directory.
 - name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
  community.general.filesystem:
    fstype: ext4
    dev: "{{ node_baseline_etcd_device }}"
    opts: "-L {{ node_baseline_etcd_fs_label }}"
  when: node_baseline_etcd_source == 'device'
 - name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
  ansible.posix.mount:
    path: "{{ node_baseline_etcd_mount }}"
    src: "LABEL={{ node_baseline_etcd_fs_label }}"
    fstype: ext4
    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
    opts: "defaults,noatime"
    dump: "0"
    passno: "2"
    state: mounted
  when: node_baseline_etcd_source == 'device'
 # --- The no-dedicated-device paths (prod) ---------------------------------------------
 # Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
 # populated — so etcd gets no device of its own. That is an operator decision
 # (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
 # UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
 # on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
 # latency rather than collapsing it. Accepted to be MEASURED, not assumed.
 #
 # WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
 # filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
 # 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
 # 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
 # could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
 # partition and etcd moved with it: `shared-data-partition`. Both enum values are still
 # supported, and the role now PROVES which one is true instead of printing a claim.
 #
 # There is nothing to CREATE here for either shape — the data-partition section above
 # already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
 # role's job is to verify the assumption rather than act on it.
 - name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
  ansible.builtin.command: >-
    findmnt --noheadings --first-only --output SOURCE
    --mountpoint {{ node_baseline_etcd_mount }}
  register: node_baseline_etcd_existing_mount
  changed_when: false
  failed_when: false
  check_mode: false
  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
 - name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
  ansible.builtin.assert:
    that:
      - node_baseline_etcd_existing_mount.rc != 0
    fail_msg: >-
      node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
      NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
      ALREADY a separate mount, backed by
      {{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
      contradiction the role will not resolve on its own: either this node was previously
      built with a dedicated etcd device (in which case unmount it deliberately, after
      accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
      the node in a state where the declared layout and the real one disagree.
    quiet: true
  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
 # Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
 # WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
 # existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
 # that does exist — the same filesystem the data dir will be created on. TARGET is captured
 # separately from the human-readable line because the assert below judges on it.
 - name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
  ansible.builtin.shell: |
    set -eu
    p={{ node_baseline_etcd_mount | quote }}
    while [ ! -d "$p" ]; do p=$(dirname "$p"); done
    findmnt --noheadings --first-only --output TARGET --target "$p"
    findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
  register: node_baseline_etcd_backing_fs
  changed_when: false
  check_mode: false
  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
 # THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
 # `shared-data-partition` is a distinct enum value rather than a comment. Declaring
 # k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
 # above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
 # of container images would land back on the OS filesystem — the exact failure this change
 # exists to prevent, and completely invisible until the disk filled. So the role checks that
 # the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
 # run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
 # which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
 # whole record for that shape.
 - name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
  ansible.builtin.assert:
    that:
      - >-
        (node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
        == node_baseline_data_mount
    fail_msg: >-
      node_baseline_etcd_source is `shared-data-partition`, which asserts that
      {{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
      {{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
      '{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
      (full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
      The data partition is therefore NOT mounted where this layout requires, and k3s would
      put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
      filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
      image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
      this node until the mount is correct.
    quiet: true
  when: node_baseline_etcd_source == 'shared-data-partition'
 - name: "etcd (no dedicated device): show which filesystem etcd will live on"
  ansible.builtin.debug:
    msg: >-
      etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
      (source={{ node_baseline_etcd_source }}); it lands on
      {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
      (SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
      measured risk — §2.
  when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
@@ -1,6 +1,6 @@
 ---
-# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
+# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
-# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
+# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
 # and the DATA-DISK work have different readiness: packages are safe on any node from
 # first contact, while the disk tasks need block devices that a new environment may not
 # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -12,6 +12,10 @@
 # mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
 # at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
 #   ansible-playbook playbooks/node-baseline.yml --tags dns
 # `kernel` (the backports-kernel block below) is separated for the same reason `packages`
 # is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
 # it must be runnable without touching a disk decision that is still open:
 #   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
  ansible.builtin.apt:
@@ -21,6 +25,237 @@
    cache_valid_time: 3600
  tags: [packages]
 # =====================================================================================
 # BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
 # =====================================================================================
 # A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
 # FUTURE prod node gets it from provisioning instead of from someone remembering.
 #
 # WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
 # Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
 # `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
 # signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
 # through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
 # address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
 # Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
 # (first released in v6.13) validates the size and rejects the table instead. It is absent
 # from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
 # stable feed will ever pick it up. There is NO kernel cmdline workaround —
 # `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
 # take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
 # same lottery — node 1 and node 3 have merely drawn safe tickets so far.
 #
 # WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
 # defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
 # Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
 # LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
 # change for no benefit. See defaults/main.yml for why this is a variable rather than a
 # `k3s-prod-*` hostname test.
 #
 # IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
 # deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
 # off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
 # therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
 # before touching the next, never node 2 first — and a role cannot enforce that. The last
 # task here reports the pending reboot loudly instead of letting a green play imply the node
 # is already fixed.
 - name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
  when: node_baseline_backports_kernel | bool
  tags: [kernel]
  block:
    # The suite name and the running release must agree, or apt resolves no candidate and
    # `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
    # a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
    # dist-upgraded to forky while group_vars still says trixie-backports).
    - name: Assert the configured backports suite matches this node's Debian release
      ansible.builtin.assert:
        that:
          - ansible_facts['distribution'] == 'Debian'
          - node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
        fail_msg: >-
          node_baseline_backports_kernel is enabled but node_baseline_backports_suite
          ({{ node_baseline_backports_suite }}) is not the backports suite for this node's
          release ({{ ansible_facts['distribution'] }}
          {{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
          candidate and this play would report success while leaving the node on the
          vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
        quiet: true
    # deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
    # prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
    # rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
    # idempotent in a way `apt_repository` — which rewrites and reformats — is not.
    #
    # THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
    # backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
    # apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
    # 500 for a package that was explicitly installed from there so it keeps getting updates.
    # Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
    # HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
    # be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
    - name: Enable the Debian backports apt source (deb822)
      ansible.builtin.copy:
        dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
        owner: root
        group: root
        mode: "0644"
        content: |
          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
          # A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
          # machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
          # Table its firmware publishes (upstream 8fbe4c49c0cc). See
          # infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
          # on this node.
          #
          # Backports is NotAutomatic, so nothing is taken from here unless asked for by name
          # with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
          # ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
          # Pin-Priority would switch that off.
          Types: deb
          URIs: {{ node_baseline_backports_mirror }}
          Suites: {{ node_baseline_backports_suite }}
          Components: {{ node_baseline_backports_components | join(' ') }}
          Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
      register: node_baseline_backports_source
    - name: Refresh the apt cache after enabling backports
      ansible.builtin.apt:
        update_cache: true
      when: node_baseline_backports_source is changed
    # `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
    # nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
    # nothing at all — the single most likely way to write this task and have it silently fix
    # nothing forever. `latest` + `default_release` resolves the candidate inside backports
    # and upgrades the metapackage to it. Once the node is current a re-run is a genuine
    # no-op: no change, no churn, no spurious reboot flag. Scope is exactly
    # node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
    # is dragged out of the main archive.
    - name: Install the fixed kernel from backports (>= 6.13)
      ansible.builtin.apt:
        name: "{{ node_baseline_backports_kernel_packages }}"
        state: latest
        default_release: "{{ node_baseline_backports_suite }}"
        update_cache: true
        cache_valid_time: 3600
      register: node_baseline_backports_kernel_apt
    # WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
    # metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
    # answer anybody can roll back from six months later.
    - name: Read back the installed backports kernel metapackage version
      ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
      register: node_baseline_backports_kernel_version
      changed_when: false
      check_mode: false
    # `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
    # neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
    # list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
    # GRUB's "Advanced options" without uninstalling anything.
    - name: List every kernel image installed on this node, newest first
      ansible.builtin.shell:
        cmd: set -o pipefail; linux-version list | linux-version sort --reverse
        executable: /bin/bash
      register: node_baseline_installed_kernels
      changed_when: false
      check_mode: false
    # The node must be able to explain itself to an operator standing in front of it with no
    # repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
    # role is a real no-op rather than a permanent daily "changed".
    - name: Record the deviation, the exact version, and the rollback path on the node
      ansible.builtin.copy:
        dest: "{{ node_baseline_backports_kernel_stamp }}"
        owner: root
        group: root
        mode: "0644"
        content: |
          # Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
          #
          # THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
          #
          # WHY
          #   This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
          #   Linux < 6.13 reserves memory from it without validating its size, and an unlucky
          #   boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
          #   k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
          #   is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
          #   v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
          #   Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
          #
          # PROOF THE FIX IS ACTIVE (not merely a lucky boot)
          #   BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
          #   >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
          #     zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
          #   (roles/node_health does the version check for you; see the health report's
          #   firmware.efi_memattr finding.)
          #
          #   FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
          #   and each prints a DIFFERENT line — all three are rejections:
          #     tbl->version > 2         'Unexpected EFI Memory Attributes table version %d'
          #                              (NOTE: printed with %d, so garbage may read NEGATIVE)
          #     bad desc_size            'Unexpected EFI Memory Attributes descriptor size ...'
          #     num_entries > 64K        '[Firmware Bug]: Corrupted EFI Memory Attributes
          #                               Table detected! (version == ..., num_entries == ...)'
          #   Which one appears depends on WHICH field this boot's garbage corrupted, so do
          #   NOT treat the absence of the FW_BUG line as the fix being absent — all three
          #   prod nodes printed the *version* line on 2026-09-04.
          #   The one line that means the fix is NOT active:
          #     'Failed to map EFI Memory Attributes table @ 0x...'  — the kernel trusted the
          #     corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
          #     escalate, do not shrug.
          #
          # INSTALLED
          #   source   /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
          #   package  {{ node_baseline_backports_kernel_packages | join(' ') }}
          #   version  {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
          #   kernels installed on this node, newest first:
          {% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
          #     {{ kv }}
          {% endfor %}
          #
          # SECURITY COST
          #   Packages from *-backports are provided AS IS. They are NOT covered by the Debian
          #   Security Team's guarantees for the main archive: no security-tracker commitment,
          #   no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
          #
          # ROLLBACK — no repo checkout needed
          #   1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
          #      and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
          #      GNU/Linux". That answers "is the new kernel the problem?" without
          #      uninstalling anything. Note it reinstates the memattr bug for that boot.
          #   2. PERMANENT. Set node_baseline_backports_kernel: false in
          #      inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
          #      puts the backports kernel straight back — then, on this node:
          #        apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
          #        apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
          #        rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
          #        apt-get update && update-grub && reboot
          #
          # EXIT CONDITION
          #   If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
          #   rebase), this deviation can be removed entirely: flip the variable to false,
          #   roll back as above, and delete the `kernel` block from roles/node_baseline.
    # Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
    # it means the fixed kernel is on disk. See the block header for why the role will not
    # reboot on its own.
    - name: Report whether this node is running the kernel it now has installed
      ansible.builtin.debug:
        msg: >-
          {{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
              ~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
              prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
              node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
              with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
             if newest != ansible_facts['kernel']
             else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
                   ~ '); no reboot pending for the memattr fix.') }}
      vars:
        newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
 - name: Persist kernel modules for k3s/Cilium/Longhorn
  ansible.builtin.copy:
    dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
@@ -79,70 +314,32 @@
    state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
  tags: [system]
-# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
+# =====================================================================================
-# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
+# DATA DISKS — tag `disks`, in tasks/disks.yml
-# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
+# =====================================================================================
-# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
+# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
-# which do not exist on bare metal either — a confusing failure that looks like a wiring
+# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
-# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
+# should be readable end-to-end on its own; and both environments' storage shapes — dev's
-# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
+# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
- name: Assert this environment's data-disk devices have actually been decided
+# the OS filesystem — are declared there as an explicit enum rather than inferred from
-  ansible.builtin.assert:
+# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
-    that:
+#
-      - "'BLOCKED' not in node_baseline_longhorn_device"
+# The `disks` tag is applied ONCE here, in BOTH places it is needed:
-      - "'BLOCKED' not in node_baseline_etcd_device"
+#   * `tags:` on the include itself, so `--tags packages` never even reaches the include
-    fail_msg: >-
+#     and a placeholder device path is not so much as templated; and
-      node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
+#   * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
-      placeholders for this host's environment
+#     optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
-      (longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
+#     its own tags to its children the way `import_tasks` does, so without `apply` the
-      The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
+#     included tasks are untagged and `--tags disks` selects the include and then runs
-      and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
+#     NOTHING inside it. That failure is silent and reports success, which is the worst
-      the SMART read that decision waits on can be taken) re-run this play with
+#     shape a guard can have.
-      `--tags packages`, which skips every disk task including this assert.
+# A FULL run against a node whose layout is undeclared stops on the assert at the top of
-    quiet: true
+# that file instead of failing obscurely at mkfs time.
-  tags: [disks]
+- name: Provision the data disks (Longhorn store, etcd data dir)
-
+  ansible.builtin.include_tasks:
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
+    file: disks.yml
-  community.general.filesystem:
+    apply:
-    fstype: ext4
+      tags: [disks]
    dev: "{{ node_baseline_longhorn_device }}"
    opts: "-L {{ node_baseline_longhorn_fs_label }}"
  tags: [disks]
 - name: Mount the Longhorn data disk
  ansible.posix.mount:
    path: "{{ node_baseline_longhorn_mount }}"
    src: "LABEL={{ node_baseline_longhorn_fs_label }}"
    fstype: ext4
    opts: "defaults,nofail,x-systemd.device-timeout=5s"
    state: mounted
  tags: [disks]
 # Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
 # mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
 # etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
 # a disk that already carries a filesystem, so re-running against an already-migrated
 # node leaves the etcd data intact. On EXISTING nodes the one-time data migration
 # (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
 # role would mount over a populated directory.
 - name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
  community.general.filesystem:
    fstype: ext4
    dev: "{{ node_baseline_etcd_device }}"
    opts: "-L {{ node_baseline_etcd_fs_label }}"
  tags: [disks]
 - name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
  ansible.posix.mount:
    path: "{{ node_baseline_etcd_mount }}"
    src: "LABEL={{ node_baseline_etcd_fs_label }}"
    fstype: ext4
    # noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
    # missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
    opts: "defaults,noatime"
    dump: "0"
    passno: "2"
    state: mounted
  tags: [disks]
 - name: Disable swap now
@@ -88,15 +88,42 @@ measured memory, never on the log line alone:**
 |---|---|---|
 | `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
 | `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
-| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | (not consulted) | **WARN** — the garbage `version` read `> 2` so the kernel bailed out first: exposed, but this boot took the safe path |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path |
-| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 validated and rejected it: this *is* the fix, working |
+| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic |
 | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working |
 | no memattr line, log reaches back to boot | (not consulted) | PASS |
 | no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
 | `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
-The WARN is the point. All three prod nodes publish the same corrupt table and differ only
+The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt
-by **luck**, boot to boot. A node whose kernel just trusted a corrupt table is not passing a
+table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt
-health check even when it kept its RAM.
+table is not passing a health check even when it kept its RAM.
 ### The FW_BUG line is proof, but its ABSENCE proves nothing
 `efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG
 line:
 | gate | message | reachable when |
 |---|---|---|
 | `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
 | implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not |
 | `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not |
 **All three are rejections.** Which one a boot prints is decided by *which field* that boot's
 garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very
 often prints the *first* message, and looking only for the FW_BUG line will make a working
 fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed
 the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version
 alongside the log line, and why the check must never demand the FW_BUG string as its
 evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in
 the kernel image itself, not a line in `dmesg`.
 **The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged
 *negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that
 until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected
 a corrupt table was reported as one that "accepted it without complaint" — a false PASS on
 the exact bug this check exists to find.
 **Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
 `trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
@@ -497,8 +497,15 @@ BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Co
 EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
 MEMATTR_FAILED_RE = re.compile(
    r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
 # NOTE the `-?`. The kernel prints this version with %d — SIGNED — so a garbage field with
 # bit 31 set is logged NEGATIVE (k3s-prod-2, 2026-09-04: "version -1569980416"). Without the
 # minus sign this regex missed those lines entirely and the state fell through to "clean",
 # i.e. a node whose kernel had just REJECTED a corrupt table was reported as one whose
 # kernel "accepted it without complaint" — a false PASS on the exact bug this check exists
 # to find. Anchor future edits on that: the field is signed, and every diagnostic that
 # quotes it must survive a negative value.
 MEMATTR_VERSION_RE = re.compile(
-    r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I)
+    r"Unexpected EFI Memory Attributes table version\s+(-?\d+)", re.I)
 MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
 # The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
 # line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
@@ -820,12 +820,22 @@
 #       catastrophically missing — memory is lost NOW.
 #       WARN otherwise — this boot drew a survivable garbage value; the next may not.
 #   "Unexpected ... version <N>"  version read > 2, so the kernel bailed out before
-#       reserving anything. WARN: exposed, and this boot took the safe path.
+#       reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
 #       safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
 #       gates and no draw can reach the fatal arithmetic any more. Same line, opposite
 #       meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
 #   FW_BUG "Corrupted EFI Memory Attributes Table detected!"  the >= 6.13 THIRD gate
 #       (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
 #       must not be treated as such: it is only reachable on a boot whose garbage leaves the
 #       version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
 #       line instead. All three prod nodes did exactly that on 2026-09-04.
 #
 # The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
 # a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
-# measurement separates the last two. Never PASS on either line: the firmware is publishing
+# measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
-# a corrupt table and the draw is repeated on every boot.
+# firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
 # the draw still happens but every outcome is a rejection, so the same lines mean the
 # validation is working.
 #
 # WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
 # so it cannot be diagnosed once. It has to be measured on every health run.
@@ -857,15 +867,39 @@
    # near memory.physical_total. This check contributes the kernel-log facts only.
    ms: "{{ node_health_mem_shortfall }}"
    ma_state: "{{ k.memattr_state | default(none) }}"
    # DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
    # carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
    # THREE checks before it computes tbl_size at all:
    #     tbl->version > 2                       -> "Unexpected ... table version %d"
    #     desc_size implausible                  -> "Unexpected ... descriptor size %u ..."
    #     num_entries > SZ_64K                   -> FW_BUG "Corrupted EFI Memory Attributes
    #                                               Table detected! ..."
    # ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
    # given boot produces is decided by WHICH field that boot's garbage lands in — it is not
    # a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
    # version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
    # memblock_reserve(). That is the whole difference, and it is why the verdict for
    # `bad_version` below is opposite on the two kernels: same log line, completely different
    # meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
    # something the `version` test can parse.
    memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
    memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
    # `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
    # COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
    # this check and memory.physical_total can no longer disagree about lost memory.
    # `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
    # boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
    # WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
    # forever would make the check unable to ever show the fix landing, which is exactly what
    # it is for.
    memattr_status: >-
      {{ 'SKIP' if not (k.available | bool)
         else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
-               else ('WARN' if ma_state in ['failed_to_map', 'bad_version']
+               else ('WARN' if ma_state == 'failed_to_map'
                     else ('PASS' if ma_state == 'corrupt_rejected'
-                           else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
+                           else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
                                 if ma_state == 'bad_version'
                                 else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
    # One remediation, quoted by both findings. NOTE what it must NOT say: there is no
    # kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
    # could take effect. Suggesting one would be a fix that quietly does nothing.
@@ -944,6 +978,21 @@
      been seen to lose 13.3 GiB and to keep its full memory at one identical table address
      (infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
      {{ memattr_fix }}
      {%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
      the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
      kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
      VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
      boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
      outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
      efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
      WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
      Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
      (num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
      version field <= 2. Both messages are rejections, and which one a boot prints is the
      lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
      path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
      PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
      {{ memattr_corroboration }}
      {%- elif ma_state == 'bad_version' -%}
      EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
      CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
@@ -25,7 +25,7 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
 | Requirement | Summary | How this component serves it |
 |---|---|---|
 | **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
-| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
+| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the node's data disk at `${LONGHORN_DATA_PATH}` (a `cluster-vars` knob: dev `/var/lib/longhorn`, a dedicated disk; prod `/data/longhorn`, a subdirectory of the partition it shares with k3s's data dir). `replicaAutoBalance: best-effort` re-spreads on node return. |
 | **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
 ## Dev-scoped choices (change for prod)
@@ -48,9 +48,19 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
 ## Node prerequisites (already handled, informational)
-Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
+Each node presents ext4 storage at `${LONGHORN_DATA_PATH}` and installs `open-iscsi` +
-`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
+`nfs-common` + kernel modules — all via `roles/node_baseline` (`infra/cloud-init/` is a README
-`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
+now; provisioning became API-only). This component assumes that floor. The shape differs per
 environment, which is why the path is a knob:
 | | |
 |---|---|
 | dev | a **dedicated** disk (`/dev/vdb`, `LABEL=longhorn`) mounted at `/var/lib/longhorn` — the mount root *is* the dataPath |
 | prod | `/data/longhorn`, a **subdirectory** of the ~424 GB partition (`LABEL=k3s-data`) mounted at `/data`, whose sibling `/data/k3s` holds k3s's entire data dir (etcd + the containerd image store, moved off the 11.8 GiB `/var` — `infra/PROD-PROVISIONING.md` §2) |
 `[parity]`: the value for an environment MUST equal `node_baseline_longhorn_data_path` in that
 environment's Ansible group_vars. Ansible creates and mounts the directory; this component tells
 Longhorn to use it.
 ## Contract (consumed by everyone with state)
@@ -42,10 +42,22 @@ spec:
      defaultClass: false
    defaultSettings:
-      # --- data path (R-STATE-02) ---
+      # --- data path (R-STATE-02, R-ENV-02) ---
-      # Dedicated Longhorn data disk (/dev/vdb → ext4) is mounted here by cloud-init on every node
+      # The on-node directory every replica is written into. A cluster-vars KNOB, not a literal,
-      # (infra/cloud-init/user-data.yaml.tftpl). Longhorn stores replicas on this disk only.
+      # because the two environments mount their data disk differently and a hardcoded path would
-      defaultDataPath: "/var/lib/longhorn"
+      # put one of them on the OS filesystem:
      #   dev  /var/lib/longhorn  — a dedicated ext4 disk (/dev/vdb) mounted there by
      #                             roles/node_baseline. The mount root IS the dataPath: one disk,
      #                             one tenant. 18 live volumes; this value must never change.
      #   prod /data/longhorn     — a SUBDIRECTORY of the claimed ~424 GB partition mounted at
      #                             /data, whose sibling /data/k3s holds k3s's entire data dir
      #                             (etcd + the containerd image store, moved off the 11.8 GiB
      #                             /var). Two tenants, so Longhorn gets a subdirectory rather
      #                             than the mount root. infra/PROD-PROVISIONING.md §2.
      # [parity] MUST equal roles/node_baseline's node_baseline_longhorn_data_path for the same
      # environment — Ansible creates and mounts the directory, Helm tells Longhorn to use it.
      # Set in BOTH overlays (clusters/dev, clusters/prod); base default is the dev value.
      defaultDataPath: "${LONGHORN_DATA_PATH}"
      # --- replication (R-STATE-02) ---
      # Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly
@@ -8,7 +8,9 @@
 #   defaultClass: false                              # we own StorageClasses (storageclasses.yaml)
 #
 # defaultSettings:
-#   defaultDataPath: /var/lib/longhorn               # dedicated data disk (infra/cloud-init)
+#   defaultDataPath: ${LONGHORN_DATA_PATH}           # cluster-vars knob: dev /var/lib/longhorn
 #                                                    # (dedicated /dev/vdb), prod /data/longhorn
 #                                                    # (a subdirectory of the shared data partition)
 #   defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT}   # overlay knob (=3 in dev)
 #   replicaSoftAntiAffinity: false                   # HARD anti-affinity: one replica per node
 #   replicaZoneSoftAntiAffinity: true                # single zone (3 mini-PCs, no failure zones)