Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(prod): kernel fix + disk layout — prod nodes ready for k3s

Browse Source 
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.

=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.

roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).

Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.

THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.

§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.

TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
  - MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
    2's real value was -1569980416, which did not match, so state fell through to `clean` and
    the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
    exact bug the check exists to find. Now (-?\d+).
  - bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
    Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.

=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).

Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).

=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.

Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.

The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.

k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.

LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.

DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.

Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.

OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.

Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.

Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 00:39:10 -05:00
1 parent 8e913fdd7b
commit 05baae90ec
16 files changed
+1782 -126
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

clusters/base/cluster-vars.yaml
+11
View File
@@ -74,6 +74,17 @@ data:
# --- storage (ADR-0003) ---
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
LONGHORN_DATA_PATH: "/var/lib/longhorn"
# --- observability sizing (docs/10) ---
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
clusters/dev/kustomization.yaml
+10
View File
@@ -190,6 +190,16 @@ patches:
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
# on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
# mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
# else would orphan every one of them (they are not recoverable from the old path by
# rescheduling). Restated explicitly even though it equals the base default, exactly like the
# LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
# its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
- op: replace
path: /data/LONGHORN_DATA_PATH
value: "/var/lib/longhorn"
# smaller retention / volumes to keep the dev workstation footprint light.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
clusters/prod/kustomization.yaml
+14
View File
@@ -157,6 +157,20 @@ patches:
- op: replace
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
value: "3"
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
#
# [parity] MUST equal node_baseline_longhorn_data_path in
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
# path on the OS filesystem instead of the data partition and /var fills up.
- op: replace
path: /data/LONGHORN_DATA_PATH
value: "/data/longhorn"
# longer retention / larger volumes on prod NVMe.
- op: replace
path: /data/MONITORING_PROMETHEUS_RETENTION
infra/PROD-PROVISIONING.md
+382 -19
View File
@@ -559,7 +559,7 @@ nothing ever transmits `new_password` to firmware. It is write-only into a buffe
password in cleartext on the command line — it would land in process listings and shell history
unless wrapped. Store it in the Ansible vault **and** offline.
## 2. Disk layout — **OPEN DECISION, blocks the build**
## 2. Disk layout — **DECIDED (operator, 2026-09-03) · REVISED (operator, 2026-09-04)**
`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
@@ -576,6 +576,218 @@ sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
└─sda5 433G /srv <- the only large free space
```
### THE DECISION (operator, 2026-09-03; layout REVISED 2026-09-04) — made, applied, and measured-not-assumed
| | |
|---|---|
| **Data partition** | the **existing `/srv` partition** on each node's single physical disk (~432 GB nodes 1–2, ~422 GB node 3), re-purposed, re-labelled **`k3s-data`** and mounted at the neutral **`/data`** |
| **Longhorn store** | **`/data/longhorn`** — a *subdirectory* of that partition |
| **k3s `--data-dir`** | **`/data/k3s`** — the *sibling* subdirectory. Holds **embedded etcd AND the containerd image store** |
| **etcd** | **no dedicated device** — it lives at `/data/k3s/server/db`, i.e. on the data partition, **not** on the OS filesystem |
| **Shared-device I/O contention** | **ACCEPTED, knowingly** — unchanged by the 2026-09-04 revision |
#### THE 2026-09-04 REVISION, AND THE CAPACITY EVIDENCE THAT FORCED IT
The 2026-09-03 ruling gave this partition to Longhorn **only** and left k3s at its default
`/var/lib/rancher/k3s`. That was wrong, for a reason that is arithmetic rather than judgement.
**`/var` on these nodes is a separate partition of 11.8 GiB** (11.6 on node 3) — see the
partition map above; it is `/dev/sda3` / `/dev/nvme0n1p3`, not the 30 GiB `/`. k3s's data dir
puts **two** things on it: etcd's database and WAL (`server/db`) and, far larger, the
**containerd image store** (`agent/containerd`).
**Measured on the fully-deployed dev cluster** — the same platform and app set prod will run:
| | |
|---|---|
| `agent/containerd` per node | **21–25 GB** |
| total `/var` per node | **58–70 GB** |
| prod `/var` capacity | **11.8 GiB** |
So prod would have **filled `/var` during its first image pulls** — not eventually, not under
load, but on the way up. And because etcd was on that same partition, a full `/var` means
**etcd cannot write its WAL**. That is cluster death, not degradation: a wedged etcd on all
three servers is an unavailable control plane, and it would have happened during bring-up.
**The fix is to move k3s's whole data dir onto the 424 GB partition**, which is why that
partition's mountpoint and filesystem label are no longer Longhorn-specific — it now serves
**two tenants**. `/var` retains only the OS's own usage (503 MB, 5% used, measured 2026-09-04).
**Why now.** Nothing was installed on prod when this was decided — verified 2026-09-04: no
`/usr/local/bin/k3s`, no `/etc/rancher/k3s`, no `/var/lib/rancher` on any node. Moving a data
dir before first install is a one-line inventory change; after bring-up it means draining and
re-imaging three servers.
**Why `/data` and `LABEL=k3s-data`.**
- **`/data`** — short, neutral, tenant-agnostic, and *not* under `/var`. Deliberately **not
`/srv`**: `/srv` is the *claim mountpoint* the role discovers the device at, and the
idempotency contract is that `/srv` is gone from `/proc/mounts` and `/etc/fstab` once the
claim has happened. Re-using it as the destination would make "claimed" and "not yet
claimed" indistinguishable.
- **`LABEL=k3s-data`** — deliberately **not** the bare `data`. Guard #4 of the role's assert
wall accepts a partition whose label is empty *or already ours*, and names `data` as an
example of somebody else's named volume; labelling ours `data` would make a stranger's
`data` volume look already-claimed and weaken that guard. `k3s-data` is cluster-scoped
rather than tenant-scoped, and matches the `k3s-etcd` label convention already in the repo.
**The shared-device I/O risk is UNCHANGED and still accepted.** Longhorn and etcd were already
going to share this spindle under the 2026-09-03 ruling — the revision changes *which*
filesystem etcd sits on, not *which device*. The acceptance rationale and its exit condition
below stand verbatim.
**How prod's data dir is scoped away from dev.** `k3s_data_dir` is a **single** variable read
by **two** roles: `roles/k3s_server` turns it into `--data-dir` in the install flags, and
`roles/node_baseline` (via `node_baseline_k3s_data_dir`, which defers to it) creates the
directory `0700` on the data partition and derives etcd's location from it
(`node_baseline_etcd_mount` = `<data-dir>/server/db`). The role default is **k3s's own
default**, `/var/lib/rancher/k3s`, so **dev is unchanged** — and dev also has a *dedicated
etcd device* mounted at `server/db` under that path, which moving dev's data dir would orphan.
Prod overrides it in one line in `inventory/group_vars/k3s_prod/vars.yml`, the same seam
`kube_vip_address` / `k3s_api_host` / `node_baseline_backports_kernel` use, so a fourth prod
node inherits it from group membership rather than from anyone remembering.
**Longhorn's `defaultDataPath` follows the R-ENV-02 knob pattern, not a base edit.**
`platform/longhorn/helmrelease.yaml` referenced a hardcoded `/var/lib/longhorn`; it now
references `${LONGHORN_DATA_PATH}`, a `cluster-vars` knob with a base default and an explicit
value in **both** overlays:
| | |
|---|---|
| `clusters/base/cluster-vars.yaml` | `/var/lib/longhorn` (base default = the known-good dev value, same convention as `GATEWAY_VIP`) |
| `clusters/dev` | `/var/lib/longhorn` — **must never change**; the live dev cluster has 18 volumes whose replicas are on disk under that path |
| `clusters/prod` | `/data/longhorn` |
`kustomize-controller` runs with `--feature-gates=StrictPostBuildSubstitutions=true`, so an
undefined variable fails the Kustomization outright — hence base-first, then both overlays.
`[parity]`: the prod value MUST equal `node_baseline_longhorn_data_path` in
`group_vars/k3s_prod/vars.yml`. Ansible creates and mounts the directory; Helm tells Longhorn
to use it. If the two disagree, Longhorn writes replicas onto the OS filesystem and `/var`
fills up — the exact failure this revision exists to prevent.
**Verified 2026-09-04 that dev's rendered output is byte-identical.** All 12 Flux
Kustomizations that declare `postBuild.substituteFrom cluster-vars` were rendered
(`kubectl kustomize <spec.path> | flux envsubst --strict`, substituting the dev overlay's
*effective* cluster-vars) both before and after the change: 12/12 identical per-file SHA-256,
and one aggregate SHA-256 over every rendered byte, `b34c5d3d…89e7d`, unchanged. The only
dev-side difference anywhere is the presence of the new knob in the ConfigMap itself, carrying
dev's existing literal. The same harness shows prod's render changing on exactly one line
(`defaultDataPath: /var/lib/longhorn` → `/data/longhorn`), which is what proves the comparison
could have detected a regression.
**Per-node device paths — they are NOT the same, and that is why nothing hardcodes them.**
Verified 2026-09-03 with `findmnt --mountpoint /srv` on each node:
| Node | Boot media | `/srv` partition | Size | Contents before the claim |
|---|---|---|---|---|
| `k3s-prod-1` | SATA SSD `C400-MTFDDAC512M` | `/dev/sda5` | 432.5 GiB ext4 | `lost+found` only (`du -sh` 20K; `df` 2.1 M of 425 G) |
| `k3s-prod-2` | **NVMe** `PM9A1 Samsung` | **`/dev/nvme0n1p5`** | 432.5 GiB ext4 | `lost+found` only (same) |
| `k3s-prod-3` | SATA SSD `WDC WDS500G2B0A` | `/dev/sda5` | 422.2 GiB ext4 | `lost+found` only (same) |
The `df` "5.1% used" that made `/srv` look occupied is **ext4's default 5% root-reserved
blocks**, not data — confirmed per node by `ls -A /srv` (only `lost+found`) and
`du -sh --exclude=lost+found /srv` (20 K). Nothing was destroyed.
**How the role resolves the device — discovery, not per-host inventory.** A single
`k3s_prod` group value cannot be right when node 2 is NVMe, and per-host `host_vars` would
freeze a *fact about today's hardware* into inventory, where it goes stale the first time a
node is re-imaged or a disk is swapped — and the stale value's failure mode is `mkfs` on the
wrong partition. So `group_vars/k3s_prod/vars.yml` declares **intent**
(`node_baseline_longhorn_source: claim-mountpoint`,
`node_baseline_longhorn_claim_mountpoint: /srv`) and
`roles/node_baseline/tasks/disks.yml` asks the **node** which block device is mounted there,
then proves the answer is safe before writing anything. Six independent asserts, all
fail-closed: it is a partition or whole disk (not tmpfs/NFS/bind), it is **not** backing
`/`, `/boot`, `/boot/efi`, `/usr`, `/var`, `/home`, `/tmp` or active swap (that set is built
from the node's *own* gathered facts plus `swapon`, never a hardcoded partition list), it is
already ext4, it carries no other filesystem label, it is ≥ 100 GiB, and it is **empty**.
**Unmount → relabel, not reformat.** `/srv` is already formatted and already mounted, and
`community.general.filesystem` will not touch a device that has a filesystem — it would
silently no-op and then the `LABEL=longhorn` mount would fail. The role therefore unmounts
`/srv`, removes its `/etc/fstab` entry (installer-written, pinned by UUID — leaving it would
re-mount the partition at `/srv` underneath Longhorn on the next boot), and stamps the label
with `e2label`. Relabelling rather than reformatting is deliberate: the partition is already
ext4 and already verified empty, so `mkfs` buys nothing, and if some assert were ever too
permissive the worst case is a renamed filesystem instead of a destroyed one.
**Idempotency guard.** A second run is a no-op, not a reformat. After the claim the
partition carries the ext4 label `k3s-data` and `/srv` is gone from `/proc/mounts` *and*
`/etc/fstab` — so "what is mounted at `/srv`" would find nothing on run 2. The role pivots
on the durable on-disk evidence instead: `blkid -L k3s-data` returning a device means the
node is already claimed, and **every** destructive task is gated on that returning nothing.
**The 2026-09-04 transition, handled idempotently.** All three nodes had *already* been
claimed under the previous naming (`LABEL=longhorn` at `/var/lib/longhorn`). Because the
idempotency pivot asks "does the **current** label exist?", changing the label makes an
already-claimed node answer *no* — and it would then walk into the greenfield claim path and
fail closed at "the claim mountpoint must actually be a mountpoint", `/srv` having been
consumed by that first claim. Fail-closed is correct, but it is not convergence, so the
transition is **declared** rather than inferred:
`node_baseline_data_retired_fs_label: longhorn` + `node_baseline_data_retired_mount:
/var/lib/longhorn` in `group_vars/k3s_prod/vars.yml`. The role then unmounts the retired
mountpoint, drops its `fstab` entry and re-stamps the label with `e2label` — a **metadata
write, never a reformat**, exactly as in the claim path — having first proved that the retired
mountpoint is still a mount, that it is backed by the very device the retired label names, and
that it is **empty** so nothing is orphaned. The common `LABEL=`-based mount task then brings
it up at `/data`, and the two per-tenant subdirectories are created (`0755` Longhorn, `0700`
k3s, since the latter holds the etcd DB, the node token and the image store).
The transition block is **one-time and self-retiring**: on every subsequent run
`blkid -L k3s-data` succeeds and none of it is reachable. The two retired-* lines are safe to
delete from `group_vars` once the fleet is converged (it is, as of 2026-09-04).
**Three independent reasons the LIVE dev cluster cannot enter that transition** — it has 18
Longhorn volumes on `/var/lib/longhorn` and unmounting it would orphan every replica:
(1) both retired-* vars default to `""` and are set only in the prod-scoped `group_vars`;
(2) the path is gated on `longhorn_source == 'claim-mountpoint'`, and dev is `device` — the
same structural gate that keeps dev out of the claim path; (3) the emptiness assert, which
fails closed on a partition with replica directories even if (1) and (2) were both defeated.
Dev's mountpoint, label and k3s data dir are also simply **not changing**.
**Run evidence (2026-09-04).** Full `node-baseline.yml` against `k3s_prod`: run 1 converged
(5–6 changed per node), run 2 **`changed=0` on all three nodes** with every transition and
claim task skipping. Per node afterwards: `blkid -L k3s-data` → `/dev/sda5` (nodes 1, 3) /
`/dev/nvme0n1p5` (node 2); `blkid -L longhorn` → gone; `/data` mounted with 403 G / 394 G
available; `/var/lib/longhorn` and `/srv` no longer mountpoints; `fstab` carrying exactly
`LABEL=k3s-data /data ext4 defaults,nofail,x-systemd.device-timeout=5s 0 0`; and `/var` down
to **5% used, ~11 GiB free**.
**Dev can never reach this path.** Both role defaults stay `source: device`, so the LIVE
dev cluster keeps its dedicated `/dev/vdb` (Longhorn) and `/dev/vdc` (etcd) and the claim
tasks are structurally unreachable there — they are gated on `claim-mountpoint`, which only
the prod-scoped `group_vars` sets.
### The accepted risk, and its exit condition
Dev was burned by exactly this shape: a Longhorn rebuild starved etcd WAL fsync
(`docs/plans/ssd-etcd-isolation.md`), which is why dev has a third, dedicated etcd device.
**Prod accepts the same shared-device contention** because the failure mode is materially
cheaper here: dev's contention was on a **spinning HDD** with seek-bound queueing; every
prod node is **SSD or NVMe** end-to-end, where concurrent rebuild I/O costs latency rather
than collapsing it. The three nodes also have no second device to give etcd — one physical
disk each, already fully partitioned, no free slot verified populated.
**Accepted to be MEASURED, not assumed.** Exit condition, in order of preference:
- **Measure it before it matters.** Under a deliberate Longhorn replica rebuild, watch
`etcd_disk_wal_fsync_duration_seconds` p99 and `etcd_disk_backend_commit_duration_seconds`
p99, plus `etcd_server_leader_changes_seen_total`. If fsync p99 stays inside etcd's
practical budget (~10 ms wal fsync / ~25 ms backend commit) with no leader churn, the
acceptance holds and this becomes a recorded measurement rather than an assumption.
- **If it does not hold:** add an M.2 NVMe per node (chassis slot to be verified
physically) and flip `node_baseline_etcd_source` back to `device` — the role still
supports that path unchanged, and it is the dev shape.
**~~Known, separate consequence — `/var` is only 12 GiB.~~ RESOLVED 2026-09-04 by moving
k3s's data dir.** This was recorded here on 2026-09-03 as a capacity item "to watch or
repartition". Re-measuring it against dev turned it from a watch item into a blocker: the
container image store alone is **21–25 GB per node** on the fully-deployed dev cluster and
`/var` totals **58–70 GB**, against prod's **11.8 GiB**. So prod would have filled `/var`
during its first image pulls, and with etcd on the same partition a full `/var` means etcd
cannot write its WAL. The fix was neither "watch" nor "repartition" but `--data-dir` — see
"THE 2026-09-04 REVISION" above. `/var` now carries only the OS's own usage (503 MB, 5%).
Consequences to settle before imaging nodes 2 and 3:
- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
@@ -609,24 +821,43 @@ Consequences to settle before imaging nodes 2 and 3:
a member corrupts — this drive has already logged 32 of them.
(Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
`DECISIONS-LOG.md`. It is not part of this verdict.)
- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
(`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
prod must override them in `group_vars`. On a single-disk node there is no second device at
all, so one of:
- **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
spindle.
- **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
- **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
physical failure domain.
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
- [ ] Record the chosen device paths here and set them in
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
`/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
`node-baseline` run against a node still carrying those placeholders now stops on an explicit
assert naming this section, instead of failing obscurely at `mkfs` time.
- [x] **Layout DECIDED and APPLIED 2026-09-03** — see "THE DECISION" above. `roles/node_baseline`
no longer expects a block-device path for either store; it takes a declared **source
enum** per environment (`roles/node_baseline/tasks/disks.yml`):
- **Longhorn** — `node_baseline_longhorn_source: claim-mountpoint` +
`node_baseline_longhorn_claim_mountpoint: /srv`. The `/dev/vdb` default (a virtio
name that does not exist on bare metal) is unused on prod; the real partition is
discovered on the node and guarded, because it is `/dev/sda5` on nodes 1 and 3 and
`/dev/nvme0n1p5` on node 2.
- **etcd** — `node_baseline_etcd_source: shared-data-partition` (was `root-filesystem`
until 2026-09-04). "No dedicated device" is a first-class, explicitly-configured state
that the `disks` assert **accepts**, not a blank variable that the tasks happen to
skip — and the *name* of that state now describes reality, because etcd is no longer
on the OS filesystem. The distinction is load-bearing, not cosmetic:
`shared-data-partition` is **asserted**, per node on every run, by checking that the
filesystem backing the etcd data dir really is the data partition's mount. Declaring
`k3s_data_dir: /data/k3s` does not by itself achieve that — had the mount silently
failed, `/data` would be a plain directory on `/` and etcd plus 21–25 GB of images
would land back on the OS filesystem, invisibly, until the disk filled. Leaving the
value at `root-filesystem` would have been a lie inventory told about the cluster, and
`root-filesystem` makes no such claim and gets no such assert. The role also still
asserts nothing is already mounted at the etcd data dir (so a half-migrated node
cannot be inherited silently) and reports which filesystem etcd actually lands on
(measured 2026-09-04: `/data`, 403 G / 394 G available, vs `/var`'s 10.7 GiB).
- The two rejected options stand rejected: **add a second drive** (no verified free slot;
the exit condition above reopens it if measurement fails) and **repartition** (same
single failure domain, plus a destructive re-image of three working nodes for no
isolation the SSDs do not already give).
- [x] **All three nodes get the identical layout.** Same declaration, same mountpoint, same
resulting `/data` + `/data/longhorn` + `/data/k3s`. Not the same *device path* and not
the same *size* (432.5 / 432.5 / 422.2 GiB) — Longhorn tolerates unequal node capacity;
it is replica *placement* that assumes a symmetric layout, and the layout is symmetric.
Confirmed per node 2026-09-04 by `findmnt`/`lsblk`/`blkid` after the transition.
- [x] **Chosen paths recorded here and set in `group_vars`.** The BLOCKED placeholders in
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` are replaced — by the source
enums above, plus deliberately-unusable `/dev/NOT-A-DEVICE-…` values for the two device
variables so that flipping a source back to `device` without supplying a real path
still fails on the role's assert instead of silently inheriting dev's `/dev/vdb`.
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
tasks this decision blocks — see §5's drive-health box for the exact command.
@@ -639,6 +870,138 @@ Consequences to settle before imaging nodes 2 and 3:
- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
`roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
so the same role covers bare metal. Nothing to port.
- [x] **KERNEL: a standing, deliberate deviation from the distro baseline — `linux-image-amd64`
from `trixie-backports` (DONE on all three nodes, 2026-09-04).**
**What.** `roles/node_baseline` gained a `kernel`-tagged block that enables
`trixie-backports` as a deb822 apt source and installs the `linux-image-amd64`
**metapackage** from it. Applied 2026-09-04; all three nodes landed on
**`linux-image-amd64` 7.1.8-1~bpo13+1 → kernel `7.1.8+deb13-amd64`**, rebooted one at a
time, and are running it.
```bash
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
```
**Why.** §0b. HP's R21 firmware publishes a corrupt EFI Memory Attributes Table; Linux
< 6.13 reserves memory from it without validating the size, and `k3s-prod-2` has booted
with 2.6 GiB of its 16 GB, twice. The fix is upstream commit **`8fbe4c49c0cc`**
*"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, first
released in **v6.13**, hardened by `5d0faa8e8369`). It is absent from v6.12 and from
`linux-6.12.y`, carries neither `Cc: stable` nor `Fixes:`, and there is **no kernel
cmdline workaround**. A k3s **server** that intermittently comes up with 2.6 GiB will
fail under etcd plus its workload share on any reboot, so this is a **prerequisite to
installing k3s**, not a follow-up.
**How it is scoped.** `node_baseline_backports_kernel` defaults to **false** in
`roles/node_baseline/defaults/main.yml` and is set **true** exactly once, in
`inventory/group_vars/k3s_prod/vars.yml` — the same seam `kube_vip_address` /
`k3s_api_host` / `cluster_kubeconfig_name` use. Dev's Proxmox VMs publish no MAT and are
genuinely unaffected; they must not be dragged onto a backports kernel as a side effect
of sharing the role, and a fourth prod node must inherit the deviation from **group
membership**, not from a `k3s-prod-*` hostname test. Verified: a `--tags kernel` run
against dev skips all 8 tasks on all 3 hosts.
**Security cost, accepted.** `*-backports` packages are provided **as-is** and are
**not** covered by the Debian Security Team's guarantees for the main archive — no
security-tracker commitment, no DSAs. Recorded here as the price of the nodes keeping
their RAM.
**No apt pin, deliberately.** The backports `Release` file carries `NotAutomatic: yes`
+ `ButAutomaticUpgrades: yes`, which apt already reads as *"priority 100 so nothing is
taken from backports by accident, 500 for a package explicitly installed from there so
it keeps getting updates."* Writing an explicit `Pin-Priority: 100` in
`preferences.d` would look like belt-and-braces and would in fact **switch off the
automatic security upgrades for the very kernel this deviation installs.** Enabling the
source is the whole configuration.
**Metapackage, not a pinned ABI.** A frozen `linux-image-6.19.14+deb13-amd64` would stop
receiving even backports' own updates the moment the ABI rolls. The metapackage tracks
whatever backports ships (>= 6.13 by construction). The **exact** version each node
landed on is recorded on the node itself at `/etc/node-baseline-backports-kernel`,
written by the same role, along with the rollback path below.
**Rollback.**
1. *Non-destructive, try first.* The stock kernels are still installed
(`6.12.107+deb13-amd64` and `6.12.94+deb13-amd64` on all three) and selectable from
GRUB → *Advanced options for Debian GNU/Linux*. That answers "is the new kernel the
problem?" without uninstalling anything — it also reinstates the memattr bug for that
boot.
2. *Permanent.* Set `node_baseline_backports_kernel: false` in
`group_vars/k3s_prod/vars.yml` **first** (otherwise the next Ansible run puts it
back), then on each node:
```bash
apt-get install --reinstall linux-image-amd64/trixie
apt-get purge linux-image-7.1.8+deb13-amd64
rm /etc/apt/sources.list.d/trixie-backports.sources
apt-get update && update-grub && reboot
```
**The role does NOT reboot**, and that is deliberate — see §0b "SECOND FAULT". Reboots
are a serialized, supervised operator step: one node at a time, verify each returns
before touching the next, never node 2 first. On 2026-09-04 all three returned in
**39–41 s** (node 1 39 s, node 3 41 s, node 2 39 s); node 2 did **not** reproduce its
2026-09-03 hang, but one clean warm reboot is not a fix and that defect stays open.
**Exit condition — how this deviation ends.** If `8fbe4c49c0cc` is ever picked up into
`linux-6.12.y` (and therefore into Debian's 6.12 rebase, which reaches us by plain
`apt upgrade`), this can be removed entirely: set `node_baseline_backports_kernel:
false`, roll back per the note on each node, and delete the `kernel` block from
`roles/node_baseline`. §0b records why that pickup will not happen on its own and what
request would cause it.
- [x] **Verification of the fix, and a correction to how it is verified (2026-09-04).**
§0b says *"A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table
detected!`"*. That is true but **conditional, and it did not happen on any of the three
nodes.** `efi_memattr_init()` on >= 6.13 has **three** gates and only the third prints
that line:
| gate | message | reachable when |
|---|---|---|
| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, `desc_size` does not |
| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* `desc_size` read plausibly, entry count does not |
**All three are rejections**, and which one a boot prints is decided by *which field*
that boot's garbage corrupts. On 7.1.8 all three nodes hit the **first** gate:
| node | `MEMATTR=` | kernel line on 7.1.8 | MemTotal | reserved at boot |
|---|---|---|---|---|
| 1 | `0xbee84018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1030` | 31.1 GiB | 0.9 GiB |
| 2 | `0xbeeb0018` | `efi: memattr: Unexpected EFI Memory Attributes table version -1569980416` | 15.4 GiB | 0.6 GiB |
| 3 | `0xbeecc018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` | 31.1 GiB | 0.9 GiB |
Node 2 is the result that matters: **`0xbeeb0018` is the exact address that produced
2.6 GiB twice**, and on 7.1.8 it reserved 634,904 K instead of 13,913,856 K.
**Boot-independent proof the fix is compiled in**, since the log line is a lottery: the
running kernel image itself contains the string. Decompressing `/boot/vmlinuz-$(uname
-r)` on all three nodes yields, verbatim —
```
efi: memattr: [Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)
efi: memattr: Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)
```
— neither of which exists in 6.12. That is the positive proof; the `dmesg` line is not
available on demand.
**Two bugs in `roles/node_health` were fixed as a consequence**, because without them
the health check could not show the fix landing:
- `MEMATTR_VERSION_RE` in `files/node_health_collect.py` matched `(\d+)` only. The
kernel prints that field with `%d` — **signed** — so node 2's `-1569980416` did not
match, the state fell through to `clean`, and the node was reported **PASS "the kernel
accepted it without complaint"**: a false pass on the exact bug the check exists to
find. Now `(-?\d+)`.
- `firmware.efi_memattr` treated `bad_version` as WARN unconditionally, so a node on a
fixed kernel would have stayed WARN forever. The verdict now keys on the running
kernel version (`memattr_kernel_fixed`, `>= 6.13`): the same log line is WARN on
< 6.13 (a lucky boot) and PASS on >= 6.13 (the validation working). The pre-existing
`corrupt_rejected` PASS branch is unchanged and remains reachable.
**`playbooks/node-health.yml -e node_health_hosts=k3s_prod`, after the reboots:
`firmware.efi_memattr` = PASS on all three, `memory.physical_total` = PASS on all
three.** Dev re-run unchanged.
## 4. Network (per node)
infra/ansible/inventory/group_vars/k3s_prod/vars.yml
+148 -17
View File
@@ -87,24 +87,155 @@ kube_vip_address: "192.168.2.9"
# — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
# backports kernel as a side effect of sharing the role), and this line switches it on for
# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
# rather than from anyone remembering.
#
# The overrides below are deliberately present-but-broken: leaving them UNSET would
# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
# hardware — a confusing failure that looks like a wiring bug. This value fails just
# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
# updates these two lines. All three nodes must end up with an identical layout
# (Longhorn replica placement assumes symmetry) before any of this is used for real.
node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
# in §3.
#
# COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
# guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
# the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
# back to false, roll back per the note the role writes to
# /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
# roles/node_baseline.
#
# The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
# FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
node_baseline_backports_kernel: true
# --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
# Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
#
# DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
# disk, claimed and mounted at the NEUTRAL mountpoint `/data`
# (LABEL=k3s-data), then split between TWO tenants:
# /data/longhorn Longhorn's dataPath
# /data/k3s k3s --data-dir (etcd AND the containerd image store)
# ETCD -> NO dedicated device. It lives inside /data/k3s on that partition.
#
# ================================ THE 2026-09-04 REVISION ================================
# The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
# /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
# data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
# deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
# would therefore have filled /var during its FIRST image pulls — and with etcd on the same
# partition, a full /var means etcd cannot write its WAL. That is cluster death, not
# degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
# mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
# tenants. Nothing was installed on prod when this was decided, which is exactly why it was
# decided then — it is cheap now and very expensive after bring-up.
#
# THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
# were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
# contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
# MEASURED, not assumed — exit condition in §2.
# =========================================================================================
#
# Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
# not as a device path and a blank. That distinction is the point of this block:
#
# 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
# k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
# (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
# Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
# freezes a fact about today's hardware into inventory, where it goes stale silently the
# first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
# nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
# So inventory declares the INTENT ("the partition currently mounted at /srv") and the
# role discovers the device on the node, then proves it is safe to claim (a partition or
# disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
# unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
#
# 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
# describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
# layout etcd is inside /data/k3s on the claimed data partition, which is neither a
# dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
# Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
# and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
# on every run, that the filesystem backing the etcd data dir really is /data. If the
# mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
# WAL would land back on the OS filesystem, invisibly, until the disk filled. The
# `root-filesystem` value makes no such claim and gets no such assert — which is exactly
# why the two are distinct enum values rather than one with a comment.
#
# 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
# the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
# the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
# this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
# k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
# Longhorn volumes are untouched.
node_baseline_longhorn_source: claim-mountpoint
node_baseline_longhorn_claim_mountpoint: /srv
node_baseline_etcd_source: shared-data-partition
# --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
# `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
# CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
# /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
# as the destination would make "claimed" and "not claimed" indistinguishable.
# `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
# partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
# else's named volume — so labelling ours `data` would make a stranger's `data` volume look
# already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
# `k3s-etcd` label convention already in this repo.
node_baseline_data_mount: /data
node_baseline_data_fs_label: k3s-data
# Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
# equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
# side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
# default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
node_baseline_longhorn_data_path: /data/longhorn
# --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
# The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
# `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
# label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
# would walk into the greenfield claim path — which fails closed at "the claim mountpoint
# must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
# is right but it is not convergence, so the transition is DECLARED here: the role unmounts
# the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
# the partition is empty (nothing orphaned) and is the device the retired label names.
# Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
# SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
# then skip on the retired-label lookup and nothing else depends on them.
node_baseline_data_retired_fs_label: longhorn
node_baseline_data_retired_mount: /var/lib/longhorn
# --- k3s --data-dir (§2) -----------------------------------------------------------
# THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
# claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
# `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
# which defers to this variable) creates the directory 0700 on the data partition and derives
# etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
# Ansible side cannot disagree with itself.
#
# Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
# it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
# dir would orphan that mount under a running cluster.
k3s_data_dir: /data/k3s
# Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
# source uses a device path any more, but leaving these UNSET would let prod silently
# inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
# metal) the moment anyone flipped a source back to `device`. These values can never parse
# as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
# before mkfs is ever reached.
node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
# --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
# roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
infra/ansible/roles/k3s_server/defaults/main.yml
+32
View File
@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
# the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
k3s_disable: "traefik,servicelb,local-storage"
# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
# image store (agent/containerd) — which is by far the largest tenant.
#
# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
# running cluster's entire state directory out from under it and orphan that etcd mount.
#
# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
# node inherits it from group membership rather than from anyone remembering. It is set
# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
# this same variable). One line moves both.
#
# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
#
# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
# Restart handler for a value that is not changing. The install flags are the correct and
# sufficient place, and prod has no k3s installed yet.
k3s_data_dir: "/var/lib/rancher/k3s"
# Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
k3s_server_mode: "secondary"
infra/ansible/roles/k3s_server/tasks/main.yml
+1
View File
@@ -22,6 +22,7 @@
--write-kubeconfig-mode=644
{% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
--node-ip={{ k3s_node_ip }}
--data-dir={{ k3s_data_dir }}
# On an ALREADY-installed cluster the install task above is idempotent (creates:
# k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
infra/ansible/roles/node_baseline/defaults/main.yml
+152 -4
View File
@@ -18,22 +18,170 @@ node_baseline_packages:
# node from first baseline — package installation is the role's job, never a manual step
# (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
- smartmontools
# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
# baseline — not a one-off `apt install` someone has to remember on the next node.
#
# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
# from its contents, so an unlucky boot reserves everything above the table and the node
# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
#
# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
# encode the environment into the ROLE, would silently miss a fourth prod node named
# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
#
# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
#
# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
# delete this block. Nothing else in the repo depends on it.
node_baseline_backports_kernel: false
# The suite, mirror and components the source line is built from. Split out so a mirror
# change is a variable, not an edit to a heredoc in tasks/main.yml.
node_baseline_backports_suite: trixie-backports
node_baseline_backports_mirror: "http://deb.debian.org/debian"
# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
# surface beyond what the node already trusts is a separate decision nobody has taken.
node_baseline_backports_components:
- main
- non-free-firmware
# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
node_baseline_backports_kernel_packages:
- linux-image-amd64
# Provenance/rollback note written on the node. An operator standing in front of a prod box
# with no repo checkout must be able to answer "why is this machine not on the distro
# kernel, and how do I undo it?" from the machine itself.
node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
node_baseline_kernel_modules:
- overlay
- br_netfilter
- iscsi_tcp
- dm_crypt
- nfs
# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
#
# node_baseline_longhorn_source:
# device dedicated block device at node_baseline_longhorn_device (DEV: the
# tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
# claim-mountpoint re-purpose the partition the node ALREADY has mounted at
# node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
# path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
# and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
# is correct and a hardcoded one would eventually mkfs the wrong
# partition on a future node. Destructive; heavily guarded — read
# tasks/disks.yml before changing anything here.
#
# node_baseline_etcd_source:
# device dedicated block device at node_baseline_etcd_device (DEV:
# /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
# isolated from Longhorn rebuild I/O after dev was burned by HDD
# contention).
# root-filesystem etcd lives on the OS filesystem, with NO dedicated device and
# NO claimed data partition under it. Kept as a valid, supported
# shape; NOT what prod uses any more (see below).
# shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
# CLAIMED data partition (PROD). No dedicated device, but also NOT
# the OS filesystem: the whole point of the 2026-09-04 layout
# revision is that /var (11.8 GiB) can hold neither the container
# image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
#
# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
# var, which still fails the `disks` assert.
node_baseline_longhorn_source: device
node_baseline_longhorn_device: /dev/vdb
node_baseline_longhorn_mount: /var/lib/longhorn
node_baseline_longhorn_fs_label: longhorn
# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
#
# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
node_baseline_data_mount: /var/lib/longhorn
node_baseline_data_fs_label: longhorn
# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
# same fact, and the two are checked against each other by nothing but this comment.
node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
# install flags, this role reads the same value to create the directory and to place etcd,
# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
# value (higher precedence than any role default) is what actually resolves here, even
# though roles/k3s_server is not in the node-baseline play; the default() keeps
# node-baseline.yml runnable standalone.
node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
# default so a dev host cannot enter the claim path even by accident.
node_baseline_longhorn_claim_mountpoint: ""
# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
node_baseline_data_retired_fs_label: ""
node_baseline_data_retired_mount: ""
# Sanity floor for the claim path: a data partition smaller than this is not "a small
# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
node_baseline_longhorn_min_size_gb: 100
# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
# device renames (matches the Longhorn LABEL= approach).
# device renames (matches the data partition's LABEL= approach).
node_baseline_etcd_source: device
node_baseline_etcd_device: /dev/vdc
node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
# to /data/k3s/server/db, inside the claimed partition.
node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
node_baseline_etcd_fs_label: k3s-etcd
# Mounts whose backing devices the claim path must NEVER touch. Used to build the
# protected-device set from the node's own gathered facts (plus live swap), so a node
# partitioned differently from the fleet is judged on its own layout rather than against
# a hardcoded list of partition names.
node_baseline_protected_mounts:
- /
- /boot
- /boot/efi
- /usr
- /var
- /home
- /tmp
# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
infra/ansible/roles/node_baseline/tasks/disks.yml
+642
View File
@@ -0,0 +1,642 @@
---
# =====================================================================================
# node_baseline — DATA DISKS. Every task here carries the `disks` tag (applied once by
# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
# Split into its own file rather than living in main.yml because it is the one part of
# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
#
# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
#
# dev (group_vars/all.yml, role defaults)
# node_baseline_longhorn_source: device -> mkfs /dev/vdb, mount LABEL=longhorn
# at /var/lib/longhorn (== dataPath)
# node_baseline_etcd_source: device -> mkfs /dev/vdc, mount LABEL=k3s-etcd
# k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
# THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
# exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
#
# prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
# node_baseline_longhorn_source: claim-mountpoint (claim mountpoint /srv)
# node_baseline_data_mount: /data (LABEL=k3s-data)
# node_baseline_etcd_source: shared-data-partition
# k3s_data_dir: /data/k3s
# ONE physical disk per node, already fully partitioned by the Debian installer. The
# ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
# between TWO tenants: /data/longhorn (Longhorn replicas) + /data/k3s (the whole k3s
# data dir: etcd AND the containerd image store).
#
# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
# partition is the fix; the partition therefore serves two tenants, and neither its
# mountpoint nor its filesystem label may name only one of them.
#
# WHY /data AND LABEL k3s-data.
# * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
# Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
# at, and the idempotency contract is that /srv is gone from /proc/mounts and
# /etc/fstab once the claim has happened. Re-using it as the destination would make
# "claimed" and "not yet claimed" indistinguishable.
# * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
# accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
# `data` as an example of somebody else's named volume. Labelling ours `data` would make
# a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
# cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
# convention in this repo.
#
# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
# empty string, and the second one must never silently succeed. `root-filesystem` and
# `shared-data-partition` are explicit, greppable statements of intent that the assert
# below accepts; an unset or placeholder device is still a hard failure. The two differ in
# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
# than printing a claim it never checked (see the etcd section).
#
# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
# point at the wrong partition on some future node and mkfs it. So the role asks the NODE
# which block device is mounted at the claim mountpoint, and then proves that answer is
# safe (see the assert wall below) before touching it. The inventory declares INTENT
# ("the partition currently mounted at /srv"); the node supplies the FACT.
# =====================================================================================
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
# its data-disk layout. Two failure shapes are caught:
# * an undeclared/typo'd source enum, and
# * source: device with a device path that is a PLACEHOLDER rather than a real device.
# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
# vars now that prod uses neither: if anyone ever flips a source back to `device` without
# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
# stays in the token list because that was the pre-decision placeholder and may still be
# carried by other environments/branches.
- name: Assert this environment's data-disk layout has actually been declared
ansible.builtin.assert:
that:
- node_baseline_longhorn_source in ['device', 'claim-mountpoint']
- node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
# `shared-data-partition` is only coherent when there IS a claimed data partition to
# share, and only when the k3s data dir actually sits INSIDE it. Catch the
# contradiction here, in inventory terms, rather than three sections later when etcd
# has silently landed back on the OS filesystem.
- >-
node_baseline_etcd_source != 'shared-data-partition'
or node_baseline_longhorn_source == 'claim-mountpoint'
- >-
node_baseline_etcd_source != 'shared-data-partition'
or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
# The transition knobs are a matched PAIR: a retired label with no retired mountpoint
# (or vice versa) cannot describe a state to migrate from.
- >-
(node_baseline_data_retired_fs_label | length > 0)
== (node_baseline_data_retired_mount | length > 0)
- >-
node_baseline_longhorn_source != 'device'
or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
- >-
node_baseline_etcd_source != 'device'
or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
- >-
node_baseline_longhorn_source != 'claim-mountpoint'
or (node_baseline_longhorn_claim_mountpoint | length) > 0
fail_msg: >-
This host's data-disk layout is not declared.
longhorn_source={{ node_baseline_longhorn_source }}
(device={{ node_baseline_longhorn_device }},
claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
etcd_source={{ node_baseline_etcd_source }}
(device={{ node_baseline_etcd_device }}).
Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
`claim-mountpoint` (re-purpose the partition already mounted at
node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
`device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
data partition — prod's shape since 2026-09-04, which additionally requires
longhorn_source `claim-mountpoint` and a k3s data dir under
node_baseline_data_mount={{ node_baseline_data_mount }};
k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
Declare it in the environment's group_vars and record the reasoning in
§2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
`--tags packages`, which skips every disk task including this assert.
quiet: true
# =====================================================================================
# THE DATA PARTITION (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
# =====================================================================================
# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
# rather than a reformat. Once the claim has happened the partition carries the ext4
# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
# mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
# "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
# path means converged, rc != 0 means the claim has not been made. Everything destructive
# below is gated on rc != 0.
- name: "Data partition: look for a filesystem already carrying the data-partition label"
ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
register: node_baseline_lh_labelled
changed_when: false
failed_when: false
check_mode: false
# --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
# WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
# label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
# takes over — which then fails closed at "the claim mountpoint must actually be a
# mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
# behaviour but it is not convergence, and the prod nodes are in exactly that state today:
# claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
# So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
#
# WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
# partition with the new label. It never touches the DATA and never reformats: e2label is a
# metadata write, exactly as in the claim path below. The common mount task then mounts
# LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
# `blkid -L <new label>` succeeds and none of this is reachable.
#
# THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
# volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
# unmounting it would orphan every one of them:
# 1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
# group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
# 2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
# the same structural gate that keeps dev out of the claim path keeps it out of this one.
# 3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
# replica directories on it fails closed and nothing is unmounted.
# Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
- name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
register: node_baseline_lh_retired_labelled
changed_when: false
failed_when: false
check_mode: false
when:
- node_baseline_longhorn_source == 'claim-mountpoint'
- node_baseline_data_retired_fs_label | length > 0
- node_baseline_lh_labelled.rc != 0
- name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
ansible.builtin.set_fact:
# A node claimed under the RETIRED label. Mutually exclusive with the claim below.
node_baseline_lh_transition_needed: >-
{{ node_baseline_longhorn_source == 'claim-mountpoint'
and node_baseline_data_retired_fs_label | length > 0
and node_baseline_lh_labelled.rc != 0
and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
# True only for a claim-mountpoint environment that has NOT been claimed under EITHER
# label. A `device` environment (dev) can never enter the claim path at all — that is
# the structural guarantee that no generalisation of prod's behaviour can reach the
# LIVE dev nodes and their /dev/vdb Longhorn replicas.
node_baseline_lh_claim_needed: >-
{{ node_baseline_longhorn_source == 'claim-mountpoint'
and node_baseline_lh_labelled.rc != 0
and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
- name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
ansible.builtin.command: >-
findmnt --noheadings --first-only --output SOURCE
--mountpoint {{ node_baseline_data_retired_mount }}
register: node_baseline_lh_retired_findmnt
changed_when: false
failed_when: false
check_mode: false
when: node_baseline_lh_transition_needed | bool
# Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
# not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
# an unmount of a populated Longhorn store would orphan every replica on the node.
- name: "Data partition (transition): read the contents of the retired mountpoint"
ansible.builtin.find:
paths: "{{ node_baseline_data_retired_mount }}"
file_type: any
hidden: true
recurse: false
excludes: ['lost+found']
register: node_baseline_lh_retired_contents
when: node_baseline_lh_transition_needed | bool
- name: "Data partition (transition): prove this node is safe to re-label"
ansible.builtin.assert:
that:
# 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
# retired label names. The only state this role knows how to transition is the one
# the previous version of it produced; anything else is an operator decision.
- (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
- >-
(node_baseline_lh_retired_findmnt.stdout | default('') | trim)
== (node_baseline_lh_retired_labelled.stdout | default('') | trim)
# 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
# not the node we think it is, or a tenant has already written data — in both cases
# unmounting and re-labelling under it is destructive and the role must stop.
- (node_baseline_lh_retired_contents.files | length) == 0
fail_msg: >-
REFUSING to transition this node's data partition from
LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
(findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
{{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
If the partition holds DATA, this transition would orphan it — Longhorn replicas are
not recoverable from an unmounted, re-labelled filesystem. Migrate the data
deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
Do NOT relax this assert.
quiet: true
when: node_baseline_lh_transition_needed | bool
- name: "Data partition (transition): report exactly what is about to change"
ansible.builtin.debug:
msg: >-
Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
No filesystem is created and no data is moved.
when: node_baseline_lh_transition_needed | bool
# e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
# breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
- name: "Data partition (transition): unmount the retired mountpoint"
ansible.posix.mount:
path: "{{ node_baseline_data_retired_mount }}"
state: unmounted
when: node_baseline_lh_transition_needed | bool
- name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
ansible.posix.mount:
path: "{{ node_baseline_data_retired_mount }}"
state: absent_from_fstab
backup: true
when: node_baseline_lh_transition_needed | bool
- name: "Data partition (transition): re-stamp the partition with the new filesystem label"
ansible.builtin.command: >-
e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
{{ node_baseline_data_fs_label }}
changed_when: true
when: node_baseline_lh_transition_needed | bool
# --- The claim (prod, first run only) -------------------------------------------------
# Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
# mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
# returning the device behind `/` — which is precisely the mistake that would hand the
# root partition to e2label.
- name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
ansible.builtin.command: >-
findmnt --noheadings --first-only --output SOURCE
--mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
register: node_baseline_lh_claim_findmnt
changed_when: false
failed_when: false
check_mode: false
when: node_baseline_lh_claim_needed | bool
- name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
ansible.builtin.assert:
that:
- node_baseline_lh_claim_findmnt.rc == 0
- (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
fail_msg: >-
node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
The role will NOT go looking for "a big partition" to claim instead. Either the
node is partitioned differently from the fleet — in which case fix the node or give
this host its own declaration — or the claim already happened and the label was
removed by hand.
quiet: true
when: node_baseline_lh_claim_needed | bool
- name: "Longhorn (claim): inspect the candidate partition"
ansible.builtin.command: >-
lsblk --json --nodeps --bytes
--output PATH,TYPE,FSTYPE,LABEL,SIZE
{{ node_baseline_lh_claim_findmnt.stdout | trim }}
register: node_baseline_lh_claim_lsblk
changed_when: false
check_mode: false
when: node_baseline_lh_claim_needed | bool
# The devices this role must never, under any circumstances, hand to e2label. Built from
# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
# NODE's account of what is load-bearing, not a hardcoded list of partition names.
- name: "Longhorn (claim): list the devices currently backing active swap"
ansible.builtin.command: swapon --noheadings --show=NAME
register: node_baseline_lh_swapdevs
changed_when: false
failed_when: false
check_mode: false
when: node_baseline_lh_claim_needed | bool
# Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
# ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
- name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
ansible.builtin.find:
paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
file_type: any
hidden: true
recurse: false
excludes: ['lost+found']
register: node_baseline_lh_claim_contents
when: node_baseline_lh_claim_needed | bool
# THE GUARD WALL. Five independent conditions, every one of which must hold before the
# role modifies a filesystem it did not create. Any single failure stops the play with
# the evidence in the message — nothing is unmounted, relabelled or mounted first.
- name: "Longhorn (claim): prove the candidate partition is safe to claim"
ansible.builtin.assert:
that:
# 1. It is a real block device — a partition or a whole disk. Catches a claim
# mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
# which would have produced a non-/dev SOURCE or an lsblk row of another type.
- node_baseline_lh_candidate.type in ['part', 'disk']
# 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
# so a differently-partitioned node is caught by its own facts.
- node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
# 3. It already holds ext4. The claim path RELABELS rather than reformats (see
# below), which is only meaningful for the filesystem type we are about to
# declare in fstab. Anything else is an operator decision, not a role's.
- node_baseline_lh_candidate.fstype == 'ext4'
# 4. It carries no filesystem label, or already carries ours. A partition labelled
# `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
# it would make it unfindable by its own fstab entry.
- (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
# 5. It is big enough to be the partition we meant. A data partition of a few GB is
# not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
- (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
# 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
# ext4 5% root reserve; this is where that premise is verified per node instead
# of trusted. If any node has real data on it, the play stops here — the role
# never wipes data it cannot account for.
- (node_baseline_lh_claim_contents.files | length) == 0
fail_msg: >-
REFUSING to claim {{ node_baseline_lh_candidate.path }}
(mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
type={{ node_baseline_lh_candidate.type }},
fstype={{ node_baseline_lh_candidate.fstype }},
label='{{ node_baseline_lh_candidate.label | default('', true) }}',
size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
(minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
{{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
devices this node needs and that must never be claimed:
{{ node_baseline_lh_protected_devices }}.
This is the destructive step of the role and it fails CLOSED. If the partition
holds data, back it up or pick a different one; do NOT relax this assert.
quiet: true
vars:
node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
# The node's OWN account of what it cannot afford to lose: the devices behind the
# mounts in node_baseline_protected_mounts (from gathered facts, so a node
# partitioned differently is judged on its own layout) plus every active swap device.
# Built here rather than hardcoded so no partition-name list can go stale.
node_baseline_lh_protected_devices: >-
{{ (ansible_facts.mounts | default([])
| selectattr('mount', 'in', node_baseline_protected_mounts)
| map(attribute='device') | list)
+ (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
when: node_baseline_lh_claim_needed | bool
- name: "Longhorn (claim): report exactly what is about to be claimed"
ansible.builtin.debug:
msg: >-
Claiming {{ node_baseline_lh_candidate.path }}
({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
as the data partition: unmount -> drop its fstab entry ->
relabel LABEL={{ node_baseline_data_fs_label }} ->
mount at {{ node_baseline_data_mount }}.
vars:
node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
when: node_baseline_lh_claim_needed | bool
# --- Why the role must unmount and relabel, not just point fstab at the device ---------
# The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
# /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
# path does not. The installer-created /srv filesystem has NO label (fstab pins it by
# UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
# the unmount first — and the fstab entry must go in the same breath, or the next boot
# re-mounts the same partition at /srv underneath Longhorn.
#
# RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
# mkfs would buy nothing and would destroy anything the emptiness check got wrong.
# e2label is a metadata write: if some assert above was too permissive, the worst case is
# a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
# cannot do this job: it declines to touch a device that already has a filesystem, so it
# would silently no-op and then the LABEL= mount would fail with no useful message.)
- name: "Longhorn (claim): unmount the claim mountpoint"
ansible.posix.mount:
path: "{{ node_baseline_longhorn_claim_mountpoint }}"
state: unmounted
when: node_baseline_lh_claim_needed | bool
- name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
ansible.posix.mount:
path: "{{ node_baseline_longhorn_claim_mountpoint }}"
state: absent_from_fstab
backup: true
when: node_baseline_lh_claim_needed | bool
- name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
ansible.builtin.command: >-
e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
{{ node_baseline_data_fs_label }}
changed_when: true
when: node_baseline_lh_claim_needed | bool
# --- The dedicated-device path (dev) --------------------------------------------------
# Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
# never reformats a device that already carries a filesystem.
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
community.general.filesystem:
fstype: ext4
dev: "{{ node_baseline_longhorn_device }}"
opts: "-L {{ node_baseline_data_fs_label }}"
when: node_baseline_longhorn_source == 'device'
# --- Common to ALL paths (device, claim, transition) ----------------------------------
# Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
# mount task. Idempotent: on a converged node this reports ok, which is what makes a
# re-run a no-op rather than a reformat. It is also what completes the transition above:
# the retired mountpoint is gone from fstab and the partition now answers to the new label,
# so this task mounts it at the new path.
- name: Mount the data partition
ansible.posix.mount:
path: "{{ node_baseline_data_mount }}"
src: "LABEL={{ node_baseline_data_fs_label }}"
fstype: ext4
opts: "defaults,nofail,x-systemd.device-timeout=5s"
state: mounted
# --- The per-tenant subdirectories ---------------------------------------------------
# WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
# tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
# subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
# /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
# either there would at best be a no-op and at worst re-mode the running dev cluster's data
# dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
# mounted — i.e. only the directories it is actually responsible for. That test is derived
# from the paths themselves, so no environment has to remember to switch it off.
#
# MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
# the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
# etcd database, the node token and the containerd image store, and k3s would tighten it to
# 0700 itself on first start — doing it here means it is never briefly world-readable.
- name: Create the per-tenant data directories on the data partition
ansible.builtin.file:
path: "{{ item.path }}"
state: directory
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
- { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
loop_control:
label: "{{ item.path }} ({{ item.tenant }})"
when: item.path.startswith(node_baseline_data_mount ~ '/')
# =====================================================================================
# ETCD DATA DIR
# =====================================================================================
# --- The dedicated-device path (dev) --------------------------------------------------
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
# a disk that already carries a filesystem, so re-running against an already-migrated
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
# role would mount over a populated directory.
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
community.general.filesystem:
fstype: ext4
dev: "{{ node_baseline_etcd_device }}"
opts: "-L {{ node_baseline_etcd_fs_label }}"
when: node_baseline_etcd_source == 'device'
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
ansible.posix.mount:
path: "{{ node_baseline_etcd_mount }}"
src: "LABEL={{ node_baseline_etcd_fs_label }}"
fstype: ext4
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
opts: "defaults,noatime"
dump: "0"
passno: "2"
state: mounted
when: node_baseline_etcd_source == 'device'
# --- The no-dedicated-device paths (prod) ---------------------------------------------
# Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
# populated — so etcd gets no device of its own. That is an operator decision
# (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
# UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
# on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
# latency rather than collapsing it. Accepted to be MEASURED, not assumed.
#
# WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
# filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
# 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
# 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
# could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
# partition and etcd moved with it: `shared-data-partition`. Both enum values are still
# supported, and the role now PROVES which one is true instead of printing a claim.
#
# There is nothing to CREATE here for either shape — the data-partition section above
# already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
# role's job is to verify the assumption rather than act on it.
- name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
ansible.builtin.command: >-
findmnt --noheadings --first-only --output SOURCE
--mountpoint {{ node_baseline_etcd_mount }}
register: node_baseline_etcd_existing_mount
changed_when: false
failed_when: false
check_mode: false
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
- name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
ansible.builtin.assert:
that:
- node_baseline_etcd_existing_mount.rc != 0
fail_msg: >-
node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
ALREADY a separate mount, backed by
{{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
contradiction the role will not resolve on its own: either this node was previously
built with a dedicated etcd device (in which case unmount it deliberately, after
accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
the node in a state where the declared layout and the real one disagree.
quiet: true
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
# Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
# WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
# existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
# that does exist — the same filesystem the data dir will be created on. TARGET is captured
# separately from the human-readable line because the assert below judges on it.
- name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
ansible.builtin.shell: |
set -eu
p={{ node_baseline_etcd_mount | quote }}
while [ ! -d "$p" ]; do p=$(dirname "$p"); done
findmnt --noheadings --first-only --output TARGET --target "$p"
findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
register: node_baseline_etcd_backing_fs
changed_when: false
check_mode: false
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
# THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
# `shared-data-partition` is a distinct enum value rather than a comment. Declaring
# k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
# above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
# of container images would land back on the OS filesystem — the exact failure this change
# exists to prevent, and completely invisible until the disk filled. So the role checks that
# the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
# run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
# which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
# whole record for that shape.
- name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
ansible.builtin.assert:
that:
- >-
(node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
== node_baseline_data_mount
fail_msg: >-
node_baseline_etcd_source is `shared-data-partition`, which asserts that
{{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
{{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
'{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
(full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
The data partition is therefore NOT mounted where this layout requires, and k3s would
put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
this node until the mount is correct.
quiet: true
when: node_baseline_etcd_source == 'shared-data-partition'
- name: "etcd (no dedicated device): show which filesystem etcd will live on"
ansible.builtin.debug:
msg: >-
etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
(source={{ node_baseline_etcd_source }}); it lands on
{{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
(SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
measured risk — §2.
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
infra/ansible/roles/node_baseline/tasks/main.yml
+263 -66
View File
@@ -1,6 +1,6 @@
---
# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
# and the DATA-DISK work have different readiness: packages are safe on any node from
# first contact, while the disk tasks need block devices that a new environment may not
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -12,6 +12,10 @@
# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
# at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
# ansible-playbook playbooks/node-baseline.yml --tags dns
# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
# it must be runnable without touching a disk decision that is still open:
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
ansible.builtin.apt:
@@ -21,6 +25,237 @@
cache_valid_time: 3600
tags: [packages]
# =====================================================================================
# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
# =====================================================================================
# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
# FUTURE prod node gets it from provisioning instead of from someone remembering.
#
# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
# (first released in v6.13) validates the size and rejects the table instead. It is absent
# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
# stable feed will ever pick it up. There is NO kernel cmdline workaround —
# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
#
# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
# change for no benefit. See defaults/main.yml for why this is a variable rather than a
# `k3s-prod-*` hostname test.
#
# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
# before touching the next, never node 2 first — and a role cannot enforce that. The last
# task here reports the pending reboot loudly instead of letting a green play imply the node
# is already fixed.
- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
when: node_baseline_backports_kernel | bool
tags: [kernel]
block:
# The suite name and the running release must agree, or apt resolves no candidate and
# `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
# a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
# dist-upgraded to forky while group_vars still says trixie-backports).
- name: Assert the configured backports suite matches this node's Debian release
ansible.builtin.assert:
that:
- ansible_facts['distribution'] == 'Debian'
- node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
fail_msg: >-
node_baseline_backports_kernel is enabled but node_baseline_backports_suite
({{ node_baseline_backports_suite }}) is not the backports suite for this node's
release ({{ ansible_facts['distribution'] }}
{{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
candidate and this play would report success while leaving the node on the
vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
quiet: true
# deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
# prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
# rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
# idempotent in a way `apt_repository` — which rewrites and reformats — is not.
#
# THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
# backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
# apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
# 500 for a package that was explicitly installed from there so it keeps getting updates.
# Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
# HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
# be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
- name: Enable the Debian backports apt source (deb822)
ansible.builtin.copy:
dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
owner: root
group: root
mode: "0644"
content: |
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
# A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
# machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
# Table its firmware publishes (upstream 8fbe4c49c0cc). See
# infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
# on this node.
#
# Backports is NotAutomatic, so nothing is taken from here unless asked for by name
# with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
# ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
# Pin-Priority would switch that off.
Types: deb
URIs: {{ node_baseline_backports_mirror }}
Suites: {{ node_baseline_backports_suite }}
Components: {{ node_baseline_backports_components | join(' ') }}
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
register: node_baseline_backports_source
- name: Refresh the apt cache after enabling backports
ansible.builtin.apt:
update_cache: true
when: node_baseline_backports_source is changed
# `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
# nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
# nothing at all — the single most likely way to write this task and have it silently fix
# nothing forever. `latest` + `default_release` resolves the candidate inside backports
# and upgrades the metapackage to it. Once the node is current a re-run is a genuine
# no-op: no change, no churn, no spurious reboot flag. Scope is exactly
# node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
# is dragged out of the main archive.
- name: Install the fixed kernel from backports (>= 6.13)
ansible.builtin.apt:
name: "{{ node_baseline_backports_kernel_packages }}"
state: latest
default_release: "{{ node_baseline_backports_suite }}"
update_cache: true
cache_valid_time: 3600
register: node_baseline_backports_kernel_apt
# WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
# metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
# answer anybody can roll back from six months later.
- name: Read back the installed backports kernel metapackage version
ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
register: node_baseline_backports_kernel_version
changed_when: false
check_mode: false
# `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
# neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
# list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
# GRUB's "Advanced options" without uninstalling anything.
- name: List every kernel image installed on this node, newest first
ansible.builtin.shell:
cmd: set -o pipefail; linux-version list | linux-version sort --reverse
executable: /bin/bash
register: node_baseline_installed_kernels
changed_when: false
check_mode: false
# The node must be able to explain itself to an operator standing in front of it with no
# repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
# role is a real no-op rather than a permanent daily "changed".
- name: Record the deviation, the exact version, and the rollback path on the node
ansible.builtin.copy:
dest: "{{ node_baseline_backports_kernel_stamp }}"
owner: root
group: root
mode: "0644"
content: |
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
#
# THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
#
# WHY
# This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
# Linux < 6.13 reserves memory from it without validating its size, and an unlucky
# boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
# k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
# is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
# v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
# Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
#
# PROOF THE FIX IS ACTIVE (not merely a lucky boot)
# BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
# >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
# zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
# (roles/node_health does the version check for you; see the health report's
# firmware.efi_memattr finding.)
#
# FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
# and each prints a DIFFERENT line — all three are rejections:
# tbl->version > 2 'Unexpected EFI Memory Attributes table version %d'
# (NOTE: printed with %d, so garbage may read NEGATIVE)
# bad desc_size 'Unexpected EFI Memory Attributes descriptor size ...'
# num_entries > 64K '[Firmware Bug]: Corrupted EFI Memory Attributes
# Table detected! (version == ..., num_entries == ...)'
# Which one appears depends on WHICH field this boot's garbage corrupted, so do
# NOT treat the absence of the FW_BUG line as the fix being absent — all three
# prod nodes printed the *version* line on 2026-09-04.
# The one line that means the fix is NOT active:
# 'Failed to map EFI Memory Attributes table @ 0x...' — the kernel trusted the
# corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
# escalate, do not shrug.
#
# INSTALLED
# source /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
# package {{ node_baseline_backports_kernel_packages | join(' ') }}
# version {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
# kernels installed on this node, newest first:
{% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
# {{ kv }}
{% endfor %}
#
# SECURITY COST
# Packages from *-backports are provided AS IS. They are NOT covered by the Debian
# Security Team's guarantees for the main archive: no security-tracker commitment,
# no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
#
# ROLLBACK — no repo checkout needed
# 1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
# and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
# GNU/Linux". That answers "is the new kernel the problem?" without
# uninstalling anything. Note it reinstates the memattr bug for that boot.
# 2. PERMANENT. Set node_baseline_backports_kernel: false in
# inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
# puts the backports kernel straight back — then, on this node:
# apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
# apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
# rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
# apt-get update && update-grub && reboot
#
# EXIT CONDITION
# If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
# rebase), this deviation can be removed entirely: flip the variable to false,
# roll back as above, and delete the `kernel` block from roles/node_baseline.
# Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
# it means the fixed kernel is on disk. See the block header for why the role will not
# reboot on its own.
- name: Report whether this node is running the kernel it now has installed
ansible.builtin.debug:
msg: >-
{{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
if newest != ansible_facts['kernel']
else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
~ '); no reboot pending for the memattr fix.') }}
vars:
newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
- name: Persist kernel modules for k3s/Cilium/Longhorn
ansible.builtin.copy:
dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
@@ -79,70 +314,32 @@
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
tags: [system]
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
# which do not exist on bare metal either — a confusing failure that looks like a wiring
# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
- name: Assert this environment's data-disk devices have actually been decided
ansible.builtin.assert:
that:
- "'BLOCKED' not in node_baseline_longhorn_device"
- "'BLOCKED' not in node_baseline_etcd_device"
fail_msg: >-
node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
placeholders for this host's environment
(longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
the SMART read that decision waits on can be taken) re-run this play with
`--tags packages`, which skips every disk task including this assert.
quiet: true
tags: [disks]
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
community.general.filesystem:
fstype: ext4
dev: "{{ node_baseline_longhorn_device }}"
opts: "-L {{ node_baseline_longhorn_fs_label }}"
tags: [disks]
- name: Mount the Longhorn data disk
ansible.posix.mount:
path: "{{ node_baseline_longhorn_mount }}"
src: "LABEL={{ node_baseline_longhorn_fs_label }}"
fstype: ext4
opts: "defaults,nofail,x-systemd.device-timeout=5s"
state: mounted
tags: [disks]
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
# a disk that already carries a filesystem, so re-running against an already-migrated
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
# role would mount over a populated directory.
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
community.general.filesystem:
fstype: ext4
dev: "{{ node_baseline_etcd_device }}"
opts: "-L {{ node_baseline_etcd_fs_label }}"
tags: [disks]
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
ansible.posix.mount:
path: "{{ node_baseline_etcd_mount }}"
src: "LABEL={{ node_baseline_etcd_fs_label }}"
fstype: ext4
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
opts: "defaults,noatime"
dump: "0"
passno: "2"
state: mounted
# =====================================================================================
# DATA DISKS — tag `disks`, in tasks/disks.yml
# =====================================================================================
# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
# should be readable end-to-end on its own; and both environments' storage shapes — dev's
# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
# the OS filesystem — are declared there as an explicit enum rather than inferred from
# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
#
# The `disks` tag is applied ONCE here, in BOTH places it is needed:
# * `tags:` on the include itself, so `--tags packages` never even reaches the include
# and a placeholder device path is not so much as templated; and
# * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
# optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
# its own tags to its children the way `import_tasks` does, so without `apply` the
# included tasks are untagged and `--tags disks` selects the include and then runs
# NOTHING inside it. That failure is silent and reports success, which is the worst
# shape a guard can have.
# A FULL run against a node whose layout is undeclared stops on the assert at the top of
# that file instead of failing obscurely at mkfs time.
- name: Provision the data disks (Longhorn store, etcd data dir)
ansible.builtin.include_tasks:
file: disks.yml
apply:
tags: [disks]
tags: [disks]
- name: Disable swap now
infra/ansible/roles/node_health/README.md
+32 -5
View File
@@ -88,15 +88,42 @@ measured memory, never on the log line alone:**
|---|---|---|
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | (not consulted) | **WARN** — the garbage `version` read `> 2` so the kernel bailed out first: exposed, but this boot took the safe path |
| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 validated and rejected it: this *is* the fix, working |
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path |
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic |
| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working |
| no memattr line, log reaches back to boot | (not consulted) | PASS |
| no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
| `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
The WARN is the point. All three prod nodes publish the same corrupt table and differ only
by **luck**, boot to boot. A node whose kernel just trusted a corrupt table is not passing a
health check even when it kept its RAM.
The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt
table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt
table is not passing a health check even when it kept its RAM.
### The FW_BUG line is proof, but its ABSENCE proves nothing
`efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG
line:
| gate | message | reachable when |
|---|---|---|
| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not |
| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not |
**All three are rejections.** Which one a boot prints is decided by *which field* that boot's
garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very
often prints the *first* message, and looking only for the FW_BUG line will make a working
fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed
the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version
alongside the log line, and why the check must never demand the FW_BUG string as its
evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in
the kernel image itself, not a line in `dmesg`.
**The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged
*negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that
until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected
a corrupt table was reported as one that "accepted it without complaint" — a false PASS on
the exact bug this check exists to find.
**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
infra/ansible/roles/node_health/files/node_health_collect.py
+8 -1
View File
@@ -497,8 +497,15 @@ BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Co
EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
MEMATTR_FAILED_RE = re.compile(
r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
# NOTE the `-?`. The kernel prints this version with %d — SIGNED — so a garbage field with
# bit 31 set is logged NEGATIVE (k3s-prod-2, 2026-09-04: "version -1569980416"). Without the
# minus sign this regex missed those lines entirely and the state fell through to "clean",
# i.e. a node whose kernel had just REJECTED a corrupt table was reported as one whose
# kernel "accepted it without complaint" — a false PASS on the exact bug this check exists
# to find. Anchor future edits on that: the field is signed, and every diagnostic that
# quotes it must survive a negative value.
MEMATTR_VERSION_RE = re.compile(
r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I)
r"Unexpected EFI Memory Attributes table version\s+(-?\d+)", re.I)
MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
infra/ansible/roles/node_health/tasks/assess.yml
+54 -5
View File
@@ -820,12 +820,22 @@
# catastrophically missing — memory is lost NOW.
# WARN otherwise — this boot drew a survivable garbage value; the next may not.
# "Unexpected ... version <N>" version read > 2, so the kernel bailed out before
# reserving anything. WARN: exposed, and this boot took the safe path.
# reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
# safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
# gates and no draw can reach the fatal arithmetic any more. Same line, opposite
# meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
# FW_BUG "Corrupted EFI Memory Attributes Table detected!" the >= 6.13 THIRD gate
# (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
# must not be treated as such: it is only reachable on a boot whose garbage leaves the
# version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
# line instead. All three prod nodes did exactly that on 2026-09-04.
#
# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
# measurement separates the last two. Never PASS on either line: the firmware is publishing
# a corrupt table and the draw is repeated on every boot.
# measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
# firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
# the draw still happens but every outcome is a rejection, so the same lines mean the
# validation is working.
#
# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
# so it cannot be diagnosed once. It has to be measured on every health run.
@@ -857,15 +867,39 @@
# near memory.physical_total. This check contributes the kernel-log facts only.
ms: "{{ node_health_mem_shortfall }}"
ma_state: "{{ k.memattr_state | default(none) }}"
# DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
# carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
# THREE checks before it computes tbl_size at all:
# tbl->version > 2 -> "Unexpected ... table version %d"
# desc_size implausible -> "Unexpected ... descriptor size %u ..."
# num_entries > SZ_64K -> FW_BUG "Corrupted EFI Memory Attributes
# Table detected! ..."
# ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
# given boot produces is decided by WHICH field that boot's garbage lands in — it is not
# a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
# version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
# memblock_reserve(). That is the whole difference, and it is why the verdict for
# `bad_version` below is opposite on the two kernels: same log line, completely different
# meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
# something the `version` test can parse.
memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
# `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
# COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
# this check and memory.physical_total can no longer disagree about lost memory.
# `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
# boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
# WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
# forever would make the check unable to ever show the fix landing, which is exactly what
# it is for.
memattr_status: >-
{{ 'SKIP' if not (k.available | bool)
else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
else ('WARN' if ma_state in ['failed_to_map', 'bad_version']
else ('WARN' if ma_state == 'failed_to_map'
else ('PASS' if ma_state == 'corrupt_rejected'
else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
if ma_state == 'bad_version'
else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
# One remediation, quoted by both findings. NOTE what it must NOT say: there is no
# kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
# could take effect. Suggesting one would be a fix that quietly does nothing.
@@ -944,6 +978,21 @@
been seen to lose 13.3 GiB and to keep its full memory at one identical table address
(infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
{{ memattr_fix }}
{%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
(num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
version field <= 2. Both messages are rejections, and which one a boot prints is the
lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
{{ memattr_corroboration }}
{%- elif ma_state == 'bad_version' -%}
EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
platform/longhorn/README.md
+14 -4
View File
@@ -25,7 +25,7 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
| Requirement | Summary | How this component serves it |
|---|---|---|
| **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the node's data disk at `${LONGHORN_DATA_PATH}` (a `cluster-vars` knob: dev `/var/lib/longhorn`, a dedicated disk; prod `/data/longhorn`, a subdirectory of the partition it shares with k3s's data dir). `replicaAutoBalance: best-effort` re-spreads on node return. |
| **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
## Dev-scoped choices (change for prod)
@@ -48,9 +48,19 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
## Node prerequisites (already handled, informational)
Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
Each node presents ext4 storage at `${LONGHORN_DATA_PATH}` and installs `open-iscsi` +
`nfs-common` + kernel modules — all via `roles/node_baseline` (`infra/cloud-init/` is a README
now; provisioning became API-only). This component assumes that floor. The shape differs per
environment, which is why the path is a knob:
| | |
|---|---|
| dev | a **dedicated** disk (`/dev/vdb`, `LABEL=longhorn`) mounted at `/var/lib/longhorn` — the mount root *is* the dataPath |
| prod | `/data/longhorn`, a **subdirectory** of the ~424 GB partition (`LABEL=k3s-data`) mounted at `/data`, whose sibling `/data/k3s` holds k3s's entire data dir (etcd + the containerd image store, moved off the 11.8 GiB `/var` — `infra/PROD-PROVISIONING.md` §2) |
`[parity]`: the value for an environment MUST equal `node_baseline_longhorn_data_path` in that
environment's Ansible group_vars. Ansible creates and mounts the directory; this component tells
Longhorn to use it.
## Contract (consumed by everyone with state)
platform/longhorn/helmrelease.yaml
+16 -4
View File
@@ -42,10 +42,22 @@ spec:
defaultClass: false
defaultSettings:
# --- data path (R-STATE-02) ---
# Dedicated Longhorn data disk (/dev/vdb → ext4) is mounted here by cloud-init on every node
# (infra/cloud-init/user-data.yaml.tftpl). Longhorn stores replicas on this disk only.
defaultDataPath: "/var/lib/longhorn"
# --- data path (R-STATE-02, R-ENV-02) ---
# The on-node directory every replica is written into. A cluster-vars KNOB, not a literal,
# because the two environments mount their data disk differently and a hardcoded path would
# put one of them on the OS filesystem:
# dev /var/lib/longhorn — a dedicated ext4 disk (/dev/vdb) mounted there by
# roles/node_baseline. The mount root IS the dataPath: one disk,
# one tenant. 18 live volumes; this value must never change.
# prod /data/longhorn — a SUBDIRECTORY of the claimed ~424 GB partition mounted at
# /data, whose sibling /data/k3s holds k3s's entire data dir
# (etcd + the containerd image store, moved off the 11.8 GiB
# /var). Two tenants, so Longhorn gets a subdirectory rather
# than the mount root. infra/PROD-PROVISIONING.md §2.
# [parity] MUST equal roles/node_baseline's node_baseline_longhorn_data_path for the same
# environment — Ansible creates and mounts the directory, Helm tells Longhorn to use it.
# Set in BOTH overlays (clusters/dev, clusters/prod); base default is the dev value.
defaultDataPath: "${LONGHORN_DATA_PATH}"
# --- replication (R-STATE-02) ---
# Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly
platform/longhorn/values.yaml
+3 -1
View File
@@ -8,7 +8,9 @@
# defaultClass: false # we own StorageClasses (storageclasses.yaml)
#
# defaultSettings:
# defaultDataPath: /var/lib/longhorn # dedicated data disk (infra/cloud-init)
# defaultDataPath: ${LONGHORN_DATA_PATH} # cluster-vars knob: dev /var/lib/longhorn
# # (dedicated /dev/vdb), prod /data/longhorn
# # (a subdirectory of the shared data partition)
# defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT} # overlay knob (=3 in dev)
# replicaSoftAntiAffinity: false # HARD anti-affinity: one replica per node
# replicaZoneSoftAntiAffinity: true # single zone (3 mini-PCs, no failure zones)