feat(prod): kernel fix + disk layout — prod nodes ready for k3s

Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.

=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.

roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).

Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.

THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.

§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.

TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
  - MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
    2's real value was -1569980416, which did not match, so state fell through to `clean` and
    the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
    exact bug the check exists to find. Now (-?\d+).
  - bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
    Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.

=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).

Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).

=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.

Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.

The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.

k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.

LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.

DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.

Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.

OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.

Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.

Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 00:39:10 -05:00
1 parent 8e913fdd7b
commit 05baae90ec
16 files changed
+1782 -126

No files matched your search

+382 -19
View File
@@ -559,7 +559,7 @@ nothing ever transmits `new_password` to firmware. It is write-only into a buffe
password in cleartext on the command line — it would land in process listings and shell history
unless wrapped. Store it in the Ansible vault **and** offline.
## 2. Disk layout — **OPEN DECISION, blocks the build**
## 2. Disk layout — **DECIDED (operator, 2026-09-03) · REVISED (operator, 2026-09-04)**
`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
@@ -576,6 +576,218 @@ sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
└─sda5 433G /srv <- the only large free space
```
### THE DECISION (operator, 2026-09-03; layout REVISED 2026-09-04) — made, applied, and measured-not-assumed
| | |
|---|---|
| **Data partition** | the **existing `/srv` partition** on each node's single physical disk (~432 GB nodes 1–2, ~422 GB node 3), re-purposed, re-labelled **`k3s-data`** and mounted at the neutral **`/data`** |
| **Longhorn store** | **`/data/longhorn`** — a *subdirectory* of that partition |
| **k3s `--data-dir`** | **`/data/k3s`** — the *sibling* subdirectory. Holds **embedded etcd AND the containerd image store** |
| **etcd** | **no dedicated device** — it lives at `/data/k3s/server/db`, i.e. on the data partition, **not** on the OS filesystem |
| **Shared-device I/O contention** | **ACCEPTED, knowingly** — unchanged by the 2026-09-04 revision |
#### THE 2026-09-04 REVISION, AND THE CAPACITY EVIDENCE THAT FORCED IT
The 2026-09-03 ruling gave this partition to Longhorn **only** and left k3s at its default
`/var/lib/rancher/k3s`. That was wrong, for a reason that is arithmetic rather than judgement.
**`/var` on these nodes is a separate partition of 11.8 GiB** (11.6 on node 3) — see the
partition map above; it is `/dev/sda3` / `/dev/nvme0n1p3`, not the 30 GiB `/`. k3s's data dir
puts **two** things on it: etcd's database and WAL (`server/db`) and, far larger, the
**containerd image store** (`agent/containerd`).
**Measured on the fully-deployed dev cluster** — the same platform and app set prod will run:
| | |
|---|---|
| `agent/containerd` per node | **21–25 GB** |
| total `/var` per node | **58–70 GB** |
| prod `/var` capacity | **11.8 GiB** |
So prod would have **filled `/var` during its first image pulls** — not eventually, not under
load, but on the way up. And because etcd was on that same partition, a full `/var` means
**etcd cannot write its WAL**. That is cluster death, not degradation: a wedged etcd on all
three servers is an unavailable control plane, and it would have happened during bring-up.
**The fix is to move k3s's whole data dir onto the 424 GB partition**, which is why that
partition's mountpoint and filesystem label are no longer Longhorn-specific — it now serves
**two tenants**. `/var` retains only the OS's own usage (503 MB, 5% used, measured 2026-09-04).
**Why now.** Nothing was installed on prod when this was decided — verified 2026-09-04: no
`/usr/local/bin/k3s`, no `/etc/rancher/k3s`, no `/var/lib/rancher` on any node. Moving a data
dir before first install is a one-line inventory change; after bring-up it means draining and
re-imaging three servers.
**Why `/data` and `LABEL=k3s-data`.**
- **`/data`** — short, neutral, tenant-agnostic, and *not* under `/var`. Deliberately **not
`/srv`**: `/srv` is the *claim mountpoint* the role discovers the device at, and the
idempotency contract is that `/srv` is gone from `/proc/mounts` and `/etc/fstab` once the
claim has happened. Re-using it as the destination would make "claimed" and "not yet
claimed" indistinguishable.
- **`LABEL=k3s-data`** — deliberately **not** the bare `data`. Guard #4 of the role's assert
wall accepts a partition whose label is empty *or already ours*, and names `data` as an
example of somebody else's named volume; labelling ours `data` would make a stranger's
`data` volume look already-claimed and weaken that guard. `k3s-data` is cluster-scoped
rather than tenant-scoped, and matches the `k3s-etcd` label convention already in the repo.
**The shared-device I/O risk is UNCHANGED and still accepted.** Longhorn and etcd were already
going to share this spindle under the 2026-09-03 ruling — the revision changes *which*
filesystem etcd sits on, not *which device*. The acceptance rationale and its exit condition
below stand verbatim.
**How prod's data dir is scoped away from dev.** `k3s_data_dir` is a **single** variable read
by **two** roles: `roles/k3s_server` turns it into `--data-dir` in the install flags, and
`roles/node_baseline` (via `node_baseline_k3s_data_dir`, which defers to it) creates the
directory `0700` on the data partition and derives etcd's location from it
(`node_baseline_etcd_mount` = `<data-dir>/server/db`). The role default is **k3s's own
default**, `/var/lib/rancher/k3s`, so **dev is unchanged** — and dev also has a *dedicated
etcd device* mounted at `server/db` under that path, which moving dev's data dir would orphan.
Prod overrides it in one line in `inventory/group_vars/k3s_prod/vars.yml`, the same seam
`kube_vip_address` / `k3s_api_host` / `node_baseline_backports_kernel` use, so a fourth prod
node inherits it from group membership rather than from anyone remembering.
**Longhorn's `defaultDataPath` follows the R-ENV-02 knob pattern, not a base edit.**
`platform/longhorn/helmrelease.yaml` referenced a hardcoded `/var/lib/longhorn`; it now
references `${LONGHORN_DATA_PATH}`, a `cluster-vars` knob with a base default and an explicit
value in **both** overlays:
| | |
|---|---|
| `clusters/base/cluster-vars.yaml` | `/var/lib/longhorn` (base default = the known-good dev value, same convention as `GATEWAY_VIP`) |
| `clusters/dev` | `/var/lib/longhorn` — **must never change**; the live dev cluster has 18 volumes whose replicas are on disk under that path |
| `clusters/prod` | `/data/longhorn` |
`kustomize-controller` runs with `--feature-gates=StrictPostBuildSubstitutions=true`, so an
undefined variable fails the Kustomization outright — hence base-first, then both overlays.
`[parity]`: the prod value MUST equal `node_baseline_longhorn_data_path` in
`group_vars/k3s_prod/vars.yml`. Ansible creates and mounts the directory; Helm tells Longhorn
to use it. If the two disagree, Longhorn writes replicas onto the OS filesystem and `/var`
fills up — the exact failure this revision exists to prevent.
**Verified 2026-09-04 that dev's rendered output is byte-identical.** All 12 Flux
Kustomizations that declare `postBuild.substituteFrom cluster-vars` were rendered
(`kubectl kustomize <spec.path> | flux envsubst --strict`, substituting the dev overlay's
*effective* cluster-vars) both before and after the change: 12/12 identical per-file SHA-256,
and one aggregate SHA-256 over every rendered byte, `b34c5d3d…89e7d`, unchanged. The only
dev-side difference anywhere is the presence of the new knob in the ConfigMap itself, carrying
dev's existing literal. The same harness shows prod's render changing on exactly one line
(`defaultDataPath: /var/lib/longhorn` → `/data/longhorn`), which is what proves the comparison
could have detected a regression.
**Per-node device paths — they are NOT the same, and that is why nothing hardcodes them.**
Verified 2026-09-03 with `findmnt --mountpoint /srv` on each node:
| Node | Boot media | `/srv` partition | Size | Contents before the claim |
|---|---|---|---|---|
| `k3s-prod-1` | SATA SSD `C400-MTFDDAC512M` | `/dev/sda5` | 432.5 GiB ext4 | `lost+found` only (`du -sh` 20K; `df` 2.1 M of 425 G) |
| `k3s-prod-2` | **NVMe** `PM9A1 Samsung` | **`/dev/nvme0n1p5`** | 432.5 GiB ext4 | `lost+found` only (same) |
| `k3s-prod-3` | SATA SSD `WDC WDS500G2B0A` | `/dev/sda5` | 422.2 GiB ext4 | `lost+found` only (same) |
The `df` "5.1% used" that made `/srv` look occupied is **ext4's default 5% root-reserved
blocks**, not data — confirmed per node by `ls -A /srv` (only `lost+found`) and
`du -sh --exclude=lost+found /srv` (20 K). Nothing was destroyed.
**How the role resolves the device — discovery, not per-host inventory.** A single
`k3s_prod` group value cannot be right when node 2 is NVMe, and per-host `host_vars` would
freeze a *fact about today's hardware* into inventory, where it goes stale the first time a
node is re-imaged or a disk is swapped — and the stale value's failure mode is `mkfs` on the
wrong partition. So `group_vars/k3s_prod/vars.yml` declares **intent**
(`node_baseline_longhorn_source: claim-mountpoint`,
`node_baseline_longhorn_claim_mountpoint: /srv`) and
`roles/node_baseline/tasks/disks.yml` asks the **node** which block device is mounted there,
then proves the answer is safe before writing anything. Six independent asserts, all
fail-closed: it is a partition or whole disk (not tmpfs/NFS/bind), it is **not** backing
`/`, `/boot`, `/boot/efi`, `/usr`, `/var`, `/home`, `/tmp` or active swap (that set is built
from the node's *own* gathered facts plus `swapon`, never a hardcoded partition list), it is
already ext4, it carries no other filesystem label, it is ≥ 100 GiB, and it is **empty**.
**Unmount → relabel, not reformat.** `/srv` is already formatted and already mounted, and
`community.general.filesystem` will not touch a device that has a filesystem — it would
silently no-op and then the `LABEL=longhorn` mount would fail. The role therefore unmounts
`/srv`, removes its `/etc/fstab` entry (installer-written, pinned by UUID — leaving it would
re-mount the partition at `/srv` underneath Longhorn on the next boot), and stamps the label
with `e2label`. Relabelling rather than reformatting is deliberate: the partition is already
ext4 and already verified empty, so `mkfs` buys nothing, and if some assert were ever too
permissive the worst case is a renamed filesystem instead of a destroyed one.
**Idempotency guard.** A second run is a no-op, not a reformat. After the claim the
partition carries the ext4 label `k3s-data` and `/srv` is gone from `/proc/mounts` *and*
`/etc/fstab` — so "what is mounted at `/srv`" would find nothing on run 2. The role pivots
on the durable on-disk evidence instead: `blkid -L k3s-data` returning a device means the
node is already claimed, and **every** destructive task is gated on that returning nothing.
**The 2026-09-04 transition, handled idempotently.** All three nodes had *already* been
claimed under the previous naming (`LABEL=longhorn` at `/var/lib/longhorn`). Because the
idempotency pivot asks "does the **current** label exist?", changing the label makes an
already-claimed node answer *no* — and it would then walk into the greenfield claim path and
fail closed at "the claim mountpoint must actually be a mountpoint", `/srv` having been
consumed by that first claim. Fail-closed is correct, but it is not convergence, so the
transition is **declared** rather than inferred:
`node_baseline_data_retired_fs_label: longhorn` + `node_baseline_data_retired_mount:
/var/lib/longhorn` in `group_vars/k3s_prod/vars.yml`. The role then unmounts the retired
mountpoint, drops its `fstab` entry and re-stamps the label with `e2label` — a **metadata
write, never a reformat**, exactly as in the claim path — having first proved that the retired
mountpoint is still a mount, that it is backed by the very device the retired label names, and
that it is **empty** so nothing is orphaned. The common `LABEL=`-based mount task then brings
it up at `/data`, and the two per-tenant subdirectories are created (`0755` Longhorn, `0700`
k3s, since the latter holds the etcd DB, the node token and the image store).
The transition block is **one-time and self-retiring**: on every subsequent run
`blkid -L k3s-data` succeeds and none of it is reachable. The two retired-* lines are safe to
delete from `group_vars` once the fleet is converged (it is, as of 2026-09-04).
**Three independent reasons the LIVE dev cluster cannot enter that transition** — it has 18
Longhorn volumes on `/var/lib/longhorn` and unmounting it would orphan every replica:
(1) both retired-* vars default to `""` and are set only in the prod-scoped `group_vars`;
(2) the path is gated on `longhorn_source == 'claim-mountpoint'`, and dev is `device` — the
same structural gate that keeps dev out of the claim path; (3) the emptiness assert, which
fails closed on a partition with replica directories even if (1) and (2) were both defeated.
Dev's mountpoint, label and k3s data dir are also simply **not changing**.
**Run evidence (2026-09-04).** Full `node-baseline.yml` against `k3s_prod`: run 1 converged
(5–6 changed per node), run 2 **`changed=0` on all three nodes** with every transition and
claim task skipping. Per node afterwards: `blkid -L k3s-data` → `/dev/sda5` (nodes 1, 3) /
`/dev/nvme0n1p5` (node 2); `blkid -L longhorn` → gone; `/data` mounted with 403 G / 394 G
available; `/var/lib/longhorn` and `/srv` no longer mountpoints; `fstab` carrying exactly
`LABEL=k3s-data /data ext4 defaults,nofail,x-systemd.device-timeout=5s 0 0`; and `/var` down
to **5% used, ~11 GiB free**.
**Dev can never reach this path.** Both role defaults stay `source: device`, so the LIVE
dev cluster keeps its dedicated `/dev/vdb` (Longhorn) and `/dev/vdc` (etcd) and the claim
tasks are structurally unreachable there — they are gated on `claim-mountpoint`, which only
the prod-scoped `group_vars` sets.
### The accepted risk, and its exit condition
Dev was burned by exactly this shape: a Longhorn rebuild starved etcd WAL fsync
(`docs/plans/ssd-etcd-isolation.md`), which is why dev has a third, dedicated etcd device.
**Prod accepts the same shared-device contention** because the failure mode is materially
cheaper here: dev's contention was on a **spinning HDD** with seek-bound queueing; every
prod node is **SSD or NVMe** end-to-end, where concurrent rebuild I/O costs latency rather
than collapsing it. The three nodes also have no second device to give etcd — one physical
disk each, already fully partitioned, no free slot verified populated.
**Accepted to be MEASURED, not assumed.** Exit condition, in order of preference:
- **Measure it before it matters.** Under a deliberate Longhorn replica rebuild, watch
`etcd_disk_wal_fsync_duration_seconds` p99 and `etcd_disk_backend_commit_duration_seconds`
p99, plus `etcd_server_leader_changes_seen_total`. If fsync p99 stays inside etcd's
practical budget (~10 ms wal fsync / ~25 ms backend commit) with no leader churn, the
acceptance holds and this becomes a recorded measurement rather than an assumption.
- **If it does not hold:** add an M.2 NVMe per node (chassis slot to be verified
physically) and flip `node_baseline_etcd_source` back to `device` — the role still
supports that path unchanged, and it is the dev shape.
**~~Known, separate consequence — `/var` is only 12 GiB.~~ RESOLVED 2026-09-04 by moving
k3s's data dir.** This was recorded here on 2026-09-03 as a capacity item "to watch or
repartition". Re-measuring it against dev turned it from a watch item into a blocker: the
container image store alone is **21–25 GB per node** on the fully-deployed dev cluster and
`/var` totals **58–70 GB**, against prod's **11.8 GiB**. So prod would have filled `/var`
during its first image pulls, and with etcd on the same partition a full `/var` means etcd
cannot write its WAL. The fix was neither "watch" nor "repartition" but `--data-dir` — see
"THE 2026-09-04 REVISION" above. `/var` now carries only the OS's own usage (503 MB, 5%).
Consequences to settle before imaging nodes 2 and 3:
- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
@@ -609,24 +821,43 @@ Consequences to settle before imaging nodes 2 and 3:
a member corrupts — this drive has already logged 32 of them.
(Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
`DECISIONS-LOG.md`. It is not part of this verdict.)
- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
(`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
prod must override them in `group_vars`. On a single-disk node there is no second device at
all, so one of:
- **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
spindle.
- **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
- **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
physical failure domain.
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
- [ ] Record the chosen device paths here and set them in
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
`/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
`node-baseline` run against a node still carrying those placeholders now stops on an explicit
assert naming this section, instead of failing obscurely at `mkfs` time.
- [x] **Layout DECIDED and APPLIED 2026-09-03** — see "THE DECISION" above. `roles/node_baseline`
no longer expects a block-device path for either store; it takes a declared **source
enum** per environment (`roles/node_baseline/tasks/disks.yml`):
- **Longhorn** — `node_baseline_longhorn_source: claim-mountpoint` +
`node_baseline_longhorn_claim_mountpoint: /srv`. The `/dev/vdb` default (a virtio
name that does not exist on bare metal) is unused on prod; the real partition is
discovered on the node and guarded, because it is `/dev/sda5` on nodes 1 and 3 and
`/dev/nvme0n1p5` on node 2.
- **etcd** — `node_baseline_etcd_source: shared-data-partition` (was `root-filesystem`
until 2026-09-04). "No dedicated device" is a first-class, explicitly-configured state
that the `disks` assert **accepts**, not a blank variable that the tasks happen to
skip — and the *name* of that state now describes reality, because etcd is no longer
on the OS filesystem. The distinction is load-bearing, not cosmetic:
`shared-data-partition` is **asserted**, per node on every run, by checking that the
filesystem backing the etcd data dir really is the data partition's mount. Declaring
`k3s_data_dir: /data/k3s` does not by itself achieve that — had the mount silently
failed, `/data` would be a plain directory on `/` and etcd plus 21–25 GB of images
would land back on the OS filesystem, invisibly, until the disk filled. Leaving the
value at `root-filesystem` would have been a lie inventory told about the cluster, and
`root-filesystem` makes no such claim and gets no such assert. The role also still
asserts nothing is already mounted at the etcd data dir (so a half-migrated node
cannot be inherited silently) and reports which filesystem etcd actually lands on
(measured 2026-09-04: `/data`, 403 G / 394 G available, vs `/var`'s 10.7 GiB).
- The two rejected options stand rejected: **add a second drive** (no verified free slot;
the exit condition above reopens it if measurement fails) and **repartition** (same
single failure domain, plus a destructive re-image of three working nodes for no
isolation the SSDs do not already give).
- [x] **All three nodes get the identical layout.** Same declaration, same mountpoint, same
resulting `/data` + `/data/longhorn` + `/data/k3s`. Not the same *device path* and not
the same *size* (432.5 / 432.5 / 422.2 GiB) — Longhorn tolerates unequal node capacity;
it is replica *placement* that assumes a symmetric layout, and the layout is symmetric.
Confirmed per node 2026-09-04 by `findmnt`/`lsblk`/`blkid` after the transition.
- [x] **Chosen paths recorded here and set in `group_vars`.** The BLOCKED placeholders in
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` are replaced — by the source
enums above, plus deliberately-unusable `/dev/NOT-A-DEVICE-…` values for the two device
variables so that flipping a source back to `device` without supplying a real path
still fails on the role's assert instead of silently inheriting dev's `/dev/vdb`.
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
tasks this decision blocks — see §5's drive-health box for the exact command.
@@ -639,6 +870,138 @@ Consequences to settle before imaging nodes 2 and 3:
- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
`roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
so the same role covers bare metal. Nothing to port.
- [x] **KERNEL: a standing, deliberate deviation from the distro baseline — `linux-image-amd64`
from `trixie-backports` (DONE on all three nodes, 2026-09-04).**
**What.** `roles/node_baseline` gained a `kernel`-tagged block that enables
`trixie-backports` as a deb822 apt source and installs the `linux-image-amd64`
**metapackage** from it. Applied 2026-09-04; all three nodes landed on
**`linux-image-amd64` 7.1.8-1~bpo13+1 → kernel `7.1.8+deb13-amd64`**, rebooted one at a
time, and are running it.
```bash
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
```
**Why.** §0b. HP's R21 firmware publishes a corrupt EFI Memory Attributes Table; Linux
< 6.13 reserves memory from it without validating the size, and `k3s-prod-2` has booted
with 2.6 GiB of its 16 GB, twice. The fix is upstream commit **`8fbe4c49c0cc`**
*"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, first
released in **v6.13**, hardened by `5d0faa8e8369`). It is absent from v6.12 and from
`linux-6.12.y`, carries neither `Cc: stable` nor `Fixes:`, and there is **no kernel
cmdline workaround**. A k3s **server** that intermittently comes up with 2.6 GiB will
fail under etcd plus its workload share on any reboot, so this is a **prerequisite to
installing k3s**, not a follow-up.
**How it is scoped.** `node_baseline_backports_kernel` defaults to **false** in
`roles/node_baseline/defaults/main.yml` and is set **true** exactly once, in
`inventory/group_vars/k3s_prod/vars.yml` — the same seam `kube_vip_address` /
`k3s_api_host` / `cluster_kubeconfig_name` use. Dev's Proxmox VMs publish no MAT and are
genuinely unaffected; they must not be dragged onto a backports kernel as a side effect
of sharing the role, and a fourth prod node must inherit the deviation from **group
membership**, not from a `k3s-prod-*` hostname test. Verified: a `--tags kernel` run
against dev skips all 8 tasks on all 3 hosts.
**Security cost, accepted.** `*-backports` packages are provided **as-is** and are
**not** covered by the Debian Security Team's guarantees for the main archive — no
security-tracker commitment, no DSAs. Recorded here as the price of the nodes keeping
their RAM.
**No apt pin, deliberately.** The backports `Release` file carries `NotAutomatic: yes`
+ `ButAutomaticUpgrades: yes`, which apt already reads as *"priority 100 so nothing is
taken from backports by accident, 500 for a package explicitly installed from there so
it keeps getting updates."* Writing an explicit `Pin-Priority: 100` in
`preferences.d` would look like belt-and-braces and would in fact **switch off the
automatic security upgrades for the very kernel this deviation installs.** Enabling the
source is the whole configuration.
**Metapackage, not a pinned ABI.** A frozen `linux-image-6.19.14+deb13-amd64` would stop
receiving even backports' own updates the moment the ABI rolls. The metapackage tracks
whatever backports ships (>= 6.13 by construction). The **exact** version each node
landed on is recorded on the node itself at `/etc/node-baseline-backports-kernel`,
written by the same role, along with the rollback path below.
**Rollback.**
1. *Non-destructive, try first.* The stock kernels are still installed
(`6.12.107+deb13-amd64` and `6.12.94+deb13-amd64` on all three) and selectable from
GRUB → *Advanced options for Debian GNU/Linux*. That answers "is the new kernel the
problem?" without uninstalling anything — it also reinstates the memattr bug for that
boot.
2. *Permanent.* Set `node_baseline_backports_kernel: false` in
`group_vars/k3s_prod/vars.yml` **first** (otherwise the next Ansible run puts it
back), then on each node:
```bash
apt-get install --reinstall linux-image-amd64/trixie
apt-get purge linux-image-7.1.8+deb13-amd64
rm /etc/apt/sources.list.d/trixie-backports.sources
apt-get update && update-grub && reboot
```
**The role does NOT reboot**, and that is deliberate — see §0b "SECOND FAULT". Reboots
are a serialized, supervised operator step: one node at a time, verify each returns
before touching the next, never node 2 first. On 2026-09-04 all three returned in
**39–41 s** (node 1 39 s, node 3 41 s, node 2 39 s); node 2 did **not** reproduce its
2026-09-03 hang, but one clean warm reboot is not a fix and that defect stays open.
**Exit condition — how this deviation ends.** If `8fbe4c49c0cc` is ever picked up into
`linux-6.12.y` (and therefore into Debian's 6.12 rebase, which reaches us by plain
`apt upgrade`), this can be removed entirely: set `node_baseline_backports_kernel:
false`, roll back per the note on each node, and delete the `kernel` block from
`roles/node_baseline`. §0b records why that pickup will not happen on its own and what
request would cause it.
- [x] **Verification of the fix, and a correction to how it is verified (2026-09-04).**
§0b says *"A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table
detected!`"*. That is true but **conditional, and it did not happen on any of the three
nodes.** `efi_memattr_init()` on >= 6.13 has **three** gates and only the third prints
that line:
| gate | message | reachable when |
|---|---|---|
| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, `desc_size` does not |
| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* `desc_size` read plausibly, entry count does not |
**All three are rejections**, and which one a boot prints is decided by *which field*
that boot's garbage corrupts. On 7.1.8 all three nodes hit the **first** gate:
| node | `MEMATTR=` | kernel line on 7.1.8 | MemTotal | reserved at boot |
|---|---|---|---|---|
| 1 | `0xbee84018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1030` | 31.1 GiB | 0.9 GiB |
| 2 | `0xbeeb0018` | `efi: memattr: Unexpected EFI Memory Attributes table version -1569980416` | 15.4 GiB | 0.6 GiB |
| 3 | `0xbeecc018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` | 31.1 GiB | 0.9 GiB |
Node 2 is the result that matters: **`0xbeeb0018` is the exact address that produced
2.6 GiB twice**, and on 7.1.8 it reserved 634,904 K instead of 13,913,856 K.
**Boot-independent proof the fix is compiled in**, since the log line is a lottery: the
running kernel image itself contains the string. Decompressing `/boot/vmlinuz-$(uname
-r)` on all three nodes yields, verbatim —
```
efi: memattr: [Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)
efi: memattr: Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)
```
— neither of which exists in 6.12. That is the positive proof; the `dmesg` line is not
available on demand.
**Two bugs in `roles/node_health` were fixed as a consequence**, because without them
the health check could not show the fix landing:
- `MEMATTR_VERSION_RE` in `files/node_health_collect.py` matched `(\d+)` only. The
kernel prints that field with `%d` — **signed** — so node 2's `-1569980416` did not
match, the state fell through to `clean`, and the node was reported **PASS "the kernel
accepted it without complaint"**: a false pass on the exact bug the check exists to
find. Now `(-?\d+)`.
- `firmware.efi_memattr` treated `bad_version` as WARN unconditionally, so a node on a
fixed kernel would have stayed WARN forever. The verdict now keys on the running
kernel version (`memattr_kernel_fixed`, `>= 6.13`): the same log line is WARN on
< 6.13 (a lucky boot) and PASS on >= 6.13 (the validation working). The pre-existing
`corrupt_rejected` PASS branch is unchanged and remains reachable.
**`playbooks/node-health.yml -e node_health_hosts=k3s_prod`, after the reboots:
`firmware.efi_memattr` = PASS on all three, `memory.physical_total` = PASS on all
three.** Dev re-run unchanged.
## 4. Network (per node)