diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index d3be429..ee03139 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -74,6 +74,17 @@ data: # --- storage (ADR-0003) --- LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes) + # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every + # replica is written into. It must be the path the NODE's own data disk is actually mounted at — + # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE + # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk. + # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good + # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes + # on it — changing dev's value would orphan every replica, so the dev overlay restates it + # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data + # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the + # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root. + LONGHORN_DATA_PATH: "/var/lib/longhorn" # --- observability sizing (docs/10) --- MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index dab6047..33d180e 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -190,6 +190,16 @@ patches: - op: replace path: /data/LONGHORN_DEFAULT_REPLICA_COUNT value: "3" + # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are + # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline + # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere + # else would orphan every one of them (they are not recoverable from the old path by + # rescheduling). Restated explicitly even though it equals the base default, exactly like the + # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because + # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2. + - op: replace + path: /data/LONGHORN_DATA_PATH + value: "/var/lib/longhorn" # smaller retention / volumes to keep the dev workstation footprint light. - op: replace path: /data/MONITORING_PROMETHEUS_RETENTION diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index ec34c1a..cb7e50e 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -157,6 +157,20 @@ patches: - op: replace path: /data/LONGHORN_DEFAULT_REPLICA_COUNT value: "3" + # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not + # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data + # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir + # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var: + # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on + # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2. + # + # [parity] MUST equal node_baseline_longhorn_data_path in + # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the + # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a + # path on the OS filesystem instead of the data partition and /var fills up. + - op: replace + path: /data/LONGHORN_DATA_PATH + value: "/data/longhorn" # longer retention / larger volumes on prod NVMe. - op: replace path: /data/MONITORING_PROMETHEUS_RETENTION diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 5fcdeea..407d489 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -559,7 +559,7 @@ nothing ever transmits `new_password` to firmware. It is write-only into a buffe password in cleartext on the command line — it would land in process listings and shell history unless wrapped. Store it in the Ansible vault **and** offline. -## 2. Disk layout — **OPEN DECISION, blocks the build** +## 2. Disk layout — **DECIDED (operator, 2026-09-03) · REVISED (operator, 2026-09-04)** `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went further and added a *third* device for etcd after HDD contention starved etcd WAL fsync @@ -576,6 +576,218 @@ sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial) └─sda5 433G /srv <- the only large free space ``` +### THE DECISION (operator, 2026-09-03; layout REVISED 2026-09-04) — made, applied, and measured-not-assumed + +| | | +|---|---| +| **Data partition** | the **existing `/srv` partition** on each node's single physical disk (~432 GB nodes 1–2, ~422 GB node 3), re-purposed, re-labelled **`k3s-data`** and mounted at the neutral **`/data`** | +| **Longhorn store** | **`/data/longhorn`** — a *subdirectory* of that partition | +| **k3s `--data-dir`** | **`/data/k3s`** — the *sibling* subdirectory. Holds **embedded etcd AND the containerd image store** | +| **etcd** | **no dedicated device** — it lives at `/data/k3s/server/db`, i.e. on the data partition, **not** on the OS filesystem | +| **Shared-device I/O contention** | **ACCEPTED, knowingly** — unchanged by the 2026-09-04 revision | + +#### THE 2026-09-04 REVISION, AND THE CAPACITY EVIDENCE THAT FORCED IT + +The 2026-09-03 ruling gave this partition to Longhorn **only** and left k3s at its default +`/var/lib/rancher/k3s`. That was wrong, for a reason that is arithmetic rather than judgement. + +**`/var` on these nodes is a separate partition of 11.8 GiB** (11.6 on node 3) — see the +partition map above; it is `/dev/sda3` / `/dev/nvme0n1p3`, not the 30 GiB `/`. k3s's data dir +puts **two** things on it: etcd's database and WAL (`server/db`) and, far larger, the +**containerd image store** (`agent/containerd`). + +**Measured on the fully-deployed dev cluster** — the same platform and app set prod will run: + +| | | +|---|---| +| `agent/containerd` per node | **21–25 GB** | +| total `/var` per node | **58–70 GB** | +| prod `/var` capacity | **11.8 GiB** | + +So prod would have **filled `/var` during its first image pulls** — not eventually, not under +load, but on the way up. And because etcd was on that same partition, a full `/var` means +**etcd cannot write its WAL**. That is cluster death, not degradation: a wedged etcd on all +three servers is an unavailable control plane, and it would have happened during bring-up. + +**The fix is to move k3s's whole data dir onto the 424 GB partition**, which is why that +partition's mountpoint and filesystem label are no longer Longhorn-specific — it now serves +**two tenants**. `/var` retains only the OS's own usage (503 MB, 5% used, measured 2026-09-04). + +**Why now.** Nothing was installed on prod when this was decided — verified 2026-09-04: no +`/usr/local/bin/k3s`, no `/etc/rancher/k3s`, no `/var/lib/rancher` on any node. Moving a data +dir before first install is a one-line inventory change; after bring-up it means draining and +re-imaging three servers. + +**Why `/data` and `LABEL=k3s-data`.** + +- **`/data`** — short, neutral, tenant-agnostic, and *not* under `/var`. Deliberately **not + `/srv`**: `/srv` is the *claim mountpoint* the role discovers the device at, and the + idempotency contract is that `/srv` is gone from `/proc/mounts` and `/etc/fstab` once the + claim has happened. Re-using it as the destination would make "claimed" and "not yet + claimed" indistinguishable. +- **`LABEL=k3s-data`** — deliberately **not** the bare `data`. Guard #4 of the role's assert + wall accepts a partition whose label is empty *or already ours*, and names `data` as an + example of somebody else's named volume; labelling ours `data` would make a stranger's + `data` volume look already-claimed and weaken that guard. `k3s-data` is cluster-scoped + rather than tenant-scoped, and matches the `k3s-etcd` label convention already in the repo. + +**The shared-device I/O risk is UNCHANGED and still accepted.** Longhorn and etcd were already +going to share this spindle under the 2026-09-03 ruling — the revision changes *which* +filesystem etcd sits on, not *which device*. The acceptance rationale and its exit condition +below stand verbatim. + +**How prod's data dir is scoped away from dev.** `k3s_data_dir` is a **single** variable read +by **two** roles: `roles/k3s_server` turns it into `--data-dir` in the install flags, and +`roles/node_baseline` (via `node_baseline_k3s_data_dir`, which defers to it) creates the +directory `0700` on the data partition and derives etcd's location from it +(`node_baseline_etcd_mount` = `/server/db`). The role default is **k3s's own +default**, `/var/lib/rancher/k3s`, so **dev is unchanged** — and dev also has a *dedicated +etcd device* mounted at `server/db` under that path, which moving dev's data dir would orphan. +Prod overrides it in one line in `inventory/group_vars/k3s_prod/vars.yml`, the same seam +`kube_vip_address` / `k3s_api_host` / `node_baseline_backports_kernel` use, so a fourth prod +node inherits it from group membership rather than from anyone remembering. + +**Longhorn's `defaultDataPath` follows the R-ENV-02 knob pattern, not a base edit.** +`platform/longhorn/helmrelease.yaml` referenced a hardcoded `/var/lib/longhorn`; it now +references `${LONGHORN_DATA_PATH}`, a `cluster-vars` knob with a base default and an explicit +value in **both** overlays: + +| | | +|---|---| +| `clusters/base/cluster-vars.yaml` | `/var/lib/longhorn` (base default = the known-good dev value, same convention as `GATEWAY_VIP`) | +| `clusters/dev` | `/var/lib/longhorn` — **must never change**; the live dev cluster has 18 volumes whose replicas are on disk under that path | +| `clusters/prod` | `/data/longhorn` | + +`kustomize-controller` runs with `--feature-gates=StrictPostBuildSubstitutions=true`, so an +undefined variable fails the Kustomization outright — hence base-first, then both overlays. +`[parity]`: the prod value MUST equal `node_baseline_longhorn_data_path` in +`group_vars/k3s_prod/vars.yml`. Ansible creates and mounts the directory; Helm tells Longhorn +to use it. If the two disagree, Longhorn writes replicas onto the OS filesystem and `/var` +fills up — the exact failure this revision exists to prevent. + +**Verified 2026-09-04 that dev's rendered output is byte-identical.** All 12 Flux +Kustomizations that declare `postBuild.substituteFrom cluster-vars` were rendered +(`kubectl kustomize | flux envsubst --strict`, substituting the dev overlay's +*effective* cluster-vars) both before and after the change: 12/12 identical per-file SHA-256, +and one aggregate SHA-256 over every rendered byte, `b34c5d3d…89e7d`, unchanged. The only +dev-side difference anywhere is the presence of the new knob in the ConfigMap itself, carrying +dev's existing literal. The same harness shows prod's render changing on exactly one line +(`defaultDataPath: /var/lib/longhorn` → `/data/longhorn`), which is what proves the comparison +could have detected a regression. + +**Per-node device paths — they are NOT the same, and that is why nothing hardcodes them.** +Verified 2026-09-03 with `findmnt --mountpoint /srv` on each node: + +| Node | Boot media | `/srv` partition | Size | Contents before the claim | +|---|---|---|---|---| +| `k3s-prod-1` | SATA SSD `C400-MTFDDAC512M` | `/dev/sda5` | 432.5 GiB ext4 | `lost+found` only (`du -sh` 20K; `df` 2.1 M of 425 G) | +| `k3s-prod-2` | **NVMe** `PM9A1 Samsung` | **`/dev/nvme0n1p5`** | 432.5 GiB ext4 | `lost+found` only (same) | +| `k3s-prod-3` | SATA SSD `WDC WDS500G2B0A` | `/dev/sda5` | 422.2 GiB ext4 | `lost+found` only (same) | + +The `df` "5.1% used" that made `/srv` look occupied is **ext4's default 5% root-reserved +blocks**, not data — confirmed per node by `ls -A /srv` (only `lost+found`) and +`du -sh --exclude=lost+found /srv` (20 K). Nothing was destroyed. + +**How the role resolves the device — discovery, not per-host inventory.** A single +`k3s_prod` group value cannot be right when node 2 is NVMe, and per-host `host_vars` would +freeze a *fact about today's hardware* into inventory, where it goes stale the first time a +node is re-imaged or a disk is swapped — and the stale value's failure mode is `mkfs` on the +wrong partition. So `group_vars/k3s_prod/vars.yml` declares **intent** +(`node_baseline_longhorn_source: claim-mountpoint`, +`node_baseline_longhorn_claim_mountpoint: /srv`) and +`roles/node_baseline/tasks/disks.yml` asks the **node** which block device is mounted there, +then proves the answer is safe before writing anything. Six independent asserts, all +fail-closed: it is a partition or whole disk (not tmpfs/NFS/bind), it is **not** backing +`/`, `/boot`, `/boot/efi`, `/usr`, `/var`, `/home`, `/tmp` or active swap (that set is built +from the node's *own* gathered facts plus `swapon`, never a hardcoded partition list), it is +already ext4, it carries no other filesystem label, it is ≥ 100 GiB, and it is **empty**. + +**Unmount → relabel, not reformat.** `/srv` is already formatted and already mounted, and +`community.general.filesystem` will not touch a device that has a filesystem — it would +silently no-op and then the `LABEL=longhorn` mount would fail. The role therefore unmounts +`/srv`, removes its `/etc/fstab` entry (installer-written, pinned by UUID — leaving it would +re-mount the partition at `/srv` underneath Longhorn on the next boot), and stamps the label +with `e2label`. Relabelling rather than reformatting is deliberate: the partition is already +ext4 and already verified empty, so `mkfs` buys nothing, and if some assert were ever too +permissive the worst case is a renamed filesystem instead of a destroyed one. + +**Idempotency guard.** A second run is a no-op, not a reformat. After the claim the +partition carries the ext4 label `k3s-data` and `/srv` is gone from `/proc/mounts` *and* +`/etc/fstab` — so "what is mounted at `/srv`" would find nothing on run 2. The role pivots +on the durable on-disk evidence instead: `blkid -L k3s-data` returning a device means the +node is already claimed, and **every** destructive task is gated on that returning nothing. + +**The 2026-09-04 transition, handled idempotently.** All three nodes had *already* been +claimed under the previous naming (`LABEL=longhorn` at `/var/lib/longhorn`). Because the +idempotency pivot asks "does the **current** label exist?", changing the label makes an +already-claimed node answer *no* — and it would then walk into the greenfield claim path and +fail closed at "the claim mountpoint must actually be a mountpoint", `/srv` having been +consumed by that first claim. Fail-closed is correct, but it is not convergence, so the +transition is **declared** rather than inferred: +`node_baseline_data_retired_fs_label: longhorn` + `node_baseline_data_retired_mount: +/var/lib/longhorn` in `group_vars/k3s_prod/vars.yml`. The role then unmounts the retired +mountpoint, drops its `fstab` entry and re-stamps the label with `e2label` — a **metadata +write, never a reformat**, exactly as in the claim path — having first proved that the retired +mountpoint is still a mount, that it is backed by the very device the retired label names, and +that it is **empty** so nothing is orphaned. The common `LABEL=`-based mount task then brings +it up at `/data`, and the two per-tenant subdirectories are created (`0755` Longhorn, `0700` +k3s, since the latter holds the etcd DB, the node token and the image store). + +The transition block is **one-time and self-retiring**: on every subsequent run +`blkid -L k3s-data` succeeds and none of it is reachable. The two retired-* lines are safe to +delete from `group_vars` once the fleet is converged (it is, as of 2026-09-04). + +**Three independent reasons the LIVE dev cluster cannot enter that transition** — it has 18 +Longhorn volumes on `/var/lib/longhorn` and unmounting it would orphan every replica: +(1) both retired-* vars default to `""` and are set only in the prod-scoped `group_vars`; +(2) the path is gated on `longhorn_source == 'claim-mountpoint'`, and dev is `device` — the +same structural gate that keeps dev out of the claim path; (3) the emptiness assert, which +fails closed on a partition with replica directories even if (1) and (2) were both defeated. +Dev's mountpoint, label and k3s data dir are also simply **not changing**. + +**Run evidence (2026-09-04).** Full `node-baseline.yml` against `k3s_prod`: run 1 converged +(5–6 changed per node), run 2 **`changed=0` on all three nodes** with every transition and +claim task skipping. Per node afterwards: `blkid -L k3s-data` → `/dev/sda5` (nodes 1, 3) / +`/dev/nvme0n1p5` (node 2); `blkid -L longhorn` → gone; `/data` mounted with 403 G / 394 G +available; `/var/lib/longhorn` and `/srv` no longer mountpoints; `fstab` carrying exactly +`LABEL=k3s-data /data ext4 defaults,nofail,x-systemd.device-timeout=5s 0 0`; and `/var` down +to **5% used, ~11 GiB free**. + +**Dev can never reach this path.** Both role defaults stay `source: device`, so the LIVE +dev cluster keeps its dedicated `/dev/vdb` (Longhorn) and `/dev/vdc` (etcd) and the claim +tasks are structurally unreachable there — they are gated on `claim-mountpoint`, which only +the prod-scoped `group_vars` sets. + +### The accepted risk, and its exit condition + +Dev was burned by exactly this shape: a Longhorn rebuild starved etcd WAL fsync +(`docs/plans/ssd-etcd-isolation.md`), which is why dev has a third, dedicated etcd device. +**Prod accepts the same shared-device contention** because the failure mode is materially +cheaper here: dev's contention was on a **spinning HDD** with seek-bound queueing; every +prod node is **SSD or NVMe** end-to-end, where concurrent rebuild I/O costs latency rather +than collapsing it. The three nodes also have no second device to give etcd — one physical +disk each, already fully partitioned, no free slot verified populated. + +**Accepted to be MEASURED, not assumed.** Exit condition, in order of preference: + +- **Measure it before it matters.** Under a deliberate Longhorn replica rebuild, watch + `etcd_disk_wal_fsync_duration_seconds` p99 and `etcd_disk_backend_commit_duration_seconds` + p99, plus `etcd_server_leader_changes_seen_total`. If fsync p99 stays inside etcd's + practical budget (~10 ms wal fsync / ~25 ms backend commit) with no leader churn, the + acceptance holds and this becomes a recorded measurement rather than an assumption. +- **If it does not hold:** add an M.2 NVMe per node (chassis slot to be verified + physically) and flip `node_baseline_etcd_source` back to `device` — the role still + supports that path unchanged, and it is the dev shape. + +**~~Known, separate consequence — `/var` is only 12 GiB.~~ RESOLVED 2026-09-04 by moving +k3s's data dir.** This was recorded here on 2026-09-03 as a capacity item "to watch or +repartition". Re-measuring it against dev turned it from a watch item into a blocker: the +container image store alone is **21–25 GB per node** on the fully-deployed dev cluster and +`/var` totals **58–70 GB**, against prod's **11.8 GiB**. So prod would have filled `/var` +during its first image pulls, and with etcd on the same partition a full `/var` means etcd +cannot write its WAL. The fix was neither "watch" nor "repartition" but `--data-dir` — see +"THE 2026-09-04 REVISION" above. `/var` now carries only the OS's own usage (503 MB, 5%). + Consequences to settle before imaging nodes 2 and 3: - [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package @@ -609,24 +821,43 @@ Consequences to settle before imaging nodes 2 and 3: a member corrupts — this drive has already logged 32 of them. (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see `DECISIONS-LOG.md`. It is not part of this verdict.) -- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices* - (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them. - Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** — - prod must override them in `group_vars`. On a single-disk node there is no second device at - all, so one of: - - **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a - SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own - spindle. - - **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`. - Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by. - - **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one - physical failure domain. -- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry. -- [ ] Record the chosen device paths here and set them in - `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately - `/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full - `node-baseline` run against a node still carrying those placeholders now stops on an explicit - assert naming this section, instead of failing obscurely at `mkfs` time. +- [x] **Layout DECIDED and APPLIED 2026-09-03** — see "THE DECISION" above. `roles/node_baseline` + no longer expects a block-device path for either store; it takes a declared **source + enum** per environment (`roles/node_baseline/tasks/disks.yml`): + - **Longhorn** — `node_baseline_longhorn_source: claim-mountpoint` + + `node_baseline_longhorn_claim_mountpoint: /srv`. The `/dev/vdb` default (a virtio + name that does not exist on bare metal) is unused on prod; the real partition is + discovered on the node and guarded, because it is `/dev/sda5` on nodes 1 and 3 and + `/dev/nvme0n1p5` on node 2. + - **etcd** — `node_baseline_etcd_source: shared-data-partition` (was `root-filesystem` + until 2026-09-04). "No dedicated device" is a first-class, explicitly-configured state + that the `disks` assert **accepts**, not a blank variable that the tasks happen to + skip — and the *name* of that state now describes reality, because etcd is no longer + on the OS filesystem. The distinction is load-bearing, not cosmetic: + `shared-data-partition` is **asserted**, per node on every run, by checking that the + filesystem backing the etcd data dir really is the data partition's mount. Declaring + `k3s_data_dir: /data/k3s` does not by itself achieve that — had the mount silently + failed, `/data` would be a plain directory on `/` and etcd plus 21–25 GB of images + would land back on the OS filesystem, invisibly, until the disk filled. Leaving the + value at `root-filesystem` would have been a lie inventory told about the cluster, and + `root-filesystem` makes no such claim and gets no such assert. The role also still + asserts nothing is already mounted at the etcd data dir (so a half-migrated node + cannot be inherited silently) and reports which filesystem etcd actually lands on + (measured 2026-09-04: `/data`, 403 G / 394 G available, vs `/var`'s 10.7 GiB). + - The two rejected options stand rejected: **add a second drive** (no verified free slot; + the exit condition above reopens it if measurement fails) and **repartition** (same + single failure domain, plus a destructive re-image of three working nodes for no + isolation the SSDs do not already give). +- [x] **All three nodes get the identical layout.** Same declaration, same mountpoint, same + resulting `/data` + `/data/longhorn` + `/data/k3s`. Not the same *device path* and not + the same *size* (432.5 / 432.5 / 422.2 GiB) — Longhorn tolerates unequal node capacity; + it is replica *placement* that assumes a symmetric layout, and the layout is symmetric. + Confirmed per node 2026-09-04 by `findmnt`/`lsblk`/`blkid` after the transition. +- [x] **Chosen paths recorded here and set in `group_vars`.** The BLOCKED placeholders in + `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` are replaced — by the source + enums above, plus deliberately-unusable `/dev/NOT-A-DEVICE-…` values for the two device + variables so that flipping a source back to `device` without supplying a real path + still fails on the role's assert instead of silently inheriting dev's `/dev/vdb`. - [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline` is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk tasks this decision blocks — see §5's drive-health box for the exact command. @@ -639,6 +870,138 @@ Consequences to settle before imaging nodes 2 and 3: - [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into `roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now), so the same role covers bare metal. Nothing to port. +- [x] **KERNEL: a standing, deliberate deviation from the distro baseline — `linux-image-amd64` + from `trixie-backports` (DONE on all three nodes, 2026-09-04).** + + **What.** `roles/node_baseline` gained a `kernel`-tagged block that enables + `trixie-backports` as a deb822 apt source and installs the `linux-image-amd64` + **metapackage** from it. Applied 2026-09-04; all three nodes landed on + **`linux-image-amd64` 7.1.8-1~bpo13+1 → kernel `7.1.8+deb13-amd64`**, rebooted one at a + time, and are running it. + + ```bash + ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel + ``` + + **Why.** §0b. HP's R21 firmware publishes a corrupt EFI Memory Attributes Table; Linux + < 6.13 reserves memory from it without validating the size, and `k3s-prod-2` has booted + with 2.6 GiB of its 16 GB, twice. The fix is upstream commit **`8fbe4c49c0cc`** + *"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, first + released in **v6.13**, hardened by `5d0faa8e8369`). It is absent from v6.12 and from + `linux-6.12.y`, carries neither `Cc: stable` nor `Fixes:`, and there is **no kernel + cmdline workaround**. A k3s **server** that intermittently comes up with 2.6 GiB will + fail under etcd plus its workload share on any reboot, so this is a **prerequisite to + installing k3s**, not a follow-up. + + **How it is scoped.** `node_baseline_backports_kernel` defaults to **false** in + `roles/node_baseline/defaults/main.yml` and is set **true** exactly once, in + `inventory/group_vars/k3s_prod/vars.yml` — the same seam `kube_vip_address` / + `k3s_api_host` / `cluster_kubeconfig_name` use. Dev's Proxmox VMs publish no MAT and are + genuinely unaffected; they must not be dragged onto a backports kernel as a side effect + of sharing the role, and a fourth prod node must inherit the deviation from **group + membership**, not from a `k3s-prod-*` hostname test. Verified: a `--tags kernel` run + against dev skips all 8 tasks on all 3 hosts. + + **Security cost, accepted.** `*-backports` packages are provided **as-is** and are + **not** covered by the Debian Security Team's guarantees for the main archive — no + security-tracker commitment, no DSAs. Recorded here as the price of the nodes keeping + their RAM. + + **No apt pin, deliberately.** The backports `Release` file carries `NotAutomatic: yes` + + `ButAutomaticUpgrades: yes`, which apt already reads as *"priority 100 so nothing is + taken from backports by accident, 500 for a package explicitly installed from there so + it keeps getting updates."* Writing an explicit `Pin-Priority: 100` in + `preferences.d` would look like belt-and-braces and would in fact **switch off the + automatic security upgrades for the very kernel this deviation installs.** Enabling the + source is the whole configuration. + + **Metapackage, not a pinned ABI.** A frozen `linux-image-6.19.14+deb13-amd64` would stop + receiving even backports' own updates the moment the ABI rolls. The metapackage tracks + whatever backports ships (>= 6.13 by construction). The **exact** version each node + landed on is recorded on the node itself at `/etc/node-baseline-backports-kernel`, + written by the same role, along with the rollback path below. + + **Rollback.** + 1. *Non-destructive, try first.* The stock kernels are still installed + (`6.12.107+deb13-amd64` and `6.12.94+deb13-amd64` on all three) and selectable from + GRUB → *Advanced options for Debian GNU/Linux*. That answers "is the new kernel the + problem?" without uninstalling anything — it also reinstates the memattr bug for that + boot. + 2. *Permanent.* Set `node_baseline_backports_kernel: false` in + `group_vars/k3s_prod/vars.yml` **first** (otherwise the next Ansible run puts it + back), then on each node: + ```bash + apt-get install --reinstall linux-image-amd64/trixie + apt-get purge linux-image-7.1.8+deb13-amd64 + rm /etc/apt/sources.list.d/trixie-backports.sources + apt-get update && update-grub && reboot + ``` + + **The role does NOT reboot**, and that is deliberate — see §0b "SECOND FAULT". Reboots + are a serialized, supervised operator step: one node at a time, verify each returns + before touching the next, never node 2 first. On 2026-09-04 all three returned in + **39–41 s** (node 1 39 s, node 3 41 s, node 2 39 s); node 2 did **not** reproduce its + 2026-09-03 hang, but one clean warm reboot is not a fix and that defect stays open. + + **Exit condition — how this deviation ends.** If `8fbe4c49c0cc` is ever picked up into + `linux-6.12.y` (and therefore into Debian's 6.12 rebase, which reaches us by plain + `apt upgrade`), this can be removed entirely: set `node_baseline_backports_kernel: + false`, roll back per the note on each node, and delete the `kernel` block from + `roles/node_baseline`. §0b records why that pickup will not happen on its own and what + request would cause it. +- [x] **Verification of the fix, and a correction to how it is verified (2026-09-04).** + + §0b says *"A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table + detected!`"*. That is true but **conditional, and it did not happen on any of the three + nodes.** `efi_memattr_init()` on >= 6.13 has **three** gates and only the third prints + that line: + + | gate | message | reachable when | + |---|---|---| + | `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field | + | implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, `desc_size` does not | + | `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* `desc_size` read plausibly, entry count does not | + + **All three are rejections**, and which one a boot prints is decided by *which field* + that boot's garbage corrupts. On 7.1.8 all three nodes hit the **first** gate: + + | node | `MEMATTR=` | kernel line on 7.1.8 | MemTotal | reserved at boot | + |---|---|---|---|---| + | 1 | `0xbee84018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1030` | 31.1 GiB | 0.9 GiB | + | 2 | `0xbeeb0018` | `efi: memattr: Unexpected EFI Memory Attributes table version -1569980416` | 15.4 GiB | 0.6 GiB | + | 3 | `0xbeecc018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` | 31.1 GiB | 0.9 GiB | + + Node 2 is the result that matters: **`0xbeeb0018` is the exact address that produced + 2.6 GiB twice**, and on 7.1.8 it reserved 634,904 K instead of 13,913,856 K. + + **Boot-independent proof the fix is compiled in**, since the log line is a lottery: the + running kernel image itself contains the string. Decompressing `/boot/vmlinuz-$(uname + -r)` on all three nodes yields, verbatim — + + ``` + efi: memattr: [Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u) + efi: memattr: Unexpected EFI Memory Attributes descriptor size %u (expected: %lu) + ``` + + — neither of which exists in 6.12. That is the positive proof; the `dmesg` line is not + available on demand. + + **Two bugs in `roles/node_health` were fixed as a consequence**, because without them + the health check could not show the fix landing: + - `MEMATTR_VERSION_RE` in `files/node_health_collect.py` matched `(\d+)` only. The + kernel prints that field with `%d` — **signed** — so node 2's `-1569980416` did not + match, the state fell through to `clean`, and the node was reported **PASS "the kernel + accepted it without complaint"**: a false pass on the exact bug the check exists to + find. Now `(-?\d+)`. + - `firmware.efi_memattr` treated `bad_version` as WARN unconditionally, so a node on a + fixed kernel would have stayed WARN forever. The verdict now keys on the running + kernel version (`memattr_kernel_fixed`, `>= 6.13`): the same log line is WARN on + < 6.13 (a lucky boot) and PASS on >= 6.13 (the validation working). The pre-existing + `corrupt_rejected` PASS branch is unchanged and remains reachable. + + **`playbooks/node-health.yml -e node_health_hosts=k3s_prod`, after the reboots: + `firmware.efi_memattr` = PASS on all three, `memory.physical_total` = PASS on all + three.** Dev re-run unchanged. ## 4. Network (per node) diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 2113cf5..5e04ed9 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -87,24 +87,155 @@ kube_vip_address: "192.168.2.9" # — see group_vars/all.yml), so once real this collapses to a direct "https://:6443". cluster_controller_api_server: "https://{{ kube_vip_address }}:6443" -# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ---------------- -# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb, -# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO -# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block -# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a -# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on -# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31). +# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) -------- +# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox +# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a +# backports kernel as a side effect of sharing the role), and this line switches it on for +# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host / +# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership +# rather than from anyone remembering. # -# The overrides below are deliberately present-but-broken: leaving them UNSET would -# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this -# hardware — a confusing failure that looks like a wiring bug. This value fails just -# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching -# a real device. DO NOT replace with a guess — replace only once §2 is decided, and -# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that -# updates these two lines. All three nodes must end up with an identical layout -# (Longhorn replica placement assumes symmetry) before any of this is used for real. -node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2" -node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2" +# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT +# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating +# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that +# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail +# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw +# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far. +# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc, +# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel +# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded +# in §3. +# +# COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team +# guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep +# the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this +# back to false, roll back per the note the role writes to +# /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from +# roles/node_baseline. +# +# The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND +# FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes). +node_baseline_backports_kernel: true + +# --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2) +# Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale: +# +# DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical +# disk, claimed and mounted at the NEUTRAL mountpoint `/data` +# (LABEL=k3s-data), then split between TWO tenants: +# /data/longhorn Longhorn's dataPath +# /data/k3s k3s --data-dir (etcd AND the containerd image store) +# ETCD -> NO dedicated device. It lives inside /data/k3s on that partition. +# +# ================================ THE 2026-09-04 REVISION ================================ +# The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default +# /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's +# data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully- +# deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod +# would therefore have filled /var during its FIRST image pulls — and with etcd on the same +# partition, a full /var means etcd cannot write its WAL. That is cluster death, not +# degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the +# mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two +# tenants. Nothing was installed on prod when this was decided, which is exactly why it was +# decided then — it is cheap now and very expensive after bring-up. +# +# THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd +# were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn +# contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be +# MEASURED, not assumed — exit condition in §2. +# ========================================================================================= +# +# Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml), +# not as a device path and a blank. That distinction is the point of this block: +# +# 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right. +# k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe +# (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three. +# Per-host `host_vars` would "work" and would be WRONG in the way that matters: it +# freezes a fact about today's hardware into inventory, where it goes stale silently the +# first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller +# nobody predicted — and the stale value's failure mode is mkfs on the wrong partition. +# So inventory declares the INTENT ("the partition currently mounted at /srv") and the +# role discovers the device on the node, then proves it is safe to claim (a partition or +# disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4, +# unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed. +# +# 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that +# describes reality. It was `root-filesystem` while etcd sat on /var; under the revised +# layout etcd is inside /data/k3s on the claimed data partition, which is neither a +# dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`. +# Leaving it at `root-filesystem` would have been a lie inventory told about the cluster, +# and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node +# on every run, that the filesystem backing the etcd data dir really is /data. If the +# mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's +# WAL would land back on the OS filesystem, invisibly, until the disk filled. The +# `root-filesystem` value makes no such claim and gets no such assert — which is exactly +# why the two are distinct enum values rather than one with a comment. +# +# 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so +# the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter +# the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only +# this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and +# k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live +# Longhorn volumes are untouched. +node_baseline_longhorn_source: claim-mountpoint +node_baseline_longhorn_claim_mountpoint: /srv +node_baseline_etcd_source: shared-data-partition + +# --- the neutral mountpoint + label (role defaults are DEV's live values) ---------- +# `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the +# CLAIM mountpoint the role discovers the device at, and the idempotency contract is that +# /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it +# as the destination would make "claimed" and "not claimed" indistinguishable. +# `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a +# partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody +# else's named volume — so labelling ours `data` would make a stranger's `data` volume look +# already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the +# `k3s-etcd` label convention already in this repo. +node_baseline_data_mount: /data +node_baseline_data_fs_label: k3s-data +# Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST +# equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm +# side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role +# default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn. +node_baseline_longhorn_data_path: /data/longhorn + +# --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------ +# The three prod nodes were ALREADY claimed by the previous version of this role: LABEL +# `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT +# label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and +# would walk into the greenfield claim path — which fails closed at "the claim mountpoint +# must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed +# is right but it is not convergence, so the transition is DECLARED here: the role unmounts +# the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved +# the partition is empty (nothing orphaned) and is the device the retired label names. +# Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all. +# SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks +# then skip on the retired-label lookup and nothing else depends on them. +node_baseline_data_retired_fs_label: longhorn +node_baseline_data_retired_mount: /var/lib/longhorn + +# --- k3s --data-dir (§2) ----------------------------------------------------------- +# THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the +# claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into +# `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir, +# which defers to this variable) creates the directory 0700 on the data partition and derives +# etcd's location from it (node_baseline_etcd_mount = /server/db). One variable, so the +# Ansible side cannot disagree with itself. +# +# Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults): +# it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data +# dir would orphan that mount under a running cluster. +k3s_data_dir: /data/k3s + +# Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither +# source uses a device path any more, but leaving these UNSET would let prod silently +# inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare +# metal) the moment anyone flipped a source back to `device`. These values can never parse +# as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token +# before mkfs is ever reached. +node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2" +node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2" # --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) ----- # roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1 diff --git a/infra/ansible/roles/k3s_server/defaults/main.yml b/infra/ansible/roles/k3s_server/defaults/main.yml index b2a78ce..4875e1d 100644 --- a/infra/ansible/roles/k3s_server/defaults/main.yml +++ b/infra/ansible/roles/k3s_server/defaults/main.yml @@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh" # the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.) k3s_disable: "traefik,servicelb,local-storage" +# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------ +# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and +# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd +# image store (agent/containerd) — which is by far the largest tenant. +# +# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the +# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps +# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at +# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a +# running cluster's entire state directory out from under it and orphan that etcd mount. +# +# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam +# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod +# node inherits it from group membership rather than from anyone remembering. It is set +# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role +# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed +# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to +# this same variable). One line moves both. +# +# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a +# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd +# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its +# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death, +# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition. +# +# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at +# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to +# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling +# Restart handler for a value that is not changing. The install flags are the correct and +# sufficient place, and prod has no k3s installed yet. +k3s_data_dir: "/var/lib/rancher/k3s" + # Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary). k3s_server_mode: "secondary" diff --git a/infra/ansible/roles/k3s_server/tasks/main.yml b/infra/ansible/roles/k3s_server/tasks/main.yml index eee9d6e..f925742 100644 --- a/infra/ansible/roles/k3s_server/tasks/main.yml +++ b/infra/ansible/roles/k3s_server/tasks/main.yml @@ -22,6 +22,7 @@ --write-kubeconfig-mode=644 {% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %} --node-ip={{ k3s_node_ip }} + --data-dir={{ k3s_data_dir }} # On an ALREADY-installed cluster the install task above is idempotent (creates: # k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index 3c527ae..d353b71 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -18,22 +18,170 @@ node_baseline_packages: # node from first baseline — package installation is the role's job, never a manual step # (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo). - smartmontools +# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) --------------------- +# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13 +# baseline — not a one-off `apt install` someone has to remember on the next node. +# +# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory +# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory +# from its contents, so an unlucky boot reserves everything above the table and the node +# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit +# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel, +# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y — +# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO +# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a +# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b. +# +# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware +# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a +# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped +# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would +# encode the environment into the ROLE, would silently miss a fourth prod node named +# differently, and could not be flipped for a one-off test host. So it is a plain boolean, +# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam +# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use. +# +# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT +# covered by the Debian Security Team's guarantees for the main archive — no security-tracker +# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it +# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's. +# +# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's +# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and +# delete this block. Nothing else in the repo depends on it. +node_baseline_backports_kernel: false +# The suite, mirror and components the source line is built from. Split out so a mirror +# change is a variable, not an edit to a heredoc in tasks/main.yml. +node_baseline_backports_suite: trixie-backports +node_baseline_backports_mirror: "http://deb.debian.org/debian" +# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT +# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive +# surface beyond what the node already trusts is a separate decision nobody has taken. +node_baseline_backports_components: + - main + - non-free-firmware +# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI +# would stop receiving even backports' own updates the moment the ABI rolls, leaving the +# node on a kernel nothing patches — strictly worse than the risk this deviation accepts. +# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and +# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file +# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on +# is recorded on the node itself — see node_baseline_backports_kernel_stamp. +node_baseline_backports_kernel_packages: + - linux-image-amd64 +# Provenance/rollback note written on the node. An operator standing in front of a prod box +# with no repo checkout must be able to answer "why is this machine not on the distro +# kernel, and how do I undo it?" from the machine itself. +node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel + node_baseline_kernel_modules: - overlay - br_netfilter - iscsi_tcp - dm_crypt - nfs + +# --- DATA DISKS (tasks/disks.yml, tag `disks`) ---------------------------------------- +# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether +# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape: +# +# node_baseline_longhorn_source: +# device dedicated block device at node_baseline_longhorn_device (DEV: the +# tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label. +# claim-mountpoint re-purpose the partition the node ALREADY has mounted at +# node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device +# path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1 +# and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value +# is correct and a hardcoded one would eventually mkfs the wrong +# partition on a future node. Destructive; heavily guarded — read +# tasks/disks.yml before changing anything here. +# +# node_baseline_etcd_source: +# device dedicated block device at node_baseline_etcd_device (DEV: +# /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync +# isolated from Longhorn rebuild I/O after dev was burned by HDD +# contention). +# root-filesystem etcd lives on the OS filesystem, with NO dedicated device and +# NO claimed data partition under it. Kept as a valid, supported +# shape; NOT what prod uses any more (see below). +# shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the +# CLAIMED data partition (PROD). No dedicated device, but also NOT +# the OS filesystem: the whole point of the 2026-09-04 layout +# revision is that /var (11.8 GiB) can hold neither the container +# image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2. +# +# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision +# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device +# var, which still fails the `disks` assert. +node_baseline_longhorn_source: device node_baseline_longhorn_device: /dev/vdb -node_baseline_longhorn_mount: /var/lib/longhorn -node_baseline_longhorn_fs_label: longhorn +# --- The DATA PARTITION: mountpoint + filesystem label ------------------------------- +# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO +# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image +# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides +# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`). +# +# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18 +# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at +# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays +# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one +# partition), which is why node_baseline_longhorn_data_path below defaults to the mount. +node_baseline_data_mount: /var/lib/longhorn +node_baseline_data_fs_label: longhorn +# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount +# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a +# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob +# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the +# same fact, and the two are checked against each other by nothing but this comment. +node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}" +# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares +# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately +# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the +# install flags, this role reads the same value to create the directory and to place etcd, +# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars +# value (higher precedence than any role default) is what actually resolves here, even +# though roles/k3s_server is not in the node-baseline play; the default() keeps +# node-baseline.yml runnable standalone. +node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}" +# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by +# default so a dev host cannot enter the claim path even by accident. +node_baseline_longhorn_claim_mountpoint: "" +# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------ +# A node claimed by an EARLIER version of this role carries the OLD label at the OLD +# mountpoint, and the idempotency pivot (`blkid -L `) would therefore read +# "not claimed yet" and walk into the greenfield claim path — which then fails closed at +# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These +# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint +# are NOT changing, so dev has nothing to transition and cannot enter the transition path. +node_baseline_data_retired_fs_label: "" +node_baseline_data_retired_mount: "" +# Sanity floor for the claim path: a data partition smaller than this is not "a small +# store", it is the wrong partition. Prod's /srv is 415-425 GiB. +node_baseline_longhorn_min_size_gb: 100 # Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir # so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk. # See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across -# device renames (matches the Longhorn LABEL= approach). +# device renames (matches the data partition's LABEL= approach). +node_baseline_etcd_source: device node_baseline_etcd_device: /dev/vdc -node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db +# DERIVED, never restated: etcd's db always lives at /server/db, so moving +# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged +# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves +# to /data/k3s/server/db, inside the claimed partition. +node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db" node_baseline_etcd_fs_label: k3s-etcd +# Mounts whose backing devices the claim path must NEVER touch. Used to build the +# protected-device set from the node's own gathered facts (plus live swap), so a node +# partitioned differently from the fleet is judged on its own layout rather than against +# a hardcoded list of partition names. +node_baseline_protected_mounts: + - / + - /boot + - /boot/efi + - /usr + - /var + - /home + - /tmp # --- DNS resilience (R-DNS-01 / R-DNS-03) ------------------------------------------- # These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml diff --git a/infra/ansible/roles/node_baseline/tasks/disks.yml b/infra/ansible/roles/node_baseline/tasks/disks.yml new file mode 100644 index 0000000..ba43eec --- /dev/null +++ b/infra/ansible/roles/node_baseline/tasks/disks.yml @@ -0,0 +1,642 @@ +--- +# ===================================================================================== +# node_baseline — DATA DISKS. Every task here carries the `disks` tag (applied once by +# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file. +# Split into its own file rather than living in main.yml because it is the one part of +# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own. +# +# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred: +# +# dev (group_vars/all.yml, role defaults) +# node_baseline_longhorn_source: device -> mkfs /dev/vdb, mount LABEL=longhorn +# at /var/lib/longhorn (== dataPath) +# node_baseline_etcd_source: device -> mkfs /dev/vdc, mount LABEL=k3s-etcd +# k3s_data_dir: (k3s default, /var/lib/rancher/k3s) +# THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for +# exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision. +# +# prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2) +# node_baseline_longhorn_source: claim-mountpoint (claim mountpoint /srv) +# node_baseline_data_mount: /data (LABEL=k3s-data) +# node_baseline_etcd_source: shared-data-partition +# k3s_data_dir: /data/k3s +# ONE physical disk per node, already fully partitioned by the Debian installer. The +# ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split +# between TWO tenants: /data/longhorn (Longhorn replicas) + /data/k3s (the whole k3s +# data dir: etcd AND the containerd image store). +# +# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md +# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts +# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev +# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would +# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its +# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB +# partition is the fix; the partition therefore serves two tenants, and neither its +# mountpoint nor its filesystem label may name only one of them. +# +# WHY /data AND LABEL k3s-data. +# * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point). +# Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device +# at, and the idempotency contract is that /srv is gone from /proc/mounts and +# /etc/fstab once the claim has happened. Re-using it as the destination would make +# "claimed" and "not yet claimed" indistinguishable. +# * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below +# accepts a partition whose label is empty OR ALREADY OURS, and its own comment names +# `data` as an example of somebody else's named volume. Labelling ours `data` would make +# a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is +# cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label +# convention in this repo. +# +# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and +# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an +# empty string, and the second one must never silently succeed. `root-filesystem` and +# `shared-data-partition` are explicit, greppable statements of intent that the assert +# below accepts; an unset or placeholder device is still a hard failure. The two differ in +# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and +# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER +# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather +# than printing a claim it never checked (see the etcd section). +# +# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on +# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is +# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory +# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node +# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually +# point at the wrong partition on some future node and mkfs it. So the role asks the NODE +# which block device is mounted at the claim mountpoint, and then proves that answer is +# safe (see the assert wall below) before touching it. The inventory declares INTENT +# ("the partition currently mounted at /srv"); the node supplies the FACT. +# ===================================================================================== + +# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared +# its data-disk layout. Two failure shapes are caught: +# * an undeclared/typo'd source enum, and +# * source: device with a device path that is a PLACEHOLDER rather than a real device. +# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device +# vars now that prod uses neither: if anyone ever flips a source back to `device` without +# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does +# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED` +# stays in the token list because that was the pre-decision placeholder and may still be +# carried by other environments/branches. +- name: Assert this environment's data-disk layout has actually been declared + ansible.builtin.assert: + that: + - node_baseline_longhorn_source in ['device', 'claim-mountpoint'] + - node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition'] + # `shared-data-partition` is only coherent when there IS a claimed data partition to + # share, and only when the k3s data dir actually sits INSIDE it. Catch the + # contradiction here, in inventory terms, rather than three sections later when etcd + # has silently landed back on the OS filesystem. + - >- + node_baseline_etcd_source != 'shared-data-partition' + or node_baseline_longhorn_source == 'claim-mountpoint' + - >- + node_baseline_etcd_source != 'shared-data-partition' + or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/') + # The transition knobs are a matched PAIR: a retired label with no retired mountpoint + # (or vice versa) cannot describe a state to migrate from. + - >- + (node_baseline_data_retired_fs_label | length > 0) + == (node_baseline_data_retired_mount | length > 0) + - >- + node_baseline_longhorn_source != 'device' + or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE')) + - >- + node_baseline_etcd_source != 'device' + or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE')) + - >- + node_baseline_longhorn_source != 'claim-mountpoint' + or (node_baseline_longhorn_claim_mountpoint | length) > 0 + fail_msg: >- + This host's data-disk layout is not declared. + longhorn_source={{ node_baseline_longhorn_source }} + (device={{ node_baseline_longhorn_device }}, + claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}), + etcd_source={{ node_baseline_etcd_source }} + (device={{ node_baseline_etcd_device }}). + Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or + `claim-mountpoint` (re-purpose the partition already mounted at + node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources: + `device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under + it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED + data partition — prod's shape since 2026-09-04, which additionally requires + longhorn_source `claim-mountpoint` and a k3s data dir under + node_baseline_data_mount={{ node_baseline_data_mount }}; + k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT + operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission. + Declare it in the environment's group_vars and record the reasoning in + §2 in the SAME change. To install ONLY the package floor meanwhile, re-run with + `--tags packages`, which skips every disk task including this assert. + quiet: true + +# ===================================================================================== +# THE DATA PARTITION (prod: Longhorn store + k3s data dir; dev: Longhorn store only) +# ===================================================================================== + +# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op +# rather than a reformat. Once the claim has happened the partition carries the ext4 +# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim +# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is +# mounted at /srv" would find nothing on run 2. Asking `blkid -L