Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
05baae90ec | ||
|
jakman85andClaude Opus 5
|
8e913fdd7b | ||
|
jakman85andClaude Opus 5
|
b19c5db9ae | ||
|
jakman85
|
6e4d311545 | ||
 gitea-admin
|
4809199da6 | ||
|
gitea-admin
|
086e9091c9 |
No files matched your search
@@ -74,6 +74,17 @@ data:
|
||||
|
||||
# --- storage (ADR-0003) ---
|
||||
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
||||
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
||||
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
||||
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
||||
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
||||
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
||||
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
||||
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
||||
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
||||
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
||||
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
||||
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
||||
|
||||
# --- observability sizing (docs/10) ---
|
||||
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
||||
|
||||
@@ -190,6 +190,16 @@ patches:
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
value: "3"
|
||||
# DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
|
||||
# on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
|
||||
# mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
|
||||
# else would orphan every one of them (they are not recoverable from the old path by
|
||||
# rescheduling). Restated explicitly even though it equals the base default, exactly like the
|
||||
# LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
|
||||
# its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DATA_PATH
|
||||
value: "/var/lib/longhorn"
|
||||
# smaller retention / volumes to keep the dev workstation footprint light.
|
||||
- op: replace
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION
|
||||
|
||||
@@ -157,6 +157,20 @@ patches:
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
value: "3"
|
||||
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
|
||||
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
|
||||
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
|
||||
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
|
||||
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
|
||||
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
|
||||
#
|
||||
# [parity] MUST equal node_baseline_longhorn_data_path in
|
||||
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
|
||||
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
|
||||
# path on the OS filesystem instead of the data partition and /var fills up.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DATA_PATH
|
||||
value: "/data/longhorn"
|
||||
# longer retention / larger volumes on prod NVMe.
|
||||
- op: replace
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION
|
||||
|
||||
+670
-39
@@ -25,15 +25,15 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
|
||||
|---|---|---|---|
|
||||
| Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
|
||||
| LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
|
||||
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | |
|
||||
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | |
|
||||
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | |
|
||||
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | |
|
||||
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | |
|
||||
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | |
|
||||
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | |
|
||||
| `/dev/kvm` | present | present | |
|
||||
| Serial | MXL1052NGN | MXL0073949 | |
|
||||
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini |
|
||||
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x |
|
||||
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) |
|
||||
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
|
||||
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
|
||||
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 |
|
||||
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 |
|
||||
| `/dev/kvm` | present | present | present |
|
||||
| Serial | MXL1052NGN | MXL0073949 | 8CC015307N |
|
||||
|
||||
**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check;
|
||||
unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**),
|
||||
@@ -41,10 +41,21 @@ unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2
|
||||
but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded
|
||||
only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch.
|
||||
|
||||
**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a
|
||||
healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica
|
||||
placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's
|
||||
NVMe is the drive node 1 should have had.
|
||||
**The nodes are NOT hardware-symmetric — and with node 3 in (2026-09-03) the asymmetry is
|
||||
THREE-WAY, not a node-1 outlier.** Node 1 has a ~2011 SATA SSD with real wear; node 2 a healthy
|
||||
modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent health (100% life,
|
||||
0 grown bad blocks). **Three nodes, three different drive models, and only one is NVMe.** §2 requires
|
||||
an identical disk layout across all three because Longhorn replica placement assumes symmetry — so
|
||||
this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should
|
||||
have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs**
|
||||
(2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually
|
||||
usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table
|
||||
too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to
|
||||
the §0b bug on exactly the same terms as node 1 and needs the same >= 6.13 kernel. Node 3 therefore
|
||||
WIDENS §0b's blast radius from "both prod nodes" to **all three**; it is not a control machine.
|
||||
Every node also carries **exactly one** physical disk with **no unpartitioned spare**, so
|
||||
`node_baseline_longhorn_device` / `node_baseline_etcd_device` have nothing to claim on any of the
|
||||
three without repartitioning or adding drives — the §2 decision cannot be satisfied by node 3 either.
|
||||
|
||||
Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).
|
||||
|
||||
@@ -92,15 +103,42 @@ memblock_reserve(efi_mem_attr_table, tbl_size);
|
||||
That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` =
|
||||
DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero.
|
||||
|
||||
### BOTH prod nodes are affected — node 1 has only been lucky
|
||||
### ALL THREE prod nodes are affected — none is immune
|
||||
|
||||
Seven boots observed across the three machines. **The MAT address moves boot-to-boot on every
|
||||
node with settings held constant**, and each node has been seen on more than one path:
|
||||
|
||||
| Node | Boot | `MEMATTR=` | Kernel's read | Memory |
|
||||
|---|---|---|---|---|
|
||||
| 1 | −1 | `0xbeee6018` | *(no message — table read as VALID)* | 32.4 G ✅ |
|
||||
| 1 | 0 | `0xbeec9298` | version **1030** (> 2 → bail) | 32.4 G ✅ |
|
||||
| 2 | −2 | `0xbeadc018` | version **1463964749** (> 2 → bail) | 15.9 G ✅ |
|
||||
| 2 | −1 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ |
|
||||
| 2 | 0 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ |
|
||||
| 3 | −1 | `0xbee8a018` | version **1463964749** (> 2 → bail) | 32.4 G ✅ |
|
||||
| 3 | 0 | `0xbeece018` | *(no message — table read as VALID)* | 32.4 G ✅ |
|
||||
|
||||
So there are **three** outcomes, not two: bail-out on a garbage version > 2, a *valid* parse with a
|
||||
sane size, and the catastrophic path. `k3s-prod-3` publishes a MAT at `0xbeece018` exactly like the
|
||||
others — the *absence* of a memattr diagnostic line on its current boot means the kernel read a
|
||||
**valid** table, which is the opposite of immunity. Dev nodes publish no MAT at all (Proxmox
|
||||
SeaBIOS/OVMF) and are genuinely unaffected.
|
||||
|
||||
**All three prod nodes are on the same lottery and all three need the ≥ 6.13 kernel.**
|
||||
|
||||
### The corrupting bytes are the OS boot chain, not the firmware or the DRAM
|
||||
|
||||
The garbage version `1463964749` = `0x5742544D` appeared on **node 2 AND node 3** — two different
|
||||
machines, identical corrupting bytes. That rules out both firmware-specific behaviour and a DRAM
|
||||
fault: the overwriter is the **shared OS boot chain** (shim / GRUB / EFI stub / initrd placement),
|
||||
writing the same data into the page the firmware pointed at. This is the documented failure mode —
|
||||
the firmware places the MAT in memory the OS classifies as plain `System RAM`, and whoever allocates
|
||||
there next overwrites it.
|
||||
|
||||
It also means the boot chain's own allocation footprint — initrd size, module set, boot device — is
|
||||
part of the causal chain, which matters for the experiment design below.
|
||||
|
||||
| Node | memattr version field | Outcome |
|
||||
|---|---|---|
|
||||
| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** |
|
||||
| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB |
|
||||
| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB |
|
||||
|
||||
Node 1 is one unlucky boot away from the identical fault. **Fix both.**
|
||||
|
||||
### Mechanism, traced to the byte (second, independent confirmation)
|
||||
|
||||
@@ -130,6 +168,58 @@ The cleanest A/B tell between the two nodes is one line of dmesg:
|
||||
Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
|
||||
node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.
|
||||
|
||||
### CORRECTION (2026-09-03) — "Failed to map" does NOT mean memory was lost
|
||||
|
||||
An earlier version of this section, and the `firmware.efi_memattr` health check built from it, treated
|
||||
the log line `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` as proof of memory loss.
|
||||
**A live counter-example falsifies that.** `k3s-prod-2`, after being power-cycled on 2026-09-03:
|
||||
|
||||
```
|
||||
MEMATTR=0xbeeb0018 <-- the SAME address as both bad boots
|
||||
[ 0.106493] efi: memattr: Failed to map ... @ 0xbeeb0018 <-- the SAME line
|
||||
MemTotal: 15857732 kB <-- 15.1 GiB of 16 GB. FULL MEMORY.
|
||||
```
|
||||
|
||||
**Two different kernel functions were being conflated:**
|
||||
|
||||
- **`efi_memattr_init()`** does the `memblock_reserve()`. Catastrophic loss requires `tbl_size` to be
|
||||
**negative** — signed `int`, sign-extended through `phys_addr_t`, clamped to `PHYS_ADDR_MAX`, so
|
||||
everything above the table is reserved. A garbage-but-**positive** `tbl_size` reserves only a modest
|
||||
wrong amount and costs nothing noticeable.
|
||||
- **`efi_memattr_apply_permissions()`** does the later `memremap()`, and **this is what prints
|
||||
"Failed to map"**. It fails for *any* sufficiently wrong size — negative or merely large-positive.
|
||||
|
||||
So the line proves **the table was corrupt and the kernel trusted it**, not that memory was lost. The
|
||||
cost is set by the *value* of the garbage on that boot, which is why the same address yielded 2.6 GiB
|
||||
twice and 15.1 GiB once.
|
||||
|
||||
**This also refutes the address→outcome hypothesis outright** — it is not merely untested, it is
|
||||
disproven. The MAT address does not determine the outcome; the garbage value at it does.
|
||||
|
||||
The health check has been re-based accordingly: the measured shortfall is computed **once** into a
|
||||
shared fact and both `memory.physical_total` and `firmware.efi_memattr` read it, so they can no longer
|
||||
contradict each other. `Failed to map` + normal memory is now **WARN** ("the bug fired and this boot
|
||||
survived it"), and FAIL requires a measured shortfall.
|
||||
|
||||
### SECOND FAULT — node 2 hangs on a warm reboot, staying powered
|
||||
|
||||
On 2026-09-03 `k3s-prod-2` failed to return from the **first** scripted reboot of the survey below. Its
|
||||
journal ends cleanly mid-shutdown (services stopping, `eno1: NIC Link is Down`) and then stops. The box
|
||||
stayed **powered on but off the LAN for ~38 minutes** — no ping, no SSH, ARP INCOMPLETE — until the
|
||||
operator held the power button and cold-booted it by hand.
|
||||
|
||||
**This is arguably a worse prod defect than the memory bug.** A node that cannot survive an unattended
|
||||
warm reboot cannot be rolled in an HA cluster: every kernel update and maintenance window becomes a
|
||||
physical visit, and `roles/node_health` cannot see a machine that is off the network.
|
||||
|
||||
Leading candidate: node 2 is the **only** node with `Recover OS after Boot Failure = Enable`. A box
|
||||
parked at a firmware recovery prompt awaiting a keypress presents exactly this way. Unproven — it needs
|
||||
eyes on the screen during a hang — but it promotes that setting from "coincidence" in the BIOS diff to a
|
||||
live availability defect. **Worth disabling on all three nodes regardless.**
|
||||
|
||||
Consequence for testing: **the reboot survey is not viable unattended on node 2** until this is
|
||||
understood or out-of-band recovery exists.
|
||||
|
||||
### THERE IS NO KERNEL CMDLINE WORKAROUND
|
||||
|
||||
`efi_memattr_init()` is called from `efi_config_parse_tables()` under only
|
||||
@@ -157,6 +247,184 @@ Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.n
|
||||
table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook
|
||||
Studio G5**, 16 GiB → 1.5 GiB).
|
||||
|
||||
### Will 6.12 ever get the fix? — researched 2026-09-02. **Not on its own.**
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| In `linux-6.12.y` today? | **No.** Newest commit touching `memattr.c` on that branch is 2024-07-12. |
|
||||
| How long has it been available? | In mainline since **v6.13, 2025-01-19** — ~19 months and ~97 6.12.y point releases. |
|
||||
| Tagged `Cc: stable`? | **No.** Neither commit. |
|
||||
| Tagged `Fixes:`? | **No** — and this is structural, not an oversight. The unvalidated arithmetic has been there since the file was created in **2016**; it is not a regression, so there is no bad commit to point a `Fixes:` at. **Both automatic feeds — the `Fixes:`-scanning scripts and AUTOSEL — had nothing to grab.** |
|
||||
| Picked up by AUTOSEL? | **Never surfaced** for any branch. |
|
||||
| Has anyone asked? | **No.** Nothing on `[email protected]`, and nothing in all 8,027 open+archived Debian bugs against `src:linux`. We would be first. |
|
||||
| 6.12 LTS horizon | **EOL December 2028** — outlives Debian 13's own support window, so a backport is *not* time-barred. |
|
||||
|
||||
**Every automatic path has already had its chance and produced nothing.** The probability that 6.12.y
|
||||
acquires this commit spontaneously is close to zero.
|
||||
|
||||
**But the request is unusually strong, and the cost is one email.** The decisive card: the cherry-pick
|
||||
is **byte-clean**. Debian trixie 6.12.94-1's `drivers/firmware/efi/memattr.c` hashes to
|
||||
`164203429fa7a…`, **identical to the upstream patch's pre-image blob** — the file has not diverged at
|
||||
all between v6.12 and the commit's parent, so `git cherry-pick 8fbe4c49c0cc` applies with zero
|
||||
conflicts and zero adjustment. Neither the stable team nor Debian would have any backporting work to
|
||||
do, which removes the usual reason such requests are declined. The commit also meets every criterion
|
||||
in `stable-kernel-rules.rst`: in mainline, 17 insertions (limit 100 lines), fixes "a hardware quirk"
|
||||
and "a real bug that bothers people" — the enumerated categories.
|
||||
|
||||
**Debian rebases**, so an upstream pickup reaches us automatically: trixie carries `linux 6.12.94-1`
|
||||
in *trixie-security* whose changelog opens *"New upstream stable update: ChangeLog-6.12.91"*. No
|
||||
action beyond `apt upgrade`.
|
||||
|
||||
#### Plan
|
||||
|
||||
1. **Run `trixie-backports` on both prod nodes now**, as an explicitly *provisional* measure. Any
|
||||
6.13+ kernel carries the fix. Record the standing cost: backports packages are provided as-is,
|
||||
without the main archive's security guarantees.
|
||||
2. **Email `[email protected]`** — this is `stable-kernel-rules.rst` Option 2 for an
|
||||
already-merged patch. CC `[email protected]` and **Ard Biesheuvel `<[email protected]>`**,
|
||||
who both authored and committed the fix as EFI maintainer. Include: the commit ID and subject;
|
||||
**6.12.y only**; the byte-clean-apply proof above (it pre-empts the main objection); and the
|
||||
reproduction — three consecutive boots on one machine, two outcomes, **byte-identical
|
||||
`BIOS-e820` across all three** (the detail that proves the divergence is in the MAT, not the
|
||||
memory map), 16 GB → 2.6 GiB, plus a second identical machine taking the safe path only by
|
||||
chance. Attach both dmesgs. Corroboration exists:
|
||||
[Ubuntu LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (HP ZBook G5,
|
||||
16 GiB → 1.5 GiB) and SUSE bugzilla 1231465, cited in the commit message itself.
|
||||
3. **File a Debian BTS bug as secondary**, against `src:linux` via `reportbug` (the bug-script
|
||||
metadata is mandatory), severity `important`, referencing the lore thread. Debian's kernel
|
||||
handbook §9.1.7 allows cherry-picking a patch already accepted upstream — which this is,
|
||||
maintainer-authored with two Acked-bys — so if stable@ goes quiet Debian can act on its own.
|
||||
4. **Trigger to make the deviation permanent:** no stable@ pickup within ~2 release cycles (4–6
|
||||
weeks) and no Debian movement ⇒ encode the backports kernel in the provisioning IaC for every
|
||||
prod node and document it as a standing deviation from the distro baseline.
|
||||
|
||||
**If patching by hand, only `8fbe4c49c0cc` is needed.** The follow-up `5d0faa8e8369` loosens a
|
||||
heuristic that was rejecting *legitimate* tables (false kexec warnings in Meta's fleet); it is not
|
||||
required for this symptom and depends on the first commit anyway.
|
||||
|
||||
### Is it a BIOS setting? — three-way diff, researched 2026-09-03. **No.**
|
||||
|
||||
The operator's hypothesis, tested properly now that there are two working nodes and one broken one.
|
||||
All **240** attributes read on all three; identical key sets; identical firmware everywhere (same
|
||||
BIOS version string, ME 12.1.1.1022, GOP 9.0.1085, Super I/O 8.9.22). So every difference is settings
|
||||
or hardware, not firmware revision.
|
||||
|
||||
**The discriminating set — both working nodes agree, the broken node differs.** 26 attributes differ
|
||||
in total; after excluding per-machine noise (serials, UUIDs, MACs, asset tags, Born On Date, Feature
|
||||
Byte, SKU, memory size, storage device names) and settings falling on a different partition, seven
|
||||
remain:
|
||||
|
||||
| Attribute | nodes 1 & 3 (work) | node 2 (broken) | Verdict |
|
||||
|---|---|---|---|
|
||||
| **Fast Boot** | Disable | **Enable** | **plausible probability-shifter — the only live candidate** |
|
||||
| Legacy Boot Options | Disable | Enable | **coincidence — the setting is INERT, see below** |
|
||||
| Microsoft / Windows / MS Option ROM UEFI CA 2023 | Enable | Disable | coincidence (sub-page `db` size at most) |
|
||||
| TPM Activation Policy | F1 to Boot | Allow user to reject | coincidence |
|
||||
| Recover OS after Boot Failure | Disable | Enable | coincidence (but a policy problem in its own right) |
|
||||
|
||||
**`After Power Loss` is eliminated by the third node** — node 3 *works* and shares node 2's
|
||||
`Power Off`; node 1 is the odd one out. It was never mechanistically credible (PCH/ME power policy,
|
||||
applied before firmware executes). It remains a §1 checklist violation on **two** nodes now.
|
||||
|
||||
**`Legacy Boot Options` is affirmatively inert**, which a naive diff gets wrong. Three attributes are
|
||||
**identical on all three nodes** and settle it: `Configure Legacy Support and Secure Boot` =
|
||||
*"Legacy Support Disable and Secure Boot Enable"*, `Configure Option ROM Launch Policy` = *"All
|
||||
UEFI"*, `UEFI Boot Options` = Enable. `Legacy Boot Options` is only the boot-order checkbox; the
|
||||
master switch that actually instantiates a CSM has legacy support disabled on node 2 too. No CSM is
|
||||
built, no legacy option ROM is shadowed. The second-strongest candidate has **no boot-time
|
||||
allocation footprint at all**. Secure Boot is likewise not a differential — all three log
|
||||
`Secure boot enabled` and load the same HP DB cert.
|
||||
|
||||
**`Fast Boot` is the one credible mechanism**, and for a *different reason* than the hypothesis this
|
||||
document previously recorded and killed. HP documents it verbatim: *"reduces boot up time by
|
||||
bypassing boot to USB, CD-ROM, and PXE. **This skips some preboot initialization steps.**"* Skipped
|
||||
preboot init ⇒ fewer boot-services allocations before the MAT is installed ⇒ a different heap top ⇒
|
||||
**the MAT lands at a different address**. That is exactly the perturbation that flipped the
|
||||
near-identical Dell case in [LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382),
|
||||
where the reporter's "fix" was **unplugging a USB mouse**. (The old Fast-Boot-caches-MRC-training
|
||||
theory stays dead — byte-identical `BIOS-e820` refuted it.)
|
||||
|
||||
#### Verdict: the hypothesis is not supported, and this diff cannot support it even in principle
|
||||
|
||||
1. **No setting can be the cause.** Node 2 booted good once and bad twice with settings constant, and
|
||||
the MAT address moves every boot on all three machines.
|
||||
2. **`Fast Boot` may shift the odds.** That is the honest ceiling. There is currently zero evidence
|
||||
that it does.
|
||||
3. **A fatal confound:** the `1=3 ≠ 2` partition is *perfectly collinear* with two hardware
|
||||
differences that have at least as strong a mechanism — node 2 is the only **single-DIMM** node and
|
||||
the only **NVMe-booting** node. Nodes 1 and 3 are both dual-DIMM and both SATA. **Adding node 3
|
||||
did not break the confound; it doubled it.** A different boot device means a different initrd
|
||||
module set and size, hence a different EFI-stub allocation footprint — the very thing that decides
|
||||
whether the MAT page gets overwritten.
|
||||
|
||||
**No model-specific documentation exists.** HP publishes nothing mentioning the EFI Memory Attributes
|
||||
Table, MAT placement, or an NX/DEP option on this generation. The only memory-adjacent options
|
||||
(`Option ROM Launch Policy`, `DMA Protection`, `Pre-boot DMA protection`) are identical fleet-wide. No
|
||||
HP advisory matches. The R21 changelog never mentions memory, and these nodes run **02.26.00 dated
|
||||
05/05/2026 — newer than anything HP has indexed publicly**, so there is no BIOS update to chase.
|
||||
|
||||
#### The experiment, if it is ever worth running
|
||||
|
||||
Two statistics kill the obvious approach. Node 2's failure rate is known only to **n = 3** (2 bad of
|
||||
3); the 95% Clopper–Pearson interval on p(bad) is ≈ **0.09–0.99**. Against a null that loose, even
|
||||
8 clean boots after a change prove nothing. **A baseline arm is mandatory**, and: detecting
|
||||
*elimination* needs ~**20 boots per arm**; detecting a mere odds shift (0.67 → 0.33) needs ~**50 per
|
||||
arm** for 80% power. **Do not design for an odds shift — design for elimination and report
|
||||
"inconclusive" otherwise.**
|
||||
|
||||
**Zero-touch first step, and the one actually worth doing:** 20 scripted reboots of node 2 with
|
||||
*nothing changed*, logging `MEMATTR=` and the `Memory: …K/…K available` line per boot. That
|
||||
establishes the real p(bad) we lack, and tests the causal chain directly — **is the outcome predicted
|
||||
by where the MAT landed?** Present data hint yes: both failures at exactly `0xbeeb0018`, all five
|
||||
successes elsewhere. If the bad outcome maps onto a narrow address range, the mechanism is settled
|
||||
without touching anything.
|
||||
|
||||
**If more is wanted, swap hardware rather than BIOS settings** — it crosses settings against memory
|
||||
topology in one move, needs no firmware write (writes are refused; a BIOS change means a physical F10
|
||||
visit per arm), and is work §2 wants anyway: move node 2's single Samsung DIMM into node 3 and node
|
||||
3's matched pair into node 2, then 20 boots each. Fault follows the DIMM ⇒ topology, settings
|
||||
exonerated. Fault stays on node 2 ⇒ node 2 implicated, and only then is Fast Boot worth testing.
|
||||
Fault disappears ⇒ single-DIMM topology. ~1 hour unattended.
|
||||
|
||||
**None of this changes the action.** The fix is already known, is not in dispute, and applies to all
|
||||
three nodes: kernel ≥ 6.13.
|
||||
|
||||
### Reboot survey — 40 boots, run 2026-09-03
|
||||
|
||||
`playbooks/memattr-survey.yml` (+ `roles/memattr_survey`), reboots opt-in via
|
||||
`memattr_survey_reboots_authorised`, resumable from an on-node ledger, artifacts in
|
||||
`_artifacts/memattr-survey/`.
|
||||
|
||||
| node | bad | n | p(bad) | 95% Clopper–Pearson |
|
||||
|---|---:|---:|---:|---|
|
||||
| prod-1 | 0 | 20 | 0.00 | 0.000 – 0.168 |
|
||||
| **prod-2** | — | **0** | — | **no data — hung on its first reboot** |
|
||||
| prod-3 | 0 | 20 | 0.00 | 0.000 – 0.168 |
|
||||
| 1+3 pooled | 0 | 40 | 0.00 | 0.000 – 0.088 |
|
||||
|
||||
**The arm the experiment was designed around produced nothing.** Node 2's prior 2-of-3 is unchanged.
|
||||
And 0/20 is **not** elimination — it still permits p(bad) up to 17% per node.
|
||||
|
||||
What the 40 boots *did* establish:
|
||||
|
||||
- **Addresses are node-specific and highly concentrated.** Node 1 used only **3** distinct addresses
|
||||
across 20 boots (15 of them at `0xbeec9298`); node 3 used 8. **No address was ever shared between
|
||||
nodes.** This is a lottery with very few tickets per machine — which is exactly why 20 clean boots on
|
||||
nodes that only ever drew safe tickets says little about the node that drew a bad one.
|
||||
- **`MEMATTR` → kernel parse was a perfect function across all 7 addresses seen more than once**
|
||||
(37 boots), with both readings present in the sample. Same address, same parse, every time. Combined
|
||||
with the counter-example above, the picture is: the *address* determines what the kernel finds, and the
|
||||
*garbage value there* determines whether it is survivable.
|
||||
- **The corrupting bytes are stable per machine, not fleet-wide.** Node 1's garbage version was `1030` on
|
||||
**all 22** of its bail boots across three different addresses; node 3's was `1463964749` on all 12. So
|
||||
the earlier "shared OS boot chain" inference needs refining: the boot chain remains the plausible
|
||||
writer, but its footprint is machine-specific.
|
||||
|
||||
**Ansible defect worth knowing repo-wide:** a looped `include_tasks` under the default `linear` strategy
|
||||
is flattened per host, so hosts run **serially**. The first survey attempt rebooted node 1 twenty times
|
||||
while nodes 2 and 3 sat idle. Fixed with `strategy: free` plus a preflight assert that `forks` ≥ host
|
||||
count. It is silent and costs 3× wall clock.
|
||||
|
||||
### What NOT to do
|
||||
|
||||
- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is
|
||||
@@ -291,7 +559,7 @@ nothing ever transmits `new_password` to firmware. It is write-only into a buffe
|
||||
password in cleartext on the command line — it would land in process listings and shell history
|
||||
unless wrapped. Store it in the Ansible vault **and** offline.
|
||||
|
||||
## 2. Disk layout — **OPEN DECISION, blocks the build**
|
||||
## 2. Disk layout — **DECIDED (operator, 2026-09-03) · REVISED (operator, 2026-09-04)**
|
||||
|
||||
`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
|
||||
further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
|
||||
@@ -308,6 +576,218 @@ sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
|
||||
└─sda5 433G /srv <- the only large free space
|
||||
```
|
||||
|
||||
### THE DECISION (operator, 2026-09-03; layout REVISED 2026-09-04) — made, applied, and measured-not-assumed
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Data partition** | the **existing `/srv` partition** on each node's single physical disk (~432 GB nodes 1–2, ~422 GB node 3), re-purposed, re-labelled **`k3s-data`** and mounted at the neutral **`/data`** |
|
||||
| **Longhorn store** | **`/data/longhorn`** — a *subdirectory* of that partition |
|
||||
| **k3s `--data-dir`** | **`/data/k3s`** — the *sibling* subdirectory. Holds **embedded etcd AND the containerd image store** |
|
||||
| **etcd** | **no dedicated device** — it lives at `/data/k3s/server/db`, i.e. on the data partition, **not** on the OS filesystem |
|
||||
| **Shared-device I/O contention** | **ACCEPTED, knowingly** — unchanged by the 2026-09-04 revision |
|
||||
|
||||
#### THE 2026-09-04 REVISION, AND THE CAPACITY EVIDENCE THAT FORCED IT
|
||||
|
||||
The 2026-09-03 ruling gave this partition to Longhorn **only** and left k3s at its default
|
||||
`/var/lib/rancher/k3s`. That was wrong, for a reason that is arithmetic rather than judgement.
|
||||
|
||||
**`/var` on these nodes is a separate partition of 11.8 GiB** (11.6 on node 3) — see the
|
||||
partition map above; it is `/dev/sda3` / `/dev/nvme0n1p3`, not the 30 GiB `/`. k3s's data dir
|
||||
puts **two** things on it: etcd's database and WAL (`server/db`) and, far larger, the
|
||||
**containerd image store** (`agent/containerd`).
|
||||
|
||||
**Measured on the fully-deployed dev cluster** — the same platform and app set prod will run:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `agent/containerd` per node | **21–25 GB** |
|
||||
| total `/var` per node | **58–70 GB** |
|
||||
| prod `/var` capacity | **11.8 GiB** |
|
||||
|
||||
So prod would have **filled `/var` during its first image pulls** — not eventually, not under
|
||||
load, but on the way up. And because etcd was on that same partition, a full `/var` means
|
||||
**etcd cannot write its WAL**. That is cluster death, not degradation: a wedged etcd on all
|
||||
three servers is an unavailable control plane, and it would have happened during bring-up.
|
||||
|
||||
**The fix is to move k3s's whole data dir onto the 424 GB partition**, which is why that
|
||||
partition's mountpoint and filesystem label are no longer Longhorn-specific — it now serves
|
||||
**two tenants**. `/var` retains only the OS's own usage (503 MB, 5% used, measured 2026-09-04).
|
||||
|
||||
**Why now.** Nothing was installed on prod when this was decided — verified 2026-09-04: no
|
||||
`/usr/local/bin/k3s`, no `/etc/rancher/k3s`, no `/var/lib/rancher` on any node. Moving a data
|
||||
dir before first install is a one-line inventory change; after bring-up it means draining and
|
||||
re-imaging three servers.
|
||||
|
||||
**Why `/data` and `LABEL=k3s-data`.**
|
||||
|
||||
- **`/data`** — short, neutral, tenant-agnostic, and *not* under `/var`. Deliberately **not
|
||||
`/srv`**: `/srv` is the *claim mountpoint* the role discovers the device at, and the
|
||||
idempotency contract is that `/srv` is gone from `/proc/mounts` and `/etc/fstab` once the
|
||||
claim has happened. Re-using it as the destination would make "claimed" and "not yet
|
||||
claimed" indistinguishable.
|
||||
- **`LABEL=k3s-data`** — deliberately **not** the bare `data`. Guard #4 of the role's assert
|
||||
wall accepts a partition whose label is empty *or already ours*, and names `data` as an
|
||||
example of somebody else's named volume; labelling ours `data` would make a stranger's
|
||||
`data` volume look already-claimed and weaken that guard. `k3s-data` is cluster-scoped
|
||||
rather than tenant-scoped, and matches the `k3s-etcd` label convention already in the repo.
|
||||
|
||||
**The shared-device I/O risk is UNCHANGED and still accepted.** Longhorn and etcd were already
|
||||
going to share this spindle under the 2026-09-03 ruling — the revision changes *which*
|
||||
filesystem etcd sits on, not *which device*. The acceptance rationale and its exit condition
|
||||
below stand verbatim.
|
||||
|
||||
**How prod's data dir is scoped away from dev.** `k3s_data_dir` is a **single** variable read
|
||||
by **two** roles: `roles/k3s_server` turns it into `--data-dir` in the install flags, and
|
||||
`roles/node_baseline` (via `node_baseline_k3s_data_dir`, which defers to it) creates the
|
||||
directory `0700` on the data partition and derives etcd's location from it
|
||||
(`node_baseline_etcd_mount` = `<data-dir>/server/db`). The role default is **k3s's own
|
||||
default**, `/var/lib/rancher/k3s`, so **dev is unchanged** — and dev also has a *dedicated
|
||||
etcd device* mounted at `server/db` under that path, which moving dev's data dir would orphan.
|
||||
Prod overrides it in one line in `inventory/group_vars/k3s_prod/vars.yml`, the same seam
|
||||
`kube_vip_address` / `k3s_api_host` / `node_baseline_backports_kernel` use, so a fourth prod
|
||||
node inherits it from group membership rather than from anyone remembering.
|
||||
|
||||
**Longhorn's `defaultDataPath` follows the R-ENV-02 knob pattern, not a base edit.**
|
||||
`platform/longhorn/helmrelease.yaml` referenced a hardcoded `/var/lib/longhorn`; it now
|
||||
references `${LONGHORN_DATA_PATH}`, a `cluster-vars` knob with a base default and an explicit
|
||||
value in **both** overlays:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `clusters/base/cluster-vars.yaml` | `/var/lib/longhorn` (base default = the known-good dev value, same convention as `GATEWAY_VIP`) |
|
||||
| `clusters/dev` | `/var/lib/longhorn` — **must never change**; the live dev cluster has 18 volumes whose replicas are on disk under that path |
|
||||
| `clusters/prod` | `/data/longhorn` |
|
||||
|
||||
`kustomize-controller` runs with `--feature-gates=StrictPostBuildSubstitutions=true`, so an
|
||||
undefined variable fails the Kustomization outright — hence base-first, then both overlays.
|
||||
`[parity]`: the prod value MUST equal `node_baseline_longhorn_data_path` in
|
||||
`group_vars/k3s_prod/vars.yml`. Ansible creates and mounts the directory; Helm tells Longhorn
|
||||
to use it. If the two disagree, Longhorn writes replicas onto the OS filesystem and `/var`
|
||||
fills up — the exact failure this revision exists to prevent.
|
||||
|
||||
**Verified 2026-09-04 that dev's rendered output is byte-identical.** All 12 Flux
|
||||
Kustomizations that declare `postBuild.substituteFrom cluster-vars` were rendered
|
||||
(`kubectl kustomize <spec.path> | flux envsubst --strict`, substituting the dev overlay's
|
||||
*effective* cluster-vars) both before and after the change: 12/12 identical per-file SHA-256,
|
||||
and one aggregate SHA-256 over every rendered byte, `b34c5d3d…89e7d`, unchanged. The only
|
||||
dev-side difference anywhere is the presence of the new knob in the ConfigMap itself, carrying
|
||||
dev's existing literal. The same harness shows prod's render changing on exactly one line
|
||||
(`defaultDataPath: /var/lib/longhorn` → `/data/longhorn`), which is what proves the comparison
|
||||
could have detected a regression.
|
||||
|
||||
**Per-node device paths — they are NOT the same, and that is why nothing hardcodes them.**
|
||||
Verified 2026-09-03 with `findmnt --mountpoint /srv` on each node:
|
||||
|
||||
| Node | Boot media | `/srv` partition | Size | Contents before the claim |
|
||||
|---|---|---|---|---|
|
||||
| `k3s-prod-1` | SATA SSD `C400-MTFDDAC512M` | `/dev/sda5` | 432.5 GiB ext4 | `lost+found` only (`du -sh` 20K; `df` 2.1 M of 425 G) |
|
||||
| `k3s-prod-2` | **NVMe** `PM9A1 Samsung` | **`/dev/nvme0n1p5`** | 432.5 GiB ext4 | `lost+found` only (same) |
|
||||
| `k3s-prod-3` | SATA SSD `WDC WDS500G2B0A` | `/dev/sda5` | 422.2 GiB ext4 | `lost+found` only (same) |
|
||||
|
||||
The `df` "5.1% used" that made `/srv` look occupied is **ext4's default 5% root-reserved
|
||||
blocks**, not data — confirmed per node by `ls -A /srv` (only `lost+found`) and
|
||||
`du -sh --exclude=lost+found /srv` (20 K). Nothing was destroyed.
|
||||
|
||||
**How the role resolves the device — discovery, not per-host inventory.** A single
|
||||
`k3s_prod` group value cannot be right when node 2 is NVMe, and per-host `host_vars` would
|
||||
freeze a *fact about today's hardware* into inventory, where it goes stale the first time a
|
||||
node is re-imaged or a disk is swapped — and the stale value's failure mode is `mkfs` on the
|
||||
wrong partition. So `group_vars/k3s_prod/vars.yml` declares **intent**
|
||||
(`node_baseline_longhorn_source: claim-mountpoint`,
|
||||
`node_baseline_longhorn_claim_mountpoint: /srv`) and
|
||||
`roles/node_baseline/tasks/disks.yml` asks the **node** which block device is mounted there,
|
||||
then proves the answer is safe before writing anything. Six independent asserts, all
|
||||
fail-closed: it is a partition or whole disk (not tmpfs/NFS/bind), it is **not** backing
|
||||
`/`, `/boot`, `/boot/efi`, `/usr`, `/var`, `/home`, `/tmp` or active swap (that set is built
|
||||
from the node's *own* gathered facts plus `swapon`, never a hardcoded partition list), it is
|
||||
already ext4, it carries no other filesystem label, it is ≥ 100 GiB, and it is **empty**.
|
||||
|
||||
**Unmount → relabel, not reformat.** `/srv` is already formatted and already mounted, and
|
||||
`community.general.filesystem` will not touch a device that has a filesystem — it would
|
||||
silently no-op and then the `LABEL=longhorn` mount would fail. The role therefore unmounts
|
||||
`/srv`, removes its `/etc/fstab` entry (installer-written, pinned by UUID — leaving it would
|
||||
re-mount the partition at `/srv` underneath Longhorn on the next boot), and stamps the label
|
||||
with `e2label`. Relabelling rather than reformatting is deliberate: the partition is already
|
||||
ext4 and already verified empty, so `mkfs` buys nothing, and if some assert were ever too
|
||||
permissive the worst case is a renamed filesystem instead of a destroyed one.
|
||||
|
||||
**Idempotency guard.** A second run is a no-op, not a reformat. After the claim the
|
||||
partition carries the ext4 label `k3s-data` and `/srv` is gone from `/proc/mounts` *and*
|
||||
`/etc/fstab` — so "what is mounted at `/srv`" would find nothing on run 2. The role pivots
|
||||
on the durable on-disk evidence instead: `blkid -L k3s-data` returning a device means the
|
||||
node is already claimed, and **every** destructive task is gated on that returning nothing.
|
||||
|
||||
**The 2026-09-04 transition, handled idempotently.** All three nodes had *already* been
|
||||
claimed under the previous naming (`LABEL=longhorn` at `/var/lib/longhorn`). Because the
|
||||
idempotency pivot asks "does the **current** label exist?", changing the label makes an
|
||||
already-claimed node answer *no* — and it would then walk into the greenfield claim path and
|
||||
fail closed at "the claim mountpoint must actually be a mountpoint", `/srv` having been
|
||||
consumed by that first claim. Fail-closed is correct, but it is not convergence, so the
|
||||
transition is **declared** rather than inferred:
|
||||
`node_baseline_data_retired_fs_label: longhorn` + `node_baseline_data_retired_mount:
|
||||
/var/lib/longhorn` in `group_vars/k3s_prod/vars.yml`. The role then unmounts the retired
|
||||
mountpoint, drops its `fstab` entry and re-stamps the label with `e2label` — a **metadata
|
||||
write, never a reformat**, exactly as in the claim path — having first proved that the retired
|
||||
mountpoint is still a mount, that it is backed by the very device the retired label names, and
|
||||
that it is **empty** so nothing is orphaned. The common `LABEL=`-based mount task then brings
|
||||
it up at `/data`, and the two per-tenant subdirectories are created (`0755` Longhorn, `0700`
|
||||
k3s, since the latter holds the etcd DB, the node token and the image store).
|
||||
|
||||
The transition block is **one-time and self-retiring**: on every subsequent run
|
||||
`blkid -L k3s-data` succeeds and none of it is reachable. The two retired-* lines are safe to
|
||||
delete from `group_vars` once the fleet is converged (it is, as of 2026-09-04).
|
||||
|
||||
**Three independent reasons the LIVE dev cluster cannot enter that transition** — it has 18
|
||||
Longhorn volumes on `/var/lib/longhorn` and unmounting it would orphan every replica:
|
||||
(1) both retired-* vars default to `""` and are set only in the prod-scoped `group_vars`;
|
||||
(2) the path is gated on `longhorn_source == 'claim-mountpoint'`, and dev is `device` — the
|
||||
same structural gate that keeps dev out of the claim path; (3) the emptiness assert, which
|
||||
fails closed on a partition with replica directories even if (1) and (2) were both defeated.
|
||||
Dev's mountpoint, label and k3s data dir are also simply **not changing**.
|
||||
|
||||
**Run evidence (2026-09-04).** Full `node-baseline.yml` against `k3s_prod`: run 1 converged
|
||||
(5–6 changed per node), run 2 **`changed=0` on all three nodes** with every transition and
|
||||
claim task skipping. Per node afterwards: `blkid -L k3s-data` → `/dev/sda5` (nodes 1, 3) /
|
||||
`/dev/nvme0n1p5` (node 2); `blkid -L longhorn` → gone; `/data` mounted with 403 G / 394 G
|
||||
available; `/var/lib/longhorn` and `/srv` no longer mountpoints; `fstab` carrying exactly
|
||||
`LABEL=k3s-data /data ext4 defaults,nofail,x-systemd.device-timeout=5s 0 0`; and `/var` down
|
||||
to **5% used, ~11 GiB free**.
|
||||
|
||||
**Dev can never reach this path.** Both role defaults stay `source: device`, so the LIVE
|
||||
dev cluster keeps its dedicated `/dev/vdb` (Longhorn) and `/dev/vdc` (etcd) and the claim
|
||||
tasks are structurally unreachable there — they are gated on `claim-mountpoint`, which only
|
||||
the prod-scoped `group_vars` sets.
|
||||
|
||||
### The accepted risk, and its exit condition
|
||||
|
||||
Dev was burned by exactly this shape: a Longhorn rebuild starved etcd WAL fsync
|
||||
(`docs/plans/ssd-etcd-isolation.md`), which is why dev has a third, dedicated etcd device.
|
||||
**Prod accepts the same shared-device contention** because the failure mode is materially
|
||||
cheaper here: dev's contention was on a **spinning HDD** with seek-bound queueing; every
|
||||
prod node is **SSD or NVMe** end-to-end, where concurrent rebuild I/O costs latency rather
|
||||
than collapsing it. The three nodes also have no second device to give etcd — one physical
|
||||
disk each, already fully partitioned, no free slot verified populated.
|
||||
|
||||
**Accepted to be MEASURED, not assumed.** Exit condition, in order of preference:
|
||||
|
||||
- **Measure it before it matters.** Under a deliberate Longhorn replica rebuild, watch
|
||||
`etcd_disk_wal_fsync_duration_seconds` p99 and `etcd_disk_backend_commit_duration_seconds`
|
||||
p99, plus `etcd_server_leader_changes_seen_total`. If fsync p99 stays inside etcd's
|
||||
practical budget (~10 ms wal fsync / ~25 ms backend commit) with no leader churn, the
|
||||
acceptance holds and this becomes a recorded measurement rather than an assumption.
|
||||
- **If it does not hold:** add an M.2 NVMe per node (chassis slot to be verified
|
||||
physically) and flip `node_baseline_etcd_source` back to `device` — the role still
|
||||
supports that path unchanged, and it is the dev shape.
|
||||
|
||||
**~~Known, separate consequence — `/var` is only 12 GiB.~~ RESOLVED 2026-09-04 by moving
|
||||
k3s's data dir.** This was recorded here on 2026-09-03 as a capacity item "to watch or
|
||||
repartition". Re-measuring it against dev turned it from a watch item into a blocker: the
|
||||
container image store alone is **21–25 GB per node** on the fully-deployed dev cluster and
|
||||
`/var` totals **58–70 GB**, against prod's **11.8 GiB**. So prod would have filled `/var`
|
||||
during its first image pulls, and with etcd on the same partition a full `/var` means etcd
|
||||
cannot write its WAL. The fix was neither "watch" nor "repartition" but `--data-dir` — see
|
||||
"THE 2026-09-04 REVISION" above. `/var` now carries only the OS's own usage (503 MB, 5%).
|
||||
|
||||
Consequences to settle before imaging nodes 2 and 3:
|
||||
|
||||
- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
|
||||
@@ -341,24 +821,43 @@ Consequences to settle before imaging nodes 2 and 3:
|
||||
a member corrupts — this drive has already logged 32 of them.
|
||||
(Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
|
||||
`DECISIONS-LOG.md`. It is not part of this verdict.)
|
||||
- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
|
||||
(`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
|
||||
Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
|
||||
prod must override them in `group_vars`. On a single-disk node there is no second device at
|
||||
all, so one of:
|
||||
- **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
|
||||
SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
|
||||
spindle.
|
||||
- **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
|
||||
Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
|
||||
- **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
|
||||
physical failure domain.
|
||||
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
|
||||
- [ ] Record the chosen device paths here and set them in
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
|
||||
`/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
|
||||
`node-baseline` run against a node still carrying those placeholders now stops on an explicit
|
||||
assert naming this section, instead of failing obscurely at `mkfs` time.
|
||||
- [x] **Layout DECIDED and APPLIED 2026-09-03** — see "THE DECISION" above. `roles/node_baseline`
|
||||
no longer expects a block-device path for either store; it takes a declared **source
|
||||
enum** per environment (`roles/node_baseline/tasks/disks.yml`):
|
||||
- **Longhorn** — `node_baseline_longhorn_source: claim-mountpoint` +
|
||||
`node_baseline_longhorn_claim_mountpoint: /srv`. The `/dev/vdb` default (a virtio
|
||||
name that does not exist on bare metal) is unused on prod; the real partition is
|
||||
discovered on the node and guarded, because it is `/dev/sda5` on nodes 1 and 3 and
|
||||
`/dev/nvme0n1p5` on node 2.
|
||||
- **etcd** — `node_baseline_etcd_source: shared-data-partition` (was `root-filesystem`
|
||||
until 2026-09-04). "No dedicated device" is a first-class, explicitly-configured state
|
||||
that the `disks` assert **accepts**, not a blank variable that the tasks happen to
|
||||
skip — and the *name* of that state now describes reality, because etcd is no longer
|
||||
on the OS filesystem. The distinction is load-bearing, not cosmetic:
|
||||
`shared-data-partition` is **asserted**, per node on every run, by checking that the
|
||||
filesystem backing the etcd data dir really is the data partition's mount. Declaring
|
||||
`k3s_data_dir: /data/k3s` does not by itself achieve that — had the mount silently
|
||||
failed, `/data` would be a plain directory on `/` and etcd plus 21–25 GB of images
|
||||
would land back on the OS filesystem, invisibly, until the disk filled. Leaving the
|
||||
value at `root-filesystem` would have been a lie inventory told about the cluster, and
|
||||
`root-filesystem` makes no such claim and gets no such assert. The role also still
|
||||
asserts nothing is already mounted at the etcd data dir (so a half-migrated node
|
||||
cannot be inherited silently) and reports which filesystem etcd actually lands on
|
||||
(measured 2026-09-04: `/data`, 403 G / 394 G available, vs `/var`'s 10.7 GiB).
|
||||
- The two rejected options stand rejected: **add a second drive** (no verified free slot;
|
||||
the exit condition above reopens it if measurement fails) and **repartition** (same
|
||||
single failure domain, plus a destructive re-image of three working nodes for no
|
||||
isolation the SSDs do not already give).
|
||||
- [x] **All three nodes get the identical layout.** Same declaration, same mountpoint, same
|
||||
resulting `/data` + `/data/longhorn` + `/data/k3s`. Not the same *device path* and not
|
||||
the same *size* (432.5 / 432.5 / 422.2 GiB) — Longhorn tolerates unequal node capacity;
|
||||
it is replica *placement* that assumes a symmetric layout, and the layout is symmetric.
|
||||
Confirmed per node 2026-09-04 by `findmnt`/`lsblk`/`blkid` after the transition.
|
||||
- [x] **Chosen paths recorded here and set in `group_vars`.** The BLOCKED placeholders in
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` are replaced — by the source
|
||||
enums above, plus deliberately-unusable `/dev/NOT-A-DEVICE-…` values for the two device
|
||||
variables so that flipping a source back to `device` without supplying a real path
|
||||
still fails on the role's assert instead of silently inheriting dev's `/dev/vdb`.
|
||||
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
|
||||
is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
|
||||
tasks this decision blocks — see §5's drive-health box for the exact command.
|
||||
@@ -371,6 +870,138 @@ Consequences to settle before imaging nodes 2 and 3:
|
||||
- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
|
||||
`roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
|
||||
so the same role covers bare metal. Nothing to port.
|
||||
- [x] **KERNEL: a standing, deliberate deviation from the distro baseline — `linux-image-amd64`
|
||||
from `trixie-backports` (DONE on all three nodes, 2026-09-04).**
|
||||
|
||||
**What.** `roles/node_baseline` gained a `kernel`-tagged block that enables
|
||||
`trixie-backports` as a deb822 apt source and installs the `linux-image-amd64`
|
||||
**metapackage** from it. Applied 2026-09-04; all three nodes landed on
|
||||
**`linux-image-amd64` 7.1.8-1~bpo13+1 → kernel `7.1.8+deb13-amd64`**, rebooted one at a
|
||||
time, and are running it.
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
|
||||
```
|
||||
|
||||
**Why.** §0b. HP's R21 firmware publishes a corrupt EFI Memory Attributes Table; Linux
|
||||
< 6.13 reserves memory from it without validating the size, and `k3s-prod-2` has booted
|
||||
with 2.6 GiB of its 16 GB, twice. The fix is upstream commit **`8fbe4c49c0cc`**
|
||||
*"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, first
|
||||
released in **v6.13**, hardened by `5d0faa8e8369`). It is absent from v6.12 and from
|
||||
`linux-6.12.y`, carries neither `Cc: stable` nor `Fixes:`, and there is **no kernel
|
||||
cmdline workaround**. A k3s **server** that intermittently comes up with 2.6 GiB will
|
||||
fail under etcd plus its workload share on any reboot, so this is a **prerequisite to
|
||||
installing k3s**, not a follow-up.
|
||||
|
||||
**How it is scoped.** `node_baseline_backports_kernel` defaults to **false** in
|
||||
`roles/node_baseline/defaults/main.yml` and is set **true** exactly once, in
|
||||
`inventory/group_vars/k3s_prod/vars.yml` — the same seam `kube_vip_address` /
|
||||
`k3s_api_host` / `cluster_kubeconfig_name` use. Dev's Proxmox VMs publish no MAT and are
|
||||
genuinely unaffected; they must not be dragged onto a backports kernel as a side effect
|
||||
of sharing the role, and a fourth prod node must inherit the deviation from **group
|
||||
membership**, not from a `k3s-prod-*` hostname test. Verified: a `--tags kernel` run
|
||||
against dev skips all 8 tasks on all 3 hosts.
|
||||
|
||||
**Security cost, accepted.** `*-backports` packages are provided **as-is** and are
|
||||
**not** covered by the Debian Security Team's guarantees for the main archive — no
|
||||
security-tracker commitment, no DSAs. Recorded here as the price of the nodes keeping
|
||||
their RAM.
|
||||
|
||||
**No apt pin, deliberately.** The backports `Release` file carries `NotAutomatic: yes`
|
||||
+ `ButAutomaticUpgrades: yes`, which apt already reads as *"priority 100 so nothing is
|
||||
taken from backports by accident, 500 for a package explicitly installed from there so
|
||||
it keeps getting updates."* Writing an explicit `Pin-Priority: 100` in
|
||||
`preferences.d` would look like belt-and-braces and would in fact **switch off the
|
||||
automatic security upgrades for the very kernel this deviation installs.** Enabling the
|
||||
source is the whole configuration.
|
||||
|
||||
**Metapackage, not a pinned ABI.** A frozen `linux-image-6.19.14+deb13-amd64` would stop
|
||||
receiving even backports' own updates the moment the ABI rolls. The metapackage tracks
|
||||
whatever backports ships (>= 6.13 by construction). The **exact** version each node
|
||||
landed on is recorded on the node itself at `/etc/node-baseline-backports-kernel`,
|
||||
written by the same role, along with the rollback path below.
|
||||
|
||||
**Rollback.**
|
||||
1. *Non-destructive, try first.* The stock kernels are still installed
|
||||
(`6.12.107+deb13-amd64` and `6.12.94+deb13-amd64` on all three) and selectable from
|
||||
GRUB → *Advanced options for Debian GNU/Linux*. That answers "is the new kernel the
|
||||
problem?" without uninstalling anything — it also reinstates the memattr bug for that
|
||||
boot.
|
||||
2. *Permanent.* Set `node_baseline_backports_kernel: false` in
|
||||
`group_vars/k3s_prod/vars.yml` **first** (otherwise the next Ansible run puts it
|
||||
back), then on each node:
|
||||
```bash
|
||||
apt-get install --reinstall linux-image-amd64/trixie
|
||||
apt-get purge linux-image-7.1.8+deb13-amd64
|
||||
rm /etc/apt/sources.list.d/trixie-backports.sources
|
||||
apt-get update && update-grub && reboot
|
||||
```
|
||||
|
||||
**The role does NOT reboot**, and that is deliberate — see §0b "SECOND FAULT". Reboots
|
||||
are a serialized, supervised operator step: one node at a time, verify each returns
|
||||
before touching the next, never node 2 first. On 2026-09-04 all three returned in
|
||||
**39–41 s** (node 1 39 s, node 3 41 s, node 2 39 s); node 2 did **not** reproduce its
|
||||
2026-09-03 hang, but one clean warm reboot is not a fix and that defect stays open.
|
||||
|
||||
**Exit condition — how this deviation ends.** If `8fbe4c49c0cc` is ever picked up into
|
||||
`linux-6.12.y` (and therefore into Debian's 6.12 rebase, which reaches us by plain
|
||||
`apt upgrade`), this can be removed entirely: set `node_baseline_backports_kernel:
|
||||
false`, roll back per the note on each node, and delete the `kernel` block from
|
||||
`roles/node_baseline`. §0b records why that pickup will not happen on its own and what
|
||||
request would cause it.
|
||||
- [x] **Verification of the fix, and a correction to how it is verified (2026-09-04).**
|
||||
|
||||
§0b says *"A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table
|
||||
detected!`"*. That is true but **conditional, and it did not happen on any of the three
|
||||
nodes.** `efi_memattr_init()` on >= 6.13 has **three** gates and only the third prints
|
||||
that line:
|
||||
|
||||
| gate | message | reachable when |
|
||||
|---|---|---|
|
||||
| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
|
||||
| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, `desc_size` does not |
|
||||
| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* `desc_size` read plausibly, entry count does not |
|
||||
|
||||
**All three are rejections**, and which one a boot prints is decided by *which field*
|
||||
that boot's garbage corrupts. On 7.1.8 all three nodes hit the **first** gate:
|
||||
|
||||
| node | `MEMATTR=` | kernel line on 7.1.8 | MemTotal | reserved at boot |
|
||||
|---|---|---|---|---|
|
||||
| 1 | `0xbee84018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1030` | 31.1 GiB | 0.9 GiB |
|
||||
| 2 | `0xbeeb0018` | `efi: memattr: Unexpected EFI Memory Attributes table version -1569980416` | 15.4 GiB | 0.6 GiB |
|
||||
| 3 | `0xbeecc018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` | 31.1 GiB | 0.9 GiB |
|
||||
|
||||
Node 2 is the result that matters: **`0xbeeb0018` is the exact address that produced
|
||||
2.6 GiB twice**, and on 7.1.8 it reserved 634,904 K instead of 13,913,856 K.
|
||||
|
||||
**Boot-independent proof the fix is compiled in**, since the log line is a lottery: the
|
||||
running kernel image itself contains the string. Decompressing `/boot/vmlinuz-$(uname
|
||||
-r)` on all three nodes yields, verbatim —
|
||||
|
||||
```
|
||||
efi: memattr: [Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)
|
||||
efi: memattr: Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)
|
||||
```
|
||||
|
||||
— neither of which exists in 6.12. That is the positive proof; the `dmesg` line is not
|
||||
available on demand.
|
||||
|
||||
**Two bugs in `roles/node_health` were fixed as a consequence**, because without them
|
||||
the health check could not show the fix landing:
|
||||
- `MEMATTR_VERSION_RE` in `files/node_health_collect.py` matched `(\d+)` only. The
|
||||
kernel prints that field with `%d` — **signed** — so node 2's `-1569980416` did not
|
||||
match, the state fell through to `clean`, and the node was reported **PASS "the kernel
|
||||
accepted it without complaint"**: a false pass on the exact bug the check exists to
|
||||
find. Now `(-?\d+)`.
|
||||
- `firmware.efi_memattr` treated `bad_version` as WARN unconditionally, so a node on a
|
||||
fixed kernel would have stayed WARN forever. The verdict now keys on the running
|
||||
kernel version (`memattr_kernel_fixed`, `>= 6.13`): the same log line is WARN on
|
||||
< 6.13 (a lucky boot) and PASS on >= 6.13 (the validation working). The pre-existing
|
||||
`corrupt_rejected` PASS branch is unchanged and remains reachable.
|
||||
|
||||
**`playbooks/node-health.yml -e node_health_hosts=k3s_prod`, after the reboots:
|
||||
`firmware.efi_memattr` = PASS on all three, `memory.physical_total` = PASS on all
|
||||
three.** Dev re-run unchanged.
|
||||
|
||||
## 4. Network (per node)
|
||||
|
||||
|
||||
@@ -40,6 +40,15 @@ ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOP
|
||||
> ```
|
||||
> (Both take the same target-group override as the rest — see **Targeting prod** below.)
|
||||
|
||||
> **Standalone diagnostics** (not part of any bring-up; safe to point anywhere):
|
||||
> ```
|
||||
> ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod # read-only hardware health
|
||||
> ansible-playbook playbooks/memattr-survey.yml # EFI memattr reboot survey
|
||||
> ```
|
||||
> `memattr-survey.yml` is the only play here that **reboots** its targets, and it will not do so
|
||||
> until `-e memattr_survey_reboots_authorised=true` is passed. See
|
||||
> `roles/memattr_survey/README.md` and `infra/PROD-PROVISIONING.md` §0b.
|
||||
|
||||
## Re-provisioning a node (tofu -replace / fresh OS)
|
||||
A node rebuild wipes **everything on the host filesystem** — including host-level config
|
||||
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
|
||||
@@ -109,6 +118,10 @@ command plus an override. Two separate mechanisms, for two separate reasons:
|
||||
`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
|
||||
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
|
||||
bring-up chain.
|
||||
`playbooks/memattr-survey.yml` follows it (`-e memattr_survey_hosts=…`, defaulting to `k3s_prod`)
|
||||
for the same reason, and adds a second interlock on top: because it **reboots** its targets, the
|
||||
role also refuses any host that is not a member of `memattr_survey_required_group`. A host pattern
|
||||
can be mistyped; inventory membership cannot.
|
||||
|
||||
```bash
|
||||
# dev — completely unchanged, no extra vars
|
||||
|
||||
@@ -87,24 +87,155 @@ kube_vip_address: "192.168.2.9"
|
||||
# — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
|
||||
cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
|
||||
|
||||
# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
|
||||
# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
|
||||
# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
|
||||
# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
|
||||
# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
|
||||
# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
|
||||
# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
|
||||
# --- node_baseline backports kernel — DECIDED, and a PREREQUISITE (§0b/§3) --------
|
||||
# THE ONE PLACE THE §0b DEVIATION IS TURNED ON. The role default is false (dev's Proxmox
|
||||
# VMs publish no EFI Memory Attributes Table at all and must NOT be dragged onto a
|
||||
# backports kernel as a side effect of sharing the role), and this line switches it on for
|
||||
# the whole [k3s_prod] group — the same seam kube_vip_address / k3s_api_host /
|
||||
# cluster_kubeconfig_name use, so a FOURTH prod node inherits it from group membership
|
||||
# rather than from anyone remembering.
|
||||
#
|
||||
# The overrides below are deliberately present-but-broken: leaving them UNSET would
|
||||
# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
|
||||
# hardware — a confusing failure that looks like a wiring bug. This value fails just
|
||||
# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
|
||||
# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
|
||||
# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
|
||||
# updates these two lines. All three nodes must end up with an identical layout
|
||||
# (Longhorn replica placement assumes symmetry) before any of this is used for real.
|
||||
node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
|
||||
node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
|
||||
# WHY IT IS A PREREQUISITE AND NOT A NICE-TO-HAVE. HP's R21 firmware publishes a CORRUPT
|
||||
# EFI Memory Attributes Table and Linux < 6.13 reserves memory from it without validating
|
||||
# the size. k3s-prod-2 has booted with 2.6 GiB of its 16 GB, twice. A k3s SERVER that
|
||||
# intermittently comes up with 2.6 GiB is not a degraded node — it is a node that will fail
|
||||
# under etcd plus its workload share, unpredictably, on any reboot. All three nodes draw
|
||||
# from the same lottery on every boot; nodes 1 and 3 have only drawn safe tickets so far.
|
||||
# So this must be true BEFORE k3s is installed, not after. Fixed upstream by 8fbe4c49c0cc,
|
||||
# first released in v6.13; absent from v6.12 and linux-6.12.y, and there is no kernel
|
||||
# cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b, deviation recorded
|
||||
# in §3.
|
||||
#
|
||||
# COST, ACCEPTED KNOWINGLY: packages from trixie-backports carry NO Debian Security Team
|
||||
# guarantee — no security-tracker commitment, no DSAs. The exchange is that the nodes keep
|
||||
# the RAM they have. EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y, set this
|
||||
# back to false, roll back per the note the role writes to
|
||||
# /etc/node-baseline-backports-kernel on each node, and delete the `kernel` block from
|
||||
# roles/node_baseline.
|
||||
#
|
||||
# The role does NOT reboot. Reboots are serialized and supervised by hand (§0b "SECOND
|
||||
# FAULT": k3s-prod-2 hung on a warm reboot, powered but off the LAN for ~38 minutes).
|
||||
node_baseline_backports_kernel: true
|
||||
|
||||
# --- node_baseline data-disk layout — DECIDED (operator, 2026-09-03; REVISED 2026-09-04; §2)
|
||||
# Was BLOCKED. The operator has ruled, and §2 records the ruling and its rationale:
|
||||
#
|
||||
# DATA PARTITION -> the existing ~424 GB `/srv` partition on each node's single physical
|
||||
# disk, claimed and mounted at the NEUTRAL mountpoint `/data`
|
||||
# (LABEL=k3s-data), then split between TWO tenants:
|
||||
# /data/longhorn Longhorn's dataPath
|
||||
# /data/k3s k3s --data-dir (etcd AND the containerd image store)
|
||||
# ETCD -> NO dedicated device. It lives inside /data/k3s on that partition.
|
||||
#
|
||||
# ================================ THE 2026-09-04 REVISION ================================
|
||||
# The 2026-09-03 ruling put ONLY Longhorn on this partition and left k3s at its default
|
||||
# /var/lib/rancher/k3s. `/var` on these nodes is a SEPARATE 11.8 GiB partition, and k3s's
|
||||
# data dir holds both etcd's WAL and the containerd IMAGE STORE. Measured on the fully-
|
||||
# deployed dev cluster: containerd alone is 21-25 GB per node and /var totals 58-70 GB. Prod
|
||||
# would therefore have filled /var during its FIRST image pulls — and with etcd on the same
|
||||
# partition, a full /var means etcd cannot write its WAL. That is cluster death, not
|
||||
# degradation. So k3s's whole data dir moved onto the 424 GB partition, which is why the
|
||||
# mountpoint and the filesystem label are no longer Longhorn-specific: one partition, two
|
||||
# tenants. Nothing was installed on prod when this was decided, which is exactly why it was
|
||||
# decided then — it is cheap now and very expensive after bring-up.
|
||||
#
|
||||
# THE SHARED-DEVICE I/O RISK IS UNCHANGED and still accepted knowingly: Longhorn and etcd
|
||||
# were already going to share this spindle under the 2026-09-03 ruling. Dev's etcd/Longhorn
|
||||
# contention incident was on a spinning HDD; every prod node is SSD or NVMe. Accepted to be
|
||||
# MEASURED, not assumed — exit condition in §2.
|
||||
# =========================================================================================
|
||||
#
|
||||
# Both sources are expressed as an explicit SOURCE enum (roles/node_baseline/tasks/disks.yml),
|
||||
# not as a device path and a blank. That distinction is the point of this block:
|
||||
#
|
||||
# 1. THE DEVICE PATH IS NOT THE SAME ON EVERY NODE, so no group value could be right.
|
||||
# k3s-prod-1 and k3s-prod-3 boot from SATA (`/dev/sda5`); k3s-prod-2 boots from NVMe
|
||||
# (`/dev/nvme0n1p5`). Verified 2026-09-03 by `findmnt --mountpoint /srv` on all three.
|
||||
# Per-host `host_vars` would "work" and would be WRONG in the way that matters: it
|
||||
# freezes a fact about today's hardware into inventory, where it goes stale silently the
|
||||
# first time a node is re-imaged, a disk is swapped, or node 4 arrives on a controller
|
||||
# nobody predicted — and the stale value's failure mode is mkfs on the wrong partition.
|
||||
# So inventory declares the INTENT ("the partition currently mounted at /srv") and the
|
||||
# role discovers the device on the node, then proves it is safe to claim (a partition or
|
||||
# disk, not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or swap, already ext4,
|
||||
# unlabelled, >= 100 GiB, and EMPTY) before it writes anything. Fails closed.
|
||||
#
|
||||
# 2. "etcd has no device" IS A STATE, NOT A MISSING VALUE — and the state has a NAME that
|
||||
# describes reality. It was `root-filesystem` while etcd sat on /var; under the revised
|
||||
# layout etcd is inside /data/k3s on the claimed data partition, which is neither a
|
||||
# dedicated device nor the OS filesystem, so the honest value is `shared-data-partition`.
|
||||
# Leaving it at `root-filesystem` would have been a lie inventory told about the cluster,
|
||||
# and a load-bearing one: the role ASSERTS `shared-data-partition` by checking, per node
|
||||
# on every run, that the filesystem backing the etcd data dir really is /data. If the
|
||||
# mount ever failed, /data would be a directory on `/` and 21-25 GB of images plus etcd's
|
||||
# WAL would land back on the OS filesystem, invisibly, until the disk filled. The
|
||||
# `root-filesystem` value makes no such claim and gets no such assert — which is exactly
|
||||
# why the two are distinct enum values rather than one with a comment.
|
||||
#
|
||||
# 3. DEV CANNOT REACH ANY OF THIS. The role defaults are `source: device` for both stores, so
|
||||
# the LIVE dev cluster keeps its dedicated /dev/vdb + /dev/vdc disks and can never enter
|
||||
# the destructive claim path — the claim tasks are gated on `claim-mountpoint`, which only
|
||||
# this prod-scoped file sets. Dev's mountpoint (/var/lib/longhorn), label (`longhorn`) and
|
||||
# k3s data dir (k3s's own default) are ALL unchanged by this revision, and dev's 18 live
|
||||
# Longhorn volumes are untouched.
|
||||
node_baseline_longhorn_source: claim-mountpoint
|
||||
node_baseline_longhorn_claim_mountpoint: /srv
|
||||
node_baseline_etcd_source: shared-data-partition
|
||||
|
||||
# --- the neutral mountpoint + label (role defaults are DEV's live values) ----------
|
||||
# `/data`: short, tenant-agnostic, and NOT under /var. Deliberately not `/srv` — /srv is the
|
||||
# CLAIM mountpoint the role discovers the device at, and the idempotency contract is that
|
||||
# /srv is gone from /proc/mounts and /etc/fstab once the claim has happened, so re-using it
|
||||
# as the destination would make "claimed" and "not claimed" indistinguishable.
|
||||
# `k3s-data`: deliberately NOT the bare `data`. Guard #4 in the role's assert wall accepts a
|
||||
# partition whose label is empty OR ALREADY OURS, and names `data` as an example of somebody
|
||||
# else's named volume — so labelling ours `data` would make a stranger's `data` volume look
|
||||
# already-claimed. `k3s-data` is cluster-scoped rather than tenant-scoped and matches the
|
||||
# `k3s-etcd` label convention already in this repo.
|
||||
node_baseline_data_mount: /data
|
||||
node_baseline_data_fs_label: k3s-data
|
||||
# Longhorn's dataPath: a SUBDIRECTORY now, because k3s owns a sibling one. [parity] MUST
|
||||
# equal the LONGHORN_DATA_PATH knob in clusters/prod/kustomization.yaml — that is the Helm
|
||||
# side of the same fact (platform/longhorn/helmrelease.yaml defaultDataPath). Dev's role
|
||||
# default keeps dataPath == the mount root, and dev's overlay keeps /var/lib/longhorn.
|
||||
node_baseline_longhorn_data_path: /data/longhorn
|
||||
|
||||
# --- ONE-TIME TRANSITION (delete these two lines once all three nodes are converged) ------
|
||||
# The three prod nodes were ALREADY claimed by the previous version of this role: LABEL
|
||||
# `longhorn`, mounted at /var/lib/longhorn. The role's idempotency pivot is "does the CURRENT
|
||||
# label exist?", so with the label now `k3s-data` an already-claimed node answers "no" and
|
||||
# would walk into the greenfield claim path — which fails closed at "the claim mountpoint
|
||||
# must actually be a mountpoint" because /srv was consumed by that first claim. Fail-closed
|
||||
# is right but it is not convergence, so the transition is DECLARED here: the role unmounts
|
||||
# the retired mountpoint, drops its fstab entry and re-stamps the label, having first proved
|
||||
# the partition is empty (nothing orphaned) and is the device the retired label names.
|
||||
# Dev sets neither var — the role defaults are "" — so dev cannot enter that path at all.
|
||||
# SAFE TO DELETE once `blkid -L k3s-data` succeeds on every prod node; the transition tasks
|
||||
# then skip on the retired-label lookup and nothing else depends on them.
|
||||
node_baseline_data_retired_fs_label: longhorn
|
||||
node_baseline_data_retired_mount: /var/lib/longhorn
|
||||
|
||||
# --- k3s --data-dir (§2) -----------------------------------------------------------
|
||||
# THE ONE LINE THAT MOVES k3s's ENTIRE STATE DIRECTORY off the 11.8 GiB /var and onto the
|
||||
# claimed 424 GB partition. Read by TWO roles, deliberately: roles/k3s_server turns it into
|
||||
# `--data-dir` in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir,
|
||||
# which defers to this variable) creates the directory 0700 on the data partition and derives
|
||||
# etcd's location from it (node_baseline_etcd_mount = <this>/server/db). One variable, so the
|
||||
# Ansible side cannot disagree with itself.
|
||||
#
|
||||
# Dev keeps k3s's own default (/var/lib/rancher/k3s, restated in roles/k3s_server/defaults):
|
||||
# it has a DEDICATED etcd device mounted at server/db under that path, and moving dev's data
|
||||
# dir would orphan that mount under a running cluster.
|
||||
k3s_data_dir: /data/k3s
|
||||
|
||||
# Deliberately NOT-A-DEVICE, exactly as they were deliberately BLOCKED before. Neither
|
||||
# source uses a device path any more, but leaving these UNSET would let prod silently
|
||||
# inherit the role defaults (/dev/vdb, /dev/vdc — virtio names that do not exist on bare
|
||||
# metal) the moment anyone flipped a source back to `device`. These values can never parse
|
||||
# as a device, and the role's `disks` assert stops the play on the NOT-A-DEVICE token
|
||||
# before mkfs is ever reached.
|
||||
node_baseline_longhorn_device: "/dev/NOT-A-DEVICE-the-data-partition-is-claimed-from-srv-see-PROD-PROVISIONING-md-2"
|
||||
node_baseline_etcd_device: "/dev/NOT-A-DEVICE-etcd-lives-in-data-k3s-on-the-shared-data-partition-see-PROD-PROVISIONING-md-2"
|
||||
|
||||
# --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
|
||||
# roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
|
||||
|
||||
@@ -48,13 +48,14 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
|
||||
|
||||
# =============================================================================
|
||||
# PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
|
||||
# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
|
||||
# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
|
||||
# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
|
||||
# nothing has been run against node 1, and the operator has decided the CLUSTER stays
|
||||
# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
|
||||
# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
|
||||
# later member-add dance.
|
||||
# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
|
||||
# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
|
||||
# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
|
||||
# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
|
||||
# check and the node-baseline PACKAGE floor has been run, and the operator has decided
|
||||
# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
|
||||
# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
|
||||
# single-server-etcd window, no later member-add dance.
|
||||
#
|
||||
# KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
|
||||
# hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
|
||||
@@ -92,9 +93,18 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
|
||||
# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
|
||||
# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
|
||||
# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
|
||||
# node 3 remains commented until it is racked.
|
||||
# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
|
||||
# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
|
||||
# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
|
||||
# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
|
||||
# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
|
||||
# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
|
||||
# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
|
||||
# three-way, not a node-1 outlier.
|
||||
# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
|
||||
# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
|
||||
k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
|
||||
# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
|
||||
k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
|
||||
|
||||
# Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
|
||||
# pattern; named to match the group_vars directory so Ansible auto-loads it).
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
# EFI Memory Attributes Table — BASELINE reboot survey (WS-TEST/WS-INFRA · R-ENV-01,
|
||||
# R-ENV-03, R-TEST-02).
|
||||
#
|
||||
# WHAT QUESTION THIS ANSWERS
|
||||
# infra/PROD-PROVISIONING.md §0b settles the MECHANISM: HP's R21 firmware publishes the
|
||||
# EFI Memory Attributes Table into plain System RAM, the boot chain overwrites it, and
|
||||
# Linux 6.12 reserves everything from the table address to the top of RAM without
|
||||
# validating it (fixed upstream in 6.13). §0b does NOT settle two things, and this
|
||||
# playbook measures them:
|
||||
#
|
||||
# 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Node 2's two observed
|
||||
# failures were both at exactly MEMATTR=0xbeeb0018; all five observed successes
|
||||
# fleet-wide were at other addresses. If bad outcomes map onto a narrow address
|
||||
# range, the causal chain closes with nothing touched.
|
||||
# 2. Is the fault node-2-specific, or does ANY node reach the bad path given enough
|
||||
# boots? Nodes 1 and 3 have each been observed exactly twice.
|
||||
#
|
||||
# §0b's own power analysis is why this is 20 rounds and not five: node 2's failure rate
|
||||
# is known to n = 3 (2 bad of 3), a 95% Clopper-Pearson interval of ~0.09-0.99. Against
|
||||
# a null that loose a handful of boots is worthless. Detecting ELIMINATION needs ~20
|
||||
# boots per arm; detecting a mere odds shift needs ~50.
|
||||
#
|
||||
# THIS IS THE BASELINE ARM. NOTHING IS CHANGED — no BIOS writes, no kernel change, no
|
||||
# package installs, no hardware moves, no node_baseline run. Its job is to establish real
|
||||
# p(bad) per node and to test the address-outcome correlation. Any later arm (a DIMM swap,
|
||||
# a Fast Boot flip, a 6.13 kernel) compares against it.
|
||||
#
|
||||
# HOW TO RUN IT
|
||||
# Dry run first — every stage except the reboots, so the pipeline is proven before 45
|
||||
# minutes of wall clock are committed:
|
||||
#
|
||||
# ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
|
||||
# ansible-playbook playbooks/memattr-survey.yml'
|
||||
#
|
||||
# The real thing (REBOOTS ARE OPT-IN — the interlock is deliberate):
|
||||
#
|
||||
# ./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
|
||||
# ansible-playbook playbooks/memattr-survey.yml \
|
||||
# -e memattr_survey_reboots_authorised=true'
|
||||
#
|
||||
# Fewer rounds, or one node:
|
||||
# -e memattr_survey_rounds=5 --limit k3s-prod-2
|
||||
#
|
||||
# RESUMABLE: re-running after an interruption continues from the on-node ledger; earlier
|
||||
# rounds are never lost or double-counted. Analysis alone, over existing artifacts:
|
||||
# ansible-playbook playbooks/memattr-survey.yml --tags analyse
|
||||
#
|
||||
# TARGET GROUP IS A VARIABLE (`memattr_survey_hosts`), following playbooks/node-health.yml.
|
||||
# It defaults to k3s_prod and the role additionally REFUSES to run against a host that is
|
||||
# not a member of memattr_survey_required_group — the dev cluster is live, and a survey
|
||||
# that reboots it 20 times because of a mistyped --limit is exactly the accident worth
|
||||
# engineering against.
|
||||
#
|
||||
# HOW TO READ THE OUTPUT
|
||||
# infra/ansible/_artifacts/memattr-survey/ (gitignored, regenerated every run)
|
||||
# <host>.rounds.jsonl one JSON line per round, mirrored off the node after EVERY round
|
||||
# <host>.boots.json every boot the persistent journal holds, re-derived in one pass
|
||||
# at the end — the AUTHORITATIVE record
|
||||
# summary.md the analysis, also printed at the end of the run
|
||||
# See roles/memattr_survey/README.md for the field-by-field meaning and for why a boot
|
||||
# with NO memattr log line is a healthy third state and not a missing measurement.
|
||||
|
||||
# =====================================================================================
|
||||
# PLAY 1 — SAFETY. any_errors_fatal: a violation on ANY node stops the WHOLE playbook
|
||||
# before a single reboot. That is the right asymmetry: the cost of a false stop is a
|
||||
# re-run, the cost of a false start is rebooting a machine that was doing something.
|
||||
# =====================================================================================
|
||||
- name: memattr survey — preflight (refuse to reboot anything that is not idle prod)
|
||||
hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
|
||||
gather_facts: false
|
||||
become: true
|
||||
any_errors_fatal: true
|
||||
tags: [survey]
|
||||
tasks:
|
||||
- name: Preflight and baseline probe
|
||||
ansible.builtin.include_role:
|
||||
name: memattr_survey
|
||||
tasks_from: preflight
|
||||
|
||||
# =====================================================================================
|
||||
# PLAY 2 — THE SURVEY.
|
||||
#
|
||||
# any_errors_fatal deliberately OFF: here a node that fails to come back must stop only
|
||||
# ITSELF (roles/memattr_survey/tasks/round.yml has the block/rescue), while the others
|
||||
# keep collecting.
|
||||
#
|
||||
# `strategy: free` IS LOAD-BEARING, NOT A TUNING KNOB — and this was measured, not
|
||||
# assumed. The rounds are a `loop` over a dynamic `include_tasks`, and Ansible expands
|
||||
# that into one include PER (host, item), appended per host: all 20 of node 1's rounds,
|
||||
# then all 20 of node 2's, then node 3's. Under the default LINEAR strategy each of those
|
||||
# blocks belongs to exactly one host, so the other two sit idle through it and the survey
|
||||
# runs SERIALLY — 60 reboots end to end, ~3x the wall clock, for machines that are
|
||||
# completely independent. Observed on the first run of this playbook: only k3s-prod-1 ever
|
||||
# appeared in the output. `free` lets each host walk its own task list at its own pace,
|
||||
# which restores the intended concurrency: 20 rounds costs the wall clock of 20 reboots.
|
||||
#
|
||||
# Consequences worth knowing before editing: output INTERLEAVES between hosts (every
|
||||
# progress line is host-tagged for exactly this reason), and `run_once` is unreliable
|
||||
# under `free` — roles/memattr_survey/tasks/harvest.yml therefore uses an idempotent
|
||||
# directory create on every host instead. Concurrency is also capped by `forks`
|
||||
# (ansible.cfg leaves the default of 5); preflight asserts forks >= host count rather
|
||||
# than letting a bigger fleet silently serialise again.
|
||||
# =====================================================================================
|
||||
- name: memattr survey — reboot rounds and journal harvest
|
||||
hosts: "{{ memattr_survey_hosts | default('k3s_prod') }}"
|
||||
gather_facts: false
|
||||
become: true
|
||||
strategy: free
|
||||
tags: [survey]
|
||||
tasks:
|
||||
- name: Run the reboot rounds
|
||||
ansible.builtin.include_role:
|
||||
name: memattr_survey
|
||||
tasks_from: rounds
|
||||
|
||||
- name: Harvest every retained boot from the persistent journal
|
||||
ansible.builtin.include_role:
|
||||
name: memattr_survey
|
||||
tasks_from: harvest
|
||||
|
||||
# =====================================================================================
|
||||
# PLAY 3 — ANALYSIS, on the controller. A SEPARATE PLAY ON localhost on purpose: it runs
|
||||
# even if every surveyed node died, because the artifacts are already on disk and the
|
||||
# analysis counts rounds from THOSE rather than from Ansible's host state. A node that
|
||||
# vanished shows up as a short arm with a wider interval, not as a missing report.
|
||||
# =====================================================================================
|
||||
- name: memattr survey — analyse
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
become: false
|
||||
tags: [analyse]
|
||||
tasks:
|
||||
- name: Produce the analysis
|
||||
ansible.builtin.include_role:
|
||||
name: memattr_survey
|
||||
tasks_from: analyse
|
||||
@@ -17,6 +17,38 @@ k3s_install_script: "/usr/local/bin/k3s-install.sh"
|
||||
# the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
|
||||
k3s_disable: "traefik,servicelb,local-storage"
|
||||
|
||||
# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
|
||||
# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
|
||||
# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
|
||||
# image store (agent/containerd) — which is by far the largest tenant.
|
||||
#
|
||||
# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
|
||||
# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
|
||||
# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
|
||||
# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
|
||||
# running cluster's entire state directory out from under it and orphan that etcd mount.
|
||||
#
|
||||
# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
|
||||
# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
|
||||
# node inherits it from group membership rather than from anyone remembering. It is set
|
||||
# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
|
||||
# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
|
||||
# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
|
||||
# this same variable). One line moves both.
|
||||
#
|
||||
# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
|
||||
# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
|
||||
# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
|
||||
# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
|
||||
# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
|
||||
#
|
||||
# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
|
||||
# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
|
||||
# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
|
||||
# Restart handler for a value that is not changing. The install flags are the correct and
|
||||
# sufficient place, and prod has no k3s installed yet.
|
||||
k3s_data_dir: "/var/lib/rancher/k3s"
|
||||
|
||||
# Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
|
||||
k3s_server_mode: "secondary"
|
||||
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
--write-kubeconfig-mode=644
|
||||
{% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
|
||||
--node-ip={{ k3s_node_ip }}
|
||||
--data-dir={{ k3s_data_dir }}
|
||||
|
||||
# On an ALREADY-installed cluster the install task above is idempotent (creates:
|
||||
# k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
# `memattr_survey` — baseline reboot survey of the EFI Memory Attributes Table lottery
|
||||
|
||||
**Workstream:** WS-TEST / WS-INFRA · **Serves:** R-ENV-01, R-ENV-03, R-TEST-02
|
||||
**Read first:** [`infra/PROD-PROVISIONING.md` §0b](../../PROD-PROVISIONING.md) — the mechanism
|
||||
is settled there and this role does **not** re-derive it.
|
||||
|
||||
## The question
|
||||
|
||||
§0b establishes *what* goes wrong: HP's R21 firmware publishes the EFI Memory Attributes
|
||||
Table into memory the OS classifies as plain `System RAM`, the boot chain overwrites it,
|
||||
and Linux 6.12 multiplies two unvalidated fields and `memblock_reserve()`s everything from
|
||||
the table address to the top of RAM. Fixed upstream in 6.13; no cmdline workaround exists.
|
||||
|
||||
Two things §0b leaves unmeasured, and this role measures:
|
||||
|
||||
1. **Is the boot outcome predicted by *where the table landed*?** Node 2's two observed
|
||||
failures were both at exactly `MEMATTR=0xbeeb0018`; all five observed successes
|
||||
fleet-wide were at other addresses. If bad outcomes map onto a narrow address range,
|
||||
the causal chain closes without touching hardware or firmware.
|
||||
2. **Is the fault node-2-specific?** Nodes 1 and 3 had been observed exactly twice each.
|
||||
|
||||
## Why 20 rounds
|
||||
|
||||
Node 2's failure rate was known only to **n = 3** (2 bad of 3) — a 95% Clopper–Pearson
|
||||
interval on p(bad) of ≈ **0.09–0.99**. Against a null that loose, a handful of boots is
|
||||
worthless. §0b's power note: detecting *elimination* needs ~20 boots per arm; detecting a
|
||||
mere odds shift needs ~50. 20 is the floor, not a round number. `memattr_survey_rounds`
|
||||
is a variable; lowering it below 20 produces an arm that cannot answer the question it was
|
||||
built for, and the analysis will say so.
|
||||
|
||||
## This is the BASELINE arm
|
||||
|
||||
**Nothing is changed.** No BIOS writes, no kernel change, no package installs, no hardware
|
||||
moves, no `node_baseline` run. The role installs nothing and configures nothing; its only
|
||||
write to a node is its own append-only ledger under `/var/lib`. Per §0b the boot chain's
|
||||
own allocation footprint — initrd size, module set, boot device — is *part of the causal
|
||||
chain*, so anything that perturbs it voids the arm. §6 of the generated report checks that
|
||||
the kernel release and cmdline were constant on every node, for exactly this reason.
|
||||
|
||||
## Running it
|
||||
|
||||
```bash
|
||||
# DRY RUN — every stage except the reboots. Proves the whole pipeline in ~30 s.
|
||||
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
|
||||
ansible-playbook playbooks/memattr-survey.yml'
|
||||
|
||||
# The real thing. Reboots are OPT-IN; the interlock is deliberate.
|
||||
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
|
||||
ansible-playbook playbooks/memattr-survey.yml \
|
||||
-e memattr_survey_reboots_authorised=true'
|
||||
|
||||
# Re-analyse existing artifacts, no hardware touched.
|
||||
./infra/devcontainer/dev.sh run bash -c 'cd /work/infra/ansible && \
|
||||
ansible-playbook playbooks/memattr-survey.yml --tags analyse'
|
||||
```
|
||||
|
||||
Useful overrides: `-e memattr_survey_rounds=5`, `--limit k3s-prod-2`,
|
||||
`-e memattr_survey_hosts=<group>`.
|
||||
|
||||
20 rounds × ~90 s ≈ 30–45 minutes. Rounds run **in parallel** across nodes — they are
|
||||
independent machines — so 20 rounds costs the wall clock of 20 reboots, not 60. That
|
||||
parallelism depends on `strategy: free` in the play and on `forks` ≥ the host count;
|
||||
preflight asserts the latter. See the playbook comment for why the default linear
|
||||
strategy silently serialises a looped `include_tasks` (it was measured, not assumed).
|
||||
|
||||
## Safety — verified, not assumed
|
||||
|
||||
The reboots are sanctioned only because the prod boxes carry no k3s and run nothing. That
|
||||
premise is re-checked on **every** run, because a document and a machine drift apart:
|
||||
|
||||
| Check | Failure mode it prevents |
|
||||
|---|---|
|
||||
| Host is in `memattr_survey_required_group` (default `k3s_prod`) | A mistyped `--limit` rebooting the **live dev cluster** 20 times. Read from inventory membership, which a host pattern cannot fake. |
|
||||
| `systemctl is-active k3s k3s-agent kubelet containerd docker` all inactive | Rebooting a node that is actually serving something. |
|
||||
| `/etc/rancher/k3s`, `/var/lib/rancher/k3s`, `/usr/local/bin/k3s` absent | k3s installed but stopped — still a machine with state. |
|
||||
| `/var/log/journal` exists | A **volatile** journal, where every reboot erases the evidence and the harvest silently returns one boot: a survey that looks complete and holds nothing. |
|
||||
| `memattr_survey_reboots_authorised` | Reboots at all. Defaults `false`. |
|
||||
|
||||
Preflight runs with `any_errors_fatal: true`, so a violation on **any** node stops the
|
||||
whole playbook before a single reboot. The cost of a false stop is a re-run; the cost of a
|
||||
false start is a rebooted production machine.
|
||||
|
||||
## Fail-safe and resume
|
||||
|
||||
* **Per-node isolation.** `tasks/round.yml` wraps each round in `block`/`rescue`. A node
|
||||
that does not return within `memattr_survey_reboot_timeout` (default 600 s, ~6× the
|
||||
observed POST-to-SSH time) sets an abort flag; every later round is skipped **for that
|
||||
host only** and the others keep collecting. The abort is announced immediately and again
|
||||
in the report — which counts rounds from the **artifacts**, not from Ansible's host
|
||||
state, so even a node that goes fully unreachable (which `rescue` cannot catch) still
|
||||
appears as a short arm with a wider interval, never as a silently missing report.
|
||||
* **Resumable.** Round numbers come from the on-node ledger, so re-running after an
|
||||
interruption continues where it stopped and can neither lose nor duplicate a round. The
|
||||
boot found on a resume is *not* re-appended: it is already round N of the interrupted
|
||||
run, and double-counting it would bias the sample toward whichever outcome survived the
|
||||
interruption.
|
||||
* **The abort is durable, not scrollback.** `<host>.aborted.json` records the round, the
|
||||
failing task and the reason, so the report can say *why* an arm is short instead of
|
||||
leaving "0 rounds" looking like a targeting mistake. It is cleared at the start of every
|
||||
run and re-written only if that run aborts. Its *absence* is not proof of success — a
|
||||
node can be lost abruptly enough that Ansible calls it unreachable before the rescue
|
||||
runs — so completeness is still judged on round count, and the marker only supplies the
|
||||
reason when there is one.
|
||||
* **Analysis survives everything.** It is a separate play on `localhost`, so it runs even
|
||||
if every surveyed node died. With a node down, `--tags analyse` is the way in: a full
|
||||
run stops at preflight (correctly — `any_errors_fatal`) on the unreachable host.
|
||||
|
||||
## Belt and braces on the data
|
||||
|
||||
Two independent records, joined on `boot_id`:
|
||||
|
||||
| | written by | authoritative for |
|
||||
|---|---|---|
|
||||
| `<host>.rounds.jsonl` | the driver, appended + `fsync`ed on the node after each round, and **mirrored to the controller after every round** | **round numbers** |
|
||||
| `<host>.boots.json` | one end-of-run pass over the **persistent journal**, re-deriving every retained boot from scratch | **outcomes** |
|
||||
| `<host>.aborted.json` | the per-round rescue, when a node stops coming back | **why an arm is short** |
|
||||
|
||||
The journal does not care whether the driver was alive: the boot happened, the kernel
|
||||
logged it, journald persisted it. So the harvest can only ever be a *superset* of the
|
||||
ledger, and a round the driver failed to append still shows up.
|
||||
|
||||
## Reading the output
|
||||
|
||||
Artifacts land in `infra/ansible/_artifacts/memattr-survey/` — same convention as
|
||||
`roles/node_health` and `conformance/_artifacts/`: generated output, `.gitignore`d, never
|
||||
committed. `summary.md` is also printed at the end of the run.
|
||||
|
||||
### The verdict is measured, never inferred from a log line
|
||||
|
||||
This is the subtlest part of the design. There are **three** kernel readings, not two:
|
||||
|
||||
| `kernel_read` | What the kernel logged | Memory |
|
||||
|---|---|---|
|
||||
| `valid` | **nothing at all** — it parsed a table it considered valid | intact |
|
||||
| `bail-version` | `Unexpected EFI Memory Attributes table version <garbage>` | intact — the *lucky* path, and it looks alarming |
|
||||
| `failed-map` | `Failed to map EFI Memory Attributes table @ 0x…` | **gone** |
|
||||
|
||||
Silence is therefore *evidence of health*, not a missing measurement — which is why the
|
||||
record carries an explicit `memattr_diagnostic_present: false` rather than an empty field.
|
||||
And per §0b the `Failed to map` message is emitted by `efi_memattr_apply_permissions()`,
|
||||
*later* than the `memblock_reserve()` that did the damage: it is a consequence, not the
|
||||
cause. So the **verdict comes from `MemTotal` versus SMBIOS installed size**, and
|
||||
`kernel_read` is recorded alongside it as an independent observation. §5a of the report
|
||||
cross-tabulates the two; if they ever disagree, that disagreement is itself the finding.
|
||||
|
||||
Thresholds (`files/memattr_probe.py`): good ≥ 0.85 usable/installed, bad ≤ 0.60. Observed
|
||||
clusters are 0.95–0.98 and 0.15–0.17, so the thresholds sit in a two-order gap. Anything
|
||||
landing between them is reported as `indeterminate` and excluded from `n` — a new third
|
||||
thing deserves human eyes, not a coin flip into a bucket.
|
||||
|
||||
### What is and is not pooled
|
||||
|
||||
Boots the survey did **not cause** are shown in the per-round table for context and
|
||||
excluded from every statistic: `pre` rows (already in the journal) and round `0` (the boot
|
||||
each node was found sitting on — for node 2 that is literally §0b's own "boot 0" row, so
|
||||
pooling it would double-count prior evidence into a run whose point is fresh trials).
|
||||
|
||||
### The statistics
|
||||
|
||||
`files/memattr_analyse.py`, pure stdlib, so it runs in the dev container as-is.
|
||||
|
||||
* **Clopper–Pearson** (exact binomial), by bisection on the exact tails rather than the
|
||||
Beta/F identity — no SciPy, and the boundary cases k = 0 and k = n fall out correctly
|
||||
instead of needing a special case. A bare fraction is what this exercise exists to stop
|
||||
reporting: 2/3 and 13/20 are the same fraction and utterly different evidence.
|
||||
* **The address↔outcome question is tested three ways**, because the tempting version is
|
||||
circular:
|
||||
* **4b determinism** — was any single address seen with *both* outcomes? One such
|
||||
address falsifies "the address determines the outcome" outright.
|
||||
* **4c separation** — do the ranges interleave? Reported descriptively and *not* dressed
|
||||
up as a p-value: the bad set is defined after seeing the data.
|
||||
* **4d the pre-registered prediction** — §0b, written before this run, names
|
||||
`0xbeeb0018`. Fisher exact against this run's boots only. This is the one
|
||||
non-circular test available.
|
||||
* **4e address → kernel reading** — the fallback that has power when 4a–4d do not. If a
|
||||
run produces no bad boot, `verdict` has no variance to explain, but `kernel_read`
|
||||
does (`valid` vs `bail-version`), and it sits one step upstream of the outcome in
|
||||
§0b's mechanism. An address that reproduces the same *parse* boot after boot is the
|
||||
address doing causal work, even when none of those parses was catastrophic.
|
||||
|
||||
## Files
|
||||
|
||||
| Path | What it is |
|
||||
|---|---|
|
||||
| `tasks/preflight.yml` | Safety gate, resume point, baseline probe |
|
||||
| `tasks/rounds.yml` | The loop — resume bounds and the reboot interlock |
|
||||
| `tasks/round.yml` | One round: reboot → measure → append → mirror, with the per-node rescue |
|
||||
| `tasks/harvest.yml` | The authoritative journal pass |
|
||||
| `tasks/analyse.yml` | Controller-side analysis |
|
||||
| `files/memattr_probe.py` | Reads one boot (or every retained boot) into one JSON record. Same code both ways, so live and harvested records are directly comparable |
|
||||
| `files/memattr_analyse.py` | Clopper–Pearson, Fisher, and the report |
|
||||
|
||||
## What this cannot tell you
|
||||
|
||||
It is one arm. It measures p(bad) and tests the address correlation; it cannot attribute
|
||||
the difference between nodes, because node 2 is simultaneously the only single-DIMM node
|
||||
and the only NVMe-booting node — a confound §0b calls out and that adding node 3 doubled
|
||||
rather than broke. If more is wanted, §0b's recommendation is to **swap hardware, not BIOS
|
||||
settings**: move node 2's DIMM into node 3 and node 3's pair into node 2, then run this
|
||||
role again on both. **None of it changes the action** — the fix is a kernel ≥ 6.13 on all
|
||||
three nodes, and that is not in dispute.
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
# memattr_survey — a BASELINE reboot survey of the EFI Memory Attributes Table lottery
|
||||
# described in infra/PROD-PROVISIONING.md §0b. Read that section first: the mechanism is
|
||||
# settled and is NOT re-derived here.
|
||||
#
|
||||
# THE QUESTION THIS ROLE ANSWERS
|
||||
# §0b establishes WHAT goes wrong (Linux 6.12 reserves everything above a corrupt EFI
|
||||
# Memory Attributes Table, fixed upstream in 6.13) but leaves two things unmeasured:
|
||||
# 1. Is the boot outcome PREDICTED BY WHERE THE TABLE LANDED? Both of node 2's
|
||||
# observed failures were at exactly MEMATTR=0xbeeb0018 and all five observed
|
||||
# successes fleet-wide were at other addresses. If bad outcomes map onto a narrow
|
||||
# address range, the causal chain is closed with no hardware or firmware touched.
|
||||
# 2. Is the fault node-2-specific, or does any node reach the bad path given enough
|
||||
# boots? Nodes 1 and 3 have each been observed exactly twice.
|
||||
#
|
||||
# WHY 20 ROUNDS AND NOT FIVE
|
||||
# Node 2's failure rate is known only to n = 3 (2 bad of 3). The 95% Clopper-Pearson
|
||||
# interval on p(bad) is roughly 0.09-0.99 — against a null that loose a handful of
|
||||
# boots proves nothing at all. §0b's own power note: detecting ELIMINATION of the fault
|
||||
# needs ~20 boots per arm, detecting a mere odds shift ~50. 20 is therefore the floor,
|
||||
# not a round number.
|
||||
#
|
||||
# THIS RUN IS THE BASELINE ARM: NOTHING IS CHANGED.
|
||||
# No BIOS writes, no kernel change, no package installs, no hardware moves. The role
|
||||
# installs nothing and configures nothing; its only write to a node is its own append-
|
||||
# only ledger under /var/lib. Anything that alters the boot chain's allocation
|
||||
# footprint — and per §0b that includes the initrd, the module set and the boot device —
|
||||
# changes the variable under test and voids the arm.
|
||||
|
||||
# --- Sample size --------------------------------------------------------------------
|
||||
# Reboot rounds per node. Rounds run in PARALLEL across nodes (they are independent
|
||||
# machines), so 20 rounds costs the wall clock of 20 reboots, not 60.
|
||||
memattr_survey_rounds: 20
|
||||
|
||||
# --- Safety interlock ----------------------------------------------------------------
|
||||
# REBOOTS ARE OPT-IN. Left false the role does everything EXCEPT reboot: safety
|
||||
# preflight, a baseline probe of the current boot, the journal harvest and the analysis.
|
||||
# That is a genuinely useful dry run and it is also how you prove the pipeline works
|
||||
# before committing 45 minutes of wall clock to it.
|
||||
memattr_survey_reboots_authorised: false
|
||||
|
||||
# The role refuses to run against a host that is not in this group. The prod boxes carry
|
||||
# no k3s and nothing is running on them, which is the entire reason rebooting them is
|
||||
# sanctioned; the dev cluster is live and must never be caught by a stray --limit.
|
||||
memattr_survey_required_group: k3s_prod
|
||||
|
||||
# Unit names whose presence or activity means something IS running on this node. Any hit
|
||||
# aborts the whole playbook before the first reboot (see tasks/preflight.yml).
|
||||
memattr_survey_forbidden_units:
|
||||
- k3s
|
||||
- k3s-agent
|
||||
- kubelet
|
||||
- containerd
|
||||
- docker
|
||||
# Paths whose existence means k3s has been installed here, even if not currently running.
|
||||
memattr_survey_forbidden_paths:
|
||||
- /etc/rancher/k3s
|
||||
- /var/lib/rancher/k3s
|
||||
- /usr/local/bin/k3s
|
||||
|
||||
# --- Reboot behaviour -----------------------------------------------------------------
|
||||
# A node that does not answer within this is declared lost: the role stops rebooting THAT
|
||||
# node and the survey continues on the others (tasks/round.yml). 10 minutes is ~6x the
|
||||
# observed ~90 s POST-to-SSH time on this hardware, so a trip is a real failure and not a
|
||||
# slow boot.
|
||||
memattr_survey_reboot_timeout: 600
|
||||
memattr_survey_connect_timeout: 20
|
||||
# Let journald settle before reading the new boot's kernel log. The kernel log itself is
|
||||
# complete long before SSH is up; this is purely margin.
|
||||
memattr_survey_post_reboot_delay: 15
|
||||
|
||||
# --- On-node ledger (belt) -------------------------------------------------------------
|
||||
# Appended one JSON line per round, fsync'd, so a round survives the next reboot even if
|
||||
# the controller dies. This is what makes the survey resumable: re-running skips rounds
|
||||
# already recorded here rather than starting over.
|
||||
memattr_survey_state_dir: /var/lib/memattr-survey
|
||||
memattr_survey_state_file: "{{ memattr_survey_state_dir }}/rounds.jsonl"
|
||||
|
||||
# --- Controller artifacts (braces) ------------------------------------------------------
|
||||
# Same convention as roles/node_health and conformance/_artifacts/: generated output under
|
||||
# a `_artifacts/` directory that .gitignore excludes, never committed.
|
||||
# <host>.rounds.jsonl the per-round ledger, re-fetched AFTER EVERY ROUND so a crash
|
||||
# mid-survey still leaves every completed round on the controller
|
||||
# <host>.boots.json the journal harvest — every retained boot re-derived in one
|
||||
# pass. THIS is the authoritative record; the ledger is the backup.
|
||||
# summary.md the analysis (also printed)
|
||||
memattr_survey_artifact_dir: "{{ playbook_dir }}/../_artifacts/memattr-survey"
|
||||
memattr_survey_write_artifacts: true
|
||||
|
||||
# Print each round's one-line result as it happens. A 45-minute unattended run with no
|
||||
# output is indistinguishable from a hung one.
|
||||
memattr_survey_show_progress: true
|
||||
@@ -0,0 +1,521 @@
|
||||
#!/usr/bin/env python3
|
||||
"""memattr_analyse.py — turn a memattr survey's artifacts into the answer.
|
||||
|
||||
Runs on the CONTROLLER, over infra/ansible/_artifacts/memattr-survey/. Reads
|
||||
<host>.boots.json the journal harvest — authoritative, one entry per retained boot
|
||||
<host>.rounds.jsonl the per-round ledger — authoritative for ROUND NUMBERS
|
||||
and joins them on boot_id.
|
||||
|
||||
WHAT IT ANSWERS, in order:
|
||||
|
||||
1. COMPLETENESS. How many rounds each node actually contributed. A short arm is stated
|
||||
loudly and up front, because every number below it is weaker than it looks.
|
||||
|
||||
2. p(bad) PER NODE, with a 95% Clopper-Pearson interval. A bare fraction is the thing
|
||||
this whole exercise exists to stop reporting: 2/3 and 13/20 are the same fraction and
|
||||
completely different evidence. Clopper-Pearson is the exact (conservative) binomial
|
||||
interval — no normal approximation, correct at 0/n and n/n where the approximation is
|
||||
degenerate and where a clean arm is most likely to land.
|
||||
|
||||
3. THE CAUSAL QUESTION: is the outcome predicted by where the table landed? Tested three
|
||||
ways, deliberately, because the tempting version of this test is circular:
|
||||
(a) DETERMINISM — is any single address ever seen with BOTH outcomes? One such
|
||||
address falsifies "the address determines the outcome" outright, and no amount
|
||||
of correlation elsewhere repairs it.
|
||||
(b) SEPARATION — do bad addresses occupy a range disjoint from good ones? Reported
|
||||
as sorted min/max per outcome plus the count of good addresses falling inside
|
||||
the bad range. This is descriptive; it is NOT dressed up as a p-value, because
|
||||
the bad set was chosen after seeing the data.
|
||||
(c) THE PRE-REGISTERED PREDICTION — §0b was written before this run and predicts
|
||||
that MEMATTR=0xbeeb0018 is bad. That prediction is tested only against boots
|
||||
this run observed, which is the one non-circular test available.
|
||||
|
||||
4. CORROBORATION. kernel_read versus verdict (do the log line and the measured memory
|
||||
ever disagree?), and NODE_DATA placement, which §0b calls the cleanest A/B tell.
|
||||
|
||||
Everything is pure stdlib — no numpy, no scipy — so it runs in the dev container as-is.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
# §0b's pre-registered prediction: this exact address was seen twice on node 2 and was bad
|
||||
# both times, while all five successes fleet-wide were at other addresses.
|
||||
PREREGISTERED_BAD_ADDRESS = "0xbeeb0018"
|
||||
|
||||
|
||||
# --- Exact binomial interval ----------------------------------------------------------
|
||||
def binom_tail_ge(n: int, k: int, p: float) -> float:
|
||||
"""P(X >= k) for X ~ Bin(n, p)."""
|
||||
if k <= 0:
|
||||
return 1.0
|
||||
return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(k, n + 1))
|
||||
|
||||
|
||||
def binom_tail_le(n: int, k: int, p: float) -> float:
|
||||
"""P(X <= k) for X ~ Bin(n, p)."""
|
||||
if k >= n:
|
||||
return 1.0
|
||||
return sum(math.comb(n, i) * p**i * (1 - p) ** (n - i) for i in range(0, k + 1))
|
||||
|
||||
|
||||
def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
|
||||
"""Exact 95% (by default) binomial confidence interval for k successes in n trials.
|
||||
|
||||
Solved by bisection on the exact tail probabilities rather than via the Beta/F
|
||||
quantile identity, so it needs nothing outside the standard library and the
|
||||
boundary cases (k = 0, k = n) fall out correctly instead of needing special-casing:
|
||||
lower bound solves P(X >= k | p) = alpha/2, and is 0 when k = 0
|
||||
upper bound solves P(X <= k | p) = alpha/2, and is 1 when k = n
|
||||
"""
|
||||
if n == 0:
|
||||
return (0.0, 1.0)
|
||||
|
||||
def bisect(fn, target, lo, hi):
|
||||
for _ in range(200):
|
||||
mid = (lo + hi) / 2
|
||||
if fn(mid) < target:
|
||||
lo = mid
|
||||
else:
|
||||
hi = mid
|
||||
return (lo + hi) / 2
|
||||
|
||||
lower = 0.0 if k == 0 else bisect(lambda p: binom_tail_ge(n, k, p), alpha / 2, 0.0, 1.0)
|
||||
upper = 1.0 if k == n else bisect(lambda p: -binom_tail_le(n, k, p), -alpha / 2, 0.0, 1.0)
|
||||
return (lower, upper)
|
||||
|
||||
|
||||
def fisher_exact_2x2(a: int, b: int, c: int, d: int) -> float:
|
||||
"""Two-sided Fisher exact p for [[a, b], [c, d]]. Exact, stdlib only.
|
||||
|
||||
Used ONLY for the pre-registered prediction (address == the §0b address vs outcome),
|
||||
never for a bad set defined after the fact — that would be a garden of forking paths
|
||||
with a p-value stapled on.
|
||||
"""
|
||||
n = a + b + c + d
|
||||
if n == 0:
|
||||
return 1.0
|
||||
row1, col1 = a + b, a + c
|
||||
|
||||
def prob(x):
|
||||
return (
|
||||
math.comb(row1, x)
|
||||
* math.comb(n - row1, col1 - x)
|
||||
/ math.comb(n, col1)
|
||||
)
|
||||
|
||||
observed = prob(a)
|
||||
lo = max(0, col1 - (n - row1))
|
||||
hi = min(row1, col1)
|
||||
# 1e-9 slack: sum the tail of tables no more likely than the observed one, tolerating
|
||||
# float noise in the equality case.
|
||||
return min(1.0, sum(prob(x) for x in range(lo, hi + 1) if prob(x) <= observed * (1 + 1e-9)))
|
||||
|
||||
|
||||
# --- Loading ---------------------------------------------------------------------------
|
||||
def load_host(art_dir: str, host: str) -> dict:
|
||||
"""Harvest (authoritative outcomes) joined to ledger (authoritative round numbers)."""
|
||||
with open(os.path.join(art_dir, f"{host}.boots.json"), encoding="utf-8") as handle:
|
||||
harvest = json.load(handle)
|
||||
|
||||
rounds_by_boot: dict[str, int] = {}
|
||||
ledger_path = os.path.join(art_dir, f"{host}.rounds.jsonl")
|
||||
ledger: list[dict] = []
|
||||
if os.path.exists(ledger_path):
|
||||
with open(ledger_path, encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
record = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
ledger.append(record)
|
||||
if record.get("boot_id") and isinstance(record.get("round"), int):
|
||||
rounds_by_boot[record["boot_id"]] = record["round"]
|
||||
|
||||
boots = []
|
||||
for boot in harvest.get("boots", []):
|
||||
entry = dict(boot)
|
||||
entry["round"] = rounds_by_boot.get(boot.get("boot_id"))
|
||||
boots.append(entry)
|
||||
boots.sort(key=lambda b: (b.get("boot_index") is None, b.get("boot_index", 0)))
|
||||
|
||||
# Written by the per-round rescue when a node stops coming back. Its absence is not
|
||||
# proof of success — a node can be lost so abruptly that Ansible calls it unreachable
|
||||
# before the rescue runs — so completeness is still judged on round COUNT, and this
|
||||
# only supplies the reason when one exists.
|
||||
abort = None
|
||||
abort_path = os.path.join(art_dir, f"{host}.aborted.json")
|
||||
if os.path.exists(abort_path):
|
||||
try:
|
||||
with open(abort_path, encoding="utf-8") as handle:
|
||||
abort = json.load(handle)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
abort = {"reason": "abort marker present but unreadable"}
|
||||
|
||||
return {"host": host, "harvest": harvest, "boots": boots, "ledger": ledger, "abort": abort}
|
||||
|
||||
|
||||
def discover_hosts(art_dir: str) -> list[str]:
|
||||
return sorted(
|
||||
os.path.basename(p)[: -len(".boots.json")]
|
||||
for p in glob.glob(os.path.join(art_dir, "*.boots.json"))
|
||||
)
|
||||
|
||||
|
||||
# --- Rendering ---------------------------------------------------------------------------
|
||||
def gib(kb) -> str:
|
||||
return "?" if kb is None else f"{kb / 1048576:.2f}"
|
||||
|
||||
|
||||
def pct(x) -> str:
|
||||
return "?" if x is None else f"{100 * x:.0f}%"
|
||||
|
||||
|
||||
def render(hosts: list[dict], expected_rounds: int) -> str:
|
||||
out: list[str] = []
|
||||
w = out.append
|
||||
|
||||
all_boots = [b for h in hosts for b in h["boots"]]
|
||||
# Only boots this survey CAUSED count toward any statistic. Excluded, deliberately:
|
||||
# * pre-existing journal boots (round None) — observed under an unknown selection
|
||||
# process, and already counted once in §0b's table;
|
||||
# * round 0, the boot each node was found sitting on — it is a boot nobody chose,
|
||||
# but for node 2 it is literally §0b's "boot 0" row, so pooling it would double-
|
||||
# count prior evidence into a run whose entire point is fresh, independent trials.
|
||||
# Both are still SHOWN in the per-round table, because context is not contamination.
|
||||
surveyed = [b for b in all_boots if b.get("round") not in (None, 0)]
|
||||
|
||||
w("# EFI Memory Attributes Table — baseline reboot survey")
|
||||
w("")
|
||||
w("Baseline arm: NOTHING changed (no BIOS writes, no kernel change, no packages, no")
|
||||
w("hardware moves). Mechanism is settled in `infra/PROD-PROVISIONING.md` §0b; this")
|
||||
w("measures p(bad) per node and tests whether the outcome is predicted by the address")
|
||||
w("the firmware published the table at.")
|
||||
w("")
|
||||
|
||||
# --- 1. Completeness -----------------------------------------------------------------
|
||||
w("## 1. Completeness — read this before any number below it")
|
||||
w("")
|
||||
w("| node | rounds recorded | target | journal boots harvested | short? |")
|
||||
w("|---|---:|---:|---:|---|")
|
||||
short_arms = []
|
||||
for h in hosts:
|
||||
rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
|
||||
n = len(rounds)
|
||||
is_short = n < expected_rounds
|
||||
if is_short:
|
||||
short_arms.append((h["host"], n))
|
||||
w(
|
||||
f"| `{h['host']}` | {n} | {expected_rounds} | {len(h['boots'])} | "
|
||||
f"{'**YES — arm incomplete**' if is_short else 'no'} |"
|
||||
)
|
||||
w("")
|
||||
if short_arms:
|
||||
w("> **INCOMPLETE SURVEY.** " + "; ".join(f"`{h}` contributed {n} rounds" for h, n in short_arms))
|
||||
w("> Every interval below is correspondingly wider than planned.")
|
||||
w("")
|
||||
for h in hosts:
|
||||
if h.get("abort"):
|
||||
a = h["abort"]
|
||||
w(f"> **`{h['host']}` ABORTED** at round {a.get('aborted_at_round', '?')} of "
|
||||
f"{a.get('target_rounds', '?')} — {a.get('reason', 'no reason recorded')}")
|
||||
w(f"> (failing task: {a.get('failed_task', 'unknown')})")
|
||||
w("")
|
||||
|
||||
# --- 2. Per-round table ---------------------------------------------------------------
|
||||
w("## 2. Per-round results")
|
||||
w("")
|
||||
for h in hosts:
|
||||
installed = h["harvest"].get("smbios_installed_kb")
|
||||
w(f"### `{h['host']}` — {gib(installed)} GiB installed (SMBIOS)")
|
||||
w("")
|
||||
w("| round | boot id | MEMATTR | kernel read | tbl version | MemTotal GiB | usable | verdict |")
|
||||
w("|---:|---|---|---|---:|---:|---:|---|")
|
||||
for b in h["boots"]:
|
||||
rnd = b.get("round")
|
||||
label = "pre" if rnd is None else str(rnd)
|
||||
usable = b.get("mem_total_kb") if b.get("mem_total_kb") is not None else b.get("memory_available_kb")
|
||||
w(
|
||||
f"| {label} | `{(b.get('boot_id') or '?')[:8]}` | "
|
||||
f"`{b.get('memattr_address') or 'none'}` | {b.get('kernel_read')} | "
|
||||
f"{b.get('memattr_table_version') if b.get('memattr_table_version') is not None else '—'} | "
|
||||
f"{gib(usable)} | {pct(b.get('usable_fraction'))} | "
|
||||
f"{'**BAD**' if b.get('verdict') == 'bad' else b.get('verdict')} |"
|
||||
)
|
||||
w("")
|
||||
w("`round` = survey round. `pre` = a boot already in the journal before the survey;")
|
||||
w("`0` = the boot the node was found on. Both are shown for context and EXCLUDED")
|
||||
w("from every statistic below. `kernel read`: `valid` = table")
|
||||
w("parsed cleanly and the kernel logged NOTHING (healthy); `bail-version` = garbage")
|
||||
w("version > 2, kernel warned and bailed (healthy, the lucky path); `failed-map` =")
|
||||
w("the catastrophic path. Verdict is measured from MemTotal vs SMBIOS, never from")
|
||||
w("the presence of a log line.")
|
||||
w("")
|
||||
|
||||
# --- 3. p(bad) -------------------------------------------------------------------------
|
||||
w("## 3. p(bad) per node — 95% Clopper-Pearson")
|
||||
w("")
|
||||
w("| node | bad | n | p(bad) | 95% CI | reading |")
|
||||
w("|---|---:|---:|---:|---|---|")
|
||||
for h in hosts:
|
||||
rounds = [b for b in h["boots"] if b.get("round") not in (None, 0)]
|
||||
n = len(rounds)
|
||||
bad = sum(1 for b in rounds if b.get("verdict") == "bad")
|
||||
lo, hi = clopper_pearson(bad, n)
|
||||
if n == 0:
|
||||
reading = "no data"
|
||||
elif bad == 0:
|
||||
reading = f"no bad boot in {n}; p(bad) could still be as high as {hi:.2f}"
|
||||
elif bad == n:
|
||||
reading = f"bad every time in {n}; p(bad) at least {lo:.2f}"
|
||||
else:
|
||||
reading = f"p(bad) between {lo:.2f} and {hi:.2f}"
|
||||
w(
|
||||
f"| `{h['host']}` | {bad} | {n} | {f'{bad / n:.2f}' if n else '—'} | "
|
||||
f"{lo:.2f} – {hi:.2f} | {reading} |"
|
||||
)
|
||||
w("")
|
||||
indet = [b for b in surveyed if b.get("verdict") not in ("good", "bad")]
|
||||
if indet:
|
||||
w(f"> {len(indet)} boot(s) landed between the good and bad thresholds and are")
|
||||
w("> reported as `indeterminate` rather than forced into a bucket. They are")
|
||||
w("> excluded from n above. Inspect them by hand.")
|
||||
w("")
|
||||
|
||||
# --- 4. Address vs outcome -----------------------------------------------------------
|
||||
w("## 4. Is the outcome predicted by where the table landed?")
|
||||
w("")
|
||||
by_addr: dict[str, Counter] = defaultdict(Counter)
|
||||
addr_hosts: dict[str, set] = defaultdict(set)
|
||||
for b in surveyed:
|
||||
addr = b.get("memattr_address") or "none"
|
||||
by_addr[addr][b.get("verdict")] += 1
|
||||
addr_hosts[addr].add(b.get("host") or "?")
|
||||
|
||||
w("### 4a. Every address observed in the survey")
|
||||
w("")
|
||||
w("| MEMATTR | good | bad | other | nodes | verdict for this address |")
|
||||
w("|---|---:|---:|---:|---|---|")
|
||||
ambiguous = []
|
||||
for addr in sorted(by_addr, key=lambda a: (a == "none", a)):
|
||||
counts = by_addr[addr]
|
||||
good, bad = counts.get("good", 0), counts.get("bad", 0)
|
||||
other = sum(v for k, v in counts.items() if k not in ("good", "bad"))
|
||||
if good and bad:
|
||||
note = "**BOTH — address does not determine outcome**"
|
||||
ambiguous.append(addr)
|
||||
elif bad:
|
||||
note = "always bad"
|
||||
elif good:
|
||||
note = "always good"
|
||||
else:
|
||||
note = "—"
|
||||
w(
|
||||
f"| `{addr}` | {good} | {bad} | {other} | "
|
||||
f"{', '.join(sorted(h.replace('k3s-prod-', 'n') for h in addr_hosts[addr]))} | {note} |"
|
||||
)
|
||||
w("")
|
||||
|
||||
w("### 4b. Determinism")
|
||||
w("")
|
||||
if not by_addr:
|
||||
w("No surveyed boots — nothing to test.")
|
||||
elif ambiguous:
|
||||
w(f"**FALSIFIED for {len(ambiguous)} address(es):** "
|
||||
+ ", ".join(f"`{a}`" for a in ambiguous)
|
||||
+ " was seen with BOTH outcomes. The address alone cannot determine the outcome.")
|
||||
else:
|
||||
w("**Not falsified.** No address in this survey was ever seen with both outcomes —")
|
||||
w("consistent with the address determining the outcome, though with this many")
|
||||
w("distinct addresses most were seen only once, which makes the test weak.")
|
||||
singles = sum(1 for a, c in by_addr.items() if sum(c.values()) == 1)
|
||||
w(f"({singles} of {len(by_addr)} distinct addresses were observed exactly once.)")
|
||||
w("")
|
||||
|
||||
w("### 4c. Separation of the address ranges")
|
||||
w("")
|
||||
bad_addrs = sorted({b["memattr_address"] for b in surveyed
|
||||
if b.get("verdict") == "bad" and b.get("memattr_address")})
|
||||
good_addrs = sorted({b["memattr_address"] for b in surveyed
|
||||
if b.get("verdict") == "good" and b.get("memattr_address")})
|
||||
if bad_addrs and good_addrs:
|
||||
blo, bhi = int(bad_addrs[0], 16), int(bad_addrs[-1], 16)
|
||||
inside = [a for a in good_addrs if blo <= int(a, 16) <= bhi]
|
||||
w(f"- bad addresses ({len(bad_addrs)} distinct): `{bad_addrs[0]}` .. `{bad_addrs[-1]}`")
|
||||
w(f"- good addresses ({len(good_addrs)} distinct): `{good_addrs[0]}` .. `{good_addrs[-1]}`")
|
||||
w(f"- good addresses falling INSIDE the bad range: **{len(inside)}**"
|
||||
+ (" — the ranges interleave, so 'a narrow bad range' does not hold"
|
||||
if inside else " — the ranges are disjoint"))
|
||||
w("")
|
||||
w("Descriptive only. The bad set was defined after seeing the data, so this is a")
|
||||
w("picture, not a test.")
|
||||
elif not bad_addrs:
|
||||
w("No bad boot was observed in this survey, so there is no bad range to separate.")
|
||||
else:
|
||||
w("No good boot was observed in this survey.")
|
||||
w("")
|
||||
|
||||
w("### 4d. The pre-registered prediction")
|
||||
w("")
|
||||
w(f"§0b, written before this run, predicts that `{PREREGISTERED_BAD_ADDRESS}` is the")
|
||||
w("bad address (both prior failures were there; all five prior successes elsewhere).")
|
||||
w("Tested against this survey's boots only:")
|
||||
w("")
|
||||
a = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
|
||||
b_ = sum(1 for b in surveyed if b.get("memattr_address") == PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
|
||||
c = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "bad")
|
||||
d = sum(1 for b in surveyed if b.get("memattr_address") != PREREGISTERED_BAD_ADDRESS and b.get("verdict") == "good")
|
||||
w(f"| | bad | good |")
|
||||
w(f"|---|---:|---:|")
|
||||
w(f"| at `{PREREGISTERED_BAD_ADDRESS}` | {a} | {b_} |")
|
||||
w(f"| elsewhere | {c} | {d} |")
|
||||
w("")
|
||||
if a + b_ == 0:
|
||||
w(f"**Untested.** `{PREREGISTERED_BAD_ADDRESS}` never recurred in this survey, so the")
|
||||
w("prediction had no opportunity to be right or wrong. That is itself informative:")
|
||||
w("the address moves every boot, so a specific address is not a reproducible handle.")
|
||||
else:
|
||||
p = fisher_exact_2x2(a, b_, c, d)
|
||||
w(f"Fisher exact (two-sided) p = **{p:.4g}**"
|
||||
+ (" — the prediction holds." if p < 0.05 and a and not b_
|
||||
else " — not significant at 0.05." if p >= 0.05
|
||||
else " — significant, but read the cell counts before believing it."))
|
||||
w("")
|
||||
|
||||
w("### 4e. Does the address predict how the KERNEL READ the table?")
|
||||
w("")
|
||||
w("The strongest test this data can support when no bad boot occurred. `verdict` has")
|
||||
w("no variance to explain, but `kernel_read` does — `valid` and `bail-version` both")
|
||||
w("occur — and it is the step immediately upstream of the outcome in §0b's mechanism:")
|
||||
w("the kernel reads whatever bytes happen to be at the address the firmware published.")
|
||||
w("If the address determines the reading, the address is doing causal work, one step")
|
||||
w("short of the catastrophic path.")
|
||||
w("")
|
||||
reads_by_addr: dict[str, set] = defaultdict(set)
|
||||
n_by_addr: Counter = Counter()
|
||||
for b in surveyed:
|
||||
addr = b.get("memattr_address") or "none"
|
||||
reads_by_addr[addr].add(b.get("kernel_read"))
|
||||
n_by_addr[addr] += 1
|
||||
repeated = {a: r for a, r in reads_by_addr.items() if n_by_addr[a] > 1}
|
||||
inconsistent = {a: r for a, r in repeated.items() if len(r) > 1}
|
||||
distinct_reads = {r for rs in reads_by_addr.values() for r in rs}
|
||||
w(f"- distinct addresses: **{len(reads_by_addr)}**, of which "
|
||||
f"**{len(repeated)}** were observed more than once")
|
||||
w(f"- distinct kernel readings across the survey: "
|
||||
+ ", ".join(f"`{r}`" for r in sorted(distinct_reads)))
|
||||
w(f"- repeated addresses giving an INCONSISTENT reading: **{len(inconsistent)}**")
|
||||
w("")
|
||||
if len(distinct_reads) < 2:
|
||||
w("Only one reading occurred in the whole survey, so there is nothing to predict.")
|
||||
elif inconsistent:
|
||||
w("**The address does NOT determine the reading:** "
|
||||
+ "; ".join(f"`{a}` gave " + "/".join(sorted(r)) for a, r in inconsistent.items()))
|
||||
elif not repeated:
|
||||
w("No address recurred, so the test could not be run.")
|
||||
else:
|
||||
w(f"**The address determines the reading, on every address observed more than once**")
|
||||
w(f"({len(repeated)} addresses, {sum(n_by_addr[a] for a in repeated)} boots, and both")
|
||||
w("readings present in the sample). This is direct evidence for the causal chain")
|
||||
w("§0b describes: the same address reproduces the same parse, boot after boot.")
|
||||
w("It does NOT establish which addresses are catastrophic — no bad boot occurred —")
|
||||
w("only that WHERE the table lands is what decides what the kernel finds there.")
|
||||
w("")
|
||||
w("| MEMATTR | boots | kernel reading(s) |")
|
||||
w("|---|---:|---|")
|
||||
for addr in sorted(reads_by_addr, key=lambda a: (a == "none", a)):
|
||||
marker = " **INCONSISTENT**" if len(reads_by_addr[addr]) > 1 else ""
|
||||
w(f"| `{addr}` | {n_by_addr[addr]} | "
|
||||
f"{', '.join(sorted(str(r) for r in reads_by_addr[addr]))}{marker} |")
|
||||
w("")
|
||||
|
||||
# --- 5. Corroboration -------------------------------------------------------------------
|
||||
w("## 5. Corroborating evidence")
|
||||
w("")
|
||||
w("### 5a. Kernel's own reading vs the measured outcome")
|
||||
w("")
|
||||
cross: dict[tuple, int] = Counter()
|
||||
for b in surveyed:
|
||||
cross[(b.get("kernel_read"), b.get("verdict"))] += 1
|
||||
w("| kernel read | verdict | boots |")
|
||||
w("|---|---|---:|")
|
||||
for (read, verdict), count in sorted(cross.items(), key=lambda kv: (-kv[1], str(kv[0]))):
|
||||
w(f"| {read} | {verdict} | {count} |")
|
||||
w("")
|
||||
disagreements = [
|
||||
(read, verdict, count)
|
||||
for (read, verdict), count in cross.items()
|
||||
if (read == "failed-map") != (verdict == "bad")
|
||||
]
|
||||
if disagreements:
|
||||
w("> **The log line and the measurement DISAGREE** in "
|
||||
f"{sum(c for _, _, c in disagreements)} boot(s): "
|
||||
+ "; ".join(f"`{r}` yet {v}" for r, v, _ in disagreements)
|
||||
+ ". That is a finding in its own right — §0b's model says `failed-map` and only")
|
||||
w("> `failed-map` costs memory.")
|
||||
else:
|
||||
w("`failed-map` and only `failed-map` cost memory, in every boot observed —")
|
||||
w("exactly as §0b's mechanism predicts, and independent of the address question.")
|
||||
w("")
|
||||
|
||||
w("### 5b. NODE_DATA(0) placement (§0b's A/B tell)")
|
||||
w("")
|
||||
w("| verdict | NODE_DATA end address | boots |")
|
||||
w("|---|---|---:|")
|
||||
nd: Counter = Counter()
|
||||
for b in surveyed:
|
||||
line = b.get("node_data_line") or ""
|
||||
end = line.split("-")[-1].rstrip("]") if "-" in line else "?"
|
||||
nd[(b.get("verdict"), end)] += 1
|
||||
for (verdict, end), count in sorted(nd.items(), key=lambda kv: (str(kv[0][0]), kv[0][1])):
|
||||
w(f"| {verdict} | `{end}` | {count} |")
|
||||
w("")
|
||||
|
||||
# --- 6. Invariants ------------------------------------------------------------------
|
||||
w("## 6. Nothing under test changed")
|
||||
w("")
|
||||
w("| node | kernels seen | distinct cmdlines | SMBIOS installed |")
|
||||
w("|---|---|---:|---|")
|
||||
for h in hosts:
|
||||
kernels = sorted({b.get("kernel_release") for b in h["boots"] if b.get("kernel_release")})
|
||||
cmdlines = {b.get("kernel_cmdline") for b in h["boots"] if b.get("kernel_cmdline")}
|
||||
w(f"| `{h['host']}` | {', '.join(kernels) or '?'} | {len(cmdlines)} | "
|
||||
f"{gib(h['harvest'].get('smbios_installed_kb'))} GiB |")
|
||||
w("")
|
||||
w("More than one kernel or more than one cmdline on a node would void that node's arm.")
|
||||
w("")
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--artifact-dir", required=True)
|
||||
parser.add_argument("--rounds", type=int, default=20, help="rounds the survey targeted")
|
||||
parser.add_argument("--out", default=None, help="also write the report here")
|
||||
args = parser.parse_args()
|
||||
|
||||
names = discover_hosts(args.artifact_dir)
|
||||
if not names:
|
||||
print(f"No *.boots.json under {args.artifact_dir} — nothing to analyse.", file=sys.stderr)
|
||||
return 1
|
||||
hosts = [load_host(args.artifact_dir, name) for name in names]
|
||||
report = render(hosts, args.rounds)
|
||||
if args.out:
|
||||
with open(args.out, "w", encoding="utf-8") as handle:
|
||||
handle.write(report)
|
||||
sys.stdout.write(report)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,489 @@
|
||||
#!/usr/bin/env python3
|
||||
"""memattr_probe.py — read one boot's EFI Memory Attributes Table outcome off a node.
|
||||
|
||||
Part of roles/memattr_survey (infra/ansible). See that role's README.md for the
|
||||
question this answers; see infra/PROD-PROVISIONING.md §0b for the settled mechanism.
|
||||
|
||||
WHY A SCRIPT AND NOT A HANDFUL OF `command:` TASKS
|
||||
The record for one boot needs six sources cross-referenced (kernel log config-table
|
||||
line, kernel log memattr diagnostic, kernel log `Memory:` line, kernel log
|
||||
`NODE_DATA` line, /proc/meminfo, SMBIOS) and one derived verdict. Doing that in
|
||||
Jinja across six registered results is unreadable and would put the classification
|
||||
logic in a template instead of in one testable place. It also has to be exactly the
|
||||
same logic for a live boot and for a boot harvested out of the journal, which a
|
||||
single --all-boots mode gives for free.
|
||||
|
||||
MODES
|
||||
(default) one JSON record for the CURRENT boot, on stdout
|
||||
--all-boots one JSON object with a `boots` list: EVERY boot the persistent
|
||||
journal still holds, re-derived from scratch. This is the
|
||||
AUTHORITATIVE record — it survives the driver script dying mid-run.
|
||||
--status {"completed_rounds": N, "records": M} from the on-node ledger, so the
|
||||
playbook can resume without re-deriving anything.
|
||||
|
||||
--round N stamp the record with the survey round number
|
||||
--append PATH ALSO append the record to PATH as one JSON line (the on-node ledger),
|
||||
created 0644 with parents. Appended with O_APPEND in one write() so a
|
||||
concurrent reader never sees a half line.
|
||||
|
||||
VERDICT IS MEASURED, NOT INFERRED FROM A LOG LINE
|
||||
A node whose kernel reads a VALID table logs no memattr diagnostic at all and is
|
||||
perfectly healthy; a node whose kernel bails out on a garbage version > 2 logs a
|
||||
scary warning and is ALSO perfectly healthy. Only the third state — "Failed to map"
|
||||
— costs memory, and even that message is emitted by a LATER function than the one
|
||||
that did the damage. So the verdict comes from usable RAM versus SMBIOS installed
|
||||
RAM, and `kernel_read` is recorded ALONGSIDE it as an independent observation. If
|
||||
the two ever disagree, that disagreement is itself the finding.
|
||||
|
||||
EXIT STATUS is 0 whenever a record could be produced, even a mostly-null one: a field
|
||||
this script could not read comes back as null with a reason in `notes`, never as an
|
||||
empty string and never as a hard failure. A survey that aborts on one unreadable field
|
||||
loses the other 19 rounds.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
SCHEMA = "memattr-survey/1"
|
||||
|
||||
# --- Verdict thresholds -------------------------------------------------------------
|
||||
# Observed on this fleet: a good boot yields usable/installed of 0.95-0.98 (kernel code,
|
||||
# reserved firmware regions and struct page overhead account for the rest); the bad path
|
||||
# yields 0.15-0.17 (everything above the table address is reserved). Two orders of
|
||||
# separation, so the thresholds are nowhere near either cluster. Anything landing in the
|
||||
# gap is deliberately NOT called: it is a new third thing and wants human eyes.
|
||||
VERDICT_GOOD_MIN_FRACTION = 0.85
|
||||
VERDICT_BAD_MAX_FRACTION = 0.60
|
||||
|
||||
# `Memory: 2568544K/16546088K available (... 13958980K reserved ...)`
|
||||
RE_MEMORY_LINE = re.compile(
|
||||
r"^Memory:\s+(?P<available>\d+)K/(?P<present>\d+)K available\b.*?"
|
||||
r"(?:(?P<reserved>\d+)K reserved)?"
|
||||
)
|
||||
# `efi: ACPI=0x... SMBIOS=0x... MEMATTR=0xbeeb0018 MOKvar=0x... INITRD=0x...`
|
||||
RE_MEMATTR_ADDR = re.compile(r"\bMEMATTR=(?P<addr>0x[0-9a-fA-F]+)")
|
||||
# `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749`
|
||||
RE_TABLE_VERSION = re.compile(r"table version\s+(?P<version>-?\d+)")
|
||||
# `NODE_DATA(0) allocated [mem 0xbee85680-0xbeeaffff]` — per §0b the cleanest A/B tell:
|
||||
# on the bad path pgdat is forced immediately BELOW the table address instead of landing
|
||||
# at the top of RAM. Captured because it is free and it is independent evidence.
|
||||
RE_NODE_DATA = re.compile(r"^NODE_DATA\(\d+\) allocated \[mem (?P<range>[0-9a-fx]+-[0-9a-fx]+)\]")
|
||||
RE_CMDLINE = re.compile(r"^Command line:\s*(?P<cmdline>.*)$")
|
||||
# dmidecode type 17 `Size: 16 GB` / `Size: 8192 MB` / `Size: No Module Installed`
|
||||
RE_DMI_SIZE = re.compile(r"^\s*Size:\s+(?P<n>\d+)\s+(?P<unit>[kKmMgGtT]B)\s*$")
|
||||
|
||||
DMI_UNIT_KB = {"kb": 1, "mb": 1024, "gb": 1024 * 1024, "tb": 1024 * 1024 * 1024}
|
||||
|
||||
|
||||
def run(argv: list[str], timeout: int = 60) -> tuple[int, str, str]:
|
||||
"""Run a command, never raise. Returns (rc, stdout, stderr); rc 127 if absent."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
argv, capture_output=True, text=True, timeout=timeout, check=False
|
||||
)
|
||||
except FileNotFoundError:
|
||||
return 127, "", f"{argv[0]}: not found"
|
||||
except subprocess.TimeoutExpired:
|
||||
return 124, "", f"{' '.join(argv)}: timed out after {timeout}s"
|
||||
except OSError as exc: # pragma: no cover - defensive
|
||||
return 126, "", f"{' '.join(argv)}: {exc}"
|
||||
return proc.returncode, proc.stdout, proc.stderr
|
||||
|
||||
|
||||
def read_text(path: str) -> str | None:
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as handle:
|
||||
return handle.read()
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def kernel_log(boot: str) -> tuple[list[str], str | None]:
|
||||
"""Kernel ring buffer for `boot` ('0' = current, '-1' = previous, or a boot ID).
|
||||
|
||||
`-o cat` is deliberate: it yields the message VERBATIM with no host/facility
|
||||
decoration, which is what the survey wants to record, and it is stable across
|
||||
systemd versions. `-k` restricts to _TRANSPORT=kernel.
|
||||
"""
|
||||
rc, out, err = run(["journalctl", "-k", "-b", boot, "-o", "cat", "--no-pager"], timeout=180)
|
||||
if rc != 0:
|
||||
return [], f"journalctl -k -b {boot} failed (rc={rc}): {err.strip()[:200]}"
|
||||
return out.splitlines(), None
|
||||
|
||||
|
||||
def list_boots() -> tuple[list[dict], str | None]:
|
||||
"""Every boot the journal still holds, oldest first.
|
||||
|
||||
Prefers `--list-boots -o json` (systemd >= 250, so always on Debian 13) and falls
|
||||
back to parsing the table, because a survey that cannot enumerate boots has lost
|
||||
its authoritative record and must say so rather than silently reporting one boot.
|
||||
"""
|
||||
rc, out, err = run(["journalctl", "--list-boots", "-o", "json", "--no-pager"], timeout=120)
|
||||
if rc == 0:
|
||||
try:
|
||||
parsed = json.loads(out)
|
||||
except json.JSONDecodeError as exc:
|
||||
return [], f"--list-boots -o json unparseable: {exc}"
|
||||
boots = [
|
||||
{
|
||||
"index": int(entry.get("index", 0)),
|
||||
"boot_id": entry.get("boot_id"),
|
||||
"first_entry_usec": entry.get("first_entry"),
|
||||
"last_entry_usec": entry.get("last_entry"),
|
||||
}
|
||||
for entry in parsed
|
||||
]
|
||||
return sorted(boots, key=lambda b: b["index"]), None
|
||||
|
||||
rc, out, err = run(["journalctl", "--list-boots", "--no-pager"], timeout=120)
|
||||
if rc != 0:
|
||||
return [], f"journalctl --list-boots failed (rc={rc}): {err.strip()[:200]}"
|
||||
boots = []
|
||||
for line in out.splitlines():
|
||||
fields = line.split()
|
||||
if len(fields) < 2 or not re.fullmatch(r"-?\d+", fields[0]):
|
||||
continue # header row
|
||||
boots.append(
|
||||
{
|
||||
"index": int(fields[0]),
|
||||
"boot_id": fields[1],
|
||||
"first_entry_usec": None,
|
||||
"last_entry_usec": None,
|
||||
}
|
||||
)
|
||||
return sorted(boots, key=lambda b: b["index"]), None
|
||||
|
||||
|
||||
def journal_storage() -> dict:
|
||||
"""Is the journal PERSISTENT? The harvest is only authoritative if it is.
|
||||
|
||||
/var/log/journal existing is what makes journald write to disk; /run/log/journal is
|
||||
the volatile fallback and would mean every reboot erases the record.
|
||||
"""
|
||||
return {
|
||||
"var_log_journal": os.path.isdir("/var/log/journal"),
|
||||
"run_log_journal": os.path.isdir("/run/log/journal"),
|
||||
"persistent": os.path.isdir("/var/log/journal"),
|
||||
}
|
||||
|
||||
|
||||
def smbios_installed_kb() -> tuple[int | None, str | None]:
|
||||
"""Installed RAM per SMBIOS type-17 records — the GROUND TRUTH for the verdict.
|
||||
|
||||
Deliberately not /proc/meminfo (that is the number under test) and not dmesg's
|
||||
`Memory: x/yK` denominator (which is already post-e820 and post-reservation).
|
||||
Hardware does not change between boots, so one read applies to harvested boots too.
|
||||
"""
|
||||
rc, out, err = run(["dmidecode", "-t", "17"], timeout=60)
|
||||
if rc != 0:
|
||||
return None, f"dmidecode -t 17 failed (rc={rc}): {err.strip()[:200]}"
|
||||
total = 0
|
||||
seen = 0
|
||||
for line in out.splitlines():
|
||||
match = RE_DMI_SIZE.match(line)
|
||||
if not match:
|
||||
continue # "No Module Installed" and every non-Size line
|
||||
total += int(match.group("n")) * DMI_UNIT_KB[match.group("unit").lower()]
|
||||
seen += 1
|
||||
if seen == 0:
|
||||
return None, "dmidecode -t 17 reported no populated memory devices"
|
||||
return total, None
|
||||
|
||||
|
||||
def meminfo_total_kb() -> tuple[int | None, str | None]:
|
||||
text = read_text("/proc/meminfo")
|
||||
if text is None:
|
||||
return None, "/proc/meminfo unreadable"
|
||||
for line in text.splitlines():
|
||||
if line.startswith("MemTotal:"):
|
||||
return int(line.split()[1]), None
|
||||
return None, "/proc/meminfo has no MemTotal"
|
||||
|
||||
|
||||
def classify_kernel_read(memattr_addr, diagnostics: list[str]) -> tuple[str, int | None]:
|
||||
"""What did the kernel make of the table? THREE outcomes, not two (§0b).
|
||||
|
||||
Returns (classification, table_version_if_reported).
|
||||
|
||||
no-table firmware published no MEMATTR pointer at all (the dev nodes)
|
||||
valid a MEMATTR pointer AND no diagnostic => the kernel parsed a table it
|
||||
considered valid. This is SILENCE AS EVIDENCE, and it is a healthy
|
||||
state, not a missing measurement.
|
||||
bail-version garbage version > 2 => kernel warned and bailed. Memory intact.
|
||||
The LUCKY path, and it looks alarming in the log.
|
||||
failed-map the catastrophic path. The message comes from
|
||||
efi_memattr_apply_permissions() and is therefore PROOF the bogus
|
||||
memblock_reserve() already happened upstream of it.
|
||||
bogus-size a kernel >= 6.13 rejecting the table. Should never appear on 6.12;
|
||||
if it does, the kernel under test changed and the arm is void.
|
||||
"""
|
||||
version = None
|
||||
for line in diagnostics:
|
||||
match = RE_TABLE_VERSION.search(line)
|
||||
if match:
|
||||
version = int(match.group("version"))
|
||||
joined = " | ".join(diagnostics)
|
||||
if "Failed to map" in joined:
|
||||
return "failed-map", version
|
||||
if "Corrupted EFI Memory Attributes Table" in joined:
|
||||
return "bogus-size", version
|
||||
if "Unexpected EFI Memory Attributes table version" in joined:
|
||||
return "bail-version", version
|
||||
if diagnostics:
|
||||
return "diagnostic-unrecognised", version
|
||||
if memattr_addr is None:
|
||||
return "no-table", version
|
||||
return "valid", version
|
||||
|
||||
|
||||
def derive_verdict(usable_kb, installed_kb) -> tuple[str, float | None]:
|
||||
if usable_kb is None or not installed_kb:
|
||||
return "unknown", None
|
||||
fraction = usable_kb / installed_kb
|
||||
if fraction >= VERDICT_GOOD_MIN_FRACTION:
|
||||
return "good", fraction
|
||||
if fraction <= VERDICT_BAD_MAX_FRACTION:
|
||||
return "bad", fraction
|
||||
return "indeterminate", fraction
|
||||
|
||||
|
||||
def build_record(
|
||||
*,
|
||||
boot_selector: str,
|
||||
boot_meta: dict | None,
|
||||
installed_kb,
|
||||
installed_note,
|
||||
round_number,
|
||||
live: bool,
|
||||
) -> dict:
|
||||
notes: list[str] = []
|
||||
if installed_note:
|
||||
notes.append(installed_note)
|
||||
|
||||
lines, log_note = kernel_log(boot_selector)
|
||||
if log_note:
|
||||
notes.append(log_note)
|
||||
|
||||
config_table_line = None
|
||||
memattr_addr = None
|
||||
diagnostics: list[str] = []
|
||||
memory_line = None
|
||||
node_data_line = None
|
||||
cmdline = None
|
||||
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if memattr_addr is None:
|
||||
match = RE_MEMATTR_ADDR.search(stripped)
|
||||
if match and stripped.startswith("efi:"):
|
||||
memattr_addr = match.group("addr").lower()
|
||||
config_table_line = stripped
|
||||
# Every memattr-related diagnostic, verbatim. Matched on the pr_fmt prefix the
|
||||
# kernel actually uses (`efi: memattr: `) AND on the message text, so a prefix
|
||||
# change upstream cannot silently drop the line.
|
||||
if stripped.startswith("efi: memattr:") or "EFI Memory Attributes" in stripped:
|
||||
if stripped != config_table_line:
|
||||
diagnostics.append(stripped)
|
||||
if memory_line is None and stripped.startswith("Memory:"):
|
||||
if RE_MEMORY_LINE.match(stripped):
|
||||
memory_line = stripped
|
||||
if node_data_line is None:
|
||||
match = RE_NODE_DATA.match(stripped)
|
||||
if match:
|
||||
node_data_line = stripped
|
||||
if cmdline is None:
|
||||
match = RE_CMDLINE.match(stripped)
|
||||
if match:
|
||||
cmdline = match.group("cmdline").strip()
|
||||
|
||||
available_kb = present_kb = reserved_kb = None
|
||||
if memory_line:
|
||||
match = RE_MEMORY_LINE.match(memory_line)
|
||||
available_kb = int(match.group("available"))
|
||||
present_kb = int(match.group("present"))
|
||||
reserved = match.group("reserved")
|
||||
reserved_kb = int(reserved) if reserved else None
|
||||
else:
|
||||
notes.append("no `Memory: .../... available` line found in this boot's kernel log")
|
||||
|
||||
kernel_read, table_version = classify_kernel_read(memattr_addr, diagnostics)
|
||||
|
||||
# /proc/meminfo describes the CURRENT boot only. For a harvested past boot the
|
||||
# equivalent measurement is the kernel's own `Memory: available` figure, which
|
||||
# tracks MemTotal to within ~7% on both paths. Which one was used is recorded, so
|
||||
# nobody has to guess later.
|
||||
mem_total_kb, meminfo_note = (None, None)
|
||||
if live:
|
||||
mem_total_kb, meminfo_note = meminfo_total_kb()
|
||||
if meminfo_note:
|
||||
notes.append(meminfo_note)
|
||||
usable_kb = mem_total_kb if mem_total_kb is not None else available_kb
|
||||
basis = "MemTotal" if mem_total_kb is not None else "dmesg Memory: available"
|
||||
verdict, fraction = derive_verdict(usable_kb, installed_kb)
|
||||
|
||||
record = {
|
||||
"schema": SCHEMA,
|
||||
"round": round_number,
|
||||
"host": os.uname().nodename,
|
||||
"recorded_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
|
||||
"recorded_at_epoch": int(time.time()),
|
||||
"live_boot": live,
|
||||
"boot_index": (boot_meta or {}).get("index"),
|
||||
"boot_id": (boot_meta or {}).get("boot_id"),
|
||||
"boot_first_entry_usec": (boot_meta or {}).get("first_entry_usec"),
|
||||
"kernel_release": os.uname().release,
|
||||
"kernel_cmdline": cmdline,
|
||||
# --- the independent variable -------------------------------------------------
|
||||
"efi_config_table_line": config_table_line,
|
||||
"memattr_address": memattr_addr,
|
||||
"memattr_diagnostics": diagnostics,
|
||||
"memattr_diagnostic_count": len(diagnostics),
|
||||
# An explicit marker, never an empty field: "the kernel said nothing" is a
|
||||
# measurement (it read a VALID table), not a gap in the data.
|
||||
"memattr_diagnostic_present": bool(diagnostics),
|
||||
"memattr_table_version": table_version,
|
||||
"kernel_read": kernel_read,
|
||||
# --- the dependent variable ---------------------------------------------------
|
||||
"memory_line": memory_line,
|
||||
"memory_available_kb": available_kb,
|
||||
"memory_present_kb": present_kb,
|
||||
"memory_reserved_kb": reserved_kb,
|
||||
"mem_total_kb": mem_total_kb,
|
||||
"smbios_installed_kb": installed_kb,
|
||||
"usable_kb": usable_kb,
|
||||
"usable_fraction": round(fraction, 5) if fraction is not None else None,
|
||||
"verdict": verdict,
|
||||
"verdict_basis": f"{basis} vs SMBIOS installed",
|
||||
# --- corroborating evidence ---------------------------------------------------
|
||||
"node_data_line": node_data_line,
|
||||
"notes": notes,
|
||||
}
|
||||
return record
|
||||
|
||||
|
||||
def append_record(path: str, record: dict) -> str | None:
|
||||
"""One O_APPEND write of one line, so a reader never sees a torn record."""
|
||||
try:
|
||||
parent = os.path.dirname(path) or "."
|
||||
os.makedirs(parent, mode=0o755, exist_ok=True)
|
||||
payload = (json.dumps(record, sort_keys=True) + "\n").encode()
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
|
||||
try:
|
||||
os.write(fd, payload)
|
||||
os.fsync(fd) # the point of the ledger is surviving the next reboot
|
||||
finally:
|
||||
os.close(fd)
|
||||
except OSError as exc:
|
||||
return f"could not append to {path}: {exc}"
|
||||
return None
|
||||
|
||||
|
||||
def ledger_status(path: str) -> dict:
|
||||
"""Resume support: how far did a previous run get?
|
||||
|
||||
Round numbers come from the ledger itself rather than a line count, so the
|
||||
baseline round 0 record and any hand-added line cannot shift the numbering.
|
||||
"""
|
||||
rounds: list[int] = []
|
||||
records = 0
|
||||
malformed = 0
|
||||
if os.path.exists(path):
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as handle:
|
||||
for line in handle:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
records += 1
|
||||
try:
|
||||
parsed = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
malformed += 1
|
||||
continue
|
||||
value = parsed.get("round")
|
||||
if isinstance(value, int):
|
||||
rounds.append(value)
|
||||
return {
|
||||
"schema": SCHEMA,
|
||||
"ledger": path,
|
||||
"records": records,
|
||||
"malformed_records": malformed,
|
||||
"completed_rounds": max(rounds) if rounds else 0,
|
||||
"rounds_recorded": sorted(rounds),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--round", type=int, default=None, help="survey round number to stamp")
|
||||
parser.add_argument("--append", default=None, help="on-node ledger to append to")
|
||||
parser.add_argument("--all-boots", action="store_true", help="harvest every retained boot")
|
||||
parser.add_argument("--status", default=None, metavar="LEDGER", help="report resume state")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.status:
|
||||
print(json.dumps(ledger_status(args.status), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
installed_kb, installed_note = smbios_installed_kb()
|
||||
boots, boots_note = list_boots()
|
||||
storage = journal_storage()
|
||||
|
||||
if args.all_boots:
|
||||
records = []
|
||||
for boot in boots:
|
||||
# Address boots by BOOT ID, not by relative index: indices shift under the
|
||||
# harvester's feet if journald rotates mid-harvest, and the ID is what the
|
||||
# per-round ledger recorded, so the two records join cleanly.
|
||||
selector = boot["boot_id"] or str(boot["index"])
|
||||
records.append(
|
||||
build_record(
|
||||
boot_selector=selector,
|
||||
boot_meta=boot,
|
||||
installed_kb=installed_kb,
|
||||
installed_note=installed_note,
|
||||
round_number=None,
|
||||
live=(boot["index"] == 0),
|
||||
)
|
||||
)
|
||||
payload = {
|
||||
"schema": SCHEMA,
|
||||
"host": os.uname().nodename,
|
||||
"harvested_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
|
||||
"journal_storage": storage,
|
||||
"smbios_installed_kb": installed_kb,
|
||||
"boots_listed": len(boots),
|
||||
"notes": [n for n in [boots_note, installed_note] if n],
|
||||
"boots": records,
|
||||
}
|
||||
print(json.dumps(payload, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
current = next((b for b in boots if b["index"] == 0), None)
|
||||
record = build_record(
|
||||
boot_selector="0",
|
||||
boot_meta=current,
|
||||
installed_kb=installed_kb,
|
||||
installed_note=installed_note,
|
||||
round_number=args.round,
|
||||
live=True,
|
||||
)
|
||||
record["journal_storage"] = storage
|
||||
if boots_note:
|
||||
record["notes"].append(boots_note)
|
||||
if args.append:
|
||||
note = append_record(args.append, record)
|
||||
if note:
|
||||
record["notes"].append(note)
|
||||
print(json.dumps(record, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
# Controller-side analysis. Runs on localhost over the artifact directory, so it is
|
||||
# independent of whether any surveyed node is still alive — and so it can be re-run on
|
||||
# its own (`--tags analyse`) without touching hardware.
|
||||
#
|
||||
# The statistics live in files/memattr_analyse.py, not in Jinja: Clopper-Pearson and
|
||||
# Fisher belong somewhere testable, and a confidence interval assembled out of template
|
||||
# filters is a confidence interval nobody will ever check.
|
||||
|
||||
- name: Locate the survey artifacts
|
||||
ansible.builtin.find:
|
||||
paths: "{{ memattr_survey_artifact_dir }}"
|
||||
patterns: "*.boots.json"
|
||||
register: memattr_survey_artifacts
|
||||
|
||||
- name: Report that there is nothing to analyse
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
No *.boots.json under {{ memattr_survey_artifact_dir }} — no survey has produced a
|
||||
harvest yet. Run the playbook with -e memattr_survey_reboots_authorised=true.
|
||||
when: memattr_survey_artifacts.matched | int == 0
|
||||
|
||||
- name: Analyse the survey
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /usr/bin/env
|
||||
- python3
|
||||
- "{{ role_path }}/files/memattr_analyse.py"
|
||||
- "--artifact-dir"
|
||||
- "{{ memattr_survey_artifact_dir }}"
|
||||
- "--rounds"
|
||||
- "{{ memattr_survey_rounds }}"
|
||||
- "--out"
|
||||
- "{{ memattr_survey_artifact_dir }}/summary.md"
|
||||
register: memattr_survey_analysis
|
||||
changed_when: false
|
||||
when: memattr_survey_artifacts.matched | int > 0
|
||||
|
||||
- name: Show the analysis
|
||||
ansible.builtin.debug:
|
||||
msg: "{{ memattr_survey_analysis.stdout.split('\n') }}"
|
||||
when: memattr_survey_analysis is not skipped
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
# The AUTHORITATIVE record. Every boot the persistent journal still holds, re-derived from
|
||||
# scratch in one pass by the same code that produced the per-round ledger.
|
||||
#
|
||||
# WHY BOTH. The ledger is written by the driver; if the driver dies between the reboot and
|
||||
# the append, that round exists on the machine but not in the ledger. The journal does not
|
||||
# care whether the driver was alive — the boot happened, the kernel logged it, journald
|
||||
# persisted it. So the harvest can only ever be a superset of the ledger, and the analysis
|
||||
# joins the two on boot_id: rounds come from the ledger, outcomes from the harvest, and
|
||||
# any boot the ledger missed still shows up.
|
||||
#
|
||||
# This runs even for a node that aborted mid-survey, which is exactly when it matters most.
|
||||
|
||||
- name: Harvest every retained boot from the persistent journal
|
||||
ansible.builtin.script:
|
||||
cmd: memattr_probe.py --all-boots
|
||||
executable: /usr/bin/python3
|
||||
register: memattr_survey_harvest_raw
|
||||
changed_when: false # strictly a read of the journal
|
||||
|
||||
- name: Parse the harvest
|
||||
ansible.builtin.set_fact:
|
||||
memattr_survey_harvest: "{{ memattr_survey_harvest_raw.stdout | from_json }}"
|
||||
|
||||
- name: Ensure the artifact directory exists (on the controller)
|
||||
ansible.builtin.file:
|
||||
path: "{{ memattr_survey_artifact_dir }}"
|
||||
state: directory
|
||||
mode: "0755"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
# NOT run_once: the survey play uses `strategy: free` (see the playbook for why) and
|
||||
# run_once is unreliable there. `state: directory` is idempotent, so three hosts each
|
||||
# asserting the directory exists is cheaper than a conditional that might not fire.
|
||||
changed_when: false
|
||||
when: memattr_survey_write_artifacts | bool
|
||||
|
||||
- name: Write the harvested boot record
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.boots.json"
|
||||
mode: "0644"
|
||||
content: "{{ memattr_survey_harvest | to_nice_json }}"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
changed_when: false
|
||||
when: memattr_survey_write_artifacts | bool
|
||||
|
||||
- name: Mirror the final ledger to the controller
|
||||
ansible.builtin.fetch:
|
||||
src: "{{ memattr_survey_state_file }}"
|
||||
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
|
||||
flat: true
|
||||
changed_when: false
|
||||
when: memattr_survey_write_artifacts | bool
|
||||
|
||||
- name: Harvest summary
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ inventory_hostname }}: harvested {{ memattr_survey_harvest.boots | length }} boots
|
||||
from the journal
|
||||
({{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'bad') | list | length }} bad,
|
||||
{{ memattr_survey_harvest.boots | selectattr('verdict', 'equalto', 'good') | list | length }} good,
|
||||
{{ memattr_survey_harvest.boots | rejectattr('verdict', 'in', ['bad', 'good']) | list | length }} other)
|
||||
{%- if memattr_survey_aborted | default(false) | bool %}
|
||||
— NOTE: this node ABORTED ({{ memattr_survey_abort_reason }})
|
||||
{%- endif %}
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
# memattr_survey — see defaults/main.yml for the question, README.md for how to run and
|
||||
# read it, and infra/PROD-PROVISIONING.md §0b for the settled mechanism this measures.
|
||||
#
|
||||
# ORDER OF OPERATIONS
|
||||
# 1. preflight.yml — refuse to touch a node that is not prod, not idle, or has no
|
||||
# persistent journal; read the resume point; record the boot we found it on.
|
||||
# 2. round.yml x N — reboot, measure, append, mirror. Per-node fail-safe.
|
||||
# 3. harvest.yml — re-derive every retained boot from the journal (authoritative).
|
||||
#
|
||||
# The playbook splits these across plays so that preflight is any_errors_fatal (one unsafe
|
||||
# node stops everything) while the rounds are not (one dead node stops only itself), and
|
||||
# so that the controller-side analysis runs even if every node dies. This file is the
|
||||
# whole-role entry point kept for completeness; the playbook calls the stages directly.
|
||||
|
||||
- name: Preflight and baseline
|
||||
ansible.builtin.include_tasks: preflight.yml
|
||||
|
||||
- name: Run the survey rounds
|
||||
ansible.builtin.include_tasks: rounds.yml
|
||||
|
||||
- name: Harvest the authoritative record
|
||||
ansible.builtin.include_tasks: harvest.yml
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
# Safety gate. Runs with any_errors_fatal at the play level (playbooks/memattr-survey.yml),
|
||||
# so a violation on ANY node stops the entire playbook before a single reboot — the
|
||||
# correct posture when the failure mode is "rebooted a machine that was doing something".
|
||||
#
|
||||
# VERIFY, DO NOT ASSUME. The premise that makes these reboots safe is that prod carries no
|
||||
# k3s and runs nothing. That premise is checked here on every run rather than trusted from
|
||||
# a document, because the document and the machine drift independently.
|
||||
|
||||
- name: Assert the target host is in the prod group
|
||||
# The dev cluster is LIVE. This is the one check that a mistyped --limit or a copied
|
||||
# command line cannot slip past: membership is read from the inventory, not the pattern.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- memattr_survey_required_group in (group_names | default([]))
|
||||
fail_msg: >-
|
||||
REFUSING TO SURVEY {{ inventory_hostname }}: it is not a member of
|
||||
'{{ memattr_survey_required_group }}' (groups: {{ group_names | default([]) | join(', ') }}).
|
||||
This role reboots its targets. Point it at prod, or override
|
||||
memattr_survey_required_group deliberately.
|
||||
quiet: true
|
||||
|
||||
- name: Check whether any cluster or container runtime unit is active
|
||||
# `is-active` returns non-zero for inactive/unknown units, which is the expected answer
|
||||
# here, so the rc is data and not a failure.
|
||||
ansible.builtin.command:
|
||||
argv: "{{ ['systemctl', 'is-active'] + memattr_survey_forbidden_units }}"
|
||||
register: memattr_survey_units
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
|
||||
- name: Check whether k3s has been installed on this node
|
||||
ansible.builtin.stat:
|
||||
path: "{{ item }}"
|
||||
loop: "{{ memattr_survey_forbidden_paths }}"
|
||||
register: memattr_survey_paths
|
||||
changed_when: false
|
||||
|
||||
- name: Assert nothing is running on this node
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- memattr_survey_units.stdout_lines | reject('equalto', 'inactive') | reject('equalto', 'unknown') | list | length == 0
|
||||
- memattr_survey_paths.results | selectattr('stat.exists') | list | length == 0
|
||||
fail_msg: >-
|
||||
REFUSING TO REBOOT {{ inventory_hostname }} — it is not idle.
|
||||
units {{ memattr_survey_forbidden_units | zip(memattr_survey_units.stdout_lines) | map('join', '=') | join(' ') }};
|
||||
present paths: {{ memattr_survey_paths.results | selectattr('stat.exists') | map(attribute='item') | list | join(', ') | default('none', true) }}.
|
||||
The survey is only sanctioned because these boxes carry no k3s and run nothing.
|
||||
quiet: true
|
||||
|
||||
- name: Clear any abort marker from a previous run
|
||||
# A marker left by a run that has since been resumed successfully would make this run's
|
||||
# report describe the LAST failure rather than this one. The marker means "this run
|
||||
# aborted", so it is cleared at the start of every run and re-written only if it does.
|
||||
ansible.builtin.file:
|
||||
path: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
|
||||
state: absent
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
changed_when: false
|
||||
when: memattr_survey_write_artifacts | bool
|
||||
|
||||
- name: Assert forks allows every node to be surveyed concurrently
|
||||
# The survey play runs `strategy: free` so the nodes reboot in parallel, but forks is a
|
||||
# hard cap on that: with fewer forks than hosts the survey quietly serialises and takes
|
||||
# a multiple of the wall clock it should. Cheap to satisfy (`-f <n>`), so assert rather
|
||||
# than discover it three hours later.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- (lookup('ansible.builtin.config', 'DEFAULT_FORKS') | int) >= (ansible_play_hosts_all | length)
|
||||
fail_msg: >-
|
||||
forks={{ lookup('ansible.builtin.config', 'DEFAULT_FORKS') }} but this survey targets
|
||||
{{ ansible_play_hosts_all | length }} hosts. The rounds would serialise. Re-run with
|
||||
-f {{ ansible_play_hosts_all | length }} (or higher).
|
||||
quiet: true
|
||||
run_once: true
|
||||
|
||||
- name: Read the node's resume state and baseline boot
|
||||
# One call does double duty: it reports how far a previous run got (so this one resumes)
|
||||
# and, with --round 0, records the boot we found the node on. Round 0 is a free data
|
||||
# point — it is a boot nobody chose, so it is not selected for by the survey.
|
||||
ansible.builtin.script:
|
||||
cmd: >-
|
||||
memattr_probe.py --status {{ memattr_survey_state_file | quote }}
|
||||
executable: /usr/bin/python3
|
||||
register: memattr_survey_status_raw
|
||||
changed_when: false
|
||||
|
||||
- name: Record the resume point
|
||||
ansible.builtin.set_fact:
|
||||
memattr_survey_status: "{{ memattr_survey_status_raw.stdout | from_json }}"
|
||||
memattr_survey_aborted: false
|
||||
memattr_survey_abort_reason: ""
|
||||
|
||||
- name: Assert the probe schema is the one this role understands
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- memattr_survey_status.schema == 'memattr-survey/1'
|
||||
fail_msg: >-
|
||||
Probe returned schema {{ memattr_survey_status.schema | default('<none>') }};
|
||||
this role understands memattr-survey/1. Role and probe are out of step.
|
||||
quiet: true
|
||||
|
||||
- name: Probe the boot the node is currently on
|
||||
# On a FIRST run this is stamped round 0 and appended: the boot the node was already
|
||||
# sitting on is a free, unselected data point. On a RESUME it is read but NOT appended,
|
||||
# because that boot is already round N of the interrupted run and double-counting it
|
||||
# would bias the sample toward whatever outcome happened to survive the interruption.
|
||||
ansible.builtin.script:
|
||||
cmd: >-
|
||||
memattr_probe.py
|
||||
{% if memattr_survey_status.completed_rounds | int == 0 %}
|
||||
--round 0 --append {{ memattr_survey_state_file | quote }}
|
||||
{% endif %}
|
||||
executable: /usr/bin/python3
|
||||
register: memattr_survey_baseline_raw
|
||||
# The append is the experiment's own ledger, not a change to the node under test: this
|
||||
# role reports `changed` only for things that could affect a boot, and a file written
|
||||
# under /var/lib long after the kernel parsed the table cannot.
|
||||
changed_when: false
|
||||
|
||||
- name: Parse the baseline record
|
||||
ansible.builtin.set_fact:
|
||||
memattr_survey_baseline: "{{ memattr_survey_baseline_raw.stdout | from_json }}"
|
||||
|
||||
- name: Assert the journal is PERSISTENT
|
||||
# The end-of-run harvest is the authoritative record precisely because it re-derives
|
||||
# every boot from the journal. On a volatile journal (/run/log/journal only) each
|
||||
# reboot erases the evidence and the harvest would silently return one boot — a survey
|
||||
# that looks complete and holds nothing. Verified, not assumed.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- memattr_survey_baseline.journal_storage.persistent | bool
|
||||
fail_msg: >-
|
||||
{{ inventory_hostname }} has NO persistent journal (/var/log/journal missing).
|
||||
Every reboot would erase the record this survey depends on. Refusing to run.
|
||||
quiet: true
|
||||
|
||||
- name: Report the starting state
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ inventory_hostname }}: idle, prod, persistent journal.
|
||||
kernel {{ memattr_survey_baseline.kernel_release | default('?') }},
|
||||
installed {{ (memattr_survey_baseline.smbios_installed_kb | default(0) | int / 1048576) | round(1) }} GiB.
|
||||
baseline boot MEMATTR={{ memattr_survey_baseline.memattr_address | default('none') }}
|
||||
kernel_read={{ memattr_survey_baseline.kernel_read | default('?') }}
|
||||
verdict={{ memattr_survey_baseline.verdict | default('?') | upper }}.
|
||||
resuming from round {{ memattr_survey_status.completed_rounds }}
|
||||
of {{ memattr_survey_rounds }}.
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
# ONE round on ONE node: reboot, wait for it back, measure the new boot, record it twice.
|
||||
# Included in a loop from tasks/main.yml with loop_var `memattr_survey_round`.
|
||||
#
|
||||
# FAIL-SAFE, PER NODE. The block/rescue below is what keeps one dead node from ending the
|
||||
# survey: on any failure this host sets memattr_survey_aborted and every later round is
|
||||
# skipped FOR THIS HOST ONLY, while the other nodes keep going (Ansible runs the loop for
|
||||
# all hosts in lockstep, so the reboots are genuinely concurrent). The abort is reported
|
||||
# loudly here AND again in the final analysis, which counts rounds from the artifacts
|
||||
# rather than from Ansible's host state — so a node that vanishes so hard that it is
|
||||
# marked unreachable (which rescue cannot catch) still shows up as a short arm.
|
||||
|
||||
- name: "Round {{ memattr_survey_round }} — survey step"
|
||||
block:
|
||||
- name: "Round {{ memattr_survey_round }} — reboot and wait for the node to return"
|
||||
ansible.builtin.reboot:
|
||||
reboot_timeout: "{{ memattr_survey_reboot_timeout | int }}"
|
||||
connect_timeout: "{{ memattr_survey_connect_timeout | int }}"
|
||||
post_reboot_delay: "{{ memattr_survey_post_reboot_delay | int }}"
|
||||
msg: "memattr survey round {{ memattr_survey_round }} of {{ memattr_survey_rounds }}"
|
||||
|
||||
- name: "Round {{ memattr_survey_round }} — measure the new boot and append to the ledger"
|
||||
ansible.builtin.script:
|
||||
cmd: >-
|
||||
memattr_probe.py --round {{ memattr_survey_round }}
|
||||
--append {{ memattr_survey_state_file | quote }}
|
||||
executable: /usr/bin/python3
|
||||
register: memattr_survey_round_raw
|
||||
changed_when: false # see the note in preflight.yml — the ledger is not the node
|
||||
|
||||
- name: "Round {{ memattr_survey_round }} — parse the record"
|
||||
ansible.builtin.set_fact:
|
||||
memattr_survey_record: "{{ memattr_survey_round_raw.stdout | from_json }}"
|
||||
|
||||
- name: "Round {{ memattr_survey_round }} — result"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
[{{ inventory_hostname }} r{{ memattr_survey_round }}/{{ memattr_survey_rounds }}]
|
||||
MEMATTR={{ memattr_survey_record.memattr_address | default('none') }}
|
||||
read={{ memattr_survey_record.kernel_read }}
|
||||
{%- if memattr_survey_record.memattr_table_version is not none %}
|
||||
(version {{ memattr_survey_record.memattr_table_version }})
|
||||
{%- endif %}
|
||||
MemTotal={{ (memattr_survey_record.mem_total_kb | default(0) | int / 1048576) | round(2) }}G
|
||||
of {{ (memattr_survey_record.smbios_installed_kb | default(0) | int / 1048576) | round(0) | int }}G
|
||||
=> {{ memattr_survey_record.verdict | upper }}
|
||||
boot={{ memattr_survey_record.boot_id | default('?') | truncate(9, true, '') }}
|
||||
when: memattr_survey_show_progress | bool
|
||||
|
||||
# BRACES. The on-node ledger already survives a reboot; re-fetching it every round
|
||||
# means the controller also holds every completed round if the DRIVER dies, without
|
||||
# waiting for the end-of-run harvest. One tiny file per host, per-host destination
|
||||
# paths, so three concurrent fetches cannot collide.
|
||||
- name: "Round {{ memattr_survey_round }} — mirror the ledger to the controller"
|
||||
ansible.builtin.fetch:
|
||||
src: "{{ memattr_survey_state_file }}"
|
||||
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.rounds.jsonl"
|
||||
flat: true
|
||||
changed_when: false
|
||||
when: memattr_survey_write_artifacts | bool
|
||||
|
||||
rescue:
|
||||
- name: "Round {{ memattr_survey_round }} — ABORT this node, continue the others"
|
||||
ansible.builtin.set_fact:
|
||||
memattr_survey_aborted: true
|
||||
memattr_survey_abort_reason: >-
|
||||
round {{ memattr_survey_round }} failed:
|
||||
{{ ansible_failed_result.msg | default('unknown error') | trim }}
|
||||
|
||||
# The abort must survive the run, not live only in terminal scrollback. Without
|
||||
# this the report can see that an arm is SHORT but not WHY, and "0 rounds" reads
|
||||
# like a targeting mistake instead of a node that failed to come back.
|
||||
- name: "Round {{ memattr_survey_round }} — record the abort as an artifact"
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ memattr_survey_artifact_dir }}/{{ inventory_hostname }}.aborted.json"
|
||||
mode: "0644"
|
||||
content: "{{ memattr_survey_abort_record | to_nice_json }}"
|
||||
vars:
|
||||
memattr_survey_abort_record:
|
||||
schema: "memattr-survey/1"
|
||||
host: "{{ inventory_hostname }}"
|
||||
aborted_at_round: "{{ memattr_survey_round }}"
|
||||
target_rounds: "{{ memattr_survey_rounds }}"
|
||||
reason: "{{ memattr_survey_abort_reason | trim }}"
|
||||
failed_task: "{{ ansible_failed_task.name | default('unknown') }}"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
changed_when: false
|
||||
when: memattr_survey_write_artifacts | bool
|
||||
|
||||
- name: "Round {{ memattr_survey_round }} — announce the abort"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
*** SURVEY ABORTED ON {{ inventory_hostname }} at round
|
||||
{{ memattr_survey_round }}/{{ memattr_survey_rounds }}:
|
||||
{{ memattr_survey_abort_reason }}
|
||||
*** The other nodes continue. This node's arm is SHORT — the analysis will
|
||||
say so, and its p(bad) interval will be correspondingly wider.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
# The loop itself, kept separate from round.yml so the loop bounds (which encode the
|
||||
# RESUME logic) are readable in one place.
|
||||
#
|
||||
# RESUMABLE. Rounds are numbered from the on-node ledger, not counted from zero, so a
|
||||
# re-run after an interruption continues where it stopped and can never lose or duplicate
|
||||
# an earlier round. Already at or past the target count => nothing to do, and the run
|
||||
# collapses to a harvest.
|
||||
#
|
||||
# NOT AUTHORISED => NO REBOOTS. memattr_survey_reboots_authorised defaults false, which
|
||||
# makes the whole loop a no-op. Everything else (preflight, baseline probe, harvest,
|
||||
# analysis) still runs, so a dry run exercises the entire pipeline without touching a
|
||||
# single machine.
|
||||
|
||||
- name: Announce the plan
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ inventory_hostname }}:
|
||||
{%- if not (memattr_survey_reboots_authorised | bool) %}
|
||||
DRY RUN — reboots not authorised (set memattr_survey_reboots_authorised=true to arm).
|
||||
{%- elif memattr_survey_status.completed_rounds | int >= memattr_survey_rounds | int %}
|
||||
already at {{ memattr_survey_status.completed_rounds }} rounds; nothing to do.
|
||||
{%- else %}
|
||||
rounds {{ memattr_survey_status.completed_rounds | int + 1 }}..{{ memattr_survey_rounds }}
|
||||
to run ({{ memattr_survey_rounds | int - memattr_survey_status.completed_rounds | int }} reboots).
|
||||
{%- endif %}
|
||||
|
||||
- name: Survey round
|
||||
ansible.builtin.include_tasks: round.yml
|
||||
loop: >-
|
||||
{{ range(memattr_survey_status.completed_rounds | int + 1,
|
||||
memattr_survey_rounds | int + 1) | list }}
|
||||
loop_control:
|
||||
loop_var: memattr_survey_round
|
||||
label: "{{ inventory_hostname }} round {{ memattr_survey_round }}"
|
||||
# Evaluated per iteration, so the abort flag set by a rescue in one round skips every
|
||||
# round after it on THIS host and no others.
|
||||
when:
|
||||
- memattr_survey_reboots_authorised | bool
|
||||
- not (memattr_survey_aborted | bool)
|
||||
@@ -18,22 +18,170 @@ node_baseline_packages:
|
||||
# node from first baseline — package installation is the role's job, never a manual step
|
||||
# (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo).
|
||||
- smartmontools
|
||||
# --- BACKPORTS KERNEL (EFI memattr, PROD-PROVISIONING.md §0b/§3) ---------------------
|
||||
# OFF BY DEFAULT, ON FOR PROD. This is a STANDING, DELIBERATE DEVIATION from the Debian 13
|
||||
# baseline — not a one-off `apt install` someone has to remember on the next node.
|
||||
#
|
||||
# WHY: HP's R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI Memory
|
||||
# Attributes Table. Linux < 6.13 does not validate it before `memblock_reserve()`ing memory
|
||||
# from its contents, so an unlucky boot reserves everything above the table and the node
|
||||
# comes up with a fraction of its RAM (k3s-prod-2: 2.6 GiB of 16 GB, twice). Upstream commit
|
||||
# 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel,
|
||||
# first released in v6.13) adds the validation. It is NOT in v6.12 and NOT in linux-6.12.y —
|
||||
# no `Cc: stable`, no `Fixes:`, so no automatic feed will ever pick it up. There is also NO
|
||||
# kernel cmdline workaround: efi_memattr_init() runs before any `efi=` bail-out. Booting a
|
||||
# >= 6.13 kernel is the only fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
|
||||
#
|
||||
# WHY A VARIABLE AND NOT A HOSTNAME TEST. The dev nodes are Proxmox VMs whose firmware
|
||||
# publishes no MAT at all — they are genuinely unaffected and must NOT be dragged onto a
|
||||
# backports kernel as a side effect of sharing this role (R-ENV-02 is "same role, scoped
|
||||
# overrides", not "same kernel"). A `when: inventory_hostname is match('k3s-prod-')` would
|
||||
# encode the environment into the ROLE, would silently miss a fourth prod node named
|
||||
# differently, and could not be flipped for a one-off test host. So it is a plain boolean,
|
||||
# default false, turned on in inventory/group_vars/k3s_prod/vars.yml — the SAME seam
|
||||
# kube_vip_address / k3s_api_host / cluster_kubeconfig_name already use.
|
||||
#
|
||||
# SECURITY COST, STATED PLAINLY: packages in *-backports are provided AS-IS. They are NOT
|
||||
# covered by the Debian Security Team's guarantees for the main archive — no security-tracker
|
||||
# commitment, no DSAs. In exchange for that the node keeps the RAM it paid for. Accepting it
|
||||
# is the operator's recorded decision (PROD-PROVISIONING.md §0b/§3), not this role's.
|
||||
#
|
||||
# EXIT CONDITION: if 8fbe4c49c0cc ever lands in linux-6.12.y (and therefore in Debian's
|
||||
# 6.12 rebase), flip this back to false everywhere, `apt purge` the backports kernel, and
|
||||
# delete this block. Nothing else in the repo depends on it.
|
||||
node_baseline_backports_kernel: false
|
||||
# The suite, mirror and components the source line is built from. Split out so a mirror
|
||||
# change is a variable, not an edit to a heredoc in tasks/main.yml.
|
||||
node_baseline_backports_suite: trixie-backports
|
||||
node_baseline_backports_mirror: "http://deb.debian.org/debian"
|
||||
# Matches /etc/apt/sources.list on these nodes (main + non-free-firmware). Deliberately NOT
|
||||
# contrib/non-free: this deviation exists to get ONE fixed kernel, and widening the archive
|
||||
# surface beyond what the node already trusts is a separate decision nobody has taken.
|
||||
node_baseline_backports_components:
|
||||
- main
|
||||
- non-free-firmware
|
||||
# The METApackage, not a pinned ABI (e.g. linux-image-6.19.14+deb13-amd64). A frozen ABI
|
||||
# would stop receiving even backports' own updates the moment the ABI rolls, leaving the
|
||||
# node on a kernel nothing patches — strictly worse than the risk this deviation accepts.
|
||||
# The metapackage tracks whatever backports currently ships (>= 6.13 by construction, and
|
||||
# far above it in practice), and `ButAutomaticUpgrades: yes` in the backports Release file
|
||||
# means `apt upgrade` keeps it current once installed. The EXACT version each node landed on
|
||||
# is recorded on the node itself — see node_baseline_backports_kernel_stamp.
|
||||
node_baseline_backports_kernel_packages:
|
||||
- linux-image-amd64
|
||||
# Provenance/rollback note written on the node. An operator standing in front of a prod box
|
||||
# with no repo checkout must be able to answer "why is this machine not on the distro
|
||||
# kernel, and how do I undo it?" from the machine itself.
|
||||
node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel
|
||||
|
||||
node_baseline_kernel_modules:
|
||||
- overlay
|
||||
- br_netfilter
|
||||
- iscsi_tcp
|
||||
- dm_crypt
|
||||
- nfs
|
||||
|
||||
# --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
|
||||
# The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
|
||||
# a variable happens to be set. Two enums, defaulting to dev's two-dedicated-disks shape:
|
||||
#
|
||||
# node_baseline_longhorn_source:
|
||||
# device dedicated block device at node_baseline_longhorn_device (DEV: the
|
||||
# tofu-provisioned virtio disk /dev/vdb). mkfs + mount by label.
|
||||
# claim-mountpoint re-purpose the partition the node ALREADY has mounted at
|
||||
# node_baseline_longhorn_claim_mountpoint (PROD: /srv). The device
|
||||
# path is discovered ON THE NODE — it is /dev/sda5 on prod nodes 1
|
||||
# and 3 but /dev/nvme0n1p5 on node 2, so no group-wide device value
|
||||
# is correct and a hardcoded one would eventually mkfs the wrong
|
||||
# partition on a future node. Destructive; heavily guarded — read
|
||||
# tasks/disks.yml before changing anything here.
|
||||
#
|
||||
# node_baseline_etcd_source:
|
||||
# device dedicated block device at node_baseline_etcd_device (DEV:
|
||||
# /dev/vdc, docs/plans/ssd-etcd-isolation.md — etcd WAL fsync
|
||||
# isolated from Longhorn rebuild I/O after dev was burned by HDD
|
||||
# contention).
|
||||
# root-filesystem etcd lives on the OS filesystem, with NO dedicated device and
|
||||
# NO claimed data partition under it. Kept as a valid, supported
|
||||
# shape; NOT what prod uses any more (see below).
|
||||
# shared-data-partition etcd lives inside node_baseline_k3s_data_dir, which sits on the
|
||||
# CLAIMED data partition (PROD). No dedicated device, but also NOT
|
||||
# the OS filesystem: the whole point of the 2026-09-04 layout
|
||||
# revision is that /var (11.8 GiB) can hold neither the container
|
||||
# image store nor etcd's WAL. See infra/PROD-PROVISIONING.md §2.
|
||||
#
|
||||
# Every one of these is a FIRST-CLASS, explicitly-configured state — an operator decision
|
||||
# recorded in infra/PROD-PROVISIONING.md §2 — and NOT the same thing as an unset device
|
||||
# var, which still fails the `disks` assert.
|
||||
node_baseline_longhorn_source: device
|
||||
node_baseline_longhorn_device: /dev/vdb
|
||||
node_baseline_longhorn_mount: /var/lib/longhorn
|
||||
node_baseline_longhorn_fs_label: longhorn
|
||||
# --- The DATA PARTITION: mountpoint + filesystem label -------------------------------
|
||||
# NEUTRAL NAMES ON PURPOSE (revised 2026-09-04). On prod this ONE partition now serves TWO
|
||||
# tenants — Longhorn's replica store AND k3s's whole data dir (etcd + the containerd image
|
||||
# store) — so neither the mountpoint nor the label may name only one of them. Prod overrides
|
||||
# both in group_vars/k3s_prod/vars.yml (`/data`, label `k3s-data`).
|
||||
#
|
||||
# THE DEFAULTS BELOW ARE DEV'S LIVE VALUES AND MUST NOT CHANGE. The dev cluster has 18
|
||||
# Longhorn volumes whose replicas live on the partition labelled `longhorn` mounted at
|
||||
# /var/lib/longhorn; dev's etcd has its own dedicated device and dev's k3s data dir stays
|
||||
# at k3s's own default. Dev's mountpoint IS Longhorn's dataPath there (one tenant, one
|
||||
# partition), which is why node_baseline_longhorn_data_path below defaults to the mount.
|
||||
node_baseline_data_mount: /var/lib/longhorn
|
||||
node_baseline_data_fs_label: longhorn
|
||||
# Longhorn's dataPath — the directory Longhorn puts replicas in. On dev that IS the mount
|
||||
# root (single tenant). On prod it is a SUBDIRECTORY of the mount, because k3s owns a
|
||||
# sibling subdirectory of the same partition. Must agree with the LONGHORN_DATA_PATH knob
|
||||
# in clusters/base/cluster-vars.yaml (patched per overlay) — that is the Helm side of the
|
||||
# same fact, and the two are checked against each other by nothing but this comment.
|
||||
node_baseline_longhorn_data_path: "{{ node_baseline_data_mount }}"
|
||||
# k3s's --data-dir. Defers to the CLUSTER-WIDE `k3s_data_dir` when an environment declares
|
||||
# one (group_vars), and otherwise restates k3s's own built-in default. It is deliberately
|
||||
# NOT a second independent variable: roles/k3s_server reads `k3s_data_dir` to build the
|
||||
# install flags, this role reads the same value to create the directory and to place etcd,
|
||||
# and one line in group_vars/k3s_prod/vars.yml moves BOTH. Lazy Jinja means the group_vars
|
||||
# value (higher precedence than any role default) is what actually resolves here, even
|
||||
# though roles/k3s_server is not in the node-baseline play; the default() keeps
|
||||
# node-baseline.yml runnable standalone.
|
||||
node_baseline_k3s_data_dir: "{{ k3s_data_dir | default('/var/lib/rancher/k3s') }}"
|
||||
# Only consulted when node_baseline_longhorn_source is `claim-mountpoint`. Empty by
|
||||
# default so a dev host cannot enter the claim path even by accident.
|
||||
node_baseline_longhorn_claim_mountpoint: ""
|
||||
# --- ONE-TIME TRANSITION KNOBS (delete once the fleet is converged) ------------------
|
||||
# A node claimed by an EARLIER version of this role carries the OLD label at the OLD
|
||||
# mountpoint, and the idempotency pivot (`blkid -L <current label>`) would therefore read
|
||||
# "not claimed yet" and walk into the greenfield claim path — which then fails closed at
|
||||
# "the claim mountpoint must actually be a mountpoint", because /srv is long gone. These
|
||||
# two vars declare the state to migrate FROM. Empty by default: dev's label and mountpoint
|
||||
# are NOT changing, so dev has nothing to transition and cannot enter the transition path.
|
||||
node_baseline_data_retired_fs_label: ""
|
||||
node_baseline_data_retired_mount: ""
|
||||
# Sanity floor for the claim path: a data partition smaller than this is not "a small
|
||||
# store", it is the wrong partition. Prod's /srv is 415-425 GiB.
|
||||
node_baseline_longhorn_min_size_gb: 100
|
||||
# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
|
||||
# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
|
||||
# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
|
||||
# device renames (matches the Longhorn LABEL= approach).
|
||||
# device renames (matches the data partition's LABEL= approach).
|
||||
node_baseline_etcd_source: device
|
||||
node_baseline_etcd_device: /dev/vdc
|
||||
node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
|
||||
# DERIVED, never restated: etcd's db always lives at <k3s data dir>/server/db, so moving
|
||||
# the data dir moves etcd with it and the two cannot drift. Dev resolves to the unchanged
|
||||
# /var/lib/rancher/k3s/server/db (where dev's dedicated etcd SSD is mounted); prod resolves
|
||||
# to /data/k3s/server/db, inside the claimed partition.
|
||||
node_baseline_etcd_mount: "{{ node_baseline_k3s_data_dir }}/server/db"
|
||||
node_baseline_etcd_fs_label: k3s-etcd
|
||||
# Mounts whose backing devices the claim path must NEVER touch. Used to build the
|
||||
# protected-device set from the node's own gathered facts (plus live swap), so a node
|
||||
# partitioned differently from the fleet is judged on its own layout rather than against
|
||||
# a hardcoded list of partition names.
|
||||
node_baseline_protected_mounts:
|
||||
- /
|
||||
- /boot
|
||||
- /boot/efi
|
||||
- /usr
|
||||
- /var
|
||||
- /home
|
||||
- /tmp
|
||||
|
||||
# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
|
||||
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
|
||||
|
||||
@@ -0,0 +1,642 @@
|
||||
---
|
||||
# =====================================================================================
|
||||
# node_baseline — DATA DISKS. Every task here carries the `disks` tag (applied once by
|
||||
# the `include_tasks` in main.yml), so `--tags packages` still skips the whole file.
|
||||
# Split into its own file rather than living in main.yml because it is the one part of
|
||||
# the role that DESTRUCTIVELY claims storage, and it deserves to be read on its own.
|
||||
#
|
||||
# TWO ENVIRONMENTS, TWO SHAPES — and the shape is DECLARED, never inferred:
|
||||
#
|
||||
# dev (group_vars/all.yml, role defaults)
|
||||
# node_baseline_longhorn_source: device -> mkfs /dev/vdb, mount LABEL=longhorn
|
||||
# at /var/lib/longhorn (== dataPath)
|
||||
# node_baseline_etcd_source: device -> mkfs /dev/vdc, mount LABEL=k3s-etcd
|
||||
# k3s_data_dir: (k3s default, /var/lib/rancher/k3s)
|
||||
# THREE devices: OS, Longhorn, etcd. Two tofu-provisioned virtio disks exist for
|
||||
# exactly this purpose. NOTHING about dev changes in the 2026-09-04 revision.
|
||||
#
|
||||
# prod (group_vars/k3s_prod/vars.yml — operator ruling, PROD-PROVISIONING.md §2)
|
||||
# node_baseline_longhorn_source: claim-mountpoint (claim mountpoint /srv)
|
||||
# node_baseline_data_mount: /data (LABEL=k3s-data)
|
||||
# node_baseline_etcd_source: shared-data-partition
|
||||
# k3s_data_dir: /data/k3s
|
||||
# ONE physical disk per node, already fully partitioned by the Debian installer. The
|
||||
# ~424 GB `/srv` partition is claimed and mounted at a NEUTRAL mountpoint, then split
|
||||
# between TWO tenants: /data/longhorn (Longhorn replicas) + /data/k3s (the whole k3s
|
||||
# data dir: etcd AND the containerd image store).
|
||||
#
|
||||
# WHY THE PARTITION IS NO LONGER LONGHORN-ONLY (revised 2026-09-04, PROD-PROVISIONING.md
|
||||
# §2). Prod's `/var` is a separate partition of just 11.8 GiB. k3s's default data dir puts
|
||||
# BOTH etcd's WAL and the containerd image store on it. Measured on the fully-deployed dev
|
||||
# cluster, containerd alone is 21-25 GB per node and /var totals 58-70 GB — so prod would
|
||||
# have filled /var during its FIRST image pulls, and a full /var means etcd cannot fsync its
|
||||
# WAL. That is cluster death, not degradation. Moving k3s's data dir onto the 424 GB
|
||||
# partition is the fix; the partition therefore serves two tenants, and neither its
|
||||
# mountpoint nor its filesystem label may name only one of them.
|
||||
#
|
||||
# WHY /data AND LABEL k3s-data.
|
||||
# * `/data` — short, neutral, tenant-agnostic, and NOT under /var (the whole point).
|
||||
# Deliberately NOT `/srv`: /srv is the CLAIM mountpoint this role discovers the device
|
||||
# at, and the idempotency contract is that /srv is gone from /proc/mounts and
|
||||
# /etc/fstab once the claim has happened. Re-using it as the destination would make
|
||||
# "claimed" and "not yet claimed" indistinguishable.
|
||||
# * label `k3s-data`, deliberately NOT the bare `data`: guard #4 in the assert wall below
|
||||
# accepts a partition whose label is empty OR ALREADY OURS, and its own comment names
|
||||
# `data` as an example of somebody else's named volume. Labelling ours `data` would make
|
||||
# a stranger's `data` volume look already-claimed and weaken that guard. `k3s-data` is
|
||||
# cluster-scoped rather than tenant-scoped and matches the existing `k3s-etcd` label
|
||||
# convention in this repo.
|
||||
#
|
||||
# WHY A "SOURCE" ENUM INSTEAD OF A BLANK VARIABLE. "etcd has no dedicated disk" and
|
||||
# "somebody forgot to set node_baseline_etcd_device" look identical if the signal is an
|
||||
# empty string, and the second one must never silently succeed. `root-filesystem` and
|
||||
# `shared-data-partition` are explicit, greppable statements of intent that the assert
|
||||
# below accepts; an unset or placeholder device is still a hard failure. The two differ in
|
||||
# a way that matters: `root-filesystem` says etcd is on the OS filesystem, and
|
||||
# `shared-data-partition` says it is on the claimed data partition. Prod was the FORMER
|
||||
# until 2026-09-04 and is now the LATTER — and the role ASSERTS which one is true rather
|
||||
# than printing a claim it never checked (see the etcd section).
|
||||
#
|
||||
# WHY DISCOVERY INSTEAD OF A PER-HOST DEVICE PATH. The prod partition is /dev/sda5 on
|
||||
# nodes 1 and 3 (SATA) but /dev/nvme0n1p5 on node 2 (NVMe), so no single group value is
|
||||
# correct — and per-host `host_vars` would encode a *guess about hardware* into inventory
|
||||
# that goes stale the moment a node is re-imaged, a disk is replaced, or a fourth node
|
||||
# arrives with a different controller. Worse, a fleet-wide `/dev/sda5` would eventually
|
||||
# point at the wrong partition on some future node and mkfs it. So the role asks the NODE
|
||||
# which block device is mounted at the claim mountpoint, and then proves that answer is
|
||||
# safe (see the assert wall below) before touching it. The inventory declares INTENT
|
||||
# ("the partition currently mounted at /srv"); the node supplies the FACT.
|
||||
# =====================================================================================
|
||||
|
||||
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet declared
|
||||
# its data-disk layout. Two failure shapes are caught:
|
||||
# * an undeclared/typo'd source enum, and
|
||||
# * source: device with a device path that is a PLACEHOLDER rather than a real device.
|
||||
# group_vars/k3s_prod/vars.yml deliberately carries `/dev/NOT-A-DEVICE-…` for both device
|
||||
# vars now that prod uses neither: if anyone ever flips a source back to `device` without
|
||||
# also supplying a real path, they get this assert instead of dev's /dev/vdb (which does
|
||||
# not exist on bare metal — a confusing failure that looks like a wiring bug). `BLOCKED`
|
||||
# stays in the token list because that was the pre-decision placeholder and may still be
|
||||
# carried by other environments/branches.
|
||||
- name: Assert this environment's data-disk layout has actually been declared
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_longhorn_source in ['device', 'claim-mountpoint']
|
||||
- node_baseline_etcd_source in ['device', 'root-filesystem', 'shared-data-partition']
|
||||
# `shared-data-partition` is only coherent when there IS a claimed data partition to
|
||||
# share, and only when the k3s data dir actually sits INSIDE it. Catch the
|
||||
# contradiction here, in inventory terms, rather than three sections later when etcd
|
||||
# has silently landed back on the OS filesystem.
|
||||
- >-
|
||||
node_baseline_etcd_source != 'shared-data-partition'
|
||||
or node_baseline_longhorn_source == 'claim-mountpoint'
|
||||
- >-
|
||||
node_baseline_etcd_source != 'shared-data-partition'
|
||||
or node_baseline_k3s_data_dir.startswith(node_baseline_data_mount ~ '/')
|
||||
# The transition knobs are a matched PAIR: a retired label with no retired mountpoint
|
||||
# (or vice versa) cannot describe a state to migrate from.
|
||||
- >-
|
||||
(node_baseline_data_retired_fs_label | length > 0)
|
||||
== (node_baseline_data_retired_mount | length > 0)
|
||||
- >-
|
||||
node_baseline_longhorn_source != 'device'
|
||||
or not (node_baseline_longhorn_device is search('BLOCKED|NOT-A-DEVICE'))
|
||||
- >-
|
||||
node_baseline_etcd_source != 'device'
|
||||
or not (node_baseline_etcd_device is search('BLOCKED|NOT-A-DEVICE'))
|
||||
- >-
|
||||
node_baseline_longhorn_source != 'claim-mountpoint'
|
||||
or (node_baseline_longhorn_claim_mountpoint | length) > 0
|
||||
fail_msg: >-
|
||||
This host's data-disk layout is not declared.
|
||||
longhorn_source={{ node_baseline_longhorn_source }}
|
||||
(device={{ node_baseline_longhorn_device }},
|
||||
claim_mountpoint={{ node_baseline_longhorn_claim_mountpoint | default('') }}),
|
||||
etcd_source={{ node_baseline_etcd_source }}
|
||||
(device={{ node_baseline_etcd_device }}).
|
||||
Valid Longhorn sources: `device` (a dedicated block device, dev's shape) or
|
||||
`claim-mountpoint` (re-purpose the partition already mounted at
|
||||
node_baseline_longhorn_claim_mountpoint, prod's shape). Valid etcd sources:
|
||||
`device`, `root-filesystem` (etcd on the OS filesystem, no claimed partition under
|
||||
it) or `shared-data-partition` (etcd inside node_baseline_k3s_data_dir on the CLAIMED
|
||||
data partition — prod's shape since 2026-09-04, which additionally requires
|
||||
longhorn_source `claim-mountpoint` and a k3s data dir under
|
||||
node_baseline_data_mount={{ node_baseline_data_mount }};
|
||||
k3s_data_dir={{ node_baseline_k3s_data_dir }}). Every one of these is an EXPLICIT
|
||||
operator decision recorded in infra/PROD-PROVISIONING.md §2, not an omission.
|
||||
Declare it in the environment's group_vars and record the reasoning in
|
||||
§2 in the SAME change. To install ONLY the package floor meanwhile, re-run with
|
||||
`--tags packages`, which skips every disk task including this assert.
|
||||
quiet: true
|
||||
|
||||
# =====================================================================================
|
||||
# THE DATA PARTITION (prod: Longhorn store + k3s data dir; dev: Longhorn store only)
|
||||
# =====================================================================================
|
||||
|
||||
# THE IDEMPOTENCY PIVOT for the claim path, and the reason a second run is a no-op
|
||||
# rather than a reformat. Once the claim has happened the partition carries the ext4
|
||||
# label node_baseline_data_fs_label and is mounted at node_baseline_data_mount — the claim
|
||||
# mountpoint (/srv) is gone from both /proc/mounts and /etc/fstab, so "discover what is
|
||||
# mounted at /srv" would find nothing on run 2. Asking `blkid -L <label>` instead answers
|
||||
# "has this node already been claimed?" from the durable, on-disk evidence: rc 0 + a device
|
||||
# path means converged, rc != 0 means the claim has not been made. Everything destructive
|
||||
# below is gated on rc != 0.
|
||||
- name: "Data partition: look for a filesystem already carrying the data-partition label"
|
||||
ansible.builtin.command: "blkid -L {{ node_baseline_data_fs_label }}"
|
||||
register: node_baseline_lh_labelled
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
|
||||
# --- The TRANSITION (2026-09-04 layout revision; one-time, then this whole block is dead) --
|
||||
# WHY THIS EXISTS. The idempotency pivot above is "does the CURRENT label exist?". When the
|
||||
# label ITSELF changes, an already-claimed node answers "no" and the greenfield claim path
|
||||
# takes over — which then fails closed at "the claim mountpoint must actually be a
|
||||
# mountpoint", because /srv was consumed by the previous claim. Fail-closed is correct
|
||||
# behaviour but it is not convergence, and the prod nodes are in exactly that state today:
|
||||
# claimed under LABEL=longhorn at /var/lib/longhorn by the previous version of this role.
|
||||
# So the role must handle the TRANSITION explicitly instead of pretending prod is greenfield.
|
||||
#
|
||||
# WHAT IT DOES: unmount the retired mountpoint -> drop its fstab entry -> re-stamp the
|
||||
# partition with the new label. It never touches the DATA and never reformats: e2label is a
|
||||
# metadata write, exactly as in the claim path below. The common mount task then mounts
|
||||
# LABEL=<new> at the new mountpoint, and the node is converged. On every subsequent run
|
||||
# `blkid -L <new label>` succeeds and none of this is reachable.
|
||||
#
|
||||
# THREE INDEPENDENT REASONS THE LIVE DEV CLUSTER CANNOT ENTER THIS PATH — it has 18 Longhorn
|
||||
# volumes whose replicas live on the partition labelled `longhorn` at /var/lib/longhorn, and
|
||||
# unmounting it would orphan every one of them:
|
||||
# 1. node_baseline_data_retired_fs_label defaults to "" and is set ONLY in
|
||||
# group_vars/k3s_prod/vars.yml. No retired label declared => no transition, ever.
|
||||
# 2. Gated on node_baseline_longhorn_source == 'claim-mountpoint'. Dev is `device`, so
|
||||
# the same structural gate that keeps dev out of the claim path keeps it out of this one.
|
||||
# 3. The emptiness assert below. Even if 1 and 2 were both defeated, a partition with
|
||||
# replica directories on it fails closed and nothing is unmounted.
|
||||
# Dev's label and mountpoint are also simply NOT CHANGING, so there is nothing to migrate.
|
||||
- name: "Data partition (transition): look for a filesystem still carrying the RETIRED label"
|
||||
ansible.builtin.command: "blkid -L {{ node_baseline_data_retired_fs_label }}"
|
||||
register: node_baseline_lh_retired_labelled
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when:
|
||||
- node_baseline_longhorn_source == 'claim-mountpoint'
|
||||
- node_baseline_data_retired_fs_label | length > 0
|
||||
- node_baseline_lh_labelled.rc != 0
|
||||
|
||||
- name: "Data partition: decide between TRANSITION, CLAIM, and already-converged"
|
||||
ansible.builtin.set_fact:
|
||||
# A node claimed under the RETIRED label. Mutually exclusive with the claim below.
|
||||
node_baseline_lh_transition_needed: >-
|
||||
{{ node_baseline_longhorn_source == 'claim-mountpoint'
|
||||
and node_baseline_data_retired_fs_label | length > 0
|
||||
and node_baseline_lh_labelled.rc != 0
|
||||
and (node_baseline_lh_retired_labelled.rc | default(1)) == 0 }}
|
||||
# True only for a claim-mountpoint environment that has NOT been claimed under EITHER
|
||||
# label. A `device` environment (dev) can never enter the claim path at all — that is
|
||||
# the structural guarantee that no generalisation of prod's behaviour can reach the
|
||||
# LIVE dev nodes and their /dev/vdb Longhorn replicas.
|
||||
node_baseline_lh_claim_needed: >-
|
||||
{{ node_baseline_longhorn_source == 'claim-mountpoint'
|
||||
and node_baseline_lh_labelled.rc != 0
|
||||
and (node_baseline_lh_retired_labelled.rc | default(1)) != 0 }}
|
||||
|
||||
- name: "Data partition (transition): resolve what is mounted at the retired mountpoint"
|
||||
ansible.builtin.command: >-
|
||||
findmnt --noheadings --first-only --output SOURCE
|
||||
--mountpoint {{ node_baseline_data_retired_mount }}
|
||||
register: node_baseline_lh_retired_findmnt
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
# Read the retired mountpoint's top level, same contract as the claim path: `lost+found` is
|
||||
# not data, hidden files cannot hide. THIS is the assert that makes the transition safe —
|
||||
# an unmount of a populated Longhorn store would orphan every replica on the node.
|
||||
- name: "Data partition (transition): read the contents of the retired mountpoint"
|
||||
ansible.builtin.find:
|
||||
paths: "{{ node_baseline_data_retired_mount }}"
|
||||
file_type: any
|
||||
hidden: true
|
||||
recurse: false
|
||||
excludes: ['lost+found']
|
||||
register: node_baseline_lh_retired_contents
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
- name: "Data partition (transition): prove this node is safe to re-label"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
# 1. The retired mountpoint is STILL A MOUNT, and it is backed by the very device the
|
||||
# retired label names. The only state this role knows how to transition is the one
|
||||
# the previous version of it produced; anything else is an operator decision.
|
||||
- (node_baseline_lh_retired_findmnt.rc | default(1)) == 0
|
||||
- >-
|
||||
(node_baseline_lh_retired_findmnt.stdout | default('') | trim)
|
||||
== (node_baseline_lh_retired_labelled.stdout | default('') | trim)
|
||||
# 2. It is EMPTY. Nothing is being orphaned. A populated store means either this is
|
||||
# not the node we think it is, or a tenant has already written data — in both cases
|
||||
# unmounting and re-labelling under it is destructive and the role must stop.
|
||||
- (node_baseline_lh_retired_contents.files | length) == 0
|
||||
fail_msg: >-
|
||||
REFUSING to transition this node's data partition from
|
||||
LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
|
||||
to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}.
|
||||
device by retired label='{{ node_baseline_lh_retired_labelled.stdout | default('') | trim }}',
|
||||
device mounted at the retired mountpoint='{{ node_baseline_lh_retired_findmnt.stdout | default('') | trim }}'
|
||||
(findmnt rc={{ node_baseline_lh_retired_findmnt.rc | default('n/a') }}),
|
||||
non-lost+found entries={{ node_baseline_lh_retired_contents.files | default([]) | length }}
|
||||
{{ node_baseline_lh_retired_contents.files | default([]) | map(attribute='path') | list }}.
|
||||
If the partition holds DATA, this transition would orphan it — Longhorn replicas are
|
||||
not recoverable from an unmounted, re-labelled filesystem. Migrate the data
|
||||
deliberately, or drop node_baseline_data_retired_fs_label and leave the node alone.
|
||||
Do NOT relax this assert.
|
||||
quiet: true
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
- name: "Data partition (transition): report exactly what is about to change"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
Transitioning {{ node_baseline_lh_retired_labelled.stdout | trim }} (verified empty)
|
||||
from LABEL={{ node_baseline_data_retired_fs_label }} at {{ node_baseline_data_retired_mount }}
|
||||
to LABEL={{ node_baseline_data_fs_label }} at {{ node_baseline_data_mount }}:
|
||||
unmount -> drop its fstab entry -> relabel -> re-mount at the new path.
|
||||
No filesystem is created and no data is moved.
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
# e2label refuses to run on a mounted filesystem, and the fstab entry must go in the same
|
||||
# breath or the next boot re-mounts the partition at the RETIRED path underneath the new one.
|
||||
- name: "Data partition (transition): unmount the retired mountpoint"
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_data_retired_mount }}"
|
||||
state: unmounted
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
- name: "Data partition (transition): remove the retired mountpoint's /etc/fstab entry"
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_data_retired_mount }}"
|
||||
state: absent_from_fstab
|
||||
backup: true
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
- name: "Data partition (transition): re-stamp the partition with the new filesystem label"
|
||||
ansible.builtin.command: >-
|
||||
e2label {{ node_baseline_lh_retired_labelled.stdout | trim }}
|
||||
{{ node_baseline_data_fs_label }}
|
||||
changed_when: true
|
||||
when: node_baseline_lh_transition_needed | bool
|
||||
|
||||
# --- The claim (prod, first run only) -------------------------------------------------
|
||||
# Ask the node, do not assume. `--mountpoint` (not `--target`) matches ONLY an exact
|
||||
# mountpoint: if /srv is not itself a mount, findmnt exits 1 rather than silently
|
||||
# returning the device behind `/` — which is precisely the mistake that would hand the
|
||||
# root partition to e2label.
|
||||
- name: "Longhorn (claim): resolve the block device mounted at the claim mountpoint"
|
||||
ansible.builtin.command: >-
|
||||
findmnt --noheadings --first-only --output SOURCE
|
||||
--mountpoint {{ node_baseline_longhorn_claim_mountpoint }}
|
||||
register: node_baseline_lh_claim_findmnt
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): the claim mountpoint must actually be a mountpoint"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_lh_claim_findmnt.rc == 0
|
||||
- (node_baseline_lh_claim_findmnt.stdout | trim) is match('^/dev/')
|
||||
fail_msg: >-
|
||||
node_baseline_longhorn_source is `claim-mountpoint` and no filesystem is claimed
|
||||
yet, but nothing is mounted at {{ node_baseline_longhorn_claim_mountpoint }} on
|
||||
this node (findmnt rc={{ node_baseline_lh_claim_findmnt.rc | default('n/a') }},
|
||||
stdout='{{ node_baseline_lh_claim_findmnt.stdout | default('') | trim }}').
|
||||
The role will NOT go looking for "a big partition" to claim instead. Either the
|
||||
node is partitioned differently from the fleet — in which case fix the node or give
|
||||
this host its own declaration — or the claim already happened and the label was
|
||||
removed by hand.
|
||||
quiet: true
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): inspect the candidate partition"
|
||||
ansible.builtin.command: >-
|
||||
lsblk --json --nodeps --bytes
|
||||
--output PATH,TYPE,FSTYPE,LABEL,SIZE
|
||||
{{ node_baseline_lh_claim_findmnt.stdout | trim }}
|
||||
register: node_baseline_lh_claim_lsblk
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# The devices this role must never, under any circumstances, hand to e2label. Built from
|
||||
# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
|
||||
# NODE's account of what is load-bearing, not a hardcoded list of partition names.
|
||||
- name: "Longhorn (claim): list the devices currently backing active swap"
|
||||
ansible.builtin.command: swapon --noheadings --show=NAME
|
||||
register: node_baseline_lh_swapdevs
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# Read the mountpoint's top level. `excludes: lost+found` because every freshly-mkfs'd
|
||||
# ext4 has one and it is not data. hidden: true so a dotfile cannot hide from the check.
|
||||
- name: "Longhorn (claim): read the contents of the mountpoint to be claimed"
|
||||
ansible.builtin.find:
|
||||
paths: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
||||
file_type: any
|
||||
hidden: true
|
||||
recurse: false
|
||||
excludes: ['lost+found']
|
||||
register: node_baseline_lh_claim_contents
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# THE GUARD WALL. Five independent conditions, every one of which must hold before the
|
||||
# role modifies a filesystem it did not create. Any single failure stops the play with
|
||||
# the evidence in the message — nothing is unmounted, relabelled or mounted first.
|
||||
- name: "Longhorn (claim): prove the candidate partition is safe to claim"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
# 1. It is a real block device — a partition or a whole disk. Catches a claim
|
||||
# mountpoint that is actually tmpfs, an NFS/CIFS export, or a bind mount, all of
|
||||
# which would have produced a non-/dev SOURCE or an lsblk row of another type.
|
||||
- node_baseline_lh_candidate.type in ['part', 'disk']
|
||||
# 2. It is not backing anything the OS needs. Derived from ansible_mounts + swapon,
|
||||
# so a differently-partitioned node is caught by its own facts.
|
||||
- node_baseline_lh_candidate.path not in node_baseline_lh_protected_devices
|
||||
# 3. It already holds ext4. The claim path RELABELS rather than reformats (see
|
||||
# below), which is only meaningful for the filesystem type we are about to
|
||||
# declare in fstab. Anything else is an operator decision, not a role's.
|
||||
- node_baseline_lh_candidate.fstype == 'ext4'
|
||||
# 4. It carries no filesystem label, or already carries ours. A partition labelled
|
||||
# `boot`, `data`, `backup`… is somebody's named volume; stamping `longhorn` over
|
||||
# it would make it unfindable by its own fstab entry.
|
||||
- (node_baseline_lh_candidate.label | default('', true)) in ['', node_baseline_data_fs_label]
|
||||
# 5. It is big enough to be the partition we meant. A data partition of a few GB is
|
||||
# not "small", it is the wrong partition — and on prod it must hold BOTH tenants.
|
||||
- (node_baseline_lh_candidate.size | int) >= (node_baseline_longhorn_min_size_gb | int) * 1024 * 1024 * 1024
|
||||
# 6. It is EMPTY. The operator's premise is that prod's /srv holds nothing but the
|
||||
# ext4 5% root reserve; this is where that premise is verified per node instead
|
||||
# of trusted. If any node has real data on it, the play stops here — the role
|
||||
# never wipes data it cannot account for.
|
||||
- (node_baseline_lh_claim_contents.files | length) == 0
|
||||
fail_msg: >-
|
||||
REFUSING to claim {{ node_baseline_lh_candidate.path }}
|
||||
(mounted at {{ node_baseline_longhorn_claim_mountpoint }}) for the Longhorn store.
|
||||
type={{ node_baseline_lh_candidate.type }},
|
||||
fstype={{ node_baseline_lh_candidate.fstype }},
|
||||
label='{{ node_baseline_lh_candidate.label | default('', true) }}',
|
||||
size={{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB
|
||||
(minimum {{ node_baseline_longhorn_min_size_gb }} GiB),
|
||||
non-lost+found entries={{ node_baseline_lh_claim_contents.files | length }}
|
||||
{{ node_baseline_lh_claim_contents.files | map(attribute='path') | list }},
|
||||
devices this node needs and that must never be claimed:
|
||||
{{ node_baseline_lh_protected_devices }}.
|
||||
This is the destructive step of the role and it fails CLOSED. If the partition
|
||||
holds data, back it up or pick a different one; do NOT relax this assert.
|
||||
quiet: true
|
||||
vars:
|
||||
node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
|
||||
# The node's OWN account of what it cannot afford to lose: the devices behind the
|
||||
# mounts in node_baseline_protected_mounts (from gathered facts, so a node
|
||||
# partitioned differently is judged on its own layout) plus every active swap device.
|
||||
# Built here rather than hardcoded so no partition-name list can go stale.
|
||||
node_baseline_lh_protected_devices: >-
|
||||
{{ (ansible_facts.mounts | default([])
|
||||
| selectattr('mount', 'in', node_baseline_protected_mounts)
|
||||
| map(attribute='device') | list)
|
||||
+ (node_baseline_lh_swapdevs.stdout_lines | default([]) | map('trim') | select | list) }}
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): report exactly what is about to be claimed"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
Claiming {{ node_baseline_lh_candidate.path }}
|
||||
({{ ((node_baseline_lh_candidate.size | int) / 1024 / 1024 / 1024) | round(1) }} GiB ext4,
|
||||
currently mounted at {{ node_baseline_longhorn_claim_mountpoint }}, verified empty)
|
||||
as the data partition: unmount -> drop its fstab entry ->
|
||||
relabel LABEL={{ node_baseline_data_fs_label }} ->
|
||||
mount at {{ node_baseline_data_mount }}.
|
||||
vars:
|
||||
node_baseline_lh_candidate: "{{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0] }}"
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# --- Why the role must unmount and relabel, not just point fstab at the device ---------
|
||||
# The mount task below mounts by `LABEL=`, deliberately: a label survives /dev/sda5 ->
|
||||
# /dev/nvme0n1p5 renaming across the fleet and across a controller change, which a device
|
||||
# path does not. The installer-created /srv filesystem has NO label (fstab pins it by
|
||||
# UUID), so it must be given one. e2label refuses to run on a mounted filesystem, hence
|
||||
# the unmount first — and the fstab entry must go in the same breath, or the next boot
|
||||
# re-mounts the same partition at /srv underneath Longhorn.
|
||||
#
|
||||
# RELABEL, NOT REFORMAT. The partition is already ext4 and already verified empty, so
|
||||
# mkfs would buy nothing and would destroy anything the emptiness check got wrong.
|
||||
# e2label is a metadata write: if some assert above was too permissive, the worst case is
|
||||
# a renamed filesystem, not a destroyed one. (This is also why `community.general.filesystem`
|
||||
# cannot do this job: it declines to touch a device that already has a filesystem, so it
|
||||
# would silently no-op and then the LABEL= mount would fail with no useful message.)
|
||||
- name: "Longhorn (claim): unmount the claim mountpoint"
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
||||
state: unmounted
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): remove the claim mountpoint's /etc/fstab entry"
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_claim_mountpoint }}"
|
||||
state: absent_from_fstab
|
||||
backup: true
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
- name: "Longhorn (claim): stamp the data-partition filesystem label onto the partition"
|
||||
ansible.builtin.command: >-
|
||||
e2label {{ (node_baseline_lh_claim_lsblk.stdout | from_json).blockdevices[0].path }}
|
||||
{{ node_baseline_data_fs_label }}
|
||||
changed_when: true
|
||||
when: node_baseline_lh_claim_needed | bool
|
||||
|
||||
# --- The dedicated-device path (dev) --------------------------------------------------
|
||||
# Unchanged behaviour for `source: device`: data-safe, because community.general.filesystem
|
||||
# never reformats a device that already carries a filesystem.
|
||||
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_longhorn_device }}"
|
||||
opts: "-L {{ node_baseline_data_fs_label }}"
|
||||
when: node_baseline_longhorn_source == 'device'
|
||||
|
||||
# --- Common to ALL paths (device, claim, transition) ----------------------------------
|
||||
# Mounted by LABEL for every shape, so dev and prod share ONE fstab convention and one
|
||||
# mount task. Idempotent: on a converged node this reports ok, which is what makes a
|
||||
# re-run a no-op rather than a reformat. It is also what completes the transition above:
|
||||
# the retired mountpoint is gone from fstab and the partition now answers to the new label,
|
||||
# so this task mounts it at the new path.
|
||||
- name: Mount the data partition
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_data_mount }}"
|
||||
src: "LABEL={{ node_baseline_data_fs_label }}"
|
||||
fstype: ext4
|
||||
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
||||
state: mounted
|
||||
|
||||
# --- The per-tenant subdirectories ---------------------------------------------------
|
||||
# WHY THIS IS CONDITIONAL RATHER THAN UNCONDITIONAL. On prod the partition serves two
|
||||
# tenants and each needs its own subdirectory carved out of it. On DEV neither path is a
|
||||
# subdirectory of the mount: Longhorn's dataPath IS the mount root, and k3s's data dir is
|
||||
# /var/lib/rancher/k3s on the OS filesystem, owned and created by a LIVE k3s. Creating
|
||||
# either there would at best be a no-op and at worst re-mode the running dev cluster's data
|
||||
# dir, so the role only creates paths that are STRICT DESCENDANTS of the partition it
|
||||
# mounted — i.e. only the directories it is actually responsible for. That test is derived
|
||||
# from the paths themselves, so no environment has to remember to switch it off.
|
||||
#
|
||||
# MODES. Longhorn's dataPath keeps 0755, matching what mkfs gives dev's mount root and what
|
||||
# the Longhorn manager expects to find. k3s's data dir is 0700 from the start: it holds the
|
||||
# etcd database, the node token and the containerd image store, and k3s would tighten it to
|
||||
# 0700 itself on first start — doing it here means it is never briefly world-readable.
|
||||
- name: Create the per-tenant data directories on the data partition
|
||||
ansible.builtin.file:
|
||||
path: "{{ item.path }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- { path: "{{ node_baseline_longhorn_data_path }}", mode: "0755", tenant: "Longhorn replicas" }
|
||||
- { path: "{{ node_baseline_k3s_data_dir }}", mode: "0700", tenant: "k3s data dir (etcd + containerd)" }
|
||||
loop_control:
|
||||
label: "{{ item.path }} ({{ item.tenant }})"
|
||||
when: item.path.startswith(node_baseline_data_mount ~ '/')
|
||||
|
||||
# =====================================================================================
|
||||
# ETCD DATA DIR
|
||||
# =====================================================================================
|
||||
|
||||
# --- The dedicated-device path (dev) --------------------------------------------------
|
||||
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
|
||||
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
|
||||
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
|
||||
# a disk that already carries a filesystem, so re-running against an already-migrated
|
||||
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
|
||||
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
|
||||
# role would mount over a populated directory.
|
||||
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_etcd_device }}"
|
||||
opts: "-L {{ node_baseline_etcd_fs_label }}"
|
||||
when: node_baseline_etcd_source == 'device'
|
||||
|
||||
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_etcd_mount }}"
|
||||
src: "LABEL={{ node_baseline_etcd_fs_label }}"
|
||||
fstype: ext4
|
||||
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
|
||||
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
|
||||
opts: "defaults,noatime"
|
||||
dump: "0"
|
||||
passno: "2"
|
||||
state: mounted
|
||||
when: node_baseline_etcd_source == 'device'
|
||||
|
||||
# --- The no-dedicated-device paths (prod) ---------------------------------------------
|
||||
# Prod has ONE physical disk per node, already fully partitioned, with no free slot verified
|
||||
# populated — so etcd gets no device of its own. That is an operator decision
|
||||
# (PROD-PROVISIONING.md §2, 2026-09-03), and the shared-device I/O contention it accepts is
|
||||
# UNCHANGED by the 2026-09-04 layout revision: dev's etcd/Longhorn contention incident was
|
||||
# on a spinning HDD, and every prod node is SSD or NVMe, where concurrent rebuild I/O costs
|
||||
# latency rather than collapsing it. Accepted to be MEASURED, not assumed.
|
||||
#
|
||||
# WHAT DID CHANGE ON 2026-09-04 is WHICH FILESYSTEM etcd shares. It used to be the OS
|
||||
# filesystem (`root-filesystem`), which on these nodes means /var — a separate partition of
|
||||
# 11.8 GiB carrying the containerd image store too. Measured on dev, containerd alone is
|
||||
# 21-25 GB per node, so prod would have filled /var during its first image pulls and etcd
|
||||
# could not have fsynced its WAL. So k3s's whole data dir moved onto the claimed 424 GB
|
||||
# partition and etcd moved with it: `shared-data-partition`. Both enum values are still
|
||||
# supported, and the role now PROVES which one is true instead of printing a claim.
|
||||
#
|
||||
# There is nothing to CREATE here for either shape — the data-partition section above
|
||||
# already created node_baseline_k3s_data_dir, and k3s creates server/db inside it — so the
|
||||
# role's job is to verify the assumption rather than act on it.
|
||||
- name: "etcd (no dedicated device): check for anything already mounted at the etcd data dir"
|
||||
ansible.builtin.command: >-
|
||||
findmnt --noheadings --first-only --output SOURCE
|
||||
--mountpoint {{ node_baseline_etcd_mount }}
|
||||
register: node_baseline_etcd_existing_mount
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
||||
|
||||
- name: "etcd (no dedicated device): assert no dedicated device is already mounted there"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- node_baseline_etcd_existing_mount.rc != 0
|
||||
fail_msg: >-
|
||||
node_baseline_etcd_source is `{{ node_baseline_etcd_source }}` (etcd deliberately has
|
||||
NO dedicated device — PROD-PROVISIONING.md §2) but {{ node_baseline_etcd_mount }} is
|
||||
ALREADY a separate mount, backed by
|
||||
{{ node_baseline_etcd_existing_mount.stdout | default('') | trim }}. That is a
|
||||
contradiction the role will not resolve on its own: either this node was previously
|
||||
built with a dedicated etcd device (in which case unmount it deliberately, after
|
||||
accounting for the etcd data on it) or the declaration is wrong. Refusing to leave
|
||||
the node in a state where the declared layout and the real one disagree.
|
||||
quiet: true
|
||||
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
||||
|
||||
# Evidence, not decoration: §2 records which filesystem etcd shares, and this establishes
|
||||
# WHICH one per node so the claim is auditable from a run log. `findmnt --target` needs an
|
||||
# existing path and k3s has not created server/db yet, so walk up to the deepest ancestor
|
||||
# that does exist — the same filesystem the data dir will be created on. TARGET is captured
|
||||
# separately from the human-readable line because the assert below judges on it.
|
||||
- name: "etcd (no dedicated device): resolve the filesystem etcd will actually live on"
|
||||
ansible.builtin.shell: |
|
||||
set -eu
|
||||
p={{ node_baseline_etcd_mount | quote }}
|
||||
while [ ! -d "$p" ]; do p=$(dirname "$p"); done
|
||||
findmnt --noheadings --first-only --output TARGET --target "$p"
|
||||
findmnt --noheadings --first-only --output SOURCE,TARGET,SIZE,AVAIL --target "$p"
|
||||
register: node_baseline_etcd_backing_fs
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
||||
|
||||
# THE ASSERT THAT MAKES THE WHOLE LAYOUT REVISION SELF-PROVING, and the reason
|
||||
# `shared-data-partition` is a distinct enum value rather than a comment. Declaring
|
||||
# k3s_data_dir: /data/k3s does NOT by itself put etcd on the data partition: if the mount
|
||||
# above had silently failed, /data would be a plain directory on `/` and etcd plus 21-25 GB
|
||||
# of container images would land back on the OS filesystem — the exact failure this change
|
||||
# exists to prevent, and completely invisible until the disk filled. So the role checks that
|
||||
# the filesystem backing the etcd data dir IS the data partition's mount, per node, on every
|
||||
# run. `root-filesystem` is deliberately NOT asserted this way: it makes no claim about
|
||||
# which filesystem it lands on beyond "not a dedicated one", and the debug line below is the
|
||||
# whole record for that shape.
|
||||
- name: "etcd (shared-data-partition): assert etcd really lands on the data partition"
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- >-
|
||||
(node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim)
|
||||
== node_baseline_data_mount
|
||||
fail_msg: >-
|
||||
node_baseline_etcd_source is `shared-data-partition`, which asserts that
|
||||
{{ node_baseline_etcd_mount }} sits on the claimed data partition mounted at
|
||||
{{ node_baseline_data_mount }} — but the filesystem actually backing it is mounted at
|
||||
'{{ node_baseline_etcd_backing_fs.stdout_lines[0] | default('') | trim }}'
|
||||
(full: {{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}).
|
||||
The data partition is therefore NOT mounted where this layout requires, and k3s would
|
||||
put etcd's WAL AND the containerd image store (21-25 GB measured on dev) on that
|
||||
filesystem instead. On prod that is /var at 11.8 GiB: it would fill during the first
|
||||
image pulls and etcd would stop being able to fsync its WAL. DO NOT install k3s on
|
||||
this node until the mount is correct.
|
||||
quiet: true
|
||||
when: node_baseline_etcd_source == 'shared-data-partition'
|
||||
|
||||
- name: "etcd (no dedicated device): show which filesystem etcd will live on"
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
etcd data dir {{ node_baseline_etcd_mount }} has NO dedicated device
|
||||
(source={{ node_baseline_etcd_source }}); it lands on
|
||||
{{ node_baseline_etcd_backing_fs.stdout_lines[1] | default('') | trim }}
|
||||
(SOURCE TARGET SIZE AVAIL). Shared-device I/O contention is an accepted, to-be-
|
||||
measured risk — §2.
|
||||
when: node_baseline_etcd_source in ['root-filesystem', 'shared-data-partition']
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
# TAGS — every task here carries one of `packages`, `system`, `disks` or `dns`. An untagged
|
||||
# run is unchanged (all four run, in this order). The split exists because the PACKAGE floor
|
||||
# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
|
||||
# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
|
||||
# and the DATA-DISK work have different readiness: packages are safe on any node from
|
||||
# first contact, while the disk tasks need block devices that a new environment may not
|
||||
# have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
|
||||
@@ -12,6 +12,10 @@
|
||||
# mkfs. A FULL run against such a node stops at the assert below instead. `dns` (the block
|
||||
# at the bottom of this file) works the same way and guards its own BLOCKED placeholder:
|
||||
# ansible-playbook playbooks/node-baseline.yml --tags dns
|
||||
# `kernel` (the backports-kernel block below) is separated for the same reason `packages`
|
||||
# is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
|
||||
# it must be runnable without touching a disk decision that is still open:
|
||||
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
|
||||
|
||||
- name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
|
||||
ansible.builtin.apt:
|
||||
@@ -21,6 +25,237 @@
|
||||
cache_valid_time: 3600
|
||||
tags: [packages]
|
||||
|
||||
# =====================================================================================
|
||||
# BACKPORTS KERNEL — tag `kernel` (EFI memattr; infra/PROD-PROVISIONING.md §0b and §3)
|
||||
# =====================================================================================
|
||||
# A STANDING DEVIATION from the Debian 13 baseline, encoded here so every current AND
|
||||
# FUTURE prod node gets it from provisioning instead of from someone remembering.
|
||||
#
|
||||
# WHAT IT FIXES. HP R21 firmware on the prod EliteDesk 800 G5 minis publishes a CORRUPT EFI
|
||||
# Memory Attributes Table. `efi_memattr_init()` in Linux < 6.13 computes
|
||||
# `tbl_size = sizeof(*tbl) + num_entries * desc_size` from two UNVALIDATED u32 fields into a
|
||||
# signed int and `memblock_reserve()`s the product; when bit 31 lands the value sign-extends
|
||||
# through phys_addr_t, `memblock_cap_size()` clamps it to PHYS_ADDR_MAX, and EVERY physical
|
||||
# address above the table is reserved. k3s-prod-2 booted with 2.6 GiB of its 16 GB, twice.
|
||||
# Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus"
|
||||
# (first released in v6.13) validates the size and rejects the table instead. It is absent
|
||||
# from v6.12 and from linux-6.12.y, has no `Cc: stable` and no `Fixes:` tag, so no automatic
|
||||
# stable feed will ever pick it up. There is NO kernel cmdline workaround —
|
||||
# `efi_memattr_init()` runs from `efi_config_parse_tables()` before any `efi=` bail-out could
|
||||
# take effect. Booting a >= 6.13 kernel is the only fix, and ALL THREE prod nodes are on the
|
||||
# same lottery — node 1 and node 3 have merely drawn safe tickets so far.
|
||||
#
|
||||
# WHY IT IS SCOPED OFF FOR DEV. `node_baseline_backports_kernel` defaults to FALSE in
|
||||
# defaults/main.yml and is set true in inventory/group_vars/k3s_prod/vars.yml only. Dev is
|
||||
# Proxmox VMs whose firmware publishes no MAT at all — genuinely unaffected — and pulling a
|
||||
# LIVE cluster onto a backports kernel as a side effect of sharing this role would be a real
|
||||
# change for no benefit. See defaults/main.yml for why this is a variable rather than a
|
||||
# `k3s-prod-*` hostname test.
|
||||
#
|
||||
# IT DOES NOT REBOOT. A new kernel is inert until the node reboots, and this role
|
||||
# deliberately will not take one: k3s-prod-2 hung on a warm reboot on 2026-09-03 — powered,
|
||||
# off the LAN for ~38 minutes, power button required (§0b "SECOND FAULT"). Rebooting is
|
||||
# therefore a SERIALIZED, SUPERVISED operator step — one node at a time, verify each returns
|
||||
# before touching the next, never node 2 first — and a role cannot enforce that. The last
|
||||
# task here reports the pending reboot loudly instead of letting a green play imply the node
|
||||
# is already fixed.
|
||||
- name: Backports kernel for the EFI memattr fix (>= 6.13, commit 8fbe4c49c0cc)
|
||||
when: node_baseline_backports_kernel | bool
|
||||
tags: [kernel]
|
||||
block:
|
||||
|
||||
# The suite name and the running release must agree, or apt resolves no candidate and
|
||||
# `state: latest` reports "ok" while leaving the node on the vulnerable 6.12 kernel —
|
||||
# a green play that fixed nothing. Fail on the mismatch instead (e.g. a node
|
||||
# dist-upgraded to forky while group_vars still says trixie-backports).
|
||||
- name: Assert the configured backports suite matches this node's Debian release
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_facts['distribution'] == 'Debian'
|
||||
- node_baseline_backports_suite == (ansible_facts['distribution_release'] ~ '-backports')
|
||||
fail_msg: >-
|
||||
node_baseline_backports_kernel is enabled but node_baseline_backports_suite
|
||||
({{ node_baseline_backports_suite }}) is not the backports suite for this node's
|
||||
release ({{ ansible_facts['distribution'] }}
|
||||
{{ ansible_facts['distribution_release'] | default('?') }}). apt would resolve no
|
||||
candidate and this play would report success while leaving the node on the
|
||||
vulnerable kernel. Fix the suite in group_vars — do not remove this assert.
|
||||
quiet: true
|
||||
|
||||
# deb822 `.sources`, not a legacy one-line `deb` entry: it is the format Debian 13
|
||||
# prefers, it carries `Signed-By` explicitly (binding this source to the archive keyring
|
||||
# rather than to whatever is in /etc/apt/trusted.gpg.d), and a `copy` of exact content is
|
||||
# idempotent in a way `apt_repository` — which rewrites and reformats — is not.
|
||||
#
|
||||
# THERE IS DELIBERATELY NO /etc/apt/preferences.d PIN, and that is not an oversight. The
|
||||
# backports Release file carries `NotAutomatic: yes` + `ButAutomaticUpgrades: yes`, which
|
||||
# apt already reads as: priority 100 so NOTHING is pulled from backports by accident, but
|
||||
# 500 for a package that was explicitly installed from there so it keeps getting updates.
|
||||
# Writing an explicit `Pin-Priority: 100` looks like belt-and-braces and is actively
|
||||
# HARMFUL — it overrides the ButAutomaticUpgrades half, and the kernel would then never
|
||||
# be upgraded again by `apt upgrade`. Enabling the source IS the whole configuration.
|
||||
- name: Enable the Debian backports apt source (deb822)
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
|
||||
# A DELIBERATE, STANDING deviation from the Debian 13 baseline, enabled ONLY so this
|
||||
# machine can boot a kernel >= 6.13 that validates the corrupt EFI Memory Attributes
|
||||
# Table its firmware publishes (upstream 8fbe4c49c0cc). See
|
||||
# infra/PROD-PROVISIONING.md §0b/§3, and {{ node_baseline_backports_kernel_stamp }}
|
||||
# on this node.
|
||||
#
|
||||
# Backports is NotAutomatic, so nothing is taken from here unless asked for by name
|
||||
# with `-t {{ node_baseline_backports_suite }}`. Do NOT add an apt pin for it:
|
||||
# ButAutomaticUpgrades is what keeps the installed kernel patched, and an explicit
|
||||
# Pin-Priority would switch that off.
|
||||
Types: deb
|
||||
URIs: {{ node_baseline_backports_mirror }}
|
||||
Suites: {{ node_baseline_backports_suite }}
|
||||
Components: {{ node_baseline_backports_components | join(' ') }}
|
||||
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
|
||||
register: node_baseline_backports_source
|
||||
|
||||
- name: Refresh the apt cache after enabling backports
|
||||
ansible.builtin.apt:
|
||||
update_cache: true
|
||||
when: node_baseline_backports_source is changed
|
||||
|
||||
# `state: latest`, NOT `state: present`. linux-image-amd64 is ALREADY installed on these
|
||||
# nodes from trixie main (6.12.107-1), so `present` would find the name satisfied and do
|
||||
# nothing at all — the single most likely way to write this task and have it silently fix
|
||||
# nothing forever. `latest` + `default_release` resolves the candidate inside backports
|
||||
# and upgrades the metapackage to it. Once the node is current a re-run is a genuine
|
||||
# no-op: no change, no churn, no spurious reboot flag. Scope is exactly
|
||||
# node_baseline_backports_kernel_packages — one metapackage — so nothing else on the node
|
||||
# is dragged out of the main archive.
|
||||
- name: Install the fixed kernel from backports (>= 6.13)
|
||||
ansible.builtin.apt:
|
||||
name: "{{ node_baseline_backports_kernel_packages }}"
|
||||
state: latest
|
||||
default_release: "{{ node_baseline_backports_suite }}"
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
register: node_baseline_backports_kernel_apt
|
||||
|
||||
# WHICH VERSION DID THIS NODE ACTUALLY LAND ON. Read back rather than assumed: the
|
||||
# metapackage tracks a moving target, so "we installed linux-image-amd64" is not an
|
||||
# answer anybody can roll back from six months later.
|
||||
- name: Read back the installed backports kernel metapackage version
|
||||
ansible.builtin.command: dpkg-query -W -f '${Version}' {{ node_baseline_backports_kernel_packages | first }}
|
||||
register: node_baseline_backports_kernel_version
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
|
||||
# `linux-version` ships in linux-base and knows Debian's kernel version ordering, which
|
||||
# neither `sort -V` nor a Jinja sort gets right across +deb13 / ABI suffixes. The full
|
||||
# list is the ROLLBACK INVENTORY: every entry is still installed and selectable from
|
||||
# GRUB's "Advanced options" without uninstalling anything.
|
||||
- name: List every kernel image installed on this node, newest first
|
||||
ansible.builtin.shell:
|
||||
cmd: set -o pipefail; linux-version list | linux-version sort --reverse
|
||||
executable: /bin/bash
|
||||
register: node_baseline_installed_kernels
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
|
||||
# The node must be able to explain itself to an operator standing in front of it with no
|
||||
# repo checkout. Content is deliberately time-INVARIANT (no timestamp) so re-running the
|
||||
# role is a real no-op rather than a permanent daily "changed".
|
||||
- name: Record the deviation, the exact version, and the rollback path on the node
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ node_baseline_backports_kernel_stamp }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by the Ansible node_baseline role (tags: kernel) — do not edit by hand.
|
||||
#
|
||||
# THIS NODE DELIBERATELY DOES NOT RUN THE DEBIAN 13 STOCK KERNEL.
|
||||
#
|
||||
# WHY
|
||||
# This machine's HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table.
|
||||
# Linux < 6.13 reserves memory from it without validating its size, and an unlucky
|
||||
# boot reserves everything above the table: 2.6 GiB of 16 GB, observed twice on
|
||||
# k3s-prod-2. Upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size
|
||||
# is clearly bogus" (first released in v6.13) adds the validation; it is NOT in
|
||||
# v6.12 and NOT in linux-6.12.y. There is NO kernel cmdline workaround.
|
||||
# Full write-up: infra/PROD-PROVISIONING.md §0b in the home-network-k3s-impl repo.
|
||||
#
|
||||
# PROOF THE FIX IS ACTIVE (not merely a lucky boot)
|
||||
# BOOT-INDEPENDENT, and this is the one to trust — the string only exists in
|
||||
# >= 6.13, so its presence in the RUNNING image is proof regardless of the boot:
|
||||
# zstd -dc < <(tail -c +$(...)) ... | grep 'Corrupted EFI Memory Attributes'
|
||||
# (roles/node_health does the version check for you; see the health report's
|
||||
# firmware.efi_memattr finding.)
|
||||
#
|
||||
# FROM dmesg, WHICH IS A LOTTERY. efi_memattr_init() on >= 6.13 has THREE gates
|
||||
# and each prints a DIFFERENT line — all three are rejections:
|
||||
# tbl->version > 2 'Unexpected EFI Memory Attributes table version %d'
|
||||
# (NOTE: printed with %d, so garbage may read NEGATIVE)
|
||||
# bad desc_size 'Unexpected EFI Memory Attributes descriptor size ...'
|
||||
# num_entries > 64K '[Firmware Bug]: Corrupted EFI Memory Attributes
|
||||
# Table detected! (version == ..., num_entries == ...)'
|
||||
# Which one appears depends on WHICH field this boot's garbage corrupted, so do
|
||||
# NOT treat the absence of the FW_BUG line as the fix being absent — all three
|
||||
# prod nodes printed the *version* line on 2026-09-04.
|
||||
# The one line that means the fix is NOT active:
|
||||
# 'Failed to map EFI Memory Attributes table @ 0x...' — the kernel trusted the
|
||||
# corrupt table. On a >= 6.13 kernel this should be unreachable; if it appears,
|
||||
# escalate, do not shrug.
|
||||
#
|
||||
# INSTALLED
|
||||
# source /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
|
||||
# package {{ node_baseline_backports_kernel_packages | join(' ') }}
|
||||
# version {{ node_baseline_backports_kernel_version.stdout | default('(not read)') }}
|
||||
# kernels installed on this node, newest first:
|
||||
{% for kv in (node_baseline_installed_kernels.stdout_lines | default([])) %}
|
||||
# {{ kv }}
|
||||
{% endfor %}
|
||||
#
|
||||
# SECURITY COST
|
||||
# Packages from *-backports are provided AS IS. They are NOT covered by the Debian
|
||||
# Security Team's guarantees for the main archive: no security-tracker commitment,
|
||||
# no DSAs. Accepted deliberately, in exchange for the node keeping its RAM.
|
||||
#
|
||||
# ROLLBACK — no repo checkout needed
|
||||
# 1. NON-DESTRUCTIVE, try this first. Every kernel listed above is still installed
|
||||
# and bootable: pick the 6.12 one from GRUB -> "Advanced options for Debian
|
||||
# GNU/Linux". That answers "is the new kernel the problem?" without
|
||||
# uninstalling anything. Note it reinstates the memattr bug for that boot.
|
||||
# 2. PERMANENT. Set node_baseline_backports_kernel: false in
|
||||
# inventory/group_vars/k3s_prod/vars.yml FIRST — otherwise the next Ansible run
|
||||
# puts the backports kernel straight back — then, on this node:
|
||||
# apt-get install --reinstall linux-image-amd64/{{ ansible_facts['distribution_release'] }}
|
||||
# apt-get purge <the backports linux-image-*+deb13-amd64 listed above>
|
||||
# rm /etc/apt/sources.list.d/{{ node_baseline_backports_suite }}.sources
|
||||
# apt-get update && update-grub && reboot
|
||||
#
|
||||
# EXIT CONDITION
|
||||
# If 8fbe4c49c0cc is ever picked up into linux-6.12.y (and so into Debian's 6.12
|
||||
# rebase), this deviation can be removed entirely: flip the variable to false,
|
||||
# roll back as above, and delete the `kernel` block from roles/node_baseline.
|
||||
|
||||
# Say the pending reboot out loud. A green play here does NOT mean the node is fixed —
|
||||
# it means the fixed kernel is on disk. See the block header for why the role will not
|
||||
# reboot on its own.
|
||||
- name: Report whether this node is running the kernel it now has installed
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
{{ ('REBOOT REQUIRED — running ' ~ ansible_facts['kernel'] ~ ', newest installed is '
|
||||
~ newest ~ '. The EFI memattr fix is NOT active until this node reboots. Reboot
|
||||
prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER
|
||||
node 2 first (infra/PROD-PROVISIONING.md §0b "SECOND FAULT"). Afterwards confirm
|
||||
with: dmesg | grep -i "Corrupted EFI Memory Attributes Table"')
|
||||
if newest != ansible_facts['kernel']
|
||||
else ('Running the newest installed kernel (' ~ ansible_facts['kernel']
|
||||
~ '); no reboot pending for the memattr fix.') }}
|
||||
vars:
|
||||
newest: "{{ (node_baseline_installed_kernels.stdout_lines | default([]) | first) | default('', true) }}"
|
||||
|
||||
- name: Persist kernel modules for k3s/Cilium/Longhorn
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/modules-load.d/k3s-cilium-longhorn.conf
|
||||
@@ -79,70 +314,32 @@
|
||||
state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
|
||||
tags: [system]
|
||||
|
||||
# Fail LOUD and EARLY rather than at mkfs time when an environment has not yet decided
|
||||
# its data-disk layout. group_vars/k3s_prod/vars.yml deliberately carries
|
||||
# `/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2` for both devices (see
|
||||
# its comments): leaving them unset would fall back to dev's virtio /dev/vdb|/dev/vdc,
|
||||
# which do not exist on bare metal either — a confusing failure that looks like a wiring
|
||||
# bug. This assert names the real blocker instead. It is a no-op on dev (/dev/vdb,
|
||||
# /dev/vdc) and is `disks`-tagged, so `--tags packages` skips it along with the disk work.
|
||||
- name: Assert this environment's data-disk devices have actually been decided
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- "'BLOCKED' not in node_baseline_longhorn_device"
|
||||
- "'BLOCKED' not in node_baseline_etcd_device"
|
||||
fail_msg: >-
|
||||
node_baseline_longhorn_device / node_baseline_etcd_device are still the BLOCKED
|
||||
placeholders for this host's environment
|
||||
(longhorn={{ node_baseline_longhorn_device }}, etcd={{ node_baseline_etcd_device }}).
|
||||
The disk layout is an OPEN operator decision — see infra/PROD-PROVISIONING.md §2 —
|
||||
and must NOT be guessed. To install ONLY the package floor (e.g. smartmontools, so
|
||||
the SMART read that decision waits on can be taken) re-run this play with
|
||||
`--tags packages`, which skips every disk task including this assert.
|
||||
quiet: true
|
||||
tags: [disks]
|
||||
|
||||
- name: Create ext4 filesystem on the Longhorn data disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_longhorn_device }}"
|
||||
opts: "-L {{ node_baseline_longhorn_fs_label }}"
|
||||
tags: [disks]
|
||||
|
||||
- name: Mount the Longhorn data disk
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_mount }}"
|
||||
src: "LABEL={{ node_baseline_longhorn_fs_label }}"
|
||||
fstype: ext4
|
||||
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
||||
state: mounted
|
||||
tags: [disks]
|
||||
|
||||
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
|
||||
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
|
||||
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
|
||||
# a disk that already carries a filesystem, so re-running against an already-migrated
|
||||
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
|
||||
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
|
||||
# role would mount over a populated directory.
|
||||
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_etcd_device }}"
|
||||
opts: "-L {{ node_baseline_etcd_fs_label }}"
|
||||
tags: [disks]
|
||||
|
||||
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_etcd_mount }}"
|
||||
src: "LABEL={{ node_baseline_etcd_fs_label }}"
|
||||
fstype: ext4
|
||||
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
|
||||
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
|
||||
opts: "defaults,noatime"
|
||||
dump: "0"
|
||||
passno: "2"
|
||||
state: mounted
|
||||
# =====================================================================================
|
||||
# DATA DISKS — tag `disks`, in tasks/disks.yml
|
||||
# =====================================================================================
|
||||
# Split into its own file for two reasons. It is the only DESTRUCTIVE part of the role
|
||||
# (prod re-purposes the partition already mounted at /srv as the Longhorn store), so it
|
||||
# should be readable end-to-end on its own; and both environments' storage shapes — dev's
|
||||
# two dedicated virtio disks, prod's single partitioned physical disk with etcd left on
|
||||
# the OS filesystem — are declared there as an explicit enum rather than inferred from
|
||||
# which variables happen to be set. See tasks/disks.yml and infra/PROD-PROVISIONING.md §2.
|
||||
#
|
||||
# The `disks` tag is applied ONCE here, in BOTH places it is needed:
|
||||
# * `tags:` on the include itself, so `--tags packages` never even reaches the include
|
||||
# and a placeholder device path is not so much as templated; and
|
||||
# * `apply: tags:`, so the tag reaches the tasks INSIDE the file. This half is not
|
||||
# optional and not decoration — a DYNAMIC include (`include_tasks`) does NOT propagate
|
||||
# its own tags to its children the way `import_tasks` does, so without `apply` the
|
||||
# included tasks are untagged and `--tags disks` selects the include and then runs
|
||||
# NOTHING inside it. That failure is silent and reports success, which is the worst
|
||||
# shape a guard can have.
|
||||
# A FULL run against a node whose layout is undeclared stops on the assert at the top of
|
||||
# that file instead of failing obscurely at mkfs time.
|
||||
- name: Provision the data disks (Longhorn store, etcd data dir)
|
||||
ansible.builtin.include_tasks:
|
||||
file: disks.yml
|
||||
apply:
|
||||
tags: [disks]
|
||||
tags: [disks]
|
||||
|
||||
- name: Disable swap now
|
||||
|
||||
@@ -54,7 +54,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
| `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
|
||||
| `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
|
||||
| `os.reboot_required`, `os.inventory`, `firmware.inventory` | pending reboot, OS/kernel/uptime, chassis + BIOS |
|
||||
| `firmware.efi_memattr` | the kernel-log signature of the corrupt EFI Memory Attributes Table that costs a node most of its RAM — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
|
||||
| `firmware.efi_memattr` | did the corrupt-EFI-Memory-Attributes-Table bug **fire on this boot**, and did it cost this node memory (per the measurement `memory.physical_total` reports) — **root only**. See [The EFI memattr check](#the-efi-memattr-check) |
|
||||
|
||||
Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
|
||||
in task logic.
|
||||
@@ -65,33 +65,81 @@ in task logic.
|
||||
`infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
|
||||
|
||||
HP's R21 firmware publishes a **corrupt EFI Memory Attributes Table**, and Linux **< 6.13**
|
||||
does not validate it before reserving memory from its contents —
|
||||
`drivers/firmware/efi/memattr.c` multiplies two never-validated fields
|
||||
(`num_entries * desc_size`) and `memblock_reserve()`s the product, i.e. everything from the
|
||||
table address to the top of RAM. Which of two kernel-log lines appears decides whether the
|
||||
node keeps its memory, and it is **decided afresh on every boot**:
|
||||
does not validate it before reserving memory from its contents. **Two different kernel
|
||||
functions matter, and the check used to conflate them:**
|
||||
|
||||
| kernel log | meaning | verdict |
|
||||
- `efi_memattr_init()` computes `tbl_size = sizeof(*tbl) + num_entries * desc_size` — both
|
||||
fields unvalidated — and `memblock_reserve()`s it. `tbl_size` is a signed `int`, so
|
||||
**catastrophic loss requires it to be NEGATIVE**: bit 31 set sign-extends through
|
||||
`phys_addr_t` and `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX`, reserving everything
|
||||
above the table. A garbage-but-**positive** `tbl_size` reserves a modest, wrong amount and
|
||||
costs nothing measurable.
|
||||
- `efi_memattr_apply_permissions()` later `memremap()`s the table. **This is what prints
|
||||
"Failed to map"**, and it fails for *any* sufficiently wrong `tbl_size` — negative or
|
||||
merely large.
|
||||
|
||||
So `Failed to map` proves the table was corrupt **and trusted by the kernel**. It does *not*
|
||||
prove memory was lost: that depends on the garbage **value** at that address on that boot,
|
||||
not on the address. `k3s-prod-2` lost 13.3 GiB twice at `0xbeeb0018` and then (2026-09-03)
|
||||
booted with all 15.1 GiB — same address, same log line. **The verdict is therefore based on
|
||||
measured memory, never on the log line alone:**
|
||||
|
||||
| kernel log | measured shortfall | verdict |
|
||||
|---|---|---|
|
||||
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | the garbage `version` field read `<= 2`, the kernel trusted the table | **FAIL** — the node has lost the memory *now* |
|
||||
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>` | the garbage `version` read `> 2`, the kernel bailed out first | **WARN** — memory intact *this boot* only; the firmware is still publishing a corrupt table |
|
||||
| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | kernel >= 6.13 validated and rejected it | PASS — this *is* the fix, working |
|
||||
| no memattr line, log reaches back to boot | no table published (VM/OVMF), or a valid one | PASS |
|
||||
| no memattr line, log does **not** reach back to boot | the ring buffer wrapped past early boot | **SKIP** — unknown, not unaffected |
|
||||
| `dmesg` unreadable | no root / `dmesg_restrict=1` | **SKIP** — unknown, not unaffected |
|
||||
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | large (`memory.physical_total` FAIL) | **FAIL** — the bug fired and memory is lost *now*, measured |
|
||||
| `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` | normal | **WARN** — the bug fired and the kernel trusted a corrupt table; this boot drew a survivable garbage value, the next may not |
|
||||
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **< 6.13** | (not consulted) | **WARN** — the garbage `version` read outside `0..2` so the kernel bailed out first: exposed, but this boot took the safe path |
|
||||
| `efi: memattr: Unexpected EFI Memory Attributes table version <N>`, kernel **>= 6.13** | (not consulted) | PASS — the same bail-out, but now the FIRST of three validation gates: no draw can reach the fatal arithmetic |
|
||||
| `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected!` | (not consulted) | PASS — kernel >= 6.13 rejected it at the `num_entries > 64K` gate: this *is* the fix, working |
|
||||
| no memattr line, log reaches back to boot | (not consulted) | PASS |
|
||||
| no memattr line, log does **not** reach back to boot | — | **SKIP** — the ring buffer wrapped past early boot: unknown, not unaffected |
|
||||
| `dmesg` unreadable | — | **SKIP** — no root / `dmesg_restrict=1`: unknown, not unaffected |
|
||||
|
||||
The WARN is the point. `k3s-prod-1` and `k3s-prod-2` publish the same corrupt table and
|
||||
differ only by **luck**: `k3s-prod-2` itself booted with its full 15.9 GiB two boots before
|
||||
it lost 13.3 GiB. A node one unlucky boot from losing 83% of its RAM is not passing.
|
||||
The WARN is the point **on a kernel < 6.13**. All three prod nodes publish the same corrupt
|
||||
table and differ only by **luck**, boot to boot. A node whose kernel just trusted a corrupt
|
||||
table is not passing a health check even when it kept its RAM.
|
||||
|
||||
### The FW_BUG line is proof, but its ABSENCE proves nothing
|
||||
|
||||
`efi_memattr_init()` on >= 6.13 has **three** gates, and only the third prints the FW_BUG
|
||||
line:
|
||||
|
||||
| gate | message | reachable when |
|
||||
|---|---|---|
|
||||
| `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
|
||||
| implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, desc_size does not |
|
||||
| `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* desc_size read plausibly, entry count does not |
|
||||
|
||||
**All three are rejections.** Which one a boot prints is decided by *which field* that boot's
|
||||
garbage corrupts — it is not a choice between "fixed" and "not fixed". So a fixed kernel very
|
||||
often prints the *first* message, and looking only for the FW_BUG line will make a working
|
||||
fix look absent. On 2026-09-04 all three prod nodes came up on 7.1.8 and all three printed
|
||||
the **version** line, not the FW_BUG one. That is why the verdict keys on the kernel version
|
||||
alongside the log line, and why the check must never demand the FW_BUG string as its
|
||||
evidence. Positive, boot-independent proof that the fixed code is compiled in is a string in
|
||||
the kernel image itself, not a line in `dmesg`.
|
||||
|
||||
**The version field is printed with `%d` — signed.** Garbage with bit 31 set is logged
|
||||
*negative* (`version -1569980416`, k3s-prod-2, 2026-09-04). The collector regex missed that
|
||||
until 2026-09-04 and such nodes fell through to `clean`, i.e. a kernel that had just rejected
|
||||
a corrupt table was reported as one that "accepted it without complaint" — a false PASS on
|
||||
the exact bug this check exists to find.
|
||||
|
||||
**Fix:** boot a kernel **>= 6.13** (Debian 13: `linux-image-amd64` from
|
||||
`trixie-backports`); upstream commit `8fbe4c49c0cc`. There is **no kernel cmdline
|
||||
workaround** — `efi_memattr_init()` runs before any `efi=` bail-out could take effect, and
|
||||
rebooting/CMOS/BIOS/USB changes only move where the table lands.
|
||||
|
||||
`memory.physical_total` measures the same *symptom* from `/proc/zoneinfo` alone, so a node
|
||||
whose kernel log cannot be read is still not unmeasured — and a future memory loss with a
|
||||
different cause is still caught.
|
||||
### The two memory checks cannot contradict each other
|
||||
|
||||
`memory.physical_total` and `firmware.efi_memattr` both need the answer to "did this machine
|
||||
lose memory?", and it is measured **once** — `/proc/zoneinfo` present-vs-managed pages plus
|
||||
the SMBIOS-vs-`MemTotal` shortfall — into the shared `node_health_mem_shortfall` fact in
|
||||
`tasks/assess.yml`. `memory.physical_total` *is* that verdict; `firmware.efi_memattr` reads
|
||||
it and may only escalate to FAIL when it says FAIL. The memattr check measures no memory of
|
||||
its own, so there is exactly one verdict on lost memory per host and both checks quote it.
|
||||
A node whose kernel log cannot be read is still measured by `memory.physical_total`, and a
|
||||
future memory loss with a different cause is still caught.
|
||||
|
||||
## Output
|
||||
|
||||
|
||||
@@ -468,19 +468,27 @@ def collect_memory_devices():
|
||||
# write-up. Short version: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes
|
||||
# Table, and Linux < 6.13 does not validate it before reserving memory from its contents
|
||||
# (drivers/firmware/efi/memattr.c multiplies two unvalidated fields and memblock_reserve()s
|
||||
# the product). The kernel prints exactly one of two lines, and WHICH one decides whether
|
||||
# the node keeps its RAM:
|
||||
# the product). The kernel prints one of two lines:
|
||||
#
|
||||
# "Failed to map EFI Memory Attributes table @ 0x..." the table's garbage version read
|
||||
# <= 2, the kernel trusted it and reserved gigabytes. THE NODE HAS LOST MEMORY NOW.
|
||||
# "Failed to map EFI Memory Attributes table @ 0x..." the garbage version read <= 2, so
|
||||
# the kernel TRUSTED the table and reserved memory from it. This is emitted later, by
|
||||
# efi_memattr_apply_permissions()'s memremap(), and it fails for ANY sufficiently
|
||||
# wrong tbl_size. So it proves the table was corrupt and trusted — it does NOT prove
|
||||
# memory was lost: only a NEGATIVE tbl_size (bit 31 set, sign-extended and clamped to
|
||||
# PHYS_ADDR_MAX) reserves everything above the table, while a garbage-but-positive
|
||||
# tbl_size costs nothing measurable. Whether memory actually went missing is decided
|
||||
# by MEASUREMENT (collect_zoneinfo below), not by this line: k3s-prod-2 lost 13.3 GiB
|
||||
# twice at 0xbeeb0018 and then booted with all 15.1 GiB at the same address, same line.
|
||||
# "Unexpected EFI Memory Attributes table version <N>" the garbage version read > 2, so
|
||||
# the kernel bailed out before reserving anything. Memory intact THIS BOOT — but the
|
||||
# firmware is still publishing a corrupt table and the outcome is decided afresh by
|
||||
# where the table lands on every boot. That is luck, not immunity.
|
||||
# what the garbage reads on every boot. That is luck, not immunity.
|
||||
#
|
||||
# It is boot-to-boot NON-DETERMINISTIC (k3s-prod-2 booted with its full 15.9 GiB two boots
|
||||
# before it lost 13.3 GiB), which is precisely why it is detected on every health run
|
||||
# instead of diagnosed once.
|
||||
# This collector reports the LINES and the addresses; it never rules on memory. Classifying
|
||||
# is roles/node_health/tasks/assess.yml, which rules on the measured shortfall.
|
||||
#
|
||||
# It is boot-to-boot NON-DETERMINISTIC, which is precisely why it is detected on every
|
||||
# health run instead of diagnosed once.
|
||||
#
|
||||
# dmesg needs root when kernel.dmesg_restrict=1 (Debian's default, and the setting on
|
||||
# every node here). The unprivileged pass therefore records WHY it could not read, and
|
||||
@@ -489,8 +497,15 @@ BOOT_ANCHOR_RE = re.compile(r"Linux version \S|BIOS-provided physical RAM map|Co
|
||||
EFI_CONFIG_RE = re.compile(r"\befi:\s.*\bMEMATTR=(0x[0-9a-fA-F]+)")
|
||||
MEMATTR_FAILED_RE = re.compile(
|
||||
r"Failed to map EFI Memory Attributes table\s*@?\s*(0x[0-9a-fA-F]+)?", re.I)
|
||||
# NOTE the `-?`. The kernel prints this version with %d — SIGNED — so a garbage field with
|
||||
# bit 31 set is logged NEGATIVE (k3s-prod-2, 2026-09-04: "version -1569980416"). Without the
|
||||
# minus sign this regex missed those lines entirely and the state fell through to "clean",
|
||||
# i.e. a node whose kernel had just REJECTED a corrupt table was reported as one whose
|
||||
# kernel "accepted it without complaint" — a false PASS on the exact bug this check exists
|
||||
# to find. Anchor future edits on that: the field is signed, and every diagnostic that
|
||||
# quotes it must survive a negative value.
|
||||
MEMATTR_VERSION_RE = re.compile(
|
||||
r"Unexpected EFI Memory Attributes table version\s+(\d+)", re.I)
|
||||
r"Unexpected EFI Memory Attributes table version\s+(-?\d+)", re.I)
|
||||
MEMATTR_CORRUPT_RE = re.compile(r"Corrupted EFI Memory Attributes Table", re.I)
|
||||
# The DIAGNOSTIC lines only. Deliberately NOT a bare "memattr": the EFI config-table
|
||||
# line lists "MEMATTR=0x..." on every UEFI boot and is not a finding — its address is
|
||||
|
||||
@@ -387,18 +387,28 @@
|
||||
~ (m.mem_available_kb | int / 1048576) | round(1) | string ~ ' GiB available ('
|
||||
~ avail_pct | string ~ '%)') }}
|
||||
|
||||
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK. Its sibling firmware.efi_memattr names the
|
||||
# one cause we have actually hit here; this one asks the blunter question — "is this
|
||||
# machine using the RAM it has?" — and answers it from two sources that share NOTHING
|
||||
# with the kernel log, so a future firmware, kernel or hypervisor losing memory for some
|
||||
# other reason still shows up. Thresholds and their measured baselines: defaults/main.yml.
|
||||
- name: Assess — usable memory vs the memory this machine physically has
|
||||
# --- THE ONE MEASUREMENT OF MEMORY THIS MACHINE CANNOT USE ----------------------------
|
||||
# Measured HERE, once, because TWO checks need the answer: memory.physical_total (directly
|
||||
# below) and firmware.efi_memattr (further down). Until 2026-09-03 each answered "did this
|
||||
# machine lose memory?" from DIFFERENT evidence — this one from measurement, the memattr one
|
||||
# from the presence of a kernel-log line — and they duly contradicted each other on the same
|
||||
# host. k3s-prod-2 booted with `efi: memattr: Failed to map EFI Memory Attributes table @
|
||||
# 0xbeeb0018` (the memattr check FAILed, asserting the node "HAS LOST MEMORY") while it was
|
||||
# measurably using 15.1 GiB of its 16 GB DIMM (this check PASSed at 4.2% unmanaged). The
|
||||
# measurement was right; the inference from the log line was wrong — the memattr block below
|
||||
# has the refined mechanism.
|
||||
#
|
||||
# THE FIX IS STRUCTURAL, NOT A RE-WORDING. The shortfall is computed in exactly this one
|
||||
# place and both checks read this fact: neither derives a shortfall of its own, and the
|
||||
# memattr check may only report memory LOST when `status` here is FAIL. So the two cannot
|
||||
# disagree about whether memory was lost — there is only one verdict on that question, and
|
||||
# it is this one. Thresholds and their measured baselines: defaults/main.yml.
|
||||
- name: Assess — measure memory the machine has and cannot use (once, shared by two checks)
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'memory.physical_total',
|
||||
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
|
||||
'status': phys_status,
|
||||
'detail': phys_detail }] }}"
|
||||
node_health_mem_shortfall: "{{ {
|
||||
'zones_pct': z_pct, 'zones_status': z_status,
|
||||
'dimm_pct': d_pct, 'dimm_status': d_status, 'installed_kb': inst_kb,
|
||||
'status': worst_status } }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
th: "{{ node_health_thresholds }}"
|
||||
@@ -430,11 +440,36 @@
|
||||
|
||||
# Worst sub-verdict wins; SKIP only when NEITHER measurement was obtained — two
|
||||
# SKIPs must never add up to a PASS.
|
||||
phys_status: >-
|
||||
worst_status: >-
|
||||
{{ (['PASS', 'WARN', 'FAIL'] | select('in', [z_status, d_status]) | list | last)
|
||||
| default('SKIP', true) }}
|
||||
# What the kernel log says, IF it was read — quoted here only as attribution for a
|
||||
# shortfall this check measured on its own. The verdict above never depends on it.
|
||||
|
||||
# THE CAUSE-INDEPENDENT MEMORY-LOSS CHECK: it reports the measurement above. Its sibling
|
||||
# firmware.efi_memattr names the one cause we have actually hit here; this one asks the
|
||||
# blunter question — "is this machine using the RAM it has?" — and answers it from two
|
||||
# sources that share NOTHING with the kernel log, so a future firmware, kernel or hypervisor
|
||||
# losing memory for some other reason still shows up.
|
||||
- name: Assess — usable memory vs the memory this machine physically has
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'memory.physical_total',
|
||||
'title': 'Usable memory vs installed/present memory (independent of the kernel log)',
|
||||
'status': phys_status,
|
||||
'detail': phys_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
ms: "{{ node_health_mem_shortfall }}"
|
||||
z: "{{ node_health_facts.memory.zones }}"
|
||||
dev: "{{ node_health_facts.memory.devices }}"
|
||||
m: "{{ node_health_facts.memory }}"
|
||||
# Straight from the shared measurement — this check re-derives nothing.
|
||||
z_pct: "{{ ms.zones_pct }}"
|
||||
inst_kb: "{{ ms.installed_kb }}"
|
||||
d_pct: "{{ ms.dimm_pct }}"
|
||||
phys_status: "{{ ms.status }}"
|
||||
# What the kernel log says, IF it was read — quoted here only as attribution for the
|
||||
# measured shortfall. The verdict above is the shared measurement and never depends on
|
||||
# the log; firmware.efi_memattr reads that same verdict, so the two always agree.
|
||||
phys_cause: >-
|
||||
{{ ('. CAUSE: the kernel logged the EFI Memory Attributes table failure — see check
|
||||
firmware.efi_memattr and infra/PROD-PROVISIONING.md §0b')
|
||||
@@ -757,24 +792,59 @@
|
||||
# what follows is only what this check needs to be read correctly.
|
||||
#
|
||||
# HP R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux < 6.13 does
|
||||
# not validate it before reserving memory from its contents: drivers/firmware/efi/memattr.c
|
||||
# multiplies two never-validated fields (num_entries x desc_size) and memblock_reserve()s
|
||||
# the product — everything from the table address to the top of RAM. The kernel prints
|
||||
# exactly one of two lines, and WHICH one is decided by whether the garbage `version` field
|
||||
# happens to read > 2:
|
||||
# not validate it before reserving memory from its contents. TWO DIFFERENT KERNEL FUNCTIONS
|
||||
# are involved, and conflating them is what made this check wrong until 2026-09-03:
|
||||
#
|
||||
# "Failed to map EFI Memory Attributes table @ 0x..." version read <= 2, the kernel
|
||||
# TRUSTED the table. The node has lost the memory. FAIL.
|
||||
# "Unexpected EFI Memory Attributes table version <N>" version read > 2, the kernel
|
||||
# bailed out first. Memory intact THIS BOOT — but the firmware is still publishing a
|
||||
# corrupt table, and the outcome is re-decided on every single boot. WARN, not PASS:
|
||||
# k3s-prod-1 and k3s-prod-2 differ by LUCK, not by immunity, and k3s-prod-2 itself
|
||||
# booted with its full 15.9 GiB two boots before it lost 13.3 GiB. A node one
|
||||
# unlucky boot from losing 83% of its RAM is not passing a health check.
|
||||
# efi_memattr_init() tbl_size = sizeof(*tbl) + num_entries * desc_size, then
|
||||
# memblock_reserve(efi_mem_attr_table, tbl_size). tbl_size
|
||||
# is a signed int. CATASTROPHIC LOSS REQUIRES IT TO BE
|
||||
# NEGATIVE: a garbage product with bit 31 set sign-extends
|
||||
# through phys_addr_t and memblock_cap_size() clamps it to
|
||||
# PHYS_ADDR_MAX — everything above the table is reserved.
|
||||
# A garbage-but-POSITIVE tbl_size reserves a modest, wrong
|
||||
# amount and costs nothing measurable.
|
||||
# efi_memattr_apply_permissions() memremap(efi_mem_attr_table, tbl_size, ...) later, and
|
||||
# THIS is what prints "Failed to map". It fails for ANY
|
||||
# sufficiently wrong tbl_size — negative OR merely large.
|
||||
#
|
||||
# SO: "Failed to map" PROVES the table was corrupt and the kernel trusted it. It does NOT
|
||||
# prove memory was lost. The outcome depends on the VALUE of the garbage at that address on
|
||||
# that boot, not on the address. k3s-prod-2 lost 13.3 GiB twice at 0xbeeb0018 and then, on
|
||||
# 2026-09-03, booted with all 15.1 GiB at the SAME address and the SAME log line. That boot
|
||||
# is the counter-example that falsified the old inference.
|
||||
#
|
||||
# THE VERDICT IS THEREFORE BASED ON MEASURED MEMORY, NOT ON THE LOG LINE:
|
||||
#
|
||||
# "Failed to map ... @ 0x..." the bug FIRED: corrupt table, trusted by the kernel.
|
||||
# FAIL when the shared measurement (node_health_mem_shortfall) says memory is
|
||||
# catastrophically missing — memory is lost NOW.
|
||||
# WARN otherwise — this boot drew a survivable garbage value; the next may not.
|
||||
# "Unexpected ... version <N>" version read > 2, so the kernel bailed out before
|
||||
# reserving anything. WARN on a kernel < 6.13 (exposed, and this boot merely took the
|
||||
# safe path) but PASS on >= 6.13, where that bail-out is the FIRST of three validation
|
||||
# gates and no draw can reach the fatal arithmetic any more. Same line, opposite
|
||||
# meaning; the kernel version is what separates them. See memattr_kernel_fixed below.
|
||||
# FW_BUG "Corrupted EFI Memory Attributes Table detected!" the >= 6.13 THIRD gate
|
||||
# (num_entries > 64K) firing. PASS. It is NOT the only proof the fix is active and
|
||||
# must not be treated as such: it is only reachable on a boot whose garbage leaves the
|
||||
# version field <= 2, so a fixed kernel very often prints the "Unexpected ... version"
|
||||
# line instead. All three prod nodes did exactly that on 2026-09-04.
|
||||
#
|
||||
# The two lines are not "lucky vs unlucky" any more — there are three outcomes (bail-out,
|
||||
# a trusted table that cost nothing, a trusted table that cost gigabytes) and only the
|
||||
# measurement separates the last two. ON A KERNEL < 6.13, never PASS on either line: the
|
||||
# firmware is publishing a corrupt table and the draw is repeated on every boot. On >= 6.13
|
||||
# the draw still happens but every outcome is a rejection, so the same lines mean the
|
||||
# validation is working.
|
||||
#
|
||||
# WHY THIS IS A CHECK AND NOT A NOTE IN A RUNBOOK: it is boot-to-boot NON-DETERMINISTIC,
|
||||
# so it cannot be diagnosed once. It has to be measured on every health run.
|
||||
#
|
||||
# IT CANNOT CONTRADICT memory.physical_total. Both checks read one fact,
|
||||
# node_health_mem_shortfall, computed once further up this file; this check does not measure
|
||||
# memory at all. Its FAIL is gated on that fact being FAIL, so "memory was lost" has exactly
|
||||
# one verdict on any host and both checks quote it.
|
||||
#
|
||||
# THE HONESTY TRAP, AND HOW IT IS AVOIDED. "No memattr line" means the node is unaffected
|
||||
# ONLY if the kernel log was actually read AND still reaches back to early boot — every
|
||||
# memattr diagnostic is printed in the first fraction of a second, and the ring buffer
|
||||
@@ -793,13 +863,43 @@
|
||||
ansible_become: false
|
||||
k: "{{ node_health_facts.klog }}"
|
||||
z: "{{ node_health_facts.memory.zones }}"
|
||||
# THE ONE MEMORY VERDICT, read (never re-derived) from the shared measurement computed
|
||||
# near memory.physical_total. This check contributes the kernel-log facts only.
|
||||
ms: "{{ node_health_mem_shortfall }}"
|
||||
ma_state: "{{ k.memattr_state | default(none) }}"
|
||||
# DOES THE RUNNING KERNEL VALIDATE THE TABLE? Everything below turns on this. >= 6.13
|
||||
# carries 8fbe4c49c0cc (hardened by 5d0faa8e8369), which gates efi_memattr_init() behind
|
||||
# THREE checks before it computes tbl_size at all:
|
||||
# tbl->version > 2 -> "Unexpected ... table version %d"
|
||||
# desc_size implausible -> "Unexpected ... descriptor size %u ..."
|
||||
# num_entries > SZ_64K -> FW_BUG "Corrupted EFI Memory Attributes
|
||||
# Table detected! ..."
|
||||
# ALL THREE ARE REJECTIONS, and only the LAST one prints the FW_BUG line. Which message a
|
||||
# given boot produces is decided by WHICH field that boot's garbage lands in — it is not
|
||||
# a choice between "fixed" and "not fixed". On < 6.13 only the first check exists, so a
|
||||
# version that reads <= 2 goes straight to the unvalidated multiply and the catastrophic
|
||||
# memblock_reserve(). That is the whole difference, and it is why the verdict for
|
||||
# `bad_version` below is opposite on the two kernels: same log line, completely different
|
||||
# meaning. Major.minor is extracted first — the raw string ("7.1.8+deb13-amd64") is not
|
||||
# something the `version` test can parse.
|
||||
memattr_kernel_mm: "{{ (node_health_facts.os.kernel | default('', true)) | regex_search('^[0-9]+\\.[0-9]+') | default('', true) }}"
|
||||
memattr_kernel_fixed: "{{ (memattr_kernel_mm | length) > 0 and (memattr_kernel_mm is version('6.13', '>=')) }}"
|
||||
# `Failed to map` means the bug FIRED — always at least a WARN, never a PASS. Whether it
|
||||
# COST anything is not in the log: only ms.status can escalate this to FAIL, which is why
|
||||
# this check and memory.physical_total can no longer disagree about lost memory.
|
||||
# `bad_version` is the one verdict that depends on the kernel: on < 6.13 it is a lucky
|
||||
# boot (WARN — the next draw may be the fatal one); on >= 6.13 it is the validation
|
||||
# WORKING (PASS — the fatal draw no longer exists). Reporting a fixed node as WARN
|
||||
# forever would make the check unable to ever show the fix landing, which is exactly what
|
||||
# it is for.
|
||||
memattr_status: >-
|
||||
{{ 'SKIP' if not (k.available | bool)
|
||||
else ('FAIL' if ma_state == 'failed_to_map'
|
||||
else ('WARN' if ma_state == 'bad_version'
|
||||
else ('FAIL' if (ma_state == 'failed_to_map' and ms.status == 'FAIL')
|
||||
else ('WARN' if ma_state == 'failed_to_map'
|
||||
else ('PASS' if ma_state == 'corrupt_rejected'
|
||||
else ('PASS' if (k.boot_covered | bool) else 'SKIP')))) }}
|
||||
else (('PASS' if (memattr_kernel_fixed | bool) else 'WARN')
|
||||
if ma_state == 'bad_version'
|
||||
else ('PASS' if (k.boot_covered | bool) else 'SKIP'))))) }}
|
||||
# One remediation, quoted by both findings. NOTE what it must NOT say: there is no
|
||||
# kernel cmdline workaround, because efi_memattr_init() runs before any efi= bail-out
|
||||
# could take effect. Suggesting one would be a fix that quietly does nothing.
|
||||
@@ -812,14 +912,32 @@
|
||||
bail-out could take effect. Rebooting, clearing CMOS, changing a BIOS setting or
|
||||
unplugging USB only change where the table lands and may APPEAR to fix it — they are
|
||||
a lottery ticket, not a fix. Full write-up: infra/PROD-PROVISIONING.md §0b.
|
||||
# The symptom, measured independently of this log (see memory.physical_total).
|
||||
memattr_corroboration: >-
|
||||
{{ (' Measured independently of this log: MemTotal '
|
||||
~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
|
||||
# The measurement, quoted from the shared fact — the ONLY thing this finding is allowed
|
||||
# to say about memory being lost or intact. Figures come from /proc/zoneinfo, which the
|
||||
# kernel log has no part in; the verdict word comes from ms.status, so it always agrees
|
||||
# with memory.physical_total. When the measurement is missing it says so, and says why.
|
||||
memattr_figures: >-
|
||||
{{ ('MemTotal ' ~ ((z.managed_kb | int) / 1048576) | round(1) | string ~ ' GiB of '
|
||||
~ ((z.present_kb | int) / 1048576) | round(1) | string ~ ' GiB present, i.e. '
|
||||
~ z.unmanaged_pct | string ~ '% of the RAM this machine has is reserved away
|
||||
from the allocator (/proc/zoneinfo) — check memory.physical_total.')
|
||||
if ((z.available | bool) and z.unmanaged_pct is number) else '' }}
|
||||
~ ms.zones_pct | string ~ '% of the RAM this machine has was never handed to the
|
||||
allocator (/proc/zoneinfo)')
|
||||
if ms.zones_pct is number
|
||||
else (('MemTotal is ' ~ ms.dimm_pct | string ~ '% short of the DIMM capacity SMBIOS
|
||||
reports as installed (/proc/zoneinfo NOT read — '
|
||||
~ (z.reason | default('no reason recorded', true)) ~ ')')
|
||||
if ms.dimm_pct is number else '') }}
|
||||
memattr_corroboration: >-
|
||||
{{ (' MEASURED THIS BOOT, independently of this log (check memory.physical_total, verdict '
|
||||
~ ms.status ~ '): ' ~ memattr_figures
|
||||
~ (' — a catastrophic shortfall: memory IS lost on this boot.' if ms.status == 'FAIL'
|
||||
else (' — an elevated shortfall, above the warn threshold.' if ms.status == 'WARN'
|
||||
else ' — the normal few percent (kernel image, memmap, firmware
|
||||
reservations, graphics stolen memory). NO measurable memory was lost
|
||||
on this boot.')))
|
||||
if (memattr_figures | trim | length) > 0
|
||||
else (' Whether memory was lost on this boot could NOT be measured here: neither
|
||||
present-vs-managed pages nor installed DIMM capacity was read — see check
|
||||
memory.physical_total for the reasons.') }}
|
||||
# The kernel log lines themselves, verbatim. A verdict the operator cannot check
|
||||
# against the evidence is not much better than a wrong one.
|
||||
memattr_evidence: >-
|
||||
@@ -837,23 +955,58 @@
|
||||
the kernel log was NOT read — {{ k.reason | default('no reason recorded', true) }}.
|
||||
Whether this node is affected is therefore UNKNOWN, not unaffected: the diagnostic
|
||||
lives only in dmesg.{{ memattr_corroboration }}
|
||||
{%- elif ma_state == 'failed_to_map' and ms.status == 'FAIL' -%}
|
||||
THE BUG FIRED AND MEMORY IS LOST ON THIS BOOT — measured, not inferred from this log
|
||||
line. The kernel failed to map the EFI Memory Attributes table at
|
||||
{{ k.memattr_table_address | default('an unreported address') }}, which proves the
|
||||
table was corrupt AND that this kernel trusted it far enough to memblock_reserve()
|
||||
from its contents; the measurement below shows that reservation was the catastrophic
|
||||
kind — a NEGATIVE tbl_size (num_entries x desc_size with bit 31 set), sign-extended
|
||||
through phys_addr_t and clamped to PHYS_ADDR_MAX, reserving everything above the
|
||||
table.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
|
||||
{%- elif ma_state == 'failed_to_map' -%}
|
||||
THIS NODE HAS LOST MEMORY TO THE BUG. The kernel failed to map the EFI Memory
|
||||
Attributes table at {{ k.memattr_table_address | default('an unreported address') }}
|
||||
and reserved memory from the corrupt table anyway — everything from that address
|
||||
upward.{{ memattr_evidence }}{{ memattr_corroboration }} {{ memattr_fix }}
|
||||
{%- elif ma_state == 'bad_version' -%}
|
||||
EXPOSED, not immune. The firmware publishes a CORRUPT EFI Memory Attributes table
|
||||
here too; this boot survived only because its garbage version field read
|
||||
{{ k.memattr_version | default('> 2') }}, which is > 2, so the kernel bailed out
|
||||
before reserving anything. The outcome is re-decided on EVERY boot: if that field
|
||||
reads <= 2 the kernel trusts the table and reserves gigabytes. Memory is intact right
|
||||
now and this node is one unlucky boot from the failure state.{{ memattr_evidence }}
|
||||
THE BUG FIRED ON THIS BOOT AND THIS BOOT SURVIVED IT — not a pass, and not memory
|
||||
loss either. The kernel failed to map the EFI Memory Attributes table at
|
||||
{{ k.memattr_table_address | default('an unreported address') }}: the table was
|
||||
corrupt and this kernel trusted it far enough to memblock_reserve() from its contents.
|
||||
WHAT THAT COST is decided by the garbage VALUE at that address on this boot, not by
|
||||
the address: catastrophic loss requires a NEGATIVE tbl_size (num_entries x desc_size
|
||||
with bit 31 set) which sign-extends and clamps to PHYS_ADDR_MAX, whereas a garbage
|
||||
tbl_size that stays POSITIVE reserves a modest, wrong amount that costs nothing
|
||||
measurable and still fails the later memremap() that printed this line. This boot drew a survivable
|
||||
value; the draw is repeated on EVERY boot, at the same address — this platform has
|
||||
been seen to lose 13.3 GiB and to keep its full memory at one identical table address
|
||||
(infra/PROD-PROVISIONING.md §0b).{{ memattr_evidence }}{{ memattr_corroboration }}
|
||||
{{ memattr_fix }}
|
||||
{%- elif ma_state == 'bad_version' and (memattr_kernel_fixed | bool) -%}
|
||||
the firmware publishes a CORRUPT EFI Memory Attributes table here, and the running
|
||||
kernel ({{ node_health_facts.os.kernel | default('unknown') }}) is >= 6.13, so it
|
||||
VALIDATES that table before reserving anything from it (upstream 8fbe4c49c0cc). On this
|
||||
boot the garbage version field read {{ k.memattr_version | default('> 2') }}, which is
|
||||
outside the 0..2 the spec allows, so the kernel rejected the table at the FIRST of
|
||||
efi_memattr_init()'s three gates and never reached the size arithmetic at all. THAT IS
|
||||
WHY THIS LOG SAYS "Unexpected ... table version" RATHER THAN THE FW_BUG "Corrupted EFI
|
||||
Memory Attributes Table detected!" LINE: the FW_BUG message comes from the THIRD gate
|
||||
(num_entries > 64K) and is only reachable on a boot whose garbage happens to leave the
|
||||
version field <= 2. Both messages are rejections, and which one a boot prints is the
|
||||
lottery — but on this kernel EVERY ticket in it is now a rejection, so the catastrophic
|
||||
path (memblock_reserve() of a negative tbl_size, sign-extended and clamped to
|
||||
PHYS_ADDR_MAX) is closed regardless of the draw. No action.{{ memattr_evidence }}
|
||||
{{ memattr_corroboration }}
|
||||
{%- elif ma_state == 'bad_version' -%}
|
||||
EXPOSED, not immune, though this boot took the safe path. The firmware publishes a
|
||||
CORRUPT EFI Memory Attributes table here too; this boot survived because its garbage
|
||||
version field read {{ k.memattr_version | default('> 2') }}, which is > 2, so the
|
||||
kernel bailed out BEFORE reserving anything from it. The outcome is re-decided on
|
||||
EVERY boot: if that field reads <= 2 the kernel trusts the table and reserves whatever
|
||||
tbl_size the garbage produces — nothing measurable if that value is positive,
|
||||
everything above the table if it is negative.{{ memattr_evidence }}
|
||||
{{ memattr_corroboration }} {{ memattr_fix }}
|
||||
{%- elif ma_state == 'corrupt_rejected' -%}
|
||||
the firmware publishes a corrupt EFI Memory Attributes table AND the running kernel
|
||||
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). Memory is
|
||||
intact and stays intact across reboots; no action.{{ memattr_evidence }}
|
||||
validated and rejected it (the >= 6.13 behaviour, commit 8fbe4c49c0cc). This bug
|
||||
cannot cost this node memory on this or any future boot; no
|
||||
action.{{ memattr_evidence }}{{ memattr_corroboration }}
|
||||
{%- elif not (k.boot_covered | bool) -%}
|
||||
NOT DETERMINED. The kernel log carries no memattr line, but it also no longer reaches
|
||||
back to early boot (no "Linux version" / "BIOS-provided physical RAM map" anchor in
|
||||
|
||||
@@ -12,6 +12,17 @@
|
||||
{% macro gib(n) -%}
|
||||
{{ (n | int / 1073741824) | round(1) }} GiB
|
||||
{%- endmacro %}
|
||||
{# smartctl OMITS `thresh` entirely for any attribute whose threshold the drive does not
|
||||
report (surfaced 2026-09-03 by k3s-prod-3's WDC WDS500G2B0A, which reports one for 1 of
|
||||
its 25 attributes) — an ABSENT key, not thresh=0, so a bare `a.thresh` raised
|
||||
"object of type 'dict' has no attribute 'thresh'" and killed the whole report. Render it
|
||||
as `n/a` (the same token this template already uses for unreported readings) rather than
|
||||
defaulting to 0, which would falsely read as a real ATA "cannot trip" threshold. The
|
||||
collector already guards this — tasks/collect_smart.yml uses selectattr('thresh',
|
||||
'defined') and `a.thresh | default(0)` — so only the template was exposed. #}
|
||||
{% macro thr(a) -%}
|
||||
{{ a.thresh | default('n/a') }}
|
||||
{%- endmacro %}
|
||||
================================================================================
|
||||
NODE HEALTH — {{ inventory_hostname }} ({{ ansible_host | default('n/a') }})
|
||||
collected {{ node_health_facts.collected_at }} as uid {{ node_health_facts.collector_uid }}
|
||||
@@ -73,19 +84,19 @@
|
||||
lifetime_remaining={{ (sm.lifetime_remaining_pct | string ~ '%') if sm.lifetime_remaining_pct is not none else 'not reported' }} temp={{ (sm.temperature_c | string ~ 'C') if sm.temperature_c is not none else 'not reported by this drive' }}
|
||||
pending(197)={{ sm.pending_sectors | default('?') }} offline_uncorrectable(198)={{ sm.offline_uncorrectable | default('?') }} reported_uncorrect(187)={{ sm.reported_uncorrect | default('?') }} crc(199)={{ sm.crc_errors | default('?') }}
|
||||
{% if sm.failing_now or sm.failed_past %}
|
||||
THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ a.thresh }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ a.thresh }}]{% endfor %}
|
||||
THRESHOLD TRIPS{% for a in sm.failing_now %} [{{ a.id }} {{ a.name }} FAILING_NOW value={{ a.value }}<=thresh={{ thr(a) }}{{ ' Pre-fail' if a.flags.prefailure | default(false) else ' Old_age' }}]{% endfor %}{% for a in sm.failed_past %} [{{ a.id }} {{ a.name }} In_the_past worst={{ a.worst }}<=thresh={{ thr(a) }}]{% endfor %}
|
||||
|
||||
{% elif sm.thresholds_present %}
|
||||
no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ sm.prefail_closest.thresh }}{% endif %}
|
||||
no attribute at/below its manufacturer threshold{% if sm.prefail_closest %}; closest Pre-fail margin: {{ sm.prefail_closest.id }} {{ sm.prefail_closest.name }} value={{ sm.prefail_closest.value }} vs thresh={{ thr(sm.prefail_closest) }}{% endif %}
|
||||
|
||||
{% endif %}
|
||||
vendor context (NOT used for the verdict — raw units are vendor-defined):
|
||||
reallocation events(196)={{ sm.realloc_events if sm.realloc_events is not none else 'n/a' }} grown bad blocks={{ sm.grown_bad_blocks if sm.grown_bad_blocks is not none else 'n/a' }} erase fails={{ sm.erase_fail_count if sm.erase_fail_count is not none else 'n/a' }} program fails={{ sm.program_fail_count if sm.program_fail_count is not none else 'n/a' }} unexpected power loss={{ sm.unexpected_power_loss if sm.unexpected_power_loss is not none else 'n/a' }}
|
||||
{% if sm.realloc_attr %}
|
||||
attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ sm.realloc_attr.thresh }} (the signal that is judged)
|
||||
attr 5 {{ sm.realloc_attr.name }} raw={{ sm.realloc_attr.raw.string | default(sm.realloc_attr.raw.value) }} (vendor-encoded) vs normalised value={{ sm.realloc_attr.value }} thresh={{ thr(sm.realloc_attr) }} (the signal that is judged)
|
||||
{% endif %}
|
||||
{% for a in sm.attributes | default([]) %}
|
||||
[{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ a.thresh }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %} <<< when_failed={{ a.when_failed }}{% endif %}
|
||||
[{{ '%3d' | format(a.id) }}] {{ '%-26s' | format(a.name) }} value={{ a.value }} worst={{ a.worst }} thresh={{ thr(a) }} raw={{ a.raw.string | default(a.raw.value) }}{% if a.when_failed | default('') %} <<< when_failed={{ a.when_failed }}{% endif %}
|
||||
|
||||
{% endfor %}
|
||||
{% else %}
|
||||
|
||||
@@ -25,7 +25,7 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
|
||||
| Requirement | Summary | How this component serves it |
|
||||
|---|---|---|
|
||||
| **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
|
||||
| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
|
||||
| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the node's data disk at `${LONGHORN_DATA_PATH}` (a `cluster-vars` knob: dev `/var/lib/longhorn`, a dedicated disk; prod `/data/longhorn`, a subdirectory of the partition it shares with k3s's data dir). `replicaAutoBalance: best-effort` re-spreads on node return. |
|
||||
| **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
|
||||
|
||||
## Dev-scoped choices (change for prod)
|
||||
@@ -48,9 +48,19 @@ A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`e
|
||||
|
||||
## Node prerequisites (already handled, informational)
|
||||
|
||||
Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
|
||||
`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
|
||||
`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
|
||||
Each node presents ext4 storage at `${LONGHORN_DATA_PATH}` and installs `open-iscsi` +
|
||||
`nfs-common` + kernel modules — all via `roles/node_baseline` (`infra/cloud-init/` is a README
|
||||
now; provisioning became API-only). This component assumes that floor. The shape differs per
|
||||
environment, which is why the path is a knob:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| dev | a **dedicated** disk (`/dev/vdb`, `LABEL=longhorn`) mounted at `/var/lib/longhorn` — the mount root *is* the dataPath |
|
||||
| prod | `/data/longhorn`, a **subdirectory** of the ~424 GB partition (`LABEL=k3s-data`) mounted at `/data`, whose sibling `/data/k3s` holds k3s's entire data dir (etcd + the containerd image store, moved off the 11.8 GiB `/var` — `infra/PROD-PROVISIONING.md` §2) |
|
||||
|
||||
`[parity]`: the value for an environment MUST equal `node_baseline_longhorn_data_path` in that
|
||||
environment's Ansible group_vars. Ansible creates and mounts the directory; this component tells
|
||||
Longhorn to use it.
|
||||
|
||||
## Contract (consumed by everyone with state)
|
||||
|
||||
|
||||
@@ -42,10 +42,22 @@ spec:
|
||||
defaultClass: false
|
||||
|
||||
defaultSettings:
|
||||
# --- data path (R-STATE-02) ---
|
||||
# Dedicated Longhorn data disk (/dev/vdb → ext4) is mounted here by cloud-init on every node
|
||||
# (infra/cloud-init/user-data.yaml.tftpl). Longhorn stores replicas on this disk only.
|
||||
defaultDataPath: "/var/lib/longhorn"
|
||||
# --- data path (R-STATE-02, R-ENV-02) ---
|
||||
# The on-node directory every replica is written into. A cluster-vars KNOB, not a literal,
|
||||
# because the two environments mount their data disk differently and a hardcoded path would
|
||||
# put one of them on the OS filesystem:
|
||||
# dev /var/lib/longhorn — a dedicated ext4 disk (/dev/vdb) mounted there by
|
||||
# roles/node_baseline. The mount root IS the dataPath: one disk,
|
||||
# one tenant. 18 live volumes; this value must never change.
|
||||
# prod /data/longhorn — a SUBDIRECTORY of the claimed ~424 GB partition mounted at
|
||||
# /data, whose sibling /data/k3s holds k3s's entire data dir
|
||||
# (etcd + the containerd image store, moved off the 11.8 GiB
|
||||
# /var). Two tenants, so Longhorn gets a subdirectory rather
|
||||
# than the mount root. infra/PROD-PROVISIONING.md §2.
|
||||
# [parity] MUST equal roles/node_baseline's node_baseline_longhorn_data_path for the same
|
||||
# environment — Ansible creates and mounts the directory, Helm tells Longhorn to use it.
|
||||
# Set in BOTH overlays (clusters/dev, clusters/prod); base default is the dev value.
|
||||
defaultDataPath: "${LONGHORN_DATA_PATH}"
|
||||
|
||||
# --- replication (R-STATE-02) ---
|
||||
# Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly
|
||||
|
||||
@@ -8,7 +8,9 @@
|
||||
# defaultClass: false # we own StorageClasses (storageclasses.yaml)
|
||||
#
|
||||
# defaultSettings:
|
||||
# defaultDataPath: /var/lib/longhorn # dedicated data disk (infra/cloud-init)
|
||||
# defaultDataPath: ${LONGHORN_DATA_PATH} # cluster-vars knob: dev /var/lib/longhorn
|
||||
# # (dedicated /dev/vdb), prod /data/longhorn
|
||||
# # (a subdirectory of the shared data partition)
|
||||
# defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT} # overlay knob (=3 in dev)
|
||||
# replicaSoftAntiAffinity: false # HARD anti-affinity: one replica per node
|
||||
# replicaZoneSoftAntiAffinity: true # single zone (3 mini-PCs, no failure zones)
|
||||
|
||||
Reference in new issue
Block a user