home-network-k3s-impl/infra/PROD-PROVISIONING.md

Prod bare-metal provisioning checklist (T-7.1)

Task: T-7.1 · Workstream: WS-INFRA · Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03 Status: Phase 7 opened by the operator 2026-08-31 — prep only. Nothing is installed on prod hardware until the blockers in §7 are closed and make -C ci gate is green.

Entry-gate note (recorded, not waived). roadmap/phases.md gates Phase 7 on "Phases 1–6 proven in dev/test; operator declares prod hardware ready." The operator has declared hardware ready. The first condition is not met: conformance is 37/51, and the uncovered set includes R-ENV-01/02/03 — the very requirements this document serves. We proceed with prep work knowingly; see DECISIONS-LOG.md (2026-08-31 Phase-7 entry).

Prod is 3× mini-PC bare metal (spec docs/02). The dev path (OpenTofu → Proxmox VM → cloud-init) does not apply: there is no hypervisor API to call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse the same Ansible roles and the same Flux base as dev — that reuse is the R-ENV-02 obligation.

---

0. Node inventory

Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.

	node 1	node 2	node 3
Hostname	k3s-prod-1	k3s-prod-2	k3s-prod-3
LAN IP	192.168.2.10	192.168.2.11	192.168.2.12
Model	HP EliteDesk 800 G5 Desktop Mini	HP EliteDesk 800 G5 Desktop Mini	HP EliteDesk 800 G5 Desktop Mini
CPU	i7-9700T, 8c/8t, VT-x	i7-9700T, 8c/8t, VT-x	i7-9700T, 8c/8t, VT-x
RAM	31 GiB (2×16 GB Micron)	2.6 GiB usable of 16 GB — see §0b	31.1 GiB usable of 32 GB (2×16 GB Crucial CT16G4SFD8266.C16FE, a matched pair)
Disks	1× 466 GB SATA SSD (Samsung SSD 850 EVO 500GB) — replaced 2026-09-07, was C400-MTFDDAC512M (~2011); see §0c	1× 477 GB NVMe (PM9A1 Samsung)	1× 466 GB SATA SSD (WDC WDS500G2B0A-00SM50)
Drive health	PASSED, 0 reallocated / 0 grown bad blocks, Wear_Leveling_Count raw 106, but 77,047 power-on hours (~8.8 y) over only 219 power cycles	96% life, 2142 h, PASSED	100% life, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC
OS	Debian 13 (trixie), kernel 7.1.8+deb13-amd64	Debian 13 (trixie), kernel 7.1.8+deb13-amd64	Debian 13 (trixie), kernel 7.1.8+deb13-amd64
Firmware	HP R21 Ver. 02.26.00	HP R21 Ver. 02.26.00	HP R21 Ver. 02.26.00
/dev/kvm	present	present	present
Serial	MXL1052NGN	MXL0073949	8CC015307N

Node 1's DIMMs are also not a matched pair (surfaced 2026-09-01 by the new health check; unrelated to §0b and not a fault): DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both are 16 GB so capacity is correct, but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch.

The nodes are NOT hardware-symmetric — and with node 3 in (2026-09-03) the asymmetry is THREE-WAY, not a node-1 outlier. Node 1 has a ~2011 SATA SSD with real wear; node 2 a healthy modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent health (100% life, 0 grown bad blocks). Three nodes, three different drive models, and only one is NVMe. §2 requires an identical disk layout across all three because Longhorn replica placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should have had; node 3 does not have it either. Node 3 is also the only node with two matched DIMMs

§0c — node 1's drive was replaced (2026-09-07)

The paragraph above is history, not current state. On 2026-09-07 node 1's kubelet stopped posting status while the host stayed up and SSH-able. Root cause was etcd starved of disk I/O, not memory: slow fdatasync at 3.5–6.7 s against an expected 1 s, apply request took too long at 8.3 s against 100 ms, then etcdserver: too many requests. No OOM, MCE or hung task in the kernel log. All three nodes place etcd's WAL (/data/k3s) and every Longhorn replica (/data/longhorn) on the SAME partition; node 1's was the 2011 C400 with 61,440 reallocated sectors and ~31% of its endurance consumed, and its /proc/pressure/io full avg300 ran 11.36 against ~1% steady state on the other two. The failure is self-reinforcing: the node dropping out degrades Longhorn volumes, whose rebuilds add write load to the very disk etcd needs.

The operator replaced the drive with a Samsung SSD 850 EVO 500GB and reinstalled Debian. Measured under the same load class — an active 18-replica Longhorn rebuild, with node 1 as the write target, i.e. the heavier side:

node	drive	full avg300 during rebuild
k3s-prod-1	Samsung 850 EVO (new)	3.79
k3s-prod-2	Samsung PM9A1 NVMe	0.33
k3s-prod-3	WDC WDS500G2B0A	2.47
k3s-prod-1 before	C400-MTFDDAC512MAM	11.36

Roughly a 3× improvement, moving node 1 from ~4× worse than its peers into the same band as node 3, and it stayed Ready throughout. The 850 EVO is not new: 77,047 power-on hours over 219 power cycles — it evidently idled in an always-on system — but its wear tells the opposite story to the C400's (0 reallocated vs 61,440; wear-leveling 106 vs 949).

§2's exit condition is NOT met by this replacement. etcd and Longhorn still share one SATA spindle. The contention is reduced, not eliminated; giving etcd its own device remains the fix, and node 1 still has exactly one disk. The three nodes also remain asymmetric — now 850 EVO / PM9A1 NVMe / WD Blue.

The rebuild itself surfaced two things worth carrying forward. Longhorn refused the new disk with DiskFilesystemChanged — "record diskUUID doesn't match the one on the disk" — because its node CR still held the old drive's diskUUID; the remedy is disable → remove the disk entry → re-add, in that order (the validating webhook enforces it). And the Ansible inventory still listed node 1 in [k3s_prod_primary], which decides both which play gets --cluster-init and what k3s_api_host resolves to — so re-provisioning it unchanged would have pointed a rebuilt, empty node at itself and founded a second cluster beside the live one. The founder now sits on node 2. (2× Crucial 2Rx8 DDR4-2666) and the only one with its full RAM complement of 32 GB actually usable. That is luck, not immunity: node 3's firmware publishes an EFI Memory Attributes Table too (MEMATTR=0xbeece018), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to the §0b bug on exactly the same terms as node 1 and needs the same >= 6.13 kernel. Node 3 therefore WIDENS §0b's blast radius from "both prod nodes" to all three; it is not a control machine. Every node also carries exactly one physical disk with no unpartitioned spare, so node_baseline_longhorn_device / node_baseline_etcd_device have nothing to claim on any of the three without repartitioning or adding drives — the §2 decision cannot be satisfied by node 3 either.

Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, two dedicated data disks (Longhorn + etcd).

---

0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01)

This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory remapping, then stale Fast Boot memory training. All three were wrong. The hardware is fine. Keeping the correction visible because the wrong framing nearly cost money and a board RMA.

k3s-prod-2 intermittently addresses only ~2.6 GiB of its 16 GB. Root cause: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before reserving memory based on its contents. Fixed upstream in 6.13.

The evidence that breaks it open

The same machine — same DIMM, same BIOS, same kernel, same cmdline — booted with the full 15.9 GiB two boots earlier:

Boot	Memory: … available	reserved	memattr line
−2	15,917,744K	615,208K	Unexpected EFI Memory Attributes table version 1463964749
−1	2,613,912K	13,913,856K	Failed to map EFI Memory Attributes table @ 0xbeeb0018
0	2,568,544K	13,958,980K	Failed to map EFI Memory Attributes table @ 0xbeeb0018

The BIOS-e820 map is byte-identical across all three. No hardware fault is intermittent in that shape, and none is fixed by a reboot.

Mechanism (verified against kernel source)

drivers/firmware/efi/memattr.c, v6.12 and current linux-6.12.y:

if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; }
tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
memblock_reserve(efi_mem_attr_table, tbl_size);

num_entries and desc_size are never validated. With a corrupt table:

• garbage version reads > 2 → kernel warns and bails → memory intact (the lucky path)
• garbage version reads ≤ 2 → kernel multiplies two garbage fields and memblock_reserve()s the product — gigabytes, from the table address to the top of RAM (the unlucky path)

That accounts for every observation: Normal zone present 3340288 / managed 0, and MemTotal = DMA (3840) + DMA32 (684481) pages = 2,753,284 kB exactly, with Normal contributing zero.

ALL THREE prod nodes are affected — none is immune

Seven boots observed across the three machines. The MAT address moves boot-to-boot on every node with settings held constant, and each node has been seen on more than one path:

Node	Boot	MEMATTR=	Kernel's read	Memory
1	−1	0xbeee6018	(no message — table read as VALID)	32.4 G ✅
1	0	0xbeec9298	version 1030 (> 2 → bail)	32.4 G ✅
2	−2	0xbeadc018	version 1463964749 (> 2 → bail)	15.9 G ✅
2	−1	0xbeeb0018	Failed to map	2.6 G ❌
2	0	0xbeeb0018	Failed to map	2.6 G ❌
3	−1	0xbee8a018	version 1463964749 (> 2 → bail)	32.4 G ✅
3	0	0xbeece018	(no message — table read as VALID)	32.4 G ✅

So there are three outcomes, not two: bail-out on a garbage version > 2, a valid parse with a sane size, and the catastrophic path. k3s-prod-3 publishes a MAT at 0xbeece018 exactly like the others — the absence of a memattr diagnostic line on its current boot means the kernel read a valid table, which is the opposite of immunity. Dev nodes publish no MAT at all (Proxmox SeaBIOS/OVMF) and are genuinely unaffected.

All three prod nodes are on the same lottery and all three need the ≥ 6.13 kernel.

The corrupting bytes are the OS boot chain, not the firmware or the DRAM

The garbage version 1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical corrupting bytes. That rules out both firmware-specific behaviour and a DRAM fault: the overwriter is the shared OS boot chain (shim / GRUB / EFI stub / initrd placement), writing the same data into the page the firmware pointed at. This is the documented failure mode — the firmware places the MAT in memory the OS classifies as plain System RAM, and whoever allocates there next overwrites it.

It also means the boot chain's own allocation footprint — initrd size, module set, boot device — is part of the causal chain, which matters for the experiment design below.

Mechanism, traced to the byte (second, independent confirmation)

A second research pass reached the same root cause by a different route and pinned the arithmetic:

• The garbage version node 2 read was 1463964749 = 0x5742544D = the ASCII bytes MTBW. The kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that /sys/firmware/memmap calls plain System RAM, so the pre-Linux boot chain overwrites it.
• tbl_size is a signed int. num_entries * desc_size is u32 * u32, wraps mod 2^32, and when bit 31 lands the result is negative.
• memblock_reserve(base, tbl_size) converts that negative int to phys_addr_t, sign-extending to ~0xFFFFFFFF_xxxxxxxx; memblock_cap_size() clamps it to PHYS_ADDR_MAX - base. Every physical address above 0xbeeb0018 is reserved.
• memblock_free_all() then hands the buddy allocator nothing above that line, while present_pages (which comes from memblock.memory) is untouched — producing present 3340288 / managed 0.
• The Failed to map EFI Memory Attributes table message is emitted later, by efi_memattr_apply_permissions(), whose memremap() of the absurd size fails. That message is therefore positive proof the reservation already happened — it is a consequence, not the cause.

The cleanest A/B tell between the two nodes is one line of dmesg:

	NODE_DATA(0) allocated	meaning
prod-2 (bad)	[mem 0xbee85680-0xbeeaffff]	ends at 0xbeeb0000 — pgdat forced immediately below the reservation
prod-1 (good)	[mem 0x82f7d5680-0x82f7fffff]	ends at 0x82f800000 — top of RAM, the normal placement

Identical 0x2a980 allocation size, opposite placement. Node 2's DMA32 managed is also 684481 vs node 1's 775777 on identical present — ~356 MiB of extra structures forced below 4 GiB.

CORRECTION (2026-09-03) — "Failed to map" does NOT mean memory was lost

An earlier version of this section, and the firmware.efi_memattr health check built from it, treated the log line efi: memattr: Failed to map EFI Memory Attributes table @ 0x… as proof of memory loss. A live counter-example falsifies that. k3s-prod-2, after being power-cycled on 2026-09-03:

MEMATTR=0xbeeb0018                                             <-- the SAME address as both bad boots
[ 0.106493] efi: memattr: Failed to map ... @ 0xbeeb0018       <-- the SAME line
MemTotal: 15857732 kB                                          <-- 15.1 GiB of 16 GB. FULL MEMORY.

Two different kernel functions were being conflated:

• efi_memattr_init() does the memblock_reserve(). Catastrophic loss requires tbl_size to be negative — signed int, sign-extended through phys_addr_t, clamped to PHYS_ADDR_MAX, so everything above the table is reserved. A garbage-but-positive tbl_size reserves only a modest wrong amount and costs nothing noticeable.
• efi_memattr_apply_permissions() does the later memremap(), and this is what prints "Failed to map". It fails for any sufficiently wrong size — negative or merely large-positive.

So the line proves the table was corrupt and the kernel trusted it, not that memory was lost. The cost is set by the value of the garbage on that boot, which is why the same address yielded 2.6 GiB twice and 15.1 GiB once.

This also refutes the address→outcome hypothesis outright — it is not merely untested, it is disproven. The MAT address does not determine the outcome; the garbage value at it does.

The health check has been re-based accordingly: the measured shortfall is computed once into a shared fact and both memory.physical_total and firmware.efi_memattr read it, so they can no longer contradict each other. Failed to map + normal memory is now WARN ("the bug fired and this boot survived it"), and FAIL requires a measured shortfall.

SECOND FAULT — node 2 hangs on a warm reboot, staying powered

On 2026-09-03 k3s-prod-2 failed to return from the first scripted reboot of the survey below. Its journal ends cleanly mid-shutdown (services stopping, eno1: NIC Link is Down) and then stops. The box stayed powered on but off the LAN for ~38 minutes — no ping, no SSH, ARP INCOMPLETE — until the operator held the power button and cold-booted it by hand.

This is arguably a worse prod defect than the memory bug. A node that cannot survive an unattended warm reboot cannot be rolled in an HA cluster: every kernel update and maintenance window becomes a physical visit, and roles/node_health cannot see a machine that is off the network.

Leading candidate: node 2 is the only node with Recover OS after Boot Failure = Enable. A box parked at a firmware recovery prompt awaiting a keypress presents exactly this way. Unproven — it needs eyes on the screen during a hang — but it promotes that setting from "coincidence" in the BIOS diff to a live availability defect. Worth disabling on all three nodes regardless.

Consequence for testing: the reboot survey is not viable unattended on node 2 until this is understood or out-of-band recovery exists.

THERE IS NO KERNEL CMDLINE WORKAROUND

efi_memattr_init() is called from efi_config_parse_tables() under only !CONFIG_X86_32 && efi_enabled(EFI_MEMMAP), and x86's efi_init() calls efi_config_init() before the efi_runtime_disabled() bail-out. So noefi, efi=noruntime and efi=novamap do not prevent the reservation. Booting a fixed kernel is the only real fix.

The fix

Boot a kernel ≥ 6.13 — upstream commit 8fbe4c49c0cc "efi/memattr: Ignore table if the size is clearly bogus" (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message describes this bug verbatim. Verified absent from v6.12 and from the current linux-6.12.y stable branch. On Debian 13 that means linux-image-amd64 from trixie-backports. A fixed kernel prints FW_BUG Corrupted EFI Memory Attributes Table detected! and keeps all the memory.

The hardening landed exactly one release too late for Debian 13: the bug has existed since the feature was introduced in 2016, 8fbe4c49c0cc first shipped in v6.13, and a follow-up 5d0faa8e8369 (2026-03-26) tightened it further to the current form, whose comment reads verbatim "This prevents a corrupted table from eating all system RAM." Debian's 6.12.107-1 demonstrably lacks it. Worth a Debian BTS report against src:linux — 6.12 is Debian 13's kernel for the release lifetime, and the fix is small and already upstream.

Same signature reported elsewhere: Ubuntu LP #2039382 (Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the table) and LP #1944019 (HP ZBook Studio G5, 16 GiB → 1.5 GiB).

Will 6.12 ever get the fix? — researched 2026-09-02. Not on its own.

Question	Answer
In linux-6.12.y today?	No. Newest commit touching memattr.c on that branch is 2024-07-12.
How long has it been available?	In mainline since v6.13, 2025-01-19 — ~19 months and ~97 6.12.y point releases.
Tagged Cc: stable?	No. Neither commit.
Tagged Fixes:?	No — and this is structural, not an oversight. The unvalidated arithmetic has been there since the file was created in 2016; it is not a regression, so there is no bad commit to point a Fixes: at. Both automatic feeds — the Fixes:-scanning scripts and AUTOSEL — had nothing to grab.
Picked up by AUTOSEL?	Never surfaced for any branch.
Has anyone asked?	No. Nothing on [email protected], and nothing in all 8,027 open+archived Debian bugs against src:linux. We would be first.
6.12 LTS horizon	EOL December 2028 — outlives Debian 13's own support window, so a backport is not time-barred.

Every automatic path has already had its chance and produced nothing. The probability that 6.12.y acquires this commit spontaneously is close to zero.

But the request is unusually strong, and the cost is one email. The decisive card: the cherry-pick is byte-clean. Debian trixie 6.12.94-1's drivers/firmware/efi/memattr.c hashes to 164203429fa7a…, identical to the upstream patch's pre-image blob — the file has not diverged at all between v6.12 and the commit's parent, so git cherry-pick 8fbe4c49c0cc applies with zero conflicts and zero adjustment. Neither the stable team nor Debian would have any backporting work to do, which removes the usual reason such requests are declined. The commit also meets every criterion in stable-kernel-rules.rst: in mainline, 17 insertions (limit 100 lines), fixes "a hardware quirk" and "a real bug that bothers people" — the enumerated categories.

Debian rebases, so an upstream pickup reaches us automatically: trixie carries linux 6.12.94-1 in trixie-security whose changelog opens "New upstream stable update: ChangeLog-6.12.91". No action beyond apt upgrade.

Plan

1. Run trixie-backports on both prod nodes now, as an explicitly provisional measure. Any 6.13+ kernel carries the fix. Record the standing cost: backports packages are provided as-is, without the main archive's security guarantees.
2. Email [email protected] — this is stable-kernel-rules.rst Option 2 for an already-merged patch. CC [email protected] and Ard Biesheuvel <[email protected]>, who both authored and committed the fix as EFI maintainer. Include: the commit ID and subject; 6.12.y only; the byte-clean-apply proof above (it pre-empts the main objection); and the reproduction — three consecutive boots on one machine, two outcomes, byte-identical BIOS-e820 across all three (the detail that proves the divergence is in the MAT, not the memory map), 16 GB → 2.6 GiB, plus a second identical machine taking the safe path only by chance. Attach both dmesgs. Corroboration exists: Ubuntu LP #1944019 (HP ZBook G5, 16 GiB → 1.5 GiB) and SUSE bugzilla 1231465, cited in the commit message itself.
3. File a Debian BTS bug as secondary, against src:linux via reportbug (the bug-script metadata is mandatory), severity important, referencing the lore thread. Debian's kernel handbook §9.1.7 allows cherry-picking a patch already accepted upstream — which this is, maintainer-authored with two Acked-bys — so if stable@ goes quiet Debian can act on its own.
4. Trigger to make the deviation permanent: no stable@ pickup within ~2 release cycles (4–6 weeks) and no Debian movement ⇒ encode the backports kernel in the provisioning IaC for every prod node and document it as a standing deviation from the distro baseline.

If patching by hand, only 8fbe4c49c0cc is needed. The follow-up 5d0faa8e8369 loosens a heuristic that was rejecting legitimate tables (false kexec warnings in Meta's fleet); it is not required for this symptom and depends on the first commit anyway.

Is it a BIOS setting? — three-way diff, researched 2026-09-03. No.

The operator's hypothesis, tested properly now that there are two working nodes and one broken one. All 240 attributes read on all three; identical key sets; identical firmware everywhere (same BIOS version string, ME 12.1.1.1022, GOP 9.0.1085, Super I/O 8.9.22). So every difference is settings or hardware, not firmware revision.

The discriminating set — both working nodes agree, the broken node differs. 26 attributes differ in total; after excluding per-machine noise (serials, UUIDs, MACs, asset tags, Born On Date, Feature Byte, SKU, memory size, storage device names) and settings falling on a different partition, seven remain:

Attribute	nodes 1 & 3 (work)	node 2 (broken)	Verdict
Fast Boot	Disable	Enable	plausible probability-shifter — the only live candidate
Legacy Boot Options	Disable	Enable	coincidence — the setting is INERT, see below
Microsoft / Windows / MS Option ROM UEFI CA 2023	Enable	Disable	coincidence (sub-page db size at most)
TPM Activation Policy	F1 to Boot	Allow user to reject	coincidence
Recover OS after Boot Failure	Disable	Enable	coincidence (but a policy problem in its own right)

After Power Loss is eliminated by the third node — node 3 works and shares node 2's Power Off; node 1 is the odd one out. It was never mechanistically credible (PCH/ME power policy, applied before firmware executes). It remains a §1 checklist violation on two nodes now.

Legacy Boot Options is affirmatively inert, which a naive diff gets wrong. Three attributes are identical on all three nodes and settle it: Configure Legacy Support and Secure Boot = "Legacy Support Disable and Secure Boot Enable", Configure Option ROM Launch Policy = "All UEFI", UEFI Boot Options = Enable. Legacy Boot Options is only the boot-order checkbox; the master switch that actually instantiates a CSM has legacy support disabled on node 2 too. No CSM is built, no legacy option ROM is shadowed. The second-strongest candidate has no boot-time allocation footprint at all. Secure Boot is likewise not a differential — all three log Secure boot enabled and load the same HP DB cert.

Fast Boot is the one credible mechanism, and for a different reason than the hypothesis this document previously recorded and killed. HP documents it verbatim: "reduces boot up time by bypassing boot to USB, CD-ROM, and PXE. This skips some preboot initialization steps." Skipped preboot init ⇒ fewer boot-services allocations before the MAT is installed ⇒ a different heap top ⇒ the MAT lands at a different address. That is exactly the perturbation that flipped the near-identical Dell case in LP #2039382, where the reporter's "fix" was unplugging a USB mouse. (The old Fast-Boot-caches-MRC-training theory stays dead — byte-identical BIOS-e820 refuted it.)

Verdict: the hypothesis is not supported, and this diff cannot support it even in principle

1. No setting can be the cause. Node 2 booted good once and bad twice with settings constant, and the MAT address moves every boot on all three machines.
2. Fast Boot may shift the odds. That is the honest ceiling. There is currently zero evidence that it does.
3. A fatal confound: the 1=3 ≠ 2 partition is perfectly collinear with two hardware differences that have at least as strong a mechanism — node 2 is the only single-DIMM node and the only NVMe-booting node. Nodes 1 and 3 are both dual-DIMM and both SATA. Adding node 3 did not break the confound; it doubled it. A different boot device means a different initrd module set and size, hence a different EFI-stub allocation footprint — the very thing that decides whether the MAT page gets overwritten.

No model-specific documentation exists. HP publishes nothing mentioning the EFI Memory Attributes Table, MAT placement, or an NX/DEP option on this generation. The only memory-adjacent options (Option ROM Launch Policy, DMA Protection, Pre-boot DMA protection) are identical fleet-wide. No HP advisory matches. The R21 changelog never mentions memory, and these nodes run 02.26.00 dated 05/05/2026 — newer than anything HP has indexed publicly, so there is no BIOS update to chase.

The experiment, if it is ever worth running

Two statistics kill the obvious approach. Node 2's failure rate is known only to n = 3 (2 bad of 3); the 95% Clopper–Pearson interval on p(bad) is ≈ 0.09–0.99. Against a null that loose, even 8 clean boots after a change prove nothing. A baseline arm is mandatory, and: detecting elimination needs ~20 boots per arm; detecting a mere odds shift (0.67 → 0.33) needs ~50 per arm for 80% power. Do not design for an odds shift — design for elimination and report "inconclusive" otherwise.

Zero-touch first step, and the one actually worth doing: 20 scripted reboots of node 2 with nothing changed, logging MEMATTR= and the Memory: …K/…K available line per boot. That establishes the real p(bad) we lack, and tests the causal chain directly — is the outcome predicted by where the MAT landed? Present data hint yes: both failures at exactly 0xbeeb0018, all five successes elsewhere. If the bad outcome maps onto a narrow address range, the mechanism is settled without touching anything.

If more is wanted, swap hardware rather than BIOS settings — it crosses settings against memory topology in one move, needs no firmware write (writes are refused; a BIOS change means a physical F10 visit per arm), and is work §2 wants anyway: move node 2's single Samsung DIMM into node 3 and node 3's matched pair into node 2, then 20 boots each. Fault follows the DIMM ⇒ topology, settings exonerated. Fault stays on node 2 ⇒ node 2 implicated, and only then is Fast Boot worth testing. Fault disappears ⇒ single-DIMM topology. ~1 hour unattended.

None of this changes the action. The fix is already known, is not in dispute, and applies to all three nodes: kernel ≥ 6.13.

Reboot survey — 40 boots, run 2026-09-03

playbooks/memattr-survey.yml (+ roles/memattr_survey), reboots opt-in via memattr_survey_reboots_authorised, resumable from an on-node ledger, artifacts in _artifacts/memattr-survey/.

node	bad	n	p(bad)	95% Clopper–Pearson
prod-1	0	20	0.00	0.000 – 0.168
prod-2	—	0	—	no data — hung on its first reboot
prod-3	0	20	0.00	0.000 – 0.168
1+3 pooled	0	40	0.00	0.000 – 0.088

The arm the experiment was designed around produced nothing. Node 2's prior 2-of-3 is unchanged. And 0/20 is not elimination — it still permits p(bad) up to 17% per node.

What the 40 boots did establish:

• Addresses are node-specific and highly concentrated. Node 1 used only 3 distinct addresses across 20 boots (15 of them at 0xbeec9298); node 3 used 8. No address was ever shared between nodes. This is a lottery with very few tickets per machine — which is exactly why 20 clean boots on nodes that only ever drew safe tickets says little about the node that drew a bad one.
• MEMATTR → kernel parse was a perfect function across all 7 addresses seen more than once (37 boots), with both readings present in the sample. Same address, same parse, every time. Combined with the counter-example above, the picture is: the address determines what the kernel finds, and the garbage value there determines whether it is survivable.
• The corrupting bytes are stable per machine, not fleet-wide. Node 1's garbage version was 1030 on all 22 of its bail boots across three different addresses; node 3's was 1463964749 on all 12. So the earlier "shared OS boot chain" inference needs refining: the boot chain remains the plausible writer, but its footprint is machine-specific.

Ansible defect worth knowing repo-wide: a looped include_tasks under the default linear strategy is flattened per host, so hosts run serially. The first survey attempt rebooted node 1 twenty times while nodes 2 and 3 sat idle. Fixed with strategy: free plus a preflight assert that forks ≥ host count. It is silent and costs 3× wall clock.

What NOT to do

• Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures. The DIMM is a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (DIMM1 must always be installed). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6.
• BIOS 02.27.00 (sp174287) helps only by accident, by shifting where the table lands. The full R21 changelog 02.10.01 → 02.27.00 contains zero memory/MRC/DIMM entries.
• Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB are all the same lottery ticket: they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They may appear to "fix" it. They are not fixes.

Hypotheses killed, and why — so they are not re-run

• Fast Boot / stale MRC training data — killed as a mechanism. Intel MRC caches are guarded by an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control.
• Memory remap / above-4G decoding — no such attribute exists among the 241 exposed by hp-bioscfg on this platform.
• Integrated graphics — Graphics Stolen Memory is exactly 64 MiB, below 4 GiB.
• EFI_MEMORY_SP / "soft reserved" memory — the strongest competing theory, excluded by measurement: /sys/firmware/memmap/10/type reads System RAM for 0x100000000-0x42f7fffff. Soft-reserved ranges become E820_TYPE_SOFT_RESERVED, print as Soft Reserved, and are never memblock_add()ed — which would have left present at ~0, not 3340288.
• SGX — ruled out at runtime: x86/cpu: SGX disabled or unsupported by BIOS.
• MTRR trim — MTRRs near-identical to node 1, no trim warning.
• A last_pfn cap — a misreading; the second value is max_low_pfn and is normal.
• The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly explained: everything from 0xbeeb0018 upward.

1. Firmware / BIOS (per node, before OS install)

These are MACHINE-CHECKABLE — but NOT machine-enforceable. The hp-bioscfg driver exposes 241 BIOS attributes at /sys/class/firmware-attributes/hp-bioscfg/attributes/ (239 have a current_value; 215 returned a non-empty value — an earlier note here said "215 attributes", which was the non-empty subset, not the total). Reads work fine: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found at all. Nobody could have spotted After Power Loss=Power Off on node 2 by eye.

Writes are refused by the firmware — verified 2026-09-01, hp_bioscfg: Returned error 0x4, "Invalid command type". HP requires BIOS-admin authentication and neither credential is provisioned (Setup Password is_enabled=0, SPM not provisioned). Setting these therefore means F10 on the box.

Follow-up worth doing: extend roles/node_health to ASSERT this checklist — that is achievable today with reads alone, and would have caught node 2's drift on its first health run. Converging it automatically additionally requires provisioning a BIOS Setup Password on every node, which is a real security decision (a stored credential that can rewrite firmware) and is the operator's call, not a default.

• Virtualization enabled — VT-x and VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with /dev/kvm present and grep -cE 'vmx|svm' /proc/cpuinfo non-zero.
• AC power recovery = Power On. A homelab node must return after a power cut without a human pressing the button. (docs/02 calls out wake/boot behaviour explicitly.)
• Wake-on-LAN enabled on the onboard NIC (remote recovery when the box is off).
• Boot order — internal disk first; USB above it only while imaging. Disable network boot unless PXE is adopted.
• Secure Boot — record the setting. k3s itself is fine either way; note it so all three nodes match.
• Firmware version recorded in §0 and consistent across all three nodes.

1b. BIOS passwords and programmatic configuration — researched 2026-09-01

There is no default BIOS password, and no factory default exists to look up. HP business desktops ship with no Setup password and no Power-On password set; the is_enabled=0 we observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5 describes both purely as things the owner configures in Computer Setup.

Owner recovery path, for the runbook: if a Setup password is ever set and forgotten, HP documents clearing it with the PSWD jumper on the system board (power off, open, remove jumper, boot once). Do NOT enable "stringent security" — HP documents that it removes the physical bypass entirely, turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The permissive default is the right setting for a homelab.

CORRECTION — error 0x4 is NOT an authorisation failure

I recorded that the firmware refused our write because BIOS-admin authentication was missing. That was wrong, and the kernel driver misled me. hp-bioscfg passes the return value of HP's WMI SetBIOSSetting method through hp_wmi_error_and_message() — which is the error table for a different interface (the "SECU" commands under HP_WMI_BIOS_GUID). The two methods have unrelated return-code spaces. Per HP's Client Management Interface white paper, SetBIOSSetting returns:

Code	Meaning
0	Success
1	Not Supported
4	Failed (generic)
6	Access Denied

So the firmware said a generic "Failed". An authorisation problem would have returned 6, and the driver has a separate BIOS_ADMIN_NOT_SET = 0x38 → -EPERM for the no-password case — neither of which we saw. The string "Invalid command type" printed in our logs is the driver looking 4 up in the wrong table. This is a genuine upstream bug worth reporting to platform-driver-x86.

Most likely actual cause (inferred, testable): with an empty password the driver emits an untagged empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when blank (Password = "<utf-16/>"). The driver only prepends that prefix for a non-empty string. So setting a Setup password may well fix writes — but not because 0x4 means "unauthorised"; it would work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this firmware already makes the driver log Prerequisites size value exceeded the maximum number of elements supported or data may be malformed at boot.

hp-bioscfg cannot set a password — verified from source. Both current_password and new_password funnel into store_password_instance(), which only strscpys into driver RAM; nothing ever transmits new_password to firmware. It is write-only into a buffer that is never read.

Recommendation

1. Land the read-only drift check now — encode this §1 checklist as expected values, diff current_value on every node in roles/node_health, fail on mismatch. That is the whole R-ENV win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10.
2. Do not provision a Setup password "to fix the auth error" — there is no auth error. Provision one only as a deliberate experiment on node 3 (not yet built): set it at F10, write it to current_password, retry one benign attribute. Return 0 confirms the <utf-16/> theory; 6 would mean a password mismatch; another 4 means the driver is incompatible with R21 and the fallback is hp-repsetup.
3. Fallback if that fails: HP's own Linux tool hp-repsetup (HP Flash and Replicated Setup Utilities, v3.26) can both dump/apply settings and set the Setup password from Linux — but it needs the out-of-tree hpuefi.ko, which requires Secure Boot disabled or the module self-signed. Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a security downgrade.
4. Skip HP Sure Admin / SPM and BCU. Sure Admin's key-based model is genuinely better in principle — the private key never leaves the workstation, each change is individually signed, with anti-replay — but provisioning and every signature require Windows-only tooling. For three machines whose firmware settings change roughly never, the operational cost far exceeds the risk reduction. BCU is Windows/WinPE-only.
5. Security trade-off if you do provision a password: it is a shared credential granting full pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) — functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via hp-bioscfg needs root on the node already (sysfs is --w-------), so the marginal loss is persistence and pre-boot control rather than initial compromise. Note hp-repsetup -p takes the password in cleartext on the command line — it would land in process listings and shell history unless wrapped. Store it in the Ansible vault and offline.

2. Disk layout — DECIDED (operator, 2026-09-03) · REVISED (operator, 2026-09-04)

docs/02 requires the OS-vs-Longhorn device split to be documented before prod build. Dev went further and added a third device for etcd after HDD contention starved etcd WAL fsync (docs/plans/ssd-etcd-isolation.md).

Node 1 as-found is a single 477 GB SATA SSD carrying everything:

sda  477G  C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
├─sda1 976M  /boot/efi
├─sda2  30G  /
├─sda3  12G  /var
├─sda4 977M  swap
└─sda5 433G  /srv          <- the only large free space

THE DECISION (operator, 2026-09-03; layout REVISED 2026-09-04) — made, applied, and measured-not-assumed

Data partition	the existing /srv partition on each node's single physical disk (~432 GB nodes 1–2, ~422 GB node 3), re-purposed, re-labelled k3s-data and mounted at the neutral /data
Longhorn store	/data/longhorn — a subdirectory of that partition
k3s --data-dir	/data/k3s — the sibling subdirectory. Holds embedded etcd AND the containerd image store
etcd	no dedicated device — it lives at /data/k3s/server/db, i.e. on the data partition, not on the OS filesystem
Shared-device I/O contention	ACCEPTED, knowingly — unchanged by the 2026-09-04 revision

THE 2026-09-04 REVISION, AND THE CAPACITY EVIDENCE THAT FORCED IT

The 2026-09-03 ruling gave this partition to Longhorn only and left k3s at its default /var/lib/rancher/k3s. That was wrong, for a reason that is arithmetic rather than judgement.

/var on these nodes is a separate partition of 11.8 GiB (11.6 on node 3) — see the partition map above; it is /dev/sda3 / /dev/nvme0n1p3, not the 30 GiB /. k3s's data dir puts two things on it: etcd's database and WAL (server/db) and, far larger, the containerd image store (agent/containerd).

Measured on the fully-deployed dev cluster — the same platform and app set prod will run:

agent/containerd per node	21–25 GB
total /var per node	58–70 GB
prod /var capacity	11.8 GiB

So prod would have filled /var during its first image pulls — not eventually, not under load, but on the way up. And because etcd was on that same partition, a full /var means etcd cannot write its WAL. That is cluster death, not degradation: a wedged etcd on all three servers is an unavailable control plane, and it would have happened during bring-up.

The fix is to move k3s's whole data dir onto the 424 GB partition, which is why that partition's mountpoint and filesystem label are no longer Longhorn-specific — it now serves two tenants. /var retains only the OS's own usage (503 MB, 5% used, measured 2026-09-04).

Why now. Nothing was installed on prod when this was decided — verified 2026-09-04: no /usr/local/bin/k3s, no /etc/rancher/k3s, no /var/lib/rancher on any node. Moving a data dir before first install is a one-line inventory change; after bring-up it means draining and re-imaging three servers.

Why /data and LABEL=k3s-data.

• /data — short, neutral, tenant-agnostic, and not under /var. Deliberately not /srv: /srv is the claim mountpoint the role discovers the device at, and the idempotency contract is that /srv is gone from /proc/mounts and /etc/fstab once the claim has happened. Re-using it as the destination would make "claimed" and "not yet claimed" indistinguishable.
• LABEL=k3s-data — deliberately not the bare data. Guard #4 of the role's assert wall accepts a partition whose label is empty or already ours, and names data as an example of somebody else's named volume; labelling ours data would make a stranger's data volume look already-claimed and weaken that guard. k3s-data is cluster-scoped rather than tenant-scoped, and matches the k3s-etcd label convention already in the repo.

The shared-device I/O risk is UNCHANGED and still accepted. Longhorn and etcd were already going to share this spindle under the 2026-09-03 ruling — the revision changes which filesystem etcd sits on, not which device. The acceptance rationale and its exit condition below stand verbatim.

How prod's data dir is scoped away from dev. k3s_data_dir is a single variable read by two roles: roles/k3s_server turns it into --data-dir in the install flags, and roles/node_baseline (via node_baseline_k3s_data_dir, which defers to it) creates the directory 0700 on the data partition and derives etcd's location from it (node_baseline_etcd_mount = <data-dir>/server/db). The role default is k3s's own default, /var/lib/rancher/k3s, so dev is unchanged — and dev also has a dedicated etcd device mounted at server/db under that path, which moving dev's data dir would orphan. Prod overrides it in one line in inventory/group_vars/k3s_prod/vars.yml, the same seam kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a fourth prod node inherits it from group membership rather than from anyone remembering.

Longhorn's defaultDataPath follows the R-ENV-02 knob pattern, not a base edit. platform/longhorn/helmrelease.yaml referenced a hardcoded /var/lib/longhorn; it now references ${LONGHORN_DATA_PATH}, a cluster-vars knob with a base default and an explicit value in both overlays:

clusters/base/cluster-vars.yaml	/var/lib/longhorn (base default = the known-good dev value, same convention as GATEWAY_VIP)
clusters/dev	/var/lib/longhorn — must never change; the live dev cluster has 18 volumes whose replicas are on disk under that path
clusters/prod	/data/longhorn

kustomize-controller runs with --feature-gates=StrictPostBuildSubstitutions=true, so an undefined variable fails the Kustomization outright — hence base-first, then both overlays. [parity]: the prod value MUST equal node_baseline_longhorn_data_path in group_vars/k3s_prod/vars.yml. Ansible creates and mounts the directory; Helm tells Longhorn to use it. If the two disagree, Longhorn writes replicas onto the OS filesystem and /var fills up — the exact failure this revision exists to prevent.

Verified 2026-09-04 that dev's rendered output is byte-identical. All 12 Flux Kustomizations that declare postBuild.substituteFrom cluster-vars were rendered (kubectl kustomize <spec.path> | flux envsubst --strict, substituting the dev overlay's effective cluster-vars) both before and after the change: 12/12 identical per-file SHA-256, and one aggregate SHA-256 over every rendered byte, b34c5d3d…89e7d, unchanged. The only dev-side difference anywhere is the presence of the new knob in the ConfigMap itself, carrying dev's existing literal. The same harness shows prod's render changing on exactly one line (defaultDataPath: /var/lib/longhorn → /data/longhorn), which is what proves the comparison could have detected a regression.

Per-node device paths — they are NOT the same, and that is why nothing hardcodes them. Verified 2026-09-03 with findmnt --mountpoint /srv on each node:

Node	Boot media	/srv partition	Size	Contents before the claim
k3s-prod-1	SATA SSD C400-MTFDDAC512M	/dev/sda5	432.5 GiB ext4	lost+found only (du -sh 20K; df 2.1 M of 425 G)
k3s-prod-2	NVMe PM9A1 Samsung	/dev/nvme0n1p5	432.5 GiB ext4	lost+found only (same)
k3s-prod-3	SATA SSD WDC WDS500G2B0A	/dev/sda5	422.2 GiB ext4	lost+found only (same)

The df "5.1% used" that made /srv look occupied is ext4's default 5% root-reserved blocks, not data — confirmed per node by ls -A /srv (only lost+found) and du -sh --exclude=lost+found /srv (20 K). Nothing was destroyed.

How the role resolves the device — discovery, not per-host inventory. A single k3s_prod group value cannot be right when node 2 is NVMe, and per-host host_vars would freeze a fact about today's hardware into inventory, where it goes stale the first time a node is re-imaged or a disk is swapped — and the stale value's failure mode is mkfs on the wrong partition. So group_vars/k3s_prod/vars.yml declares intent (node_baseline_longhorn_source: claim-mountpoint, node_baseline_longhorn_claim_mountpoint: /srv) and roles/node_baseline/tasks/disks.yml asks the node which block device is mounted there, then proves the answer is safe before writing anything. Six independent asserts, all fail-closed: it is a partition or whole disk (not tmpfs/NFS/bind), it is not backing /, /boot, /boot/efi, /usr, /var, /home, /tmp or active swap (that set is built from the node's own gathered facts plus swapon, never a hardcoded partition list), it is already ext4, it carries no other filesystem label, it is ≥ 100 GiB, and it is empty.

Unmount → relabel, not reformat. /srv is already formatted and already mounted, and community.general.filesystem will not touch a device that has a filesystem — it would silently no-op and then the LABEL=longhorn mount would fail. The role therefore unmounts /srv, removes its /etc/fstab entry (installer-written, pinned by UUID — leaving it would re-mount the partition at /srv underneath Longhorn on the next boot), and stamps the label with e2label. Relabelling rather than reformatting is deliberate: the partition is already ext4 and already verified empty, so mkfs buys nothing, and if some assert were ever too permissive the worst case is a renamed filesystem instead of a destroyed one.

Idempotency guard. A second run is a no-op, not a reformat. After the claim the partition carries the ext4 label k3s-data and /srv is gone from /proc/mounts and /etc/fstab — so "what is mounted at /srv" would find nothing on run 2. The role pivots on the durable on-disk evidence instead: blkid -L k3s-data returning a device means the node is already claimed, and every destructive task is gated on that returning nothing.

The 2026-09-04 transition, handled idempotently. All three nodes had already been claimed under the previous naming (LABEL=longhorn at /var/lib/longhorn). Because the idempotency pivot asks "does the current label exist?", changing the label makes an already-claimed node answer no — and it would then walk into the greenfield claim path and fail closed at "the claim mountpoint must actually be a mountpoint", /srv having been consumed by that first claim. Fail-closed is correct, but it is not convergence, so the transition is declared rather than inferred: node_baseline_data_retired_fs_label: longhorn + node_baseline_data_retired_mount: /var/lib/longhorn in group_vars/k3s_prod/vars.yml. The role then unmounts the retired mountpoint, drops its fstab entry and re-stamps the label with e2label — a metadata write, never a reformat, exactly as in the claim path — having first proved that the retired mountpoint is still a mount, that it is backed by the very device the retired label names, and that it is empty so nothing is orphaned. The common LABEL=-based mount task then brings it up at /data, and the two per-tenant subdirectories are created (0755 Longhorn, 0700 k3s, since the latter holds the etcd DB, the node token and the image store).

The transition block is one-time and self-retiring: on every subsequent run blkid -L k3s-data succeeds and none of it is reachable. The two retired-* lines are safe to delete from group_vars once the fleet is converged (it is, as of 2026-09-04).

Three independent reasons the LIVE dev cluster cannot enter that transition — it has 18 Longhorn volumes on /var/lib/longhorn and unmounting it would orphan every replica: (1) both retired-* vars default to "" and are set only in the prod-scoped group_vars; (2) the path is gated on longhorn_source == 'claim-mountpoint', and dev is device — the same structural gate that keeps dev out of the claim path; (3) the emptiness assert, which fails closed on a partition with replica directories even if (1) and (2) were both defeated. Dev's mountpoint, label and k3s data dir are also simply not changing.

Run evidence (2026-09-04). Full node-baseline.yml against k3s_prod: run 1 converged (5–6 changed per node), run 2 changed=0 on all three nodes with every transition and claim task skipping. Per node afterwards: blkid -L k3s-data → /dev/sda5 (nodes 1, 3) / /dev/nvme0n1p5 (node 2); blkid -L longhorn → gone; /data mounted with 403 G / 394 G available; /var/lib/longhorn and /srv no longer mountpoints; fstab carrying exactly LABEL=k3s-data /data ext4 defaults,nofail,x-systemd.device-timeout=5s 0 0; and /var down to 5% used, ~11 GiB free.

Dev can never reach this path. Both role defaults stay source: device, so the LIVE dev cluster keeps its dedicated /dev/vdb (Longhorn) and /dev/vdc (etcd) and the claim tasks are structurally unreachable there — they are gated on claim-mountpoint, which only the prod-scoped group_vars sets.

The accepted risk, and its exit condition

Dev was burned by exactly this shape: a Longhorn rebuild starved etcd WAL fsync (docs/plans/ssd-etcd-isolation.md), which is why dev has a third, dedicated etcd device. Prod accepts the same shared-device contention because the failure mode is materially cheaper here: dev's contention was on a spinning HDD with seek-bound queueing; every prod node is SSD or NVMe end-to-end, where concurrent rebuild I/O costs latency rather than collapsing it. The three nodes also have no second device to give etcd — one physical disk each, already fully partitioned, no free slot verified populated.

Accepted to be MEASURED, not assumed. Exit condition, in order of preference:

• Measure it before it matters. Under a deliberate Longhorn replica rebuild, watch etcd_disk_wal_fsync_duration_seconds p99 and etcd_disk_backend_commit_duration_seconds p99, plus etcd_server_leader_changes_seen_total. If fsync p99 stays inside etcd's practical budget (~10 ms wal fsync / ~25 ms backend commit) with no leader churn, the acceptance holds and this becomes a recorded measurement rather than an assumption.
• If it does not hold: add an M.2 NVMe per node (chassis slot to be verified physically) and flip node_baseline_etcd_source back to device — the role still supports that path unchanged, and it is the dev shape.

THE MEASUREMENT CAME BACK — 2026-09-07. It does not hold on node 1.

Not under a deliberate test: in production, 2 days 7 hours after the cluster came up. 06:05:21Z k3s-prod-1 went NotReady (Kubelet stopped posting node status) while the host stayed up, pingable and SSH-able. From the k3s journal:

slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
apply request took too long   took=8.30s                           (expects 100ms)
etcdserver: too many requests
Failed to check local etcd status for learner management: context deadline exceeded

No OOM, no MCE, no hung task in the kernel log — this is not §0b and not memory. Recovery required a manual systemctl restart k3s; 18 Longhorn volumes then rebuilt, and I/O pressure only fell once the rebuilds finished. The failure is self-reinforcing: the node dropping out degrades volumes → Longhorn rebuilds → more write load on the disk etcd needs → etcd stalls further.

/proc/pressure/io full avg300, all three nodes, during the incident:

node	device	drive	IO PSI full avg300
k3s-prod-1	/dev/sda5	C400-MTFDDAC512MAM (2011 consumer SATA SSD, no PLP)	11.36
k3s-prod-2	/dev/nvme0n1p5	Samsung PM9A1 NVMe	0.75
k3s-prod-3	/dev/sda5	WDC WDS500G2B0A (WD Blue SATA)	1.29

The prediction in the paragraph above — "every prod node is SSD or NVMe end-to-end, where concurrent rebuild I/O costs latency rather than collapsing it" — was right about node 2 and wrong about node 1. A 2011 consumer SATA SSD with 61,440 reallocated sectors, 31% of its endurance consumed and no power-loss protection is not the "SSD" that reasoning assumed. §0's "three nodes, three different drive models" is the fault line, exactly where §0 said it would be.

The exit condition is therefore OPEN and the second bullet applies. Replacing node 1's drive, or giving etcd its own device, is the fix, and it is the operator's.

Pending that, three SOFTWARE mitigations shipped on 2026-09-07 (branch fix/etcd-io-priority). None of them is the fix, and each says so where it lives:

1. Observability first — NodeIOPressureHigh / NodeIOPressureCritical in platform/monitoring/alerts.yaml, plus an I/O row on the node-pressure dashboard. The stack alerted on CPU and memory PSI and had no I/O alert at all, so the first signal of this outage was a node already out of the cluster. Thresholds (5% / 10% full IO PSI) come from the numbers in the table above. Note the etcd histograms this section's first bullet asks for are not available: kube-prometheus-stack's kubeEtcd scrape job is deliberately disabled for k3s (platform/monitoring/kube-prometheus-stack.yaml), so PSI is the signal that actually exists. Enabling the etcd scrape would make that bullet directly measurable and is worth doing.
2. Longhorn concurrency ceilings — platform/longhorn/helmrelease.yaml. Honestly: concurrentReplicaRebuildPerNodeLimit: 1 was already in force during this outage and held; the 18 rebuilds ran one at a time and the node stalled anyway. Longhorn offers no further rebuild lever for the v1 data engine (replicaRebuildingBandwidthLimit is v2/SPDK only; rebuildConcurrentSyncLimit is already at its floor of 1). What was tightened is every other Longhorn I/O source — restore, snapshot purge, backup, engine upgrade — each of which defaulted to 5 concurrent operations per node and each of which could reproduce this on its own.
3. Block-layer arbitration — roles/etcd_io_priority, enabled for [k3s_prod], applied only when the operator runs playbooks/etcd-io-priority.yml. BFQ + IOWeight so etcd's cgroup outweighs kubepods.slice. Its README is explicit that most of the obvious knobs here (IOSchedulingClass, io.latency, bare IOWeight) are inert on these nodes, and that even this one cannot arbitrate the ext4 journal both tenants share.

The last point is the one that matters for this section: no cgroup or scheduler control separates two tenants on one filesystem. etcd's fdatasync waits on a jbd2 journal commit, jbd2 is a kernel thread in the root cgroup, and Longhorn's writes dirty the same journal. The software mitigations raise the load level at which this happens. They do not remove it.

Known, separate consequence — /var is only 12 GiB. RESOLVED 2026-09-04 by moving k3s's data dir. This was recorded here on 2026-09-03 as a capacity item "to watch or repartition". Re-measuring it against dev turned it from a watch item into a blocker: the container image store alone is 21–25 GB per node on the fully-deployed dev cluster and /var totals 58–70 GB, against prod's 11.8 GiB. So prod would have filled /var during its first image pulls, and with etcd on the same partition a full /var means etcd cannot write its WAL. The fix was neither "watch" nor "repartition" but --data-dir — see "THE 2026-09-04 REVISION" above. /var now carries only the OS's own usage (503 MB, 5%).

Consequences to settle before imaging nodes 2 and 3:

• Drive health MEASURED 2026-09-01 (smartctl -a /dev/sda, via the baseline package floor — see §5). Raw attributes, so the numbers are auditable:

  | | |
  |---|---|
  | Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) |
  | SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` |
  | Power-on hours | 6,814 (~284 days) · 147 power cycles |
  | **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) |
  | **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) |
  | **Erase failures** | **187** (`172`) |
  | **Unexpected power losses** | **32** (`174`) |
  | Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) |
  | UDMA CRC errors | 0 (`199`) — cabling is clean |
  
  **Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector
  count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its
  normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**,
  not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value;
  the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent
  uncorrectable counters instead.
  
  **Verdict: usable, but not for production state.** Nothing says imminent failure — no
  pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011
  consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad
  blocks, logged **187** erase failures, and taken **32** unclean power losses **with no
  power-loss protection**, in a single failure domain. etcd is the worst possible tenant for
  it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how
  a member corrupts — this drive has already logged 32 of them.
  (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
  `DECISIONS-LOG.md`. It is not part of this verdict.)
  

• Layout DECIDED and APPLIED 2026-09-03 — see "THE DECISION" above. roles/node_baseline no longer expects a block-device path for either store; it takes a declared source enum per environment (roles/node_baseline/tasks/disks.yml): - Longhorn — node_baseline_longhorn_source: claim-mountpoint + node_baseline_longhorn_claim_mountpoint: /srv. The /dev/vdb default (a virtio name that does not exist on bare metal) is unused on prod; the real partition is discovered on the node and guarded, because it is /dev/sda5 on nodes 1 and 3 and /dev/nvme0n1p5 on node 2. - etcd — node_baseline_etcd_source: shared-data-partition (was root-filesystem until 2026-09-04). "No dedicated device" is a first-class, explicitly-configured state that the disks assert accepts, not a blank variable that the tasks happen to skip — and the name of that state now describes reality, because etcd is no longer on the OS filesystem. The distinction is load-bearing, not cosmetic: shared-data-partition is asserted, per node on every run, by checking that the filesystem backing the etcd data dir really is the data partition's mount. Declaring k3s_data_dir: /data/k3s does not by itself achieve that — had the mount silently failed, /data would be a plain directory on / and etcd plus 21–25 GB of images would land back on the OS filesystem, invisibly, until the disk filled. Leaving the value at root-filesystem would have been a lie inventory told about the cluster, and root-filesystem makes no such claim and gets no such assert. The role also still asserts nothing is already mounted at the etcd data dir (so a half-migrated node cannot be inherited silently) and reports which filesystem etcd actually lands on (measured 2026-09-04: /data, 403 G / 394 G available, vs /var's 10.7 GiB). - The two rejected options stand rejected: add a second drive (no verified free slot; the exit condition above reopens it if measurement fails) and repartition (same single failure domain, plus a destructive re-image of three working nodes for no isolation the SSDs do not already give).

• All three nodes get the identical layout. Same declaration, same mountpoint, same resulting /data + /data/longhorn + /data/k3s. Not the same device path and not the same size (432.5 / 432.5 / 422.2 GiB) — Longhorn tolerates unequal node capacity; it is replica placement that assumes a symmetric layout, and the layout is symmetric. Confirmed per node 2026-09-04 by findmnt/lsblk/blkid after the transition.

• Chosen paths recorded here and set in group_vars. The BLOCKED placeholders in infra/ansible/inventory/group_vars/k3s_prod/vars.yml are replaced — by the source enums above, plus deliberately-unusable /dev/NOT-A-DEVICE-… values for the two device variables so that flipping a source back to device without supplying a real path still fails on the role's assert instead of silently inheriting dev's /dev/vdb.

• The SMART-vs-smartmontools circular dependency is broken (2026-08-31). roles/node_baseline is tagged packages / system / disks, so the package floor can land without the disk tasks this decision blocks — see §5's drive-health box for the exact command.

3. OS install (per node)

• Debian 13 (trixie), matching dev exactly — same kernel series, same package baseline.

• Hostname k3s-prod-N.

• Partition per the §2 decision.

• No cloud-init needed. Dev's first-boot work was deliberately moved into roles/node_baseline when provisioning became API-only (infra/cloud-init/ is a README now), so the same role covers bare metal. Nothing to port.

• KERNEL: a standing, deliberate deviation from the distro baseline — linux-image-amd64 from trixie-backports (DONE on all three nodes, 2026-09-04).

  **What.** `roles/node_baseline` gained a `kernel`-tagged block that enables
  `trixie-backports` as a deb822 apt source and installs the `linux-image-amd64`
  **metapackage** from it. Applied 2026-09-04; all three nodes landed on
  **`linux-image-amd64` 7.1.8-1~bpo13+1 → kernel `7.1.8+deb13-amd64`**, rebooted one at a
  time, and are running it.
  
  ```bash
  ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
  ```
  
  **Why.** §0b. HP's R21 firmware publishes a corrupt EFI Memory Attributes Table; Linux
  < 6.13 reserves memory from it without validating the size, and `k3s-prod-2` has booted
  with 2.6 GiB of its 16 GB, twice. The fix is upstream commit **`8fbe4c49c0cc`**
  *"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, first
  released in **v6.13**, hardened by `5d0faa8e8369`). It is absent from v6.12 and from
  `linux-6.12.y`, carries neither `Cc: stable` nor `Fixes:`, and there is **no kernel
  cmdline workaround**. A k3s **server** that intermittently comes up with 2.6 GiB will
  fail under etcd plus its workload share on any reboot, so this is a **prerequisite to
  installing k3s**, not a follow-up.
  
  **How it is scoped.** `node_baseline_backports_kernel` defaults to **false** in
  `roles/node_baseline/defaults/main.yml` and is set **true** exactly once, in
  `inventory/group_vars/k3s_prod/vars.yml` — the same seam `kube_vip_address` /
  `k3s_api_host` / `cluster_kubeconfig_name` use. Dev's Proxmox VMs publish no MAT and are
  genuinely unaffected; they must not be dragged onto a backports kernel as a side effect
  of sharing the role, and a fourth prod node must inherit the deviation from **group
  membership**, not from a `k3s-prod-*` hostname test. Verified: a `--tags kernel` run
  against dev skips all 8 tasks on all 3 hosts.
  
  **Security cost, accepted.** `*-backports` packages are provided **as-is** and are
  **not** covered by the Debian Security Team's guarantees for the main archive — no
  security-tracker commitment, no DSAs. Recorded here as the price of the nodes keeping
  their RAM.
  
  **No apt pin, deliberately.** The backports `Release` file carries `NotAutomatic: yes`
  + `ButAutomaticUpgrades: yes`, which apt already reads as *"priority 100 so nothing is
  taken from backports by accident, 500 for a package explicitly installed from there so
  it keeps getting updates."* Writing an explicit `Pin-Priority: 100` in
  `preferences.d` would look like belt-and-braces and would in fact **switch off the
  automatic security upgrades for the very kernel this deviation installs.** Enabling the
  source is the whole configuration.
  
  **Metapackage, not a pinned ABI.** A frozen `linux-image-6.19.14+deb13-amd64` would stop
  receiving even backports' own updates the moment the ABI rolls. The metapackage tracks
  whatever backports ships (>= 6.13 by construction). The **exact** version each node
  landed on is recorded on the node itself at `/etc/node-baseline-backports-kernel`,
  written by the same role, along with the rollback path below.
  
  **Rollback.**
  1. *Non-destructive, try first.* The stock kernels are still installed
     (`6.12.107+deb13-amd64` and `6.12.94+deb13-amd64` on all three) and selectable from
     GRUB → *Advanced options for Debian GNU/Linux*. That answers "is the new kernel the
     problem?" without uninstalling anything — it also reinstates the memattr bug for that
     boot.
  2. *Permanent.* Set `node_baseline_backports_kernel: false` in
     `group_vars/k3s_prod/vars.yml` **first** (otherwise the next Ansible run puts it
     back), then on each node:
     ```bash
     apt-get install --reinstall linux-image-amd64/trixie
     apt-get purge linux-image-7.1.8+deb13-amd64
     rm /etc/apt/sources.list.d/trixie-backports.sources
     apt-get update && update-grub && reboot
     ```
  
  **The role does NOT reboot**, and that is deliberate — see §0b "SECOND FAULT". Reboots
  are a serialized, supervised operator step: one node at a time, verify each returns
  before touching the next, never node 2 first. On 2026-09-04 all three returned in
  **39–41 s** (node 1 39 s, node 3 41 s, node 2 39 s); node 2 did **not** reproduce its
  2026-09-03 hang, but one clean warm reboot is not a fix and that defect stays open.
  
  **Exit condition — how this deviation ends.** If `8fbe4c49c0cc` is ever picked up into
  `linux-6.12.y` (and therefore into Debian's 6.12 rebase, which reaches us by plain
  `apt upgrade`), this can be removed entirely: set `node_baseline_backports_kernel:
  false`, roll back per the note on each node, and delete the `kernel` block from
  `roles/node_baseline`. §0b records why that pickup will not happen on its own and what
  request would cause it.
  

• Verification of the fix, and a correction to how it is verified (2026-09-04).

  §0b says *"A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table
  detected!`"*. That is true but **conditional, and it did not happen on any of the three
  nodes.** `efi_memattr_init()` on >= 6.13 has **three** gates and only the third prints
  that line:
  
  | gate | message | reachable when |
  |---|---|---|
  | `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
  | implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, `desc_size` does not |
  | `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* `desc_size` read plausibly, entry count does not |
  
  **All three are rejections**, and which one a boot prints is decided by *which field*
  that boot's garbage corrupts. On 7.1.8 all three nodes hit the **first** gate:
  
  | node | `MEMATTR=` | kernel line on 7.1.8 | MemTotal | reserved at boot |
  |---|---|---|---|---|
  | 1 | `0xbee84018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1030` | 31.1 GiB | 0.9 GiB |
  | 2 | `0xbeeb0018` | `efi: memattr: Unexpected EFI Memory Attributes table version -1569980416` | 15.4 GiB | 0.6 GiB |
  | 3 | `0xbeecc018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` | 31.1 GiB | 0.9 GiB |
  
  Node 2 is the result that matters: **`0xbeeb0018` is the exact address that produced
  2.6 GiB twice**, and on 7.1.8 it reserved 634,904 K instead of 13,913,856 K.
  
  **Boot-independent proof the fix is compiled in**, since the log line is a lottery: the
  running kernel image itself contains the string. Decompressing `/boot/vmlinuz-$(uname
  -r)` on all three nodes yields, verbatim —
  
  ```
  efi: memattr: [Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)
  efi: memattr: Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)
  ```
  
  — neither of which exists in 6.12. That is the positive proof; the `dmesg` line is not
  available on demand.
  
  **Two bugs in `roles/node_health` were fixed as a consequence**, because without them
  the health check could not show the fix landing:
  - `MEMATTR_VERSION_RE` in `files/node_health_collect.py` matched `(\d+)` only. The
    kernel prints that field with `%d` — **signed** — so node 2's `-1569980416` did not
    match, the state fell through to `clean`, and the node was reported **PASS "the kernel
    accepted it without complaint"**: a false pass on the exact bug the check exists to
    find. Now `(-?\d+)`.
  - `firmware.efi_memattr` treated `bad_version` as WARN unconditionally, so a node on a
    fixed kernel would have stayed WARN forever. The verdict now keys on the running
    kernel version (`memattr_kernel_fixed`, `>= 6.13`): the same log line is WARN on
    < 6.13 (a lucky boot) and PASS on >= 6.13 (the validation working). The pre-existing
    `corrupt_rejected` PASS branch is unchanged and remains reachable.
  
  **`playbooks/node-health.yml -e node_health_hosts=k3s_prod`, after the reboots:
  `firmware.efi_memattr` = PASS on all three, `memory.physical_total` = PASS on all
  three.** Dev re-run unchanged.
  

4. Network (per node)

• Static IP on the prod LAN, recorded in §0. Node 1 is 192.168.2.10/24, gateway 192.168.2.1.

• Prod address plan — DECIDED (operator, 2026-08-31).

  | Role | Address | Owned by |
  |---|---|---|
  | `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory |
  | **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) |
  | **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) |
  | **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) |
  
  Two deliberate properties, both mirroring dev:
  - The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and
    Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way
    round (`.49` control-plane, pool `.53-.60`).
  - **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a
    defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool
    was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape
    (`GATEWAY_VIP` `.53` = first of `.53-.60`).
  

• Reserve all of 192.168.2.9 and .20-.30 outside any DHCP range on the prod LAN before the build, and confirm nothing else on the network already answers on them.

• Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks node-baseline.yml --tags dns. Node 1 has a single IPv4 nameserver, 192.168.2.1, which is also the LAN gateway. A single upstream is a known SPOF that has already bitten this project — it broke containerd image pulls and CoreDNS forwarding on dev.

  The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in
  `roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes
  `/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed
  `/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager
  owns it), and configures the ordered upstream list either way. The earlier
  `fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan
  drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie
  nodes, so it would have broken `node-baseline.yml` on every host.
  
  What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets
  `node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid
  `BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert
  stops the play on it before writing anything. Choosing the second upstream is a **network
  decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would
  resolve reliably but would also bypass any local zone and any filtering the gateway applies.
  Decide between — and record the choice HERE, in the same change that edits the group_vars:
  - a second on-LAN resolver on `192.168.2.x`;
  - the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified —
    prod sits on the workstation side of the pfSense boundary);
  - an explicit, written-down acceptance of a public resolver as fallback-only.
  
  Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`)
  are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed
  servers, so keeping them means listing them explicitly.
  `node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's
  local domain is decided.
  

• Confirm the prod LAN's relationship to the dev LAN (192.168.1.x) and to the pfSense NAT — prod on 192.168.2.x sits on the workstation side of the boundary dev lives behind.

5. Access (per node) — blocks all Ansible

• k3s user with the home_network SSH key. Working on node 1.
• Password-protected sudo, driven by Ansible become + a vaulted password. Prod does NOT use NOPASSWD sudo (operator decision, 2026-08-31). Dev nodes do; prod deliberately keeps the sudo password in place and supplies it to Ansible as an ansible-vault-encrypted var, exactly like the existing hypervisor and proxmox_hosts groups: inventory/group_vars/k3s_prod/ README.md how to (re)create the vault — committed vault.yml ansible_become_password: "..." — committed, ENCRYPTED The passphrase that decrypts it lives outside the repo at ~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays still run unattended without a plaintext secret ever touching git. bash ansible-vault create inventory/group_vars/k3s_prod/vault.yml # ansible_become_password: ... Prod nodes must therefore set ansible_become: true in their group_vars and must not inherit any passwordless-sudo assumption from group_vars/all.yml.
• Drive health baseline (§2). Do not hand-install the tooling — package installation is the node_baseline role's job, never a manual apt-get on a node (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo). smartmontools is part of node_baseline_packages, so it arrives with the baseline play — but a full baseline would hit the disk tasks §2 has not decided yet. Run the packages-only baseline instead (the disks-tagged tasks, including the BLOCKED-device assert, are not selected at all, so the placeholder device paths are never even templated): bash cd infra/ansible ansible-playbook playbooks/node-baseline.yml \ -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1 (Needs group_vars/k3s_prod/vault.yml to exist first — the become password above.) Reading the drive is then: bash sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors Record the result in §2 and close the layout decision in the same change that replaces the two node_baseline_*_device placeholders.

6. Ansible baseline & cluster join

Reuses the dev roles unchanged — that reuse is R-ENV-02. Only inventory and group_vars differ, plus a target-group override on the command line.

How prod is targeted (done, 2026-08-31)

Every node-host play used to hardcode the literal dev groups k3s_servers / k3s_primary / k3s_secondary, so nothing here could select prod at all. Two knobs now close that, deliberately kept separate because Ansible resolves them at different times:

Layer	Knob	Set via
Which hosts a play runs on	k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts (each defaults to the dev group)	-e extra vars only — a play's hosts: is templated before any host exists, so group_vars is not readable there
Which group a role means by "my primary" / "my servers"	k3s_primary_group, k3s_servers_group, cluster_kubeconfig_name	group_vars — dev in all.yml, prod in k3s_prod/vars.yml; no per-run flag

So roles/k3s_server finds the prod --cluster-init founder, builds the prod join target and tls-san list, and roles/cilium fetches the prod kubeconfig, with no role fork. Running with no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after --list-hosts diff on every playbook). Full detail: infra/ansible/README.md → Targeting prod.

• Add a k3s_prod inventory group (k3s_prod_primary / k3s_prod_secondary). Prod is reached directly on the LAN — no NAT port-forward hop, unlike dev's 192.168.2.102:225x mapping. (Nodes 2/3 stay commented out until they are physically on the bench.)
• inventory/group_vars/k3s_prod/vars.yml: kube_vip_address (192.168.2.9), cluster_controller_api_server, k3s_api_host, the group-name overrides above — and the §2 device overrides for node_baseline_longhorn_device / node_baseline_etcd_device, which are still BLOCKED placeholders.
• inventory/group_vars/k3s_prod/vault.yml — the vaulted ansible_become_password (§5). Until it exists no prod play can escalate, and --check against prod cannot work either.
• Run the packages-only baseline first, to unblock the §2 drive-health read: bash ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
• Run the full playbooks/node-baseline.yml -e k3s_hosts=k3s_prod → packages, kernel modules, sysctls, iscsid, data-disk filesystems + mounts. Blocked until §2 is decided: it asserts on the placeholder device paths and stops. (qemu-guest-agent is a no-op on bare metal — the role already only starts it when the virtio channel exists, so it degrades cleanly.)
• Run playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. Only once all three nodes are present — see §8.
• Run playbooks/kube-vip.yml -e k3s_hosts=k3s_prod → control-plane VIP (192.168.2.9) on the prod LAN.
• Run playbooks/cilium.yml -e k3s_hosts=k3s_prod → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The bootstrap values and the HelmRelease must stay in parity (kubeProxyReplacement, k8sServiceHost/Port) or the two layers fight over CNI config (docs/02). Note the prod kubeconfig is fetched to .cache/prod.kubeconfig, never over the dev one the dev container exports as $KUBECONFIG.
• Run playbooks/verify.yml -e k3s_hosts=k3s_prod.
• playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod retargets but is not yet correct: roles/flux_bootstrap shells out to bootstrap/flux/bootstrap.sh, which bootstraps --path=clusters/dev. Prod needs --path=clusters/prod — a bootstrap/ change, not a target-group change. Tracked with the other repo blockers in §7.
• Decide whether prod runs a VPN egress gateway node (dev uses k3s-dev-3, ADR-0015). playbooks/vpn.egress.gateway.yml takes -e vpn_egress_gateway_hosts=<group>, but there is deliberately no prod egress group in the inventory yet, and it would need its own vaulted PIA credentials — the existing vault is scoped to the dev group.

7. Repo blockers — prod cannot be reconciled until these land

clusters/prod is design-only today and would build wrong. R-ENV-02 promises dev and prod differ only by cluster-vars values; in practice dev LAN addresses are hardcoded into base manifests:

Hardcoded value	Where	Prod needs
k8sServiceHost: 192.168.1.50	platform/cilium/helmrelease.yaml:50, mirrored in values.yaml:16	prod primary node IP
LB pool 192.168.1.53–.60	platform-config/cilium/l2-ip-pool.yaml:15	a prod-LAN pool
192.168.1.55 (crafty)	platform/crafty/service-l4.yaml	prod-LAN address
192.168.1.54 (UniFi VM)	apps/vms/unifi/50-service-l4.yaml	prod-LAN address
192.168.1.9:9000 (S3 backup)	platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,semaphore}-db.yaml	reachable prod backup target
home.keithjackson.dev (every hostname)	HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, catalog/	CLOSED 2026-09-04 — see §10. This one was not merely "would build wrong": it already destroyed dev's public DNS. BASE_DOMAIN was a declared-but-DEAD knob; it is now real, and prod serves prod.home.keithjackson.dev.

• Tokenize each into a cluster-vars knob with a base default, then set the value in both overlays. Adding a knob before referencing ${KNOB} is the sanctioned mechanism (see the header comment in clusters/base/cluster-vars.yaml). platform, platform-config, the app waves, apps-vms and crafty all already carry postBuild.substituteFrom: cluster-vars, so the seam exists.

• clusters/prod must set GATEWAY_VIP. It currently doesn't, so prod would inherit the base default 10.73.0.240 — an address from the retired libvirt network. The base file's own comment flags this as owed "when hardware is built."

• Dev overlay sets every new knob to today's exact value so rendered dev output is byte-identical and the live cluster sees zero drift. Verify with make -C ci render-golden.

• clusters/prod is NOT bootstrappable as-is — three gaps, found 2026-09-01 while landing the dependsOn fix. Only the first is now closed: - [x] Root Kustomization — clusters/prod/sync.yaml authored, mirroring dev. This became load-bearing, not cosmetic: clusters/base/platform.yaml now carries dependsOn: [flux-system], so without a root object on prod, platform would block forever on a dependency that does not exist and nothing downstream would reconcile. - [x] SOPS decryption — DONE. Operator decision 2026-09-01: prod uses the SAME age key as dev. Eight decryption: {provider: sops, secretRef: sops-age} patches mirror dev exactly. The sops-age Secret is still created OUT OF BAND on the prod cluster (bootstrap/sops/install-age-key.sh) — the overlay references it by name only. - [~] GitRepository source — MITIGATED 2026-09-04, NOT SOLVED. Branch settled; the in-cluster URL is still unusable, so prod now points at a temporary bootstrap source. Operator decision 2026-09-01: prod tracks main, the same branch as dev, because dev is temporary and will be decommissioned once prod is stable. Recorded consequence: once prod is live, every merge to main reconciles BOTH clusters — no promotion step, no soak window.

        **Why the in-cluster URL (`https://gitea.home.keithjackson.dev/...`) cannot work on a
        cold prod cluster** — two independent reasons, found 2026-09-01, both still true:
        1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
           decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
           loses its source of truth. **This is the decommission plan's first task, not its last.**
        2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire
           `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
           prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
           `gitea.home.keithjackson.dev` to itself and never reach a repository.
  
        **The mitigation now in `clusters/prod/kustomization.yaml`** (operator directed this to be
        stood up, 2026-09-04):
  
        | | |
        |---|---|
        | GitRepository url | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` |
        | Branch | `main` (unchanged — same branch as dev) |
        | What it is | Gitea 1.22.6 in a **podman container on the operator's workstation**, started by `bootstrap/gitea/bootstrap-gitea.sh` with `HOST_NAT_IP=192.168.2.132` |
        | Reachability | All three prod nodes reach it **directly on the LAN** — no NAT hop (the libvirt network is retired), **no firewall change needed**. Verified from all three nodes. |
        | Auth | **None, deliberately.** The repo is PUBLIC on that Gitea, so Flux needs no `secretRef`. Verified anonymously: `curl .../api/v1/repos/Home-Network/home-network-k3s-impl` → `"private":false`, and `git ls-remote` returns `refs/heads/main`. |
        | Transport | **Plain HTTP, no TLS.** Accepted for the bring-up window only: LAN-local, and the repo holds no plaintext secrets (all SOPS-encrypted). |
        | Admin creds | `~/.config/home-k3s-bootstrap-gitea-admin.txt`, mode 0600, host-only, never committed. |
  
        **THIS IS A SINGLE POINT OF FAILURE FOR PROD'S RECONCILIATION, and it is not even in a
        cluster.** It is one container on one workstation, with no HA, no backup of its own, and
        no restart guarantee beyond `--restart unless-stopped`. If that workstation is powered
        off, suspended, or the podman volume `gitea-bootstrap-data` is lost:
        - already-reconciled state **keeps running** — Flux does not tear anything down;
        - but **no new revision can land on prod**, and every `GitRepository` reconcile fails.
        So prod is *not* self-sufficient while this stands. Do not treat prod as production-ready
        on this footing, and do not let it become the steady state by inattention.
  
        **One bring-up footgun:** `bootstrap-gitea.sh` pushes the operator's **current `HEAD`** to
        the bootstrap Gitea's `main`, whatever branch that HEAD is on. Before bootstrapping prod,
        confirm that remote `main` is the revision prod should run
        (`git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` vs local
        `main`). As of 2026-09-04 the remote `main` is the tip of the working branch, **not** local
        `main`.
  
        **EXIT CONDITION (the cutover) — mirrors dev's own Phase-6 cutover, `docs/05`.**
        This item is not closable until all of it is done:
        - [ ] Gitea runs **on prod**, reconciled in-cluster, answering
              `gitea.home.keithjackson.dev` through prod's Gateway VIP with a publicly-trusted cert.
        - [ ] This repo is pushed/mirrored into that in-cluster Gitea.
        - [ ] The `clusters/prod` GitRepository url is changed to
              `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`
              and the TEMPORARY comment block is **deleted** (dev's stanza is what "done" looks like).
        - [ ] `flux get sources git -n flux-system` on prod reconciles a revision from the new URL.
        - [ ] **Only then** stop the workstation container — kept stopped-not-deleted as recovery,
              exactly as dev retained its own bootstrap Gitea.
  - [x] **`roles/flux_bootstrap` could only bootstrap dev** — FIXED 2026-09-04. It invoked
        `bootstrap/flux/bootstrap.sh` without ever exporting `FLUX_ENV_PATH`, so the script fell
        through to its `clusters/dev` default and §8 step 4 (`--path=clusters/prod`) was
        unreachable without editing the role. Now parameterised on the standard group_vars seam:
        role default `flux_bootstrap_env_path: clusters/dev` (defaults/main.yml) is exported as
        `FLUX_ENV_PATH` by the role's task, and
        `inventory/group_vars/k3s_prod/vars.yml` overrides it to `clusters/prod` — the same
        mechanism as `kube_vip_address` / `k3s_data_dir` / `cluster_kubeconfig_name`. Dev has no
        override anywhere, so dev resolves `clusters/dev` exactly as before.
  

• Confirm prod's backup target: dev's S3 (versitygw on the Proxmox host, 192.168.1.9) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).

8. Build sequence (operator decision, 2026-08-31: wait for all three nodes)

Node 1 is baselined now; the cluster is not initialized until nodes 2 and 3 are on the bench. This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a single-server-etcd window, and avoids a later member-add dance.

1. Per-node §1–§5 on each of the three boxes.
2. §7 repo work lands on main with the gate green.
3. node-baseline → k3s (all three) → kube-vip → cilium.
4. flux bootstrap ... --path=clusters/prod.
5. Conformance against prod (§9).

9. Exit criteria (Phase 7)

• Prod runs the same config as dev via the prod overlay, no base edits (R-ENV-02).
• Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01).
• Full automated prod DR drill (R-STATE-03) — definition-of-done.md requires this for prod specifically.
• Monitoring and alerts live on prod.
• The conformance suite passes green against the prod cluster, not just dev (R-TEST-03). Blocked on parameterizing the L2 tests that hardcode home.keithjackson.dev — see §10 phase 2.
• Prod serves its own DNS/TLS identity and dev's records are provably untouched (§10): distinct EXTERNALDNS_TXT_OWNER_ID, prod wildcard cert *.prod.home.keithjackson.dev, and prod's own Keycloak realm + per-service client secrets.

10. DNS/TLS environment split — the 2026-09-04 incident, and phase 1 of the fix

What happened

Prod's external-dns deleted every one of dev's public DNS records.

Both clusters ran policy: sync against the same Cloudflare zone, served the same hostnames, and — because neither overlay set txtOwnerId — both inherited the external-dns chart's default owner id default. ExternalDNS decides what it may delete by reading its own owner id back out of the TXT registry. Prod read dev's TXT entries, saw its own owner id, found no local resource claiming those names, and garbage-collected the records behind them. Dev recovered once prod was stopped. Prod's platform tier and everything downstream were then suspended, and remain suspended, so it cannot recur before this change is reviewed.

policy: sync was not the bug and has not been changed. What was missing was identity.

The second bug, queued directly behind it

platform-config would have requested a *.home.keithjackson.dev wildcard Certificate from letsencrypt-prod — for a cert dev already holds — and both clusters would then have raced each other's _acme-challenge TXT records on that one zone. Neither would reliably solve DNS-01. This never fired only because the tier was suspended first.

Root cause, stated precisely

Not external-dns. Every hostname in the repo was a hardcoded literal. BASE_DOMAIN existed in clusters/base/cluster-vars.yaml but was referenced by no reconciled manifest — the repo-derived status report flagged it as a dead knob. R-ENV-02 promises dev and prod differ only by cluster-vars values; for DNS and TLS that promise was false, so clusters/prod was authored to serve dev's public identity. Two clusters that describe the same DNS name will fight over it no matter how the DNS controller is configured.

The architecture decision (operator)

dev   ->  *.dev.home.keithjackson.dev        gitea.dev.home.keithjackson.dev
prod  ->  *.prod.home.keithjackson.dev       gitea.prod.home.keithjackson.dev
apex  ->  CNAME <svc>.home.keithjackson.dev  ->  <svc>.prod.home.keithjackson.dev

PHASE 1 — prod only. Dev did not move. (this change)

BASE_DOMAIN is now real: every hostname in every Flux-reconciled manifest derives from it.

	BASE_DOMAIN	EXTERNALDNS_TXT_OWNER_ID	EXTERNALDNS_DOMAIN_FILTER
base default	home.keithjackson.dev	unset-see-cluster-vars (non-colliding sentinel)	keithjackson.dev
dev	home.keithjackson.dev (unchanged)	default (must not change)	keithjackson.dev
prod	prod.home.keithjackson.dev	prod	prod.home.keithjackson.dev

Dev's effective hostname is deliberately still the apex home.keithjackson.dev. Moving prod alone ends the collision and unblocks prod, and it is the whole of the risk that had to be taken while the operator is away. Dev's rendered output is byte-identical except for one added line, txtOwnerId: default, which is the value dev's external-dns was already using implicitly — verified by rendering all 12 substituteFrom paths through kubectl kustomize | flux envsubst --strict for both overlays and diffing.

Two independent guards now, either sufficient alone:

1. txtOwnerId — per-environment identity. A cluster only ever touches records it owns. This is the direct fix for the incident. Dev must render exactly default: its live records are already registered under that owner, and any other value would make dev's own external-dns stop recognising them and re-create duplicates beside the orphans.
2. domainFilters — per-environment name scope. Prod cannot see a name outside its own subtree. Defence in depth, added because the first guard is a single string in a single file.

The base default for the owner id is a non-colliding sentinel, deliberately not default: a future environment that forgets the knob must not silently inherit dev's identity — that is exactly what happened here. With the sentinel such a cluster owns nothing, so it can delete nothing.

What could NOT be tokenized, and why

• OIDC_ISSUER_URL / OIDC_CALLBACK_URL in platform/headlamp/headlamp-oidc-credentials.sops.yaml — SOPS-encrypted (encrypted_regex: ^(data|stringData)$), so the two hostnames are ciphertext. Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and cannot be verified by the offline render harness (kustomize does not decrypt), so a mistake would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately. Prod action required: prod's Headlamp will otherwise authenticate against dev's Keycloak. Prod needs its own copy of this Secret with keycloak.prod.home.keithjackson.dev and a client secret from prod's own realm — which it needs regardless, since every *.sops.yaml client secret is per-realm.
• oidc-issuer-url in the kube-apiserver flags (roles/k3s_server/tasks/main.yml) — a k3s server flag in /etc/rancher/k3s/config.yaml, not a manifest, so Flux post-build substitution can never reach it. Correctly tokenized on the Ansible side instead: new role default k3s_base_domain (dev value) + derived k3s_oidc_issuer_url, overridden for prod in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam as k3s_data_dir/k3s_api_host. [parity]: k3s_base_domain must equal the same cluster's BASE_DOMAIN, or the apiserver validates tokens against a different Keycloak than the cluster runs and kubectl SSO fails while the cert-based admin kubeconfig keeps working (so it looks like "SSO is broken", not like config skew).

A pre-existing prod problem this incidentally FIXED

§4 recorded that prod nodes use public resolvers, so the kube-apiserver would resolve its OIDC issuer publicly and get dev's Gateway VIP (192.168.1.53) — the wrong cluster on the wrong LAN, making prod kubectl SSO impossible, and requiring an /etc/hosts pin as a workaround. With prod on its own subdomain the public record for keycloak.prod.home.keithjackson.dev is published by prod's own external-dns pointing at prod's Gateway VIP (192.168.2.20), so the public answer and the split-horizon answer agree. No /etc/hosts pin is needed. This holds only once prod's external-dns has actually published the record; until then use the break-glass cert kubeconfig.

Verify on prod's first reconcile

• domainFilters zone selection. The Cloudflare provider uses this value for zone lookup as well as record matching, and in 2026-08 a strict-subdomain filter dropped the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no matching zone it will create nothing — fail-closed, never destructive. Fallback: set prod's EXTERNALDNS_DOMAIN_FILTER back to keithjackson.dev and rely on txtOwnerId, which is sufficient on its own.
• Prod's wildcard cert is *.prod.home.keithjackson.dev and solves DNS-01 without touching dev's _acme-challenge records.
• Dev's records are untouched after prod's tiers are resumed. Confirm dev's TXT registry entries still read owner default and that prod's read owner prod.
• Prod's own Keycloak realm + per-service client secrets exist (every *.sops.yaml client secret is realm-scoped; prod cannot reuse dev's).

PHASE 2 — dev migration + apex CNAMEs (deferred, not done)

Deliberately not attempted here: dev is live (18 Longhorn volumes, working SSO) and it hosts the Gitea that is currently this repo's GitOps source. Moving it is a migration, not a variable change. Required, roughly in order:

1. BASE_DOMAIN → dev.home.keithjackson.dev in clusters/dev, plus k3s_base_domain for the dev inventory group.
2. A new *.dev.home.keithjackson.dev wildcard cert issued and Ready before cutover.
3. Keycloak issuer change. The realm's issuer URL is baked into every token; every OIDC client (grafana, gitea, nextcloud, semaphore, headlamp, crafty, pihole, jackett, qbittorrent, radarr, sonarr, and the kubernetes public client) needs its redirect URIs updated in the same change, and the k3s apiserver oidc-issuer-url with it — which means a rolling k3s restart on the dev servers.
4. kubectl SSO re-issue for every user (issuer changed).
5. gitea.dev.home.keithjackson.dev — this is the GitOps source URL in clusters/dev/kustomization.yaml. Flux must be able to resolve and trust the new host before the URL is switched, or dev stops reconciling and the change cannot be pushed to fix itself. Sequence this last, and keep the bootstrap Gitea available as the recovery path (docs/05).
6. EXTERNALDNS_TXT_OWNER_ID for dev. Once dev's names move, its existing default-owned records are for names dev no longer serves; retire them deliberately rather than letting a renamed owner orphan them.
7. Apex CNAMEs <svc>.home.keithjackson.dev → <svc>.prod.home.keithjackson.dev. This is what actually moves user traffic to prod, and it must not land before prod serves those services.
8. The L2 behavioural conformance tests hardcode home.keithjackson.dev (dns-exposure, dns-internal-resolution, gateway-ingress-reachable, oidc-sso-grafana, sso-forward-auth, trusted-tls). They are test inputs, not reconciled manifests, so they were left alone in phase 1 — but they will fail against prod and must be parameterized before §9's "suite passes green against the prod cluster" can be claimed.