# Prod bare-metal provisioning checklist (T-7.1) **Task:** T-7.1 · **Workstream:** WS-INFRA · **Serves:** R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03 **Status:** Phase 7 **opened by the operator 2026-08-31** — *prep only*. Nothing is installed on prod hardware until the blockers in §7 are closed and `make -C ci gate` is green. > **Entry-gate note (recorded, not waived).** `roadmap/phases.md` gates Phase 7 on *"Phases 1–6 > proven in dev/test; operator declares prod hardware ready."* The operator has declared hardware > ready. The first condition is **not** met: conformance is 37/51, and the uncovered set includes > **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work > knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry). Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)). The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse the *same* Ansible roles and the *same* Flux base as dev — that reuse is the R-ENV-02 obligation. --- ## 0. Node inventory Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. | | node 1 | node 2 | node 3 | |---|---|---|---| | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` | | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` | | Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | | CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | | RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) | | Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) | | Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC | | OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 | | Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | | `/dev/kvm` | present | present | present | | Serial | MXL1052NGN | MXL0073949 | 8CC015307N | **Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check; unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**), `DIMM3` a Micron **16**ATF2G64HZ-**2G6**E1 (DDR4-**2667**). Both are 16 GB so capacity is correct, but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch. **The nodes are NOT hardware-symmetric — and with node 3 in (2026-09-03) the asymmetry is THREE-WAY, not a node-1 outlier.** Node 1 has a ~2011 SATA SSD with real wear; node 2 a healthy modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent health (100% life, 0 grown bad blocks). **Three nodes, three different drive models, and only one is NVMe.** §2 requires an identical disk layout across all three because Longhorn replica placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs** (2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to the §0b bug on exactly the same terms as node 1 and needs the same >= 6.13 kernel. Node 3 therefore WIDENS §0b's blast radius from "both prod nodes" to **all three**; it is not a control machine. Every node also carries **exactly one** physical disk with **no unpartitioned spare**, so `node_baseline_longhorn_device` / `node_baseline_etcd_device` have nothing to claim on any of the three without repartitioning or adding drives — the §2 decision cannot be satisfied by node 3 either. Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd). --- ## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01) > **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory > remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine. > Keeping the correction visible because the wrong framing nearly cost money and a board RMA. `k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before reserving memory based on its contents.** Fixed upstream in **6.13**. ### The evidence that breaks it open The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full 15.9 GiB two boots earlier**: | Boot | `Memory: … available` | reserved | memattr line | |---|---|---|---| | −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` | | −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | | 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` | The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in that shape, and none is fixed by a reboot. ### Mechanism (verified against kernel source) `drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`: ```c if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; } tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size; memblock_reserve(efi_mem_attr_table, tbl_size); ``` **`num_entries` and `desc_size` are never validated.** With a corrupt table: - garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)* - garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)* That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` = DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero. ### ALL THREE prod nodes are affected — none is immune Seven boots observed across the three machines. **The MAT address moves boot-to-boot on every node with settings held constant**, and each node has been seen on more than one path: | Node | Boot | `MEMATTR=` | Kernel's read | Memory | |---|---|---|---|---| | 1 | −1 | `0xbeee6018` | *(no message — table read as VALID)* | 32.4 G ✅ | | 1 | 0 | `0xbeec9298` | version **1030** (> 2 → bail) | 32.4 G ✅ | | 2 | −2 | `0xbeadc018` | version **1463964749** (> 2 → bail) | 15.9 G ✅ | | 2 | −1 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ | | 2 | 0 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ | | 3 | −1 | `0xbee8a018` | version **1463964749** (> 2 → bail) | 32.4 G ✅ | | 3 | 0 | `0xbeece018` | *(no message — table read as VALID)* | 32.4 G ✅ | So there are **three** outcomes, not two: bail-out on a garbage version > 2, a *valid* parse with a sane size, and the catastrophic path. `k3s-prod-3` publishes a MAT at `0xbeece018` exactly like the others — the *absence* of a memattr diagnostic line on its current boot means the kernel read a **valid** table, which is the opposite of immunity. Dev nodes publish no MAT at all (Proxmox SeaBIOS/OVMF) and are genuinely unaffected. **All three prod nodes are on the same lottery and all three need the ≥ 6.13 kernel.** ### The corrupting bytes are the OS boot chain, not the firmware or the DRAM The garbage version `1463964749` = `0x5742544D` appeared on **node 2 AND node 3** — two different machines, identical corrupting bytes. That rules out both firmware-specific behaviour and a DRAM fault: the overwriter is the **shared OS boot chain** (shim / GRUB / EFI stub / initrd placement), writing the same data into the page the firmware pointed at. This is the documented failure mode — the firmware places the MAT in memory the OS classifies as plain `System RAM`, and whoever allocates there next overwrites it. It also means the boot chain's own allocation footprint — initrd size, module set, boot device — is part of the causal chain, which matters for the experiment design below. ### Mechanism, traced to the byte (second, independent confirmation) A second research pass reached the same root cause by a different route and pinned the arithmetic: - The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it. - `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and when bit 31 lands the result is **negative**. - `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical address above `0xbeeb0018` is reserved.** - `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages` (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`. - The `Failed to map EFI Memory Attributes table` message is emitted *later*, by `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is therefore positive proof the reservation already happened** — it is a consequence, not the cause. The cleanest A/B tell between the two nodes is one line of dmesg: | | `NODE_DATA(0) allocated` | meaning | |---|---|---| | prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation | | prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement | Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB. ### THERE IS NO KERNEL CMDLINE WORKAROUND `efi_memattr_init()` is called from `efi_config_parse_tables()` under only `!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()` **before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap` **do not prevent the reservation.** Booting a fixed kernel is the only real fix. ### The fix **Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory. The hardening landed exactly one release too late for Debian 13: the bug has existed since the feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up `5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim *"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the release lifetime, and the fix is small and already upstream. Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382) (Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook Studio G5**, 16 GiB → 1.5 GiB). ### Will 6.12 ever get the fix? — researched 2026-09-02. **Not on its own.** | Question | Answer | |---|---| | In `linux-6.12.y` today? | **No.** Newest commit touching `memattr.c` on that branch is 2024-07-12. | | How long has it been available? | In mainline since **v6.13, 2025-01-19** — ~19 months and ~97 6.12.y point releases. | | Tagged `Cc: stable`? | **No.** Neither commit. | | Tagged `Fixes:`? | **No** — and this is structural, not an oversight. The unvalidated arithmetic has been there since the file was created in **2016**; it is not a regression, so there is no bad commit to point a `Fixes:` at. **Both automatic feeds — the `Fixes:`-scanning scripts and AUTOSEL — had nothing to grab.** | | Picked up by AUTOSEL? | **Never surfaced** for any branch. | | Has anyone asked? | **No.** Nothing on `stable@vger.kernel.org`, and nothing in all 8,027 open+archived Debian bugs against `src:linux`. We would be first. | | 6.12 LTS horizon | **EOL December 2028** — outlives Debian 13's own support window, so a backport is *not* time-barred. | **Every automatic path has already had its chance and produced nothing.** The probability that 6.12.y acquires this commit spontaneously is close to zero. **But the request is unusually strong, and the cost is one email.** The decisive card: the cherry-pick is **byte-clean**. Debian trixie 6.12.94-1's `drivers/firmware/efi/memattr.c` hashes to `164203429fa7a…`, **identical to the upstream patch's pre-image blob** — the file has not diverged at all between v6.12 and the commit's parent, so `git cherry-pick 8fbe4c49c0cc` applies with zero conflicts and zero adjustment. Neither the stable team nor Debian would have any backporting work to do, which removes the usual reason such requests are declined. The commit also meets every criterion in `stable-kernel-rules.rst`: in mainline, 17 insertions (limit 100 lines), fixes "a hardware quirk" and "a real bug that bothers people" — the enumerated categories. **Debian rebases**, so an upstream pickup reaches us automatically: trixie carries `linux 6.12.94-1` in *trixie-security* whose changelog opens *"New upstream stable update: ChangeLog-6.12.91"*. No action beyond `apt upgrade`. #### Plan 1. **Run `trixie-backports` on both prod nodes now**, as an explicitly *provisional* measure. Any 6.13+ kernel carries the fix. Record the standing cost: backports packages are provided as-is, without the main archive's security guarantees. 2. **Email `stable@vger.kernel.org`** — this is `stable-kernel-rules.rst` Option 2 for an already-merged patch. CC `linux-efi@vger.kernel.org` and **Ard Biesheuvel ``**, who both authored and committed the fix as EFI maintainer. Include: the commit ID and subject; **6.12.y only**; the byte-clean-apply proof above (it pre-empts the main objection); and the reproduction — three consecutive boots on one machine, two outcomes, **byte-identical `BIOS-e820` across all three** (the detail that proves the divergence is in the MAT, not the memory map), 16 GB → 2.6 GiB, plus a second identical machine taking the safe path only by chance. Attach both dmesgs. Corroboration exists: [Ubuntu LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (HP ZBook G5, 16 GiB → 1.5 GiB) and SUSE bugzilla 1231465, cited in the commit message itself. 3. **File a Debian BTS bug as secondary**, against `src:linux` via `reportbug` (the bug-script metadata is mandatory), severity `important`, referencing the lore thread. Debian's kernel handbook §9.1.7 allows cherry-picking a patch already accepted upstream — which this is, maintainer-authored with two Acked-bys — so if stable@ goes quiet Debian can act on its own. 4. **Trigger to make the deviation permanent:** no stable@ pickup within ~2 release cycles (4–6 weeks) and no Debian movement ⇒ encode the backports kernel in the provisioning IaC for every prod node and document it as a standing deviation from the distro baseline. **If patching by hand, only `8fbe4c49c0cc` is needed.** The follow-up `5d0faa8e8369` loosens a heuristic that was rejecting *legitimate* tables (false kexec warnings in Meta's fleet); it is not required for this symptom and depends on the first commit anyway. ### Is it a BIOS setting? — three-way diff, researched 2026-09-03. **No.** The operator's hypothesis, tested properly now that there are two working nodes and one broken one. All **240** attributes read on all three; identical key sets; identical firmware everywhere (same BIOS version string, ME 12.1.1.1022, GOP 9.0.1085, Super I/O 8.9.22). So every difference is settings or hardware, not firmware revision. **The discriminating set — both working nodes agree, the broken node differs.** 26 attributes differ in total; after excluding per-machine noise (serials, UUIDs, MACs, asset tags, Born On Date, Feature Byte, SKU, memory size, storage device names) and settings falling on a different partition, seven remain: | Attribute | nodes 1 & 3 (work) | node 2 (broken) | Verdict | |---|---|---|---| | **Fast Boot** | Disable | **Enable** | **plausible probability-shifter — the only live candidate** | | Legacy Boot Options | Disable | Enable | **coincidence — the setting is INERT, see below** | | Microsoft / Windows / MS Option ROM UEFI CA 2023 | Enable | Disable | coincidence (sub-page `db` size at most) | | TPM Activation Policy | F1 to Boot | Allow user to reject | coincidence | | Recover OS after Boot Failure | Disable | Enable | coincidence (but a policy problem in its own right) | **`After Power Loss` is eliminated by the third node** — node 3 *works* and shares node 2's `Power Off`; node 1 is the odd one out. It was never mechanistically credible (PCH/ME power policy, applied before firmware executes). It remains a §1 checklist violation on **two** nodes now. **`Legacy Boot Options` is affirmatively inert**, which a naive diff gets wrong. Three attributes are **identical on all three nodes** and settle it: `Configure Legacy Support and Secure Boot` = *"Legacy Support Disable and Secure Boot Enable"*, `Configure Option ROM Launch Policy` = *"All UEFI"*, `UEFI Boot Options` = Enable. `Legacy Boot Options` is only the boot-order checkbox; the master switch that actually instantiates a CSM has legacy support disabled on node 2 too. No CSM is built, no legacy option ROM is shadowed. The second-strongest candidate has **no boot-time allocation footprint at all**. Secure Boot is likewise not a differential — all three log `Secure boot enabled` and load the same HP DB cert. **`Fast Boot` is the one credible mechanism**, and for a *different reason* than the hypothesis this document previously recorded and killed. HP documents it verbatim: *"reduces boot up time by bypassing boot to USB, CD-ROM, and PXE. **This skips some preboot initialization steps.**"* Skipped preboot init ⇒ fewer boot-services allocations before the MAT is installed ⇒ a different heap top ⇒ **the MAT lands at a different address**. That is exactly the perturbation that flipped the near-identical Dell case in [LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382), where the reporter's "fix" was **unplugging a USB mouse**. (The old Fast-Boot-caches-MRC-training theory stays dead — byte-identical `BIOS-e820` refuted it.) #### Verdict: the hypothesis is not supported, and this diff cannot support it even in principle 1. **No setting can be the cause.** Node 2 booted good once and bad twice with settings constant, and the MAT address moves every boot on all three machines. 2. **`Fast Boot` may shift the odds.** That is the honest ceiling. There is currently zero evidence that it does. 3. **A fatal confound:** the `1=3 ≠ 2` partition is *perfectly collinear* with two hardware differences that have at least as strong a mechanism — node 2 is the only **single-DIMM** node and the only **NVMe-booting** node. Nodes 1 and 3 are both dual-DIMM and both SATA. **Adding node 3 did not break the confound; it doubled it.** A different boot device means a different initrd module set and size, hence a different EFI-stub allocation footprint — the very thing that decides whether the MAT page gets overwritten. **No model-specific documentation exists.** HP publishes nothing mentioning the EFI Memory Attributes Table, MAT placement, or an NX/DEP option on this generation. The only memory-adjacent options (`Option ROM Launch Policy`, `DMA Protection`, `Pre-boot DMA protection`) are identical fleet-wide. No HP advisory matches. The R21 changelog never mentions memory, and these nodes run **02.26.00 dated 05/05/2026 — newer than anything HP has indexed publicly**, so there is no BIOS update to chase. #### The experiment, if it is ever worth running Two statistics kill the obvious approach. Node 2's failure rate is known only to **n = 3** (2 bad of 3); the 95% Clopper–Pearson interval on p(bad) is ≈ **0.09–0.99**. Against a null that loose, even 8 clean boots after a change prove nothing. **A baseline arm is mandatory**, and: detecting *elimination* needs ~**20 boots per arm**; detecting a mere odds shift (0.67 → 0.33) needs ~**50 per arm** for 80% power. **Do not design for an odds shift — design for elimination and report "inconclusive" otherwise.** **Zero-touch first step, and the one actually worth doing:** 20 scripted reboots of node 2 with *nothing changed*, logging `MEMATTR=` and the `Memory: …K/…K available` line per boot. That establishes the real p(bad) we lack, and tests the causal chain directly — **is the outcome predicted by where the MAT landed?** Present data hint yes: both failures at exactly `0xbeeb0018`, all five successes elsewhere. If the bad outcome maps onto a narrow address range, the mechanism is settled without touching anything. **If more is wanted, swap hardware rather than BIOS settings** — it crosses settings against memory topology in one move, needs no firmware write (writes are refused; a BIOS change means a physical F10 visit per arm), and is work §2 wants anyway: move node 2's single Samsung DIMM into node 3 and node 3's matched pair into node 2, then 20 boots each. Fault follows the DIMM ⇒ topology, settings exonerated. Fault stays on node 2 ⇒ node 2 implicated, and only then is Fast Boot worth testing. Fault disappears ⇒ single-DIMM topology. ~1 hour unattended. **None of this changes the action.** The fix is already known, is not in dispute, and applies to all three nodes: kernel ≥ 6.13. ### What NOT to do - **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6. - **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries. - **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket: they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They may appear to "fix" it. They are not fixes. ### Hypotheses killed, and why — so they are not re-run - **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control. - **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by `hp-bioscfg` on this platform. - **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB. - **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`. Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never `memblock_add()`ed — which would have left `present` at ~0, not 3340288. - **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`. - **MTRR trim** — MTRRs near-identical to node 1, no trim warning. - **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal. - The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly explained: everything from `0xbeeb0018` upward. ## 1. Firmware / BIOS (per node, before OS install) > **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes > **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a > `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes", > which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found > at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye. > > **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4, > "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is > provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means > F10 on the box. > > **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is > achievable today with reads alone, and would have caught node 2's drift on its first health run. > Converging it automatically additionally requires provisioning a BIOS Setup Password on every > node, which is a real security decision (a stored credential that can rewrite firmware) and is > the operator's call, not a default. - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero. - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a human pressing the button. (`docs/02` calls out wake/boot behaviour explicitly.) - [ ] **Wake-on-LAN enabled** on the onboard NIC (remote recovery when the box is off). - [ ] **Boot order** — internal disk first; USB above it only while imaging. Disable network boot unless PXE is adopted. - [ ] **Secure Boot** — record the setting. k3s itself is fine either way; note it so all three nodes match. - [ ] **Firmware version recorded** in §0 and consistent across all three nodes. ### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01 **There is no default BIOS password, and no factory default exists to look up.** HP business desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5 describes both purely as things the owner configures in Computer Setup. **Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once). **Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely, turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The permissive default is the right setting for a homelab. #### CORRECTION — error `0x4` is NOT an authorisation failure I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI `SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting` returns: | Code | Meaning | |---|---| | 0 | Success | | 1 | Not Supported | | **4** | **Failed** (generic) | | 6 | Access Denied | So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`. **Most likely actual cause (inferred, testable):** with an empty password the driver emits an *untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when blank (`Password = ""`). The driver only prepends that prefix for a *non-empty* string. So setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this firmware already makes the driver log `Prerequisites size value exceeded the maximum number of elements supported or data may be malformed` at boot. **`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and `new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM; nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read. #### Recommendation 1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff `current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10. 2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to `current_password`, retry one benign attribute. Return `0` confirms the `` theory; `6` would mean a password mismatch; another `4` means the driver is incompatible with R21 and the fallback is `hp-repsetup`. 3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**. Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a security downgrade. 4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in principle — the private key never leaves the workstation, each change is individually signed, with anti-replay — but provisioning and every signature require Windows-only tooling. For three machines whose firmware settings change roughly never, the operational cost far exceeds the risk reduction. BCU is Windows/WinPE-only. 5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) — functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via `hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the password in cleartext on the command line — it would land in process listings and shell history unless wrapped. Store it in the Ansible vault **and** offline. ## 2. Disk layout — **OPEN DECISION, blocks the build** `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went further and added a *third* device for etcd after HDD contention starved etcd WAL fsync (`docs/plans/ssd-etcd-isolation.md`). Node 1 as-found is a **single** 477 GB SATA SSD carrying everything: ``` sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial) ├─sda1 976M /boot/efi ├─sda2 30G / ├─sda3 12G /var ├─sda4 977M swap └─sda5 433G /srv <- the only large free space ``` Consequences to settle before imaging nodes 2 and 3: - [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package floor — see §5). Raw attributes, so the numbers are auditable: | | | |---|---| | Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) | | SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` | | Power-on hours | 6,814 (~284 days) · 147 power cycles | | **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) | | **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) | | **Erase failures** | **187** (`172`) | | **Unexpected power losses** | **32** (`174`) | | Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) | | UDMA CRC errors | 0 (`199`) — cabling is clean | **Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**, not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value; the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent uncorrectable counters instead. **Verdict: usable, but not for production state.** Nothing says imminent failure — no pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011 consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad blocks, logged **187** erase failures, and taken **32** unclean power losses **with no power-loss protection**, in a single failure domain. etcd is the worst possible tenant for it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how a member corrupts — this drive has already logged 32 of them. (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see `DECISIONS-LOG.md`. It is not part of this verdict.) - [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices* (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them. Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** — prod must override them in `group_vars`. On a single-disk node there is no second device at all, so one of: - **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own spindle. - **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`. Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by. - **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one physical failure domain. - [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry. - [ ] Record the chosen device paths here and set them in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately `/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full `node-baseline` run against a node still carrying those placeholders now stops on an explicit assert naming this section, instead of failing obscurely at `mkfs` time. - [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline` is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk tasks this decision blocks — see §5's drive-health box for the exact command. ## 3. OS install (per node) - [ ] **Debian 13 (trixie)**, matching dev exactly — same kernel series, same package baseline. - [ ] Hostname `k3s-prod-N`. - [ ] Partition per the §2 decision. - [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into `roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now), so the same role covers bare metal. Nothing to port. ## 4. Network (per node) - [ ] **Static IP** on the prod LAN, recorded in §0. Node 1 is `192.168.2.10/24`, gateway `192.168.2.1`. - [x] **Prod address plan — DECIDED (operator, 2026-08-31).** | Role | Address | Owned by | |---|---|---| | `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory | | **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) | | **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) | | **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) | Two deliberate properties, both mirroring dev: - The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way round (`.49` control-plane, pool `.53-.60`). - **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape (`GATEWAY_VIP` `.53` = first of `.53-.60`). - [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before the build, and confirm nothing else on the network already answers on them. - [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.** Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single upstream is a known SPOF that has already bitten this project — it broke containerd image pulls and CoreDNS forwarding on dev. The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in `roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes `/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed `/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager owns it), and configures the ordered upstream list either way. The earlier `fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie nodes, so it would have broken `node-baseline.yml` on every host. What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets `node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid `BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert stops the play on it before writing anything. Choosing the second upstream is a **network decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would resolve reliably but would also bypass any local zone and any filtering the gateway applies. Decide between — and record the choice HERE, in the same change that edits the group_vars: - a second on-LAN resolver on `192.168.2.x`; - the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified — prod sits on the workstation side of the pfSense boundary); - an explicit, written-down acceptance of a public resolver as fallback-only. Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`) are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed servers, so keeping them means listing them explicitly. `node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's local domain is decided. - [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT — prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind. ## 5. Access (per node) — **blocks all Ansible** - [ ] **`k3s` user with the `home_network` SSH key.** Working on node 1. - [ ] **Password-protected sudo, driven by Ansible `become` + a vaulted password.** **Prod does NOT use NOPASSWD sudo** (operator decision, 2026-08-31). Dev nodes do; prod deliberately keeps the sudo password in place and supplies it to Ansible as an **ansible-vault-encrypted** var, exactly like the existing `hypervisor` and `proxmox_hosts` groups: ``` inventory/group_vars/k3s_prod/ README.md how to (re)create the vault — committed vault.yml ansible_become_password: "..." — committed, ENCRYPTED ``` The passphrase that decrypts it lives **outside the repo** at `~/.config/home-k3s/ansible-vault-pass` (`ansible.cfg :: vault_password_file`), so plays still run unattended without a plaintext secret ever touching git. ```bash ansible-vault create inventory/group_vars/k3s_prod/vault.yml # ansible_become_password: ... ``` Prod nodes must therefore set `ansible_become: true` in their group_vars and must **not** inherit any passwordless-sudo assumption from `group_vars/all.yml`. - [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead (the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so the placeholder device paths are never even templated): ```bash cd infra/ansible ansible-playbook playbooks/node-baseline.yml \ -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1 ``` (Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading the drive is then: ```bash sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors ``` Record the result in §2 and close the layout decision in the same change that replaces the two `node_baseline_*_device` placeholders. ## 6. Ansible baseline & cluster join Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars differ, plus a target-group override on the command line. ### How prod is targeted (done, 2026-08-31) Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` / `k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately kept separate because Ansible resolves them at different times: | Layer | Knob | Set via | |---|---|---| | Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there | | Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** | So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after `--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**. - [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached **directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping. (Nodes 2/3 stay commented out until they are physically on the bench.) - [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`), `cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the §2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which are still BLOCKED placeholders. - [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until it exists **no** prod play can escalate, and `--check` against prod cannot work either. - [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read: ```bash ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages ``` - [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare metal — the role already only *starts* it when the virtio channel exists, so it degrades cleanly.) - [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary` → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three nodes are present** — see §8. - [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the prod LAN. - [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The bootstrap values and the HelmRelease **must stay in parity** (`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config (`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the dev one the dev container exports as `$KUBECONFIG`. - [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`. - [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**: `roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a target-group change. Tracked with the other repo blockers in §7. - [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015). `playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=`, but there is deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted PIA credentials — the existing vault is scoped to the dev group. ## 7. Repo blockers — **prod cannot be reconciled until these land** `clusters/prod` is design-only today and **would build wrong**. R-ENV-02 promises dev and prod differ *only* by `cluster-vars` values; in practice dev LAN addresses are hardcoded into base manifests: | Hardcoded value | Where | Prod needs | |---|---|---| | `k8sServiceHost: 192.168.1.50` | `platform/cilium/helmrelease.yaml:50`, mirrored in `values.yaml:16` | prod primary node IP | | LB pool `192.168.1.53–.60` | `platform-config/cilium/l2-ip-pool.yaml:15` | a prod-LAN pool | | `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address | | `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address | | `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target | - [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both* overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the header comment in `clusters/base/cluster-vars.yaml`). `platform`, `platform-config`, the app waves, `apps-vms` and `crafty` all already carry `postBuild.substituteFrom: cluster-vars`, so the seam exists. - [ ] **`clusters/prod` must set `GATEWAY_VIP`.** It currently doesn't, so prod would inherit the base default `10.73.0.240` — an address from the **retired libvirt network**. The base file's own comment flags this as owed "when hardware is built." - [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`. - [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing the `dependsOn` fix. Only the first is now closed: - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block forever on a dependency that does not exist and nothing downstream would reconcile. - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only. - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and will be decommissioned once prod is stable. Recorded consequence: once prod is live, **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window. The URL is patched in but **cannot work on a cold prod cluster**, for two independent reasons found 2026-09-01: 1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth. **This is the decommission plan's first task, not its last.** 2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`: prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve `gitea.home.keithjackson.dev` to itself and never reach a repository. Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own Phase-6 cutover. - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03). ## 8. Build sequence (operator decision, 2026-08-31: **wait for all three nodes**) Node 1 is baselined now; the cluster is **not** initialized until nodes 2 and 3 are on the bench. This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a single-server-etcd window, and avoids a later member-add dance. 1. Per-node §1–§5 on each of the three boxes. 2. §7 repo work lands on `main` with the gate green. 3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`. 4. `flux bootstrap ... --path=clusters/prod`. 5. Conformance against prod (§9). ## 9. Exit criteria (Phase 7) - [ ] Prod runs the same config as dev via the prod overlay, **no base edits** (R-ENV-02). - [ ] Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01). - [ ] Full **automated** prod DR drill (R-STATE-03) — `definition-of-done.md` requires this for prod specifically. - [ ] Monitoring and alerts live on prod. - [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).