home-network-k3s-impl/infra/PROD-PROVISIONING.md

# Prod bare-metal provisioning checklist (T-7.1)

**Task:** T-7.1 · **Workstream:** WS-INFRA · **Serves:** R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03
**Status:** Phase 7 **opened by the operator 2026-08-31** — *prep only*. Nothing is installed on prod
hardware until the blockers in §7 are closed and `make -C ci gate` is green.

> **Entry-gate note (recorded, not waived).** `roadmap/phases.md` gates Phase 7 on *"Phases 1–6
> proven in dev/test; operator declares prod hardware ready."* The operator has declared hardware
> ready. The first condition is **not** met: conformance is 37/51, and the uncovered set includes
> **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work
> knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry).

> **Looking for the commands?** This document is the *decision record* — why prod is built the way it
> is, what was measured, and which hypotheses are dead. The **executable sequence** lives in
> [`PROD-STANDUP.md`](PROD-STANDUP.md): every command, in order, driven by `bootstrap/standup.sh`.
> Start there; come back here for the reasoning behind any step.

Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)).
The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to
call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse
the *same* Ansible roles and the *same* Flux base as dev — that reuse is the R-ENV-02 obligation.

---

## 0. Node inventory

Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.

| | node 1 | node 2 | node 3 |
|---|---|---|---|
| Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
| LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini |
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x |
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) |
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 |
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 |
| `/dev/kvm` | present | present | present |
| Serial | MXL1052NGN | MXL0073949 | 8CC015307N |

**Node 1's DIMMs are also not a matched pair** (surfaced 2026-09-01 by the new health check;
unrelated to §0b and **not a fault**): `DIMM1` is a Micron **8**ATF2G64HZ-**3G2**E1 (DDR4-**3200**),
`DIMM3` a Micron **16**ATF2G64HZ-**2G6**E1 (DDR4-**2667**). Both are 16 GB so capacity is correct,
but they differ in rank organisation and rated speed, so the pair runs at the slower 2667. Recorded
only because §2 requires the three nodes to end up symmetric, and "2×16 GB Micron" hid the mismatch.

**The nodes are NOT hardware-symmetric — and with node 3 in (2026-09-03) the asymmetry is
THREE-WAY, not a node-1 outlier.** Node 1 has a ~2011 SATA SSD with real wear; node 2 a healthy
modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent health (100% life,
0 grown bad blocks). **Three nodes, three different drive models, and only one is NVMe.** §2 requires
an identical disk layout across all three because Longhorn replica placement assumes symmetry — so
this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should
have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs**
(2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually
usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table
too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to
the §0b bug on exactly the same terms as node 1 and needs the same >= 6.13 kernel. Node 3 therefore
WIDENS §0b's blast radius from "both prod nodes" to **all three**; it is not a control machine.
Every node also carries **exactly one** physical disk with **no unpartitioned spare**, so
`node_baseline_longhorn_device` / `node_baseline_etcd_device` have nothing to claim on any of the
three without repartitioning or adding drives — the §2 decision cannot be satisfied by node 3 either.

Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).

---

## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01)

> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory
> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine.
> Keeping the correction visible because the wrong framing nearly cost money and a board RMA.

`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware
publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before
reserving memory based on its contents.** Fixed upstream in **6.13**.

### The evidence that breaks it open

The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full
15.9 GiB two boots earlier**:

| Boot | `Memory: … available` | reserved | memattr line |
|---|---|---|---|
| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` |
| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |

The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in
that shape, and none is fixed by a reboot.

### Mechanism (verified against kernel source)

`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`:

```c
if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; }
tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
memblock_reserve(efi_mem_attr_table, tbl_size);
```

**`num_entries` and `desc_size` are never validated.** With a corrupt table:
- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)*
- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s
  the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)*

That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` =
DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero.

### ALL THREE prod nodes are affected — none is immune

Seven boots observed across the three machines. **The MAT address moves boot-to-boot on every
node with settings held constant**, and each node has been seen on more than one path:

| Node | Boot | `MEMATTR=` | Kernel's read | Memory |
|---|---|---|---|---|
| 1 | −1 | `0xbeee6018` | *(no message — table read as VALID)* | 32.4 G ✅ |
| 1 | 0 | `0xbeec9298` | version **1030** (> 2 → bail) | 32.4 G ✅ |
| 2 | −2 | `0xbeadc018` | version **1463964749** (> 2 → bail) | 15.9 G ✅ |
| 2 | −1 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ |
| 2 | 0 | `0xbeeb0018` | **Failed to map** | 2.6 G ❌ |
| 3 | −1 | `0xbee8a018` | version **1463964749** (> 2 → bail) | 32.4 G ✅ |
| 3 | 0 | `0xbeece018` | *(no message — table read as VALID)* | 32.4 G ✅ |

So there are **three** outcomes, not two: bail-out on a garbage version > 2, a *valid* parse with a
sane size, and the catastrophic path. `k3s-prod-3` publishes a MAT at `0xbeece018` exactly like the
others — the *absence* of a memattr diagnostic line on its current boot means the kernel read a
**valid** table, which is the opposite of immunity. Dev nodes publish no MAT at all (Proxmox
SeaBIOS/OVMF) and are genuinely unaffected.

**All three prod nodes are on the same lottery and all three need the ≥ 6.13 kernel.**

### The corrupting bytes are the OS boot chain, not the firmware or the DRAM

The garbage version `1463964749` = `0x5742544D` appeared on **node 2 AND node 3** — two different
machines, identical corrupting bytes. That rules out both firmware-specific behaviour and a DRAM
fault: the overwriter is the **shared OS boot chain** (shim / GRUB / EFI stub / initrd placement),
writing the same data into the page the firmware pointed at. This is the documented failure mode —
the firmware places the MAT in memory the OS classifies as plain `System RAM`, and whoever allocates
there next overwrites it.

It also means the boot chain's own allocation footprint — initrd size, module set, boot device — is
part of the causal chain, which matters for the experiment design below.



### Mechanism, traced to the byte (second, independent confirmation)

A second research pass reached the same root cause by a different route and pinned the arithmetic:

- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The
  kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that
  `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it.
- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and
  when bit 31 lands the result is **negative**.
- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to
  ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical
  address above `0xbeeb0018` is reserved.**
- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages`
  (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`.
- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by
  `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is
  therefore positive proof the reservation already happened** — it is a consequence, not the cause.

The cleanest A/B tell between the two nodes is one line of dmesg:

| | `NODE_DATA(0) allocated` | meaning |
|---|---|---|
| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation |
| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement |

Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.

### CORRECTION (2026-09-03) — "Failed to map" does NOT mean memory was lost

An earlier version of this section, and the `firmware.efi_memattr` health check built from it, treated
the log line `efi: memattr: Failed to map EFI Memory Attributes table @ 0x…` as proof of memory loss.
**A live counter-example falsifies that.** `k3s-prod-2`, after being power-cycled on 2026-09-03:

```
MEMATTR=0xbeeb0018                                             <-- the SAME address as both bad boots
[ 0.106493] efi: memattr: Failed to map ... @ 0xbeeb0018       <-- the SAME line
MemTotal: 15857732 kB                                          <-- 15.1 GiB of 16 GB. FULL MEMORY.
```

**Two different kernel functions were being conflated:**

- **`efi_memattr_init()`** does the `memblock_reserve()`. Catastrophic loss requires `tbl_size` to be
  **negative** — signed `int`, sign-extended through `phys_addr_t`, clamped to `PHYS_ADDR_MAX`, so
  everything above the table is reserved. A garbage-but-**positive** `tbl_size` reserves only a modest
  wrong amount and costs nothing noticeable.
- **`efi_memattr_apply_permissions()`** does the later `memremap()`, and **this is what prints
  "Failed to map"**. It fails for *any* sufficiently wrong size — negative or merely large-positive.

So the line proves **the table was corrupt and the kernel trusted it**, not that memory was lost. The
cost is set by the *value* of the garbage on that boot, which is why the same address yielded 2.6 GiB
twice and 15.1 GiB once.

**This also refutes the address→outcome hypothesis outright** — it is not merely untested, it is
disproven. The MAT address does not determine the outcome; the garbage value at it does.

The health check has been re-based accordingly: the measured shortfall is computed **once** into a
shared fact and both `memory.physical_total` and `firmware.efi_memattr` read it, so they can no longer
contradict each other. `Failed to map` + normal memory is now **WARN** ("the bug fired and this boot
survived it"), and FAIL requires a measured shortfall.

### SECOND FAULT — node 2 hangs on a warm reboot, staying powered

On 2026-09-03 `k3s-prod-2` failed to return from the **first** scripted reboot of the survey below. Its
journal ends cleanly mid-shutdown (services stopping, `eno1: NIC Link is Down`) and then stops. The box
stayed **powered on but off the LAN for ~38 minutes** — no ping, no SSH, ARP INCOMPLETE — until the
operator held the power button and cold-booted it by hand.

**This is arguably a worse prod defect than the memory bug.** A node that cannot survive an unattended
warm reboot cannot be rolled in an HA cluster: every kernel update and maintenance window becomes a
physical visit, and `roles/node_health` cannot see a machine that is off the network.

Leading candidate: node 2 is the **only** node with `Recover OS after Boot Failure = Enable`. A box
parked at a firmware recovery prompt awaiting a keypress presents exactly this way. Unproven — it needs
eyes on the screen during a hang — but it promotes that setting from "coincidence" in the BIOS diff to a
live availability defect. **Worth disabling on all three nodes regardless.**

Consequence for testing: **the reboot survey is not viable unattended on node 2** until this is
understood or out-of-band recovery exists.

### THERE IS NO KERNEL CMDLINE WORKAROUND

`efi_memattr_init()` is called from `efi_config_parse_tables()` under only
`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()`
**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap`
**do not prevent the reservation.** Booting a fixed kernel is the only real fix.

### The fix

**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is
clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message
describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable
branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints
`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory.

The hardening landed exactly one release too late for Debian 13: the bug has existed since the
feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up
`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim
*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably
lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the
release lifetime, and the fix is small and already upstream.

Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382)
(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the
table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook
Studio G5**, 16 GiB → 1.5 GiB).

### Will 6.12 ever get the fix? — researched 2026-09-02. **Not on its own.**

| Question | Answer |
|---|---|
| In `linux-6.12.y` today? | **No.** Newest commit touching `memattr.c` on that branch is 2024-07-12. |
| How long has it been available? | In mainline since **v6.13, 2025-01-19** — ~19 months and ~97 6.12.y point releases. |
| Tagged `Cc: stable`? | **No.** Neither commit. |
| Tagged `Fixes:`? | **No** — and this is structural, not an oversight. The unvalidated arithmetic has been there since the file was created in **2016**; it is not a regression, so there is no bad commit to point a `Fixes:` at. **Both automatic feeds — the `Fixes:`-scanning scripts and AUTOSEL — had nothing to grab.** |
| Picked up by AUTOSEL? | **Never surfaced** for any branch. |
| Has anyone asked? | **No.** Nothing on `[email protected]`, and nothing in all 8,027 open+archived Debian bugs against `src:linux`. We would be first. |
| 6.12 LTS horizon | **EOL December 2028** — outlives Debian 13's own support window, so a backport is *not* time-barred. |

**Every automatic path has already had its chance and produced nothing.** The probability that 6.12.y
acquires this commit spontaneously is close to zero.

**But the request is unusually strong, and the cost is one email.** The decisive card: the cherry-pick
is **byte-clean**. Debian trixie 6.12.94-1's `drivers/firmware/efi/memattr.c` hashes to
`164203429fa7a…`, **identical to the upstream patch's pre-image blob** — the file has not diverged at
all between v6.12 and the commit's parent, so `git cherry-pick 8fbe4c49c0cc` applies with zero
conflicts and zero adjustment. Neither the stable team nor Debian would have any backporting work to
do, which removes the usual reason such requests are declined. The commit also meets every criterion
in `stable-kernel-rules.rst`: in mainline, 17 insertions (limit 100 lines), fixes "a hardware quirk"
and "a real bug that bothers people" — the enumerated categories.

**Debian rebases**, so an upstream pickup reaches us automatically: trixie carries `linux 6.12.94-1`
in *trixie-security* whose changelog opens *"New upstream stable update: ChangeLog-6.12.91"*. No
action beyond `apt upgrade`.

#### Plan

1. **Run `trixie-backports` on both prod nodes now**, as an explicitly *provisional* measure. Any
   6.13+ kernel carries the fix. Record the standing cost: backports packages are provided as-is,
   without the main archive's security guarantees.
2. **Email `[email protected]`** — this is `stable-kernel-rules.rst` Option 2 for an
   already-merged patch. CC `[email protected]` and **Ard Biesheuvel `<[email protected]>`**,
   who both authored and committed the fix as EFI maintainer. Include: the commit ID and subject;
   **6.12.y only**; the byte-clean-apply proof above (it pre-empts the main objection); and the
   reproduction — three consecutive boots on one machine, two outcomes, **byte-identical
   `BIOS-e820` across all three** (the detail that proves the divergence is in the MAT, not the
   memory map), 16 GB → 2.6 GiB, plus a second identical machine taking the safe path only by
   chance. Attach both dmesgs. Corroboration exists:
   [Ubuntu LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (HP ZBook G5,
   16 GiB → 1.5 GiB) and SUSE bugzilla 1231465, cited in the commit message itself.
3. **File a Debian BTS bug as secondary**, against `src:linux` via `reportbug` (the bug-script
   metadata is mandatory), severity `important`, referencing the lore thread. Debian's kernel
   handbook §9.1.7 allows cherry-picking a patch already accepted upstream — which this is,
   maintainer-authored with two Acked-bys — so if stable@ goes quiet Debian can act on its own.
4. **Trigger to make the deviation permanent:** no stable@ pickup within ~2 release cycles (4–6
   weeks) and no Debian movement ⇒ encode the backports kernel in the provisioning IaC for every
   prod node and document it as a standing deviation from the distro baseline.

**If patching by hand, only `8fbe4c49c0cc` is needed.** The follow-up `5d0faa8e8369` loosens a
heuristic that was rejecting *legitimate* tables (false kexec warnings in Meta's fleet); it is not
required for this symptom and depends on the first commit anyway.

### Is it a BIOS setting? — three-way diff, researched 2026-09-03. **No.**

The operator's hypothesis, tested properly now that there are two working nodes and one broken one.
All **240** attributes read on all three; identical key sets; identical firmware everywhere (same
BIOS version string, ME 12.1.1.1022, GOP 9.0.1085, Super I/O 8.9.22). So every difference is settings
or hardware, not firmware revision.

**The discriminating set — both working nodes agree, the broken node differs.** 26 attributes differ
in total; after excluding per-machine noise (serials, UUIDs, MACs, asset tags, Born On Date, Feature
Byte, SKU, memory size, storage device names) and settings falling on a different partition, seven
remain:

| Attribute | nodes 1 & 3 (work) | node 2 (broken) | Verdict |
|---|---|---|---|
| **Fast Boot** | Disable | **Enable** | **plausible probability-shifter — the only live candidate** |
| Legacy Boot Options | Disable | Enable | **coincidence — the setting is INERT, see below** |
| Microsoft / Windows / MS Option ROM UEFI CA 2023 | Enable | Disable | coincidence (sub-page `db` size at most) |
| TPM Activation Policy | F1 to Boot | Allow user to reject | coincidence |
| Recover OS after Boot Failure | Disable | Enable | coincidence (but a policy problem in its own right) |

**`After Power Loss` is eliminated by the third node** — node 3 *works* and shares node 2's
`Power Off`; node 1 is the odd one out. It was never mechanistically credible (PCH/ME power policy,
applied before firmware executes). It remains a §1 checklist violation on **two** nodes now.

**`Legacy Boot Options` is affirmatively inert**, which a naive diff gets wrong. Three attributes are
**identical on all three nodes** and settle it: `Configure Legacy Support and Secure Boot` =
*"Legacy Support Disable and Secure Boot Enable"*, `Configure Option ROM Launch Policy` = *"All
UEFI"*, `UEFI Boot Options` = Enable. `Legacy Boot Options` is only the boot-order checkbox; the
master switch that actually instantiates a CSM has legacy support disabled on node 2 too. No CSM is
built, no legacy option ROM is shadowed. The second-strongest candidate has **no boot-time
allocation footprint at all**. Secure Boot is likewise not a differential — all three log
`Secure boot enabled` and load the same HP DB cert.

**`Fast Boot` is the one credible mechanism**, and for a *different reason* than the hypothesis this
document previously recorded and killed. HP documents it verbatim: *"reduces boot up time by
bypassing boot to USB, CD-ROM, and PXE. **This skips some preboot initialization steps.**"* Skipped
preboot init ⇒ fewer boot-services allocations before the MAT is installed ⇒ a different heap top ⇒
**the MAT lands at a different address**. That is exactly the perturbation that flipped the
near-identical Dell case in [LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382),
where the reporter's "fix" was **unplugging a USB mouse**. (The old Fast-Boot-caches-MRC-training
theory stays dead — byte-identical `BIOS-e820` refuted it.)

#### Verdict: the hypothesis is not supported, and this diff cannot support it even in principle

1. **No setting can be the cause.** Node 2 booted good once and bad twice with settings constant, and
   the MAT address moves every boot on all three machines.
2. **`Fast Boot` may shift the odds.** That is the honest ceiling. There is currently zero evidence
   that it does.
3. **A fatal confound:** the `1=3 ≠ 2` partition is *perfectly collinear* with two hardware
   differences that have at least as strong a mechanism — node 2 is the only **single-DIMM** node and
   the only **NVMe-booting** node. Nodes 1 and 3 are both dual-DIMM and both SATA. **Adding node 3
   did not break the confound; it doubled it.** A different boot device means a different initrd
   module set and size, hence a different EFI-stub allocation footprint — the very thing that decides
   whether the MAT page gets overwritten.

**No model-specific documentation exists.** HP publishes nothing mentioning the EFI Memory Attributes
Table, MAT placement, or an NX/DEP option on this generation. The only memory-adjacent options
(`Option ROM Launch Policy`, `DMA Protection`, `Pre-boot DMA protection`) are identical fleet-wide. No
HP advisory matches. The R21 changelog never mentions memory, and these nodes run **02.26.00 dated
05/05/2026 — newer than anything HP has indexed publicly**, so there is no BIOS update to chase.

#### The experiment, if it is ever worth running

Two statistics kill the obvious approach. Node 2's failure rate is known only to **n = 3** (2 bad of
3); the 95% Clopper–Pearson interval on p(bad) is ≈ **0.09–0.99**. Against a null that loose, even
8 clean boots after a change prove nothing. **A baseline arm is mandatory**, and: detecting
*elimination* needs ~**20 boots per arm**; detecting a mere odds shift (0.67 → 0.33) needs ~**50 per
arm** for 80% power. **Do not design for an odds shift — design for elimination and report
"inconclusive" otherwise.**

**Zero-touch first step, and the one actually worth doing:** 20 scripted reboots of node 2 with
*nothing changed*, logging `MEMATTR=` and the `Memory: …K/…K available` line per boot. That
establishes the real p(bad) we lack, and tests the causal chain directly — **is the outcome predicted
by where the MAT landed?** Present data hint yes: both failures at exactly `0xbeeb0018`, all five
successes elsewhere. If the bad outcome maps onto a narrow address range, the mechanism is settled
without touching anything.

**If more is wanted, swap hardware rather than BIOS settings** — it crosses settings against memory
topology in one move, needs no firmware write (writes are refused; a BIOS change means a physical F10
visit per arm), and is work §2 wants anyway: move node 2's single Samsung DIMM into node 3 and node
3's matched pair into node 2, then 20 boots each. Fault follows the DIMM ⇒ topology, settings
exonerated. Fault stays on node 2 ⇒ node 2 implicated, and only then is Fast Boot worth testing.
Fault disappears ⇒ single-DIMM topology. ~1 hour unattended.

**None of this changes the action.** The fix is already known, is not in dispute, and applies to all
three nodes: kernel ≥ 6.13.

### Reboot survey — 40 boots, run 2026-09-03

`playbooks/memattr-survey.yml` (+ `roles/memattr_survey`), reboots opt-in via
`memattr_survey_reboots_authorised`, resumable from an on-node ledger, artifacts in
`_artifacts/memattr-survey/`.

| node | bad | n | p(bad) | 95% Clopper–Pearson |
|---|---:|---:|---:|---|
| prod-1 | 0 | 20 | 0.00 | 0.000 – 0.168 |
| **prod-2** | — | **0** | — | **no data — hung on its first reboot** |
| prod-3 | 0 | 20 | 0.00 | 0.000 – 0.168 |
| 1+3 pooled | 0 | 40 | 0.00 | 0.000 – 0.088 |

**The arm the experiment was designed around produced nothing.** Node 2's prior 2-of-3 is unchanged.
And 0/20 is **not** elimination — it still permits p(bad) up to 17% per node.

What the 40 boots *did* establish:

- **Addresses are node-specific and highly concentrated.** Node 1 used only **3** distinct addresses
  across 20 boots (15 of them at `0xbeec9298`); node 3 used 8. **No address was ever shared between
  nodes.** This is a lottery with very few tickets per machine — which is exactly why 20 clean boots on
  nodes that only ever drew safe tickets says little about the node that drew a bad one.
- **`MEMATTR` → kernel parse was a perfect function across all 7 addresses seen more than once**
  (37 boots), with both readings present in the sample. Same address, same parse, every time. Combined
  with the counter-example above, the picture is: the *address* determines what the kernel finds, and the
  *garbage value there* determines whether it is survivable.
- **The corrupting bytes are stable per machine, not fleet-wide.** Node 1's garbage version was `1030` on
  **all 22** of its bail boots across three different addresses; node 3's was `1463964749` on all 12. So
  the earlier "shared OS boot chain" inference needs refining: the boot chain remains the plausible
  writer, but its footprint is machine-specific.

**Ansible defect worth knowing repo-wide:** a looped `include_tasks` under the default `linear` strategy
is flattened per host, so hosts run **serially**. The first survey attempt rebooted node 1 twenty times
while nodes 2 and 3 sat idle. Fixed with `strategy: free` plus a preflight assert that `forks` ≥ host
count. It is silent and costs 3× wall clock.

### What NOT to do

- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is
  a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be
  installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6.
- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full
  R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries.
- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket:
  they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They
  may appear to "fix" it. They are not fixes.

### Hypotheses killed, and why — so they are not re-run

- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by
  an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP
  documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control.
- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by
  `hp-bioscfg` on this platform.
- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB.
- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by
  measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`.
  Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never
  `memblock_add()`ed — which would have left `present` at ~0, not 3340288.
- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`.
- **MTRR trim** — MTRRs near-identical to node 1, no trim warning.
- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal.
- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly
  explained: everything from `0xbeeb0018` upward.

## 1. Firmware / BIOS (per node, before OS install)

> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes
> **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a
> `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes",
> which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye.
>
> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4,
> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is
> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means
> F10 on the box.
>
> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is
> achievable today with reads alone, and would have caught node 2's drift on its first health run.
> Converging it automatically additionally requires provisioning a BIOS Setup Password on every
> node, which is a real security decision (a stored credential that can rewrite firmware) and is
> the operator's call, not a default.


- [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
      boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
- [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
      human pressing the button. (`docs/02` calls out wake/boot behaviour explicitly.)
- [ ] **Wake-on-LAN enabled** on the onboard NIC (remote recovery when the box is off).
- [ ] **Boot order** — internal disk first; USB above it only while imaging. Disable network boot
      unless PXE is adopted.
- [ ] **Secure Boot** — record the setting. k3s itself is fine either way; note it so all three
      nodes match.
- [ ] **Firmware version recorded** in §0 and consistent across all three nodes.

### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01

**There is no default BIOS password, and no factory default exists to look up.** HP business
desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we
observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5
describes both purely as things the owner configures in Computer Setup.

**Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents
clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once).
**Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely,
turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The
permissive default is the right setting for a homelab.

#### CORRECTION — error `0x4` is NOT an authorisation failure

I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That
was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI
`SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a
different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have
unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting`
returns:

| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Not Supported |
| **4** | **Failed** (generic) |
| 6 | Access Denied |

So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and
the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither
of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up
in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`.

**Most likely actual cause (inferred, testable):** with an empty password the driver emits an
*untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when
blank (`Password = "<utf-16/>"`). The driver only prepends that prefix for a *non-empty* string. So
setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would
work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this
firmware already makes the driver log `Prerequisites size value exceeded the maximum number of
elements supported or data may be malformed` at boot.

**`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and
`new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM;
nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read.

#### Recommendation

1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff
   `current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV
   win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10.
2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision
   one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to
   `current_password`, retry one benign attribute. Return `0` confirms the `<utf-16/>` theory; `6`
   would mean a password mismatch; another `4` means the driver is incompatible with R21 and the
   fallback is `hp-repsetup`.
3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup
   Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it
   needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**.
   Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a
   security downgrade.
4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in
   principle — the private key never leaves the workstation, each change is individually signed, with
   anti-replay — but provisioning and every signature require Windows-only tooling. For three
   machines whose firmware settings change roughly never, the operational cost far exceeds the risk
   reduction. BCU is Windows/WinPE-only.
5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full
   pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) —
   functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via
   `hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is
   persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the
   password in cleartext on the command line — it would land in process listings and shell history
   unless wrapped. Store it in the Ansible vault **and** offline.

## 2. Disk layout — **DECIDED (operator, 2026-09-03) · REVISED (operator, 2026-09-04)**

`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
(`docs/plans/ssd-etcd-isolation.md`).

Node 1 as-found is a **single** 477 GB SATA SSD carrying everything:

```
sda  477G  C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
├─sda1 976M  /boot/efi
├─sda2  30G  /
├─sda3  12G  /var
├─sda4 977M  swap
└─sda5 433G  /srv          <- the only large free space
```

### THE DECISION (operator, 2026-09-03; layout REVISED 2026-09-04) — made, applied, and measured-not-assumed

| | |
|---|---|
| **Data partition** | the **existing `/srv` partition** on each node's single physical disk (~432 GB nodes 1–2, ~422 GB node 3), re-purposed, re-labelled **`k3s-data`** and mounted at the neutral **`/data`** |
| **Longhorn store** | **`/data/longhorn`** — a *subdirectory* of that partition |
| **k3s `--data-dir`** | **`/data/k3s`** — the *sibling* subdirectory. Holds **embedded etcd AND the containerd image store** |
| **etcd** | **no dedicated device** — it lives at `/data/k3s/server/db`, i.e. on the data partition, **not** on the OS filesystem |
| **Shared-device I/O contention** | **ACCEPTED, knowingly** — unchanged by the 2026-09-04 revision |

#### THE 2026-09-04 REVISION, AND THE CAPACITY EVIDENCE THAT FORCED IT

The 2026-09-03 ruling gave this partition to Longhorn **only** and left k3s at its default
`/var/lib/rancher/k3s`. That was wrong, for a reason that is arithmetic rather than judgement.

**`/var` on these nodes is a separate partition of 11.8 GiB** (11.6 on node 3) — see the
partition map above; it is `/dev/sda3` / `/dev/nvme0n1p3`, not the 30 GiB `/`. k3s's data dir
puts **two** things on it: etcd's database and WAL (`server/db`) and, far larger, the
**containerd image store** (`agent/containerd`).

**Measured on the fully-deployed dev cluster** — the same platform and app set prod will run:

| | |
|---|---|
| `agent/containerd` per node | **21–25 GB** |
| total `/var` per node | **58–70 GB** |
| prod `/var` capacity | **11.8 GiB** |

So prod would have **filled `/var` during its first image pulls** — not eventually, not under
load, but on the way up. And because etcd was on that same partition, a full `/var` means
**etcd cannot write its WAL**. That is cluster death, not degradation: a wedged etcd on all
three servers is an unavailable control plane, and it would have happened during bring-up.

**The fix is to move k3s's whole data dir onto the 424 GB partition**, which is why that
partition's mountpoint and filesystem label are no longer Longhorn-specific — it now serves
**two tenants**. `/var` retains only the OS's own usage (503 MB, 5% used, measured 2026-09-04).

**Why now.** Nothing was installed on prod when this was decided — verified 2026-09-04: no
`/usr/local/bin/k3s`, no `/etc/rancher/k3s`, no `/var/lib/rancher` on any node. Moving a data
dir before first install is a one-line inventory change; after bring-up it means draining and
re-imaging three servers.

**Why `/data` and `LABEL=k3s-data`.**

- **`/data`** — short, neutral, tenant-agnostic, and *not* under `/var`. Deliberately **not
  `/srv`**: `/srv` is the *claim mountpoint* the role discovers the device at, and the
  idempotency contract is that `/srv` is gone from `/proc/mounts` and `/etc/fstab` once the
  claim has happened. Re-using it as the destination would make "claimed" and "not yet
  claimed" indistinguishable.
- **`LABEL=k3s-data`** — deliberately **not** the bare `data`. Guard #4 of the role's assert
  wall accepts a partition whose label is empty *or already ours*, and names `data` as an
  example of somebody else's named volume; labelling ours `data` would make a stranger's
  `data` volume look already-claimed and weaken that guard. `k3s-data` is cluster-scoped
  rather than tenant-scoped, and matches the `k3s-etcd` label convention already in the repo.

**The shared-device I/O risk is UNCHANGED and still accepted.** Longhorn and etcd were already
going to share this spindle under the 2026-09-03 ruling — the revision changes *which*
filesystem etcd sits on, not *which device*. The acceptance rationale and its exit condition
below stand verbatim.

**How prod's data dir is scoped away from dev.** `k3s_data_dir` is a **single** variable read
by **two** roles: `roles/k3s_server` turns it into `--data-dir` in the install flags, and
`roles/node_baseline` (via `node_baseline_k3s_data_dir`, which defers to it) creates the
directory `0700` on the data partition and derives etcd's location from it
(`node_baseline_etcd_mount` = `<data-dir>/server/db`). The role default is **k3s's own
default**, `/var/lib/rancher/k3s`, so **dev is unchanged** — and dev also has a *dedicated
etcd device* mounted at `server/db` under that path, which moving dev's data dir would orphan.
Prod overrides it in one line in `inventory/group_vars/k3s_prod/vars.yml`, the same seam
`kube_vip_address` / `k3s_api_host` / `node_baseline_backports_kernel` use, so a fourth prod
node inherits it from group membership rather than from anyone remembering.

**Longhorn's `defaultDataPath` follows the R-ENV-02 knob pattern, not a base edit.**
`platform/longhorn/helmrelease.yaml` referenced a hardcoded `/var/lib/longhorn`; it now
references `${LONGHORN_DATA_PATH}`, a `cluster-vars` knob with a base default and an explicit
value in **both** overlays:

| | |
|---|---|
| `clusters/base/cluster-vars.yaml` | `/var/lib/longhorn` (base default = the known-good dev value, same convention as `GATEWAY_VIP`) |
| `clusters/dev` | `/var/lib/longhorn` — **must never change**; the live dev cluster has 18 volumes whose replicas are on disk under that path |
| `clusters/prod` | `/data/longhorn` |

`kustomize-controller` runs with `--feature-gates=StrictPostBuildSubstitutions=true`, so an
undefined variable fails the Kustomization outright — hence base-first, then both overlays.
`[parity]`: the prod value MUST equal `node_baseline_longhorn_data_path` in
`group_vars/k3s_prod/vars.yml`. Ansible creates and mounts the directory; Helm tells Longhorn
to use it. If the two disagree, Longhorn writes replicas onto the OS filesystem and `/var`
fills up — the exact failure this revision exists to prevent.

**Verified 2026-09-04 that dev's rendered output is byte-identical.** All 12 Flux
Kustomizations that declare `postBuild.substituteFrom cluster-vars` were rendered
(`kubectl kustomize <spec.path> | flux envsubst --strict`, substituting the dev overlay's
*effective* cluster-vars) both before and after the change: 12/12 identical per-file SHA-256,
and one aggregate SHA-256 over every rendered byte, `b34c5d3d…89e7d`, unchanged. The only
dev-side difference anywhere is the presence of the new knob in the ConfigMap itself, carrying
dev's existing literal. The same harness shows prod's render changing on exactly one line
(`defaultDataPath: /var/lib/longhorn` → `/data/longhorn`), which is what proves the comparison
could have detected a regression.

**Per-node device paths — they are NOT the same, and that is why nothing hardcodes them.**
Verified 2026-09-03 with `findmnt --mountpoint /srv` on each node:

| Node | Boot media | `/srv` partition | Size | Contents before the claim |
|---|---|---|---|---|
| `k3s-prod-1` | SATA SSD `C400-MTFDDAC512M` | `/dev/sda5` | 432.5 GiB ext4 | `lost+found` only (`du -sh` 20K; `df` 2.1 M of 425 G) |
| `k3s-prod-2` | **NVMe** `PM9A1 Samsung` | **`/dev/nvme0n1p5`** | 432.5 GiB ext4 | `lost+found` only (same) |
| `k3s-prod-3` | SATA SSD `WDC WDS500G2B0A` | `/dev/sda5` | 422.2 GiB ext4 | `lost+found` only (same) |

The `df` "5.1% used" that made `/srv` look occupied is **ext4's default 5% root-reserved
blocks**, not data — confirmed per node by `ls -A /srv` (only `lost+found`) and
`du -sh --exclude=lost+found /srv` (20 K). Nothing was destroyed.

**How the role resolves the device — discovery, not per-host inventory.** A single
`k3s_prod` group value cannot be right when node 2 is NVMe, and per-host `host_vars` would
freeze a *fact about today's hardware* into inventory, where it goes stale the first time a
node is re-imaged or a disk is swapped — and the stale value's failure mode is `mkfs` on the
wrong partition. So `group_vars/k3s_prod/vars.yml` declares **intent**
(`node_baseline_longhorn_source: claim-mountpoint`,
`node_baseline_longhorn_claim_mountpoint: /srv`) and
`roles/node_baseline/tasks/disks.yml` asks the **node** which block device is mounted there,
then proves the answer is safe before writing anything. Six independent asserts, all
fail-closed: it is a partition or whole disk (not tmpfs/NFS/bind), it is **not** backing
`/`, `/boot`, `/boot/efi`, `/usr`, `/var`, `/home`, `/tmp` or active swap (that set is built
from the node's *own* gathered facts plus `swapon`, never a hardcoded partition list), it is
already ext4, it carries no other filesystem label, it is ≥ 100 GiB, and it is **empty**.

**Unmount → relabel, not reformat.** `/srv` is already formatted and already mounted, and
`community.general.filesystem` will not touch a device that has a filesystem — it would
silently no-op and then the `LABEL=longhorn` mount would fail. The role therefore unmounts
`/srv`, removes its `/etc/fstab` entry (installer-written, pinned by UUID — leaving it would
re-mount the partition at `/srv` underneath Longhorn on the next boot), and stamps the label
with `e2label`. Relabelling rather than reformatting is deliberate: the partition is already
ext4 and already verified empty, so `mkfs` buys nothing, and if some assert were ever too
permissive the worst case is a renamed filesystem instead of a destroyed one.

**Idempotency guard.** A second run is a no-op, not a reformat. After the claim the
partition carries the ext4 label `k3s-data` and `/srv` is gone from `/proc/mounts` *and*
`/etc/fstab` — so "what is mounted at `/srv`" would find nothing on run 2. The role pivots
on the durable on-disk evidence instead: `blkid -L k3s-data` returning a device means the
node is already claimed, and **every** destructive task is gated on that returning nothing.

**The 2026-09-04 transition, handled idempotently.** All three nodes had *already* been
claimed under the previous naming (`LABEL=longhorn` at `/var/lib/longhorn`). Because the
idempotency pivot asks "does the **current** label exist?", changing the label makes an
already-claimed node answer *no* — and it would then walk into the greenfield claim path and
fail closed at "the claim mountpoint must actually be a mountpoint", `/srv` having been
consumed by that first claim. Fail-closed is correct, but it is not convergence, so the
transition is **declared** rather than inferred:
`node_baseline_data_retired_fs_label: longhorn` + `node_baseline_data_retired_mount:
/var/lib/longhorn` in `group_vars/k3s_prod/vars.yml`. The role then unmounts the retired
mountpoint, drops its `fstab` entry and re-stamps the label with `e2label` — a **metadata
write, never a reformat**, exactly as in the claim path — having first proved that the retired
mountpoint is still a mount, that it is backed by the very device the retired label names, and
that it is **empty** so nothing is orphaned. The common `LABEL=`-based mount task then brings
it up at `/data`, and the two per-tenant subdirectories are created (`0755` Longhorn, `0700`
k3s, since the latter holds the etcd DB, the node token and the image store).

The transition block is **one-time and self-retiring**: on every subsequent run
`blkid -L k3s-data` succeeds and none of it is reachable. The two retired-* lines are safe to
delete from `group_vars` once the fleet is converged (it is, as of 2026-09-04).

**Three independent reasons the LIVE dev cluster cannot enter that transition** — it has 18
Longhorn volumes on `/var/lib/longhorn` and unmounting it would orphan every replica:
(1) both retired-* vars default to `""` and are set only in the prod-scoped `group_vars`;
(2) the path is gated on `longhorn_source == 'claim-mountpoint'`, and dev is `device` — the
same structural gate that keeps dev out of the claim path; (3) the emptiness assert, which
fails closed on a partition with replica directories even if (1) and (2) were both defeated.
Dev's mountpoint, label and k3s data dir are also simply **not changing**.

**Run evidence (2026-09-04).** Full `node-baseline.yml` against `k3s_prod`: run 1 converged
(5–6 changed per node), run 2 **`changed=0` on all three nodes** with every transition and
claim task skipping. Per node afterwards: `blkid -L k3s-data` → `/dev/sda5` (nodes 1, 3) /
`/dev/nvme0n1p5` (node 2); `blkid -L longhorn` → gone; `/data` mounted with 403 G / 394 G
available; `/var/lib/longhorn` and `/srv` no longer mountpoints; `fstab` carrying exactly
`LABEL=k3s-data /data ext4 defaults,nofail,x-systemd.device-timeout=5s 0 0`; and `/var` down
to **5% used, ~11 GiB free**.

**Dev can never reach this path.** Both role defaults stay `source: device`, so the LIVE
dev cluster keeps its dedicated `/dev/vdb` (Longhorn) and `/dev/vdc` (etcd) and the claim
tasks are structurally unreachable there — they are gated on `claim-mountpoint`, which only
the prod-scoped `group_vars` sets.

### The accepted risk, and its exit condition

Dev was burned by exactly this shape: a Longhorn rebuild starved etcd WAL fsync
(`docs/plans/ssd-etcd-isolation.md`), which is why dev has a third, dedicated etcd device.
**Prod accepts the same shared-device contention** because the failure mode is materially
cheaper here: dev's contention was on a **spinning HDD** with seek-bound queueing; every
prod node is **SSD or NVMe** end-to-end, where concurrent rebuild I/O costs latency rather
than collapsing it. The three nodes also have no second device to give etcd — one physical
disk each, already fully partitioned, no free slot verified populated.

**Accepted to be MEASURED, not assumed.** Exit condition, in order of preference:

- **Measure it before it matters.** Under a deliberate Longhorn replica rebuild, watch
  `etcd_disk_wal_fsync_duration_seconds` p99 and `etcd_disk_backend_commit_duration_seconds`
  p99, plus `etcd_server_leader_changes_seen_total`. If fsync p99 stays inside etcd's
  practical budget (~10 ms wal fsync / ~25 ms backend commit) with no leader churn, the
  acceptance holds and this becomes a recorded measurement rather than an assumption.
- **If it does not hold:** add an M.2 NVMe per node (chassis slot to be verified
  physically) and flip `node_baseline_etcd_source` back to `device` — the role still
  supports that path unchanged, and it is the dev shape.

**~~Known, separate consequence — `/var` is only 12 GiB.~~ RESOLVED 2026-09-04 by moving
k3s's data dir.** This was recorded here on 2026-09-03 as a capacity item "to watch or
repartition". Re-measuring it against dev turned it from a watch item into a blocker: the
container image store alone is **21–25 GB per node** on the fully-deployed dev cluster and
`/var` totals **58–70 GB**, against prod's **11.8 GiB**. So prod would have filled `/var`
during its first image pulls, and with etcd on the same partition a full `/var` means etcd
cannot write its WAL. The fix was neither "watch" nor "repartition" but `--data-dir` — see
"THE 2026-09-04 REVISION" above. `/var` now carries only the OS's own usage (503 MB, 5%).

Consequences to settle before imaging nodes 2 and 3:

- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
      floor — see §5). Raw attributes, so the numbers are auditable:

      | | |
      |---|---|
      | Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) |
      | SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` |
      | Power-on hours | 6,814 (~284 days) · 147 power cycles |
      | **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) |
      | **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) |
      | **Erase failures** | **187** (`172`) |
      | **Unexpected power losses** | **32** (`174`) |
      | Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) |
      | UDMA CRC errors | 0 (`199`) — cabling is clean |

      **Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector
      count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its
      normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**,
      not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value;
      the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent
      uncorrectable counters instead.

      **Verdict: usable, but not for production state.** Nothing says imminent failure — no
      pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011
      consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad
      blocks, logged **187** erase failures, and taken **32** unclean power losses **with no
      power-loss protection**, in a single failure domain. etcd is the worst possible tenant for
      it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how
      a member corrupts — this drive has already logged 32 of them.
      (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
      `DECISIONS-LOG.md`. It is not part of this verdict.)
- [x] **Layout DECIDED and APPLIED 2026-09-03** — see "THE DECISION" above. `roles/node_baseline`
      no longer expects a block-device path for either store; it takes a declared **source
      enum** per environment (`roles/node_baseline/tasks/disks.yml`):
      - **Longhorn** — `node_baseline_longhorn_source: claim-mountpoint` +
        `node_baseline_longhorn_claim_mountpoint: /srv`. The `/dev/vdb` default (a virtio
        name that does not exist on bare metal) is unused on prod; the real partition is
        discovered on the node and guarded, because it is `/dev/sda5` on nodes 1 and 3 and
        `/dev/nvme0n1p5` on node 2.
      - **etcd** — `node_baseline_etcd_source: shared-data-partition` (was `root-filesystem`
        until 2026-09-04). "No dedicated device" is a first-class, explicitly-configured state
        that the `disks` assert **accepts**, not a blank variable that the tasks happen to
        skip — and the *name* of that state now describes reality, because etcd is no longer
        on the OS filesystem. The distinction is load-bearing, not cosmetic:
        `shared-data-partition` is **asserted**, per node on every run, by checking that the
        filesystem backing the etcd data dir really is the data partition's mount. Declaring
        `k3s_data_dir: /data/k3s` does not by itself achieve that — had the mount silently
        failed, `/data` would be a plain directory on `/` and etcd plus 21–25 GB of images
        would land back on the OS filesystem, invisibly, until the disk filled. Leaving the
        value at `root-filesystem` would have been a lie inventory told about the cluster, and
        `root-filesystem` makes no such claim and gets no such assert. The role also still
        asserts nothing is already mounted at the etcd data dir (so a half-migrated node
        cannot be inherited silently) and reports which filesystem etcd actually lands on
        (measured 2026-09-04: `/data`, 403 G / 394 G available, vs `/var`'s 10.7 GiB).
      - The two rejected options stand rejected: **add a second drive** (no verified free slot;
        the exit condition above reopens it if measurement fails) and **repartition** (same
        single failure domain, plus a destructive re-image of three working nodes for no
        isolation the SSDs do not already give).
- [x] **All three nodes get the identical layout.** Same declaration, same mountpoint, same
      resulting `/data` + `/data/longhorn` + `/data/k3s`. Not the same *device path* and not
      the same *size* (432.5 / 432.5 / 422.2 GiB) — Longhorn tolerates unequal node capacity;
      it is replica *placement* that assumes a symmetric layout, and the layout is symmetric.
      Confirmed per node 2026-09-04 by `findmnt`/`lsblk`/`blkid` after the transition.
- [x] **Chosen paths recorded here and set in `group_vars`.** The BLOCKED placeholders in
      `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` are replaced — by the source
      enums above, plus deliberately-unusable `/dev/NOT-A-DEVICE-…` values for the two device
      variables so that flipping a source back to `device` without supplying a real path
      still fails on the role's assert instead of silently inheriting dev's `/dev/vdb`.
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
      is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
      tasks this decision blocks — see §5's drive-health box for the exact command.

## 3. OS install (per node)

- [ ] **Debian 13 (trixie)**, matching dev exactly — same kernel series, same package baseline.
- [ ] Hostname `k3s-prod-N`.
- [ ] Partition per the §2 decision.
- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
      `roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
      so the same role covers bare metal. Nothing to port.
- [x] **KERNEL: a standing, deliberate deviation from the distro baseline — `linux-image-amd64`
      from `trixie-backports` (DONE on all three nodes, 2026-09-04).**

      **What.** `roles/node_baseline` gained a `kernel`-tagged block that enables
      `trixie-backports` as a deb822 apt source and installs the `linux-image-amd64`
      **metapackage** from it. Applied 2026-09-04; all three nodes landed on
      **`linux-image-amd64` 7.1.8-1~bpo13+1 → kernel `7.1.8+deb13-amd64`**, rebooted one at a
      time, and are running it.

      ```bash
      ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
      ```

      **Why.** §0b. HP's R21 firmware publishes a corrupt EFI Memory Attributes Table; Linux
      < 6.13 reserves memory from it without validating the size, and `k3s-prod-2` has booted
      with 2.6 GiB of its 16 GB, twice. The fix is upstream commit **`8fbe4c49c0cc`**
      *"efi/memattr: Ignore table if the size is clearly bogus"* (Ard Biesheuvel, first
      released in **v6.13**, hardened by `5d0faa8e8369`). It is absent from v6.12 and from
      `linux-6.12.y`, carries neither `Cc: stable` nor `Fixes:`, and there is **no kernel
      cmdline workaround**. A k3s **server** that intermittently comes up with 2.6 GiB will
      fail under etcd plus its workload share on any reboot, so this is a **prerequisite to
      installing k3s**, not a follow-up.

      **How it is scoped.** `node_baseline_backports_kernel` defaults to **false** in
      `roles/node_baseline/defaults/main.yml` and is set **true** exactly once, in
      `inventory/group_vars/k3s_prod/vars.yml` — the same seam `kube_vip_address` /
      `k3s_api_host` / `cluster_kubeconfig_name` use. Dev's Proxmox VMs publish no MAT and are
      genuinely unaffected; they must not be dragged onto a backports kernel as a side effect
      of sharing the role, and a fourth prod node must inherit the deviation from **group
      membership**, not from a `k3s-prod-*` hostname test. Verified: a `--tags kernel` run
      against dev skips all 8 tasks on all 3 hosts.

      **Security cost, accepted.** `*-backports` packages are provided **as-is** and are
      **not** covered by the Debian Security Team's guarantees for the main archive — no
      security-tracker commitment, no DSAs. Recorded here as the price of the nodes keeping
      their RAM.

      **No apt pin, deliberately.** The backports `Release` file carries `NotAutomatic: yes`
      + `ButAutomaticUpgrades: yes`, which apt already reads as *"priority 100 so nothing is
      taken from backports by accident, 500 for a package explicitly installed from there so
      it keeps getting updates."* Writing an explicit `Pin-Priority: 100` in
      `preferences.d` would look like belt-and-braces and would in fact **switch off the
      automatic security upgrades for the very kernel this deviation installs.** Enabling the
      source is the whole configuration.

      **Metapackage, not a pinned ABI.** A frozen `linux-image-6.19.14+deb13-amd64` would stop
      receiving even backports' own updates the moment the ABI rolls. The metapackage tracks
      whatever backports ships (>= 6.13 by construction). The **exact** version each node
      landed on is recorded on the node itself at `/etc/node-baseline-backports-kernel`,
      written by the same role, along with the rollback path below.

      **Rollback.**
      1. *Non-destructive, try first.* The stock kernels are still installed
         (`6.12.107+deb13-amd64` and `6.12.94+deb13-amd64` on all three) and selectable from
         GRUB → *Advanced options for Debian GNU/Linux*. That answers "is the new kernel the
         problem?" without uninstalling anything — it also reinstates the memattr bug for that
         boot.
      2. *Permanent.* Set `node_baseline_backports_kernel: false` in
         `group_vars/k3s_prod/vars.yml` **first** (otherwise the next Ansible run puts it
         back), then on each node:
         ```bash
         apt-get install --reinstall linux-image-amd64/trixie
         apt-get purge linux-image-7.1.8+deb13-amd64
         rm /etc/apt/sources.list.d/trixie-backports.sources
         apt-get update && update-grub && reboot
         ```

      **The role does NOT reboot**, and that is deliberate — see §0b "SECOND FAULT". Reboots
      are a serialized, supervised operator step: one node at a time, verify each returns
      before touching the next, never node 2 first. On 2026-09-04 all three returned in
      **39–41 s** (node 1 39 s, node 3 41 s, node 2 39 s); node 2 did **not** reproduce its
      2026-09-03 hang, but one clean warm reboot is not a fix and that defect stays open.

      **Exit condition — how this deviation ends.** If `8fbe4c49c0cc` is ever picked up into
      `linux-6.12.y` (and therefore into Debian's 6.12 rebase, which reaches us by plain
      `apt upgrade`), this can be removed entirely: set `node_baseline_backports_kernel:
      false`, roll back per the note on each node, and delete the `kernel` block from
      `roles/node_baseline`. §0b records why that pickup will not happen on its own and what
      request would cause it.
- [x] **Verification of the fix, and a correction to how it is verified (2026-09-04).**

      §0b says *"A fixed kernel prints `FW_BUG Corrupted EFI Memory Attributes Table
      detected!`"*. That is true but **conditional, and it did not happen on any of the three
      nodes.** `efi_memattr_init()` on >= 6.13 has **three** gates and only the third prints
      that line:

      | gate | message | reachable when |
      |---|---|---|
      | `tbl->version > 2` | `Unexpected EFI Memory Attributes table version %d` | the garbage lands in the version field |
      | implausible `desc_size` | `Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)` | version reads `0..2`, `desc_size` does not |
      | `num_entries > SZ_64K` | `[Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)` | version *and* `desc_size` read plausibly, entry count does not |

      **All three are rejections**, and which one a boot prints is decided by *which field*
      that boot's garbage corrupts. On 7.1.8 all three nodes hit the **first** gate:

      | node | `MEMATTR=` | kernel line on 7.1.8 | MemTotal | reserved at boot |
      |---|---|---|---|---|
      | 1 | `0xbee84018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1030` | 31.1 GiB | 0.9 GiB |
      | 2 | `0xbeeb0018` | `efi: memattr: Unexpected EFI Memory Attributes table version -1569980416` | 15.4 GiB | 0.6 GiB |
      | 3 | `0xbeecc018` | `efi: memattr: Unexpected EFI Memory Attributes table version 1463964749` | 31.1 GiB | 0.9 GiB |

      Node 2 is the result that matters: **`0xbeeb0018` is the exact address that produced
      2.6 GiB twice**, and on 7.1.8 it reserved 634,904 K instead of 13,913,856 K.

      **Boot-independent proof the fix is compiled in**, since the log line is a lottery: the
      running kernel image itself contains the string. Decompressing `/boot/vmlinuz-$(uname
      -r)` on all three nodes yields, verbatim —

      ```
      efi: memattr: [Firmware Bug]: Corrupted EFI Memory Attributes Table detected! (version == %u, desc_size == %u, num_entries == %u)
      efi: memattr: Unexpected EFI Memory Attributes descriptor size %u (expected: %lu)
      ```

      — neither of which exists in 6.12. That is the positive proof; the `dmesg` line is not
      available on demand.

      **Two bugs in `roles/node_health` were fixed as a consequence**, because without them
      the health check could not show the fix landing:
      - `MEMATTR_VERSION_RE` in `files/node_health_collect.py` matched `(\d+)` only. The
        kernel prints that field with `%d` — **signed** — so node 2's `-1569980416` did not
        match, the state fell through to `clean`, and the node was reported **PASS "the kernel
        accepted it without complaint"**: a false pass on the exact bug the check exists to
        find. Now `(-?\d+)`.
      - `firmware.efi_memattr` treated `bad_version` as WARN unconditionally, so a node on a
        fixed kernel would have stayed WARN forever. The verdict now keys on the running
        kernel version (`memattr_kernel_fixed`, `>= 6.13`): the same log line is WARN on
        < 6.13 (a lucky boot) and PASS on >= 6.13 (the validation working). The pre-existing
        `corrupt_rejected` PASS branch is unchanged and remains reachable.

      **`playbooks/node-health.yml -e node_health_hosts=k3s_prod`, after the reboots:
      `firmware.efi_memattr` = PASS on all three, `memory.physical_total` = PASS on all
      three.** Dev re-run unchanged.

## 4. Network (per node)

- [ ] **Static IP** on the prod LAN, recorded in §0. Node 1 is `192.168.2.10/24`, gateway
      `192.168.2.1`.
- [x] **Prod address plan — DECIDED (operator, 2026-08-31).**

      | Role | Address | Owned by |
      |---|---|---|
      | `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory |
      | **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) |
      | **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) |
      | **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) |

      Two deliberate properties, both mirroring dev:
      - The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and
        Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way
        round (`.49` control-plane, pool `.53-.60`).
      - **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a
        defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool
        was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape
        (`GATEWAY_VIP` `.53` = first of `.53-.60`).
- [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before
      the build, and confirm nothing else on the network already answers on them.
- [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.**
      Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single
      upstream is a known SPOF that has already bitten this project — it broke containerd image
      pulls and CoreDNS forwarding on dev.

      The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in
      `roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes
      `/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed
      `/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager
      owns it), and configures the ordered upstream list either way. The earlier
      `fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan
      drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie
      nodes, so it would have broken `node-baseline.yml` on every host.

      What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets
      `node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid
      `BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert
      stops the play on it before writing anything. Choosing the second upstream is a **network
      decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would
      resolve reliably but would also bypass any local zone and any filtering the gateway applies.
      Decide between — and record the choice HERE, in the same change that edits the group_vars:
      - a second on-LAN resolver on `192.168.2.x`;
      - the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified —
        prod sits on the workstation side of the pfSense boundary);
      - an explicit, written-down acceptance of a public resolver as fallback-only.

      Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`)
      are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed
      servers, so keeping them means listing them explicitly.
      `node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's
      local domain is decided.
- [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT —
      prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind.

## 5. Access (per node) — **blocks all Ansible**

- [ ] **`k3s` user with the `home_network` SSH key.** Working on node 1.
- [ ] **Password-protected sudo, driven by Ansible `become` + a vaulted password.**
      **Prod does NOT use NOPASSWD sudo** (operator decision, 2026-08-31). Dev nodes do; prod
      deliberately keeps the sudo password in place and supplies it to Ansible as an
      **ansible-vault-encrypted** var, exactly like the existing `hypervisor` and `proxmox_hosts`
      groups:
      ```
      inventory/group_vars/k3s_prod/
        README.md     how to (re)create the vault — committed
        vault.yml     ansible_become_password: "..."  — committed, ENCRYPTED
      ```
      The passphrase that decrypts it lives **outside the repo** at
      `~/.config/home-k3s/ansible-vault-pass` (`ansible.cfg :: vault_password_file`), so plays still
      run unattended without a plaintext secret ever touching git.
      ```bash
      ansible-vault create inventory/group_vars/k3s_prod/vault.yml   # ansible_become_password: ...
      ```
      Prod nodes must therefore set `ansible_become: true` in their group_vars and must **not**
      inherit any passwordless-sudo assumption from `group_vars/all.yml`.
- [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is
      the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling,
      2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part
      of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline
      would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead
      (the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so
      the placeholder device paths are never even templated):
      ```bash
      cd infra/ansible
      ansible-playbook playbooks/node-baseline.yml \
        -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
      ```
      (Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading
      the drive is then:
      ```bash
      sudo smartctl -a /dev/sda   # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors
      ```
      Record the result in §2 and close the layout decision in the same change that replaces the
      two `node_baseline_*_device` placeholders.

## 6. Ansible baseline & cluster join

Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars
differ, plus a target-group override on the command line.

### How prod is targeted (done, 2026-08-31)

Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` /
`k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately
kept separate because Ansible resolves them at different times:

| Layer | Knob | Set via |
|---|---|---|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there |
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** |

So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and
tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with
no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after
`--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**.

- [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
      **directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping.
      (Nodes 2/3 stay commented out until they are physically on the bench.)
- [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`),
      `cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the
      §2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which
      are still BLOCKED placeholders.
- [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until
      it exists **no** prod play can escalate, and `--check` against prod cannot work either.
- [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read:
      ```bash
      ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
      ```
- [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel
      modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it
      asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare
      metal — the role already only *starts* it when the virtio channel exists, so it degrades
      cleanly.)
- [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`
      → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three
      nodes are present** — see §8.
- [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the
      prod LAN.
- [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts
      it afterwards. The bootstrap values and the HelmRelease **must stay in parity**
      (`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config
      (`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the
      dev one the dev container exports as `$KUBECONFIG`.
- [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`.
- [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**:
      `roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps
      `--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a
      target-group change. Tracked with the other repo blockers in §7.
- [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015).
      `playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=<group>`, but there is
      deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted
      PIA credentials — the existing vault is scoped to the dev group.

## 7. Repo blockers — **prod cannot be reconciled until these land**

`clusters/prod` is design-only today and **would build wrong**. R-ENV-02 promises dev and prod differ
*only* by `cluster-vars` values; in practice dev LAN addresses are hardcoded into base manifests:

| Hardcoded value | Where | Prod needs |
|---|---|---|
| `k8sServiceHost: 192.168.1.50` | `platform/cilium/helmrelease.yaml:50`, mirrored in `values.yaml:16` | prod primary node IP |
| LB pool `192.168.1.53–.60` | `platform-config/cilium/l2-ip-pool.yaml:15` | a prod-LAN pool |
| `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
| `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
| `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real. Prod's value was inverted the same day to the APEX `home.keithjackson.dev`, with dev moving to `dev.home.keithjackson.dev` — see §10a, which is authored but NOT yet applied. |

- [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
      overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
      header comment in `clusters/base/cluster-vars.yaml`). `platform`, `platform-config`, the app
      waves, `apps-vms` and `crafty` all already carry `postBuild.substituteFrom: cluster-vars`, so
      the seam exists.
- [ ] **`clusters/prod` must set `GATEWAY_VIP`.** It currently doesn't, so prod would inherit the
      base default `10.73.0.240` — an address from the **retired libvirt network**. The base file's
      own comment flags this as owed "when hardware is built."
- [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is
      byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`.
- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing
      the `dependsOn` fix. Only the first is now closed:
      - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became
            *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries
            `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block
            forever on a dependency that does not exist and nothing downstream would reconcile.
      - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key
            as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev
            exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster
            (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only.
      - [~] **GitRepository source** — **MITIGATED 2026-09-04, NOT SOLVED.** Branch settled; the
            in-cluster URL is still unusable, so prod now points at a **temporary bootstrap source**.
            Operator decision 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev
            is temporary and will be decommissioned once prod is stable. Recorded consequence: once
            prod is live, **every merge to `main` reconciles BOTH clusters** — no promotion step, no
            soak window.

            **Why the in-cluster URL (`https://gitea.home.keithjackson.dev/...`) cannot work on a
            cold prod cluster** — two independent reasons, found 2026-09-01, both still true:
            1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
               decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
               loses its source of truth. **This is the decommission plan's first task, not its last.**
            2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire
               `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
               prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
               `gitea.home.keithjackson.dev` to itself and never reach a repository.

            **The mitigation now in `clusters/prod/kustomization.yaml`** (operator directed this to be
            stood up, 2026-09-04):

            | | |
            |---|---|
            | GitRepository url | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` |
            | Branch | `main` (unchanged — same branch as dev) |
            | What it is | Gitea 1.22.6 in a **podman container on the operator's workstation**, started by `bootstrap/gitea/bootstrap-gitea.sh` with `HOST_NAT_IP=192.168.2.132` |
            | Reachability | All three prod nodes reach it **directly on the LAN** — no NAT hop (the libvirt network is retired), **no firewall change needed**. Verified from all three nodes. |
            | Auth | **None, deliberately.** The repo is PUBLIC on that Gitea, so Flux needs no `secretRef`. Verified anonymously: `curl .../api/v1/repos/Home-Network/home-network-k3s-impl` → `"private":false`, and `git ls-remote` returns `refs/heads/main`. |
            | Transport | **Plain HTTP, no TLS.** Accepted for the bring-up window only: LAN-local, and the repo holds no plaintext secrets (all SOPS-encrypted). |
            | Admin creds | `~/.config/home-k3s-bootstrap-gitea-admin.txt`, mode 0600, host-only, never committed. |

            **THIS IS A SINGLE POINT OF FAILURE FOR PROD'S RECONCILIATION, and it is not even in a
            cluster.** It is one container on one workstation, with no HA, no backup of its own, and
            no restart guarantee beyond `--restart unless-stopped`. If that workstation is powered
            off, suspended, or the podman volume `gitea-bootstrap-data` is lost:
            - already-reconciled state **keeps running** — Flux does not tear anything down;
            - but **no new revision can land on prod**, and every `GitRepository` reconcile fails.
            So prod is *not* self-sufficient while this stands. Do not treat prod as production-ready
            on this footing, and do not let it become the steady state by inattention.

            **One bring-up footgun:** `bootstrap-gitea.sh` pushes the operator's **current `HEAD`** to
            the bootstrap Gitea's `main`, whatever branch that HEAD is on. Before bootstrapping prod,
            confirm that remote `main` is the revision prod should run
            (`git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` vs local
            `main`). As of 2026-09-04 the remote `main` is the tip of the working branch, **not** local
            `main`.

            **EXIT CONDITION (the cutover) — mirrors dev's own Phase-6 cutover, `docs/05`.**
            This item is not closable until all of it is done:
            - [ ] Gitea runs **on prod**, reconciled in-cluster, answering
                  `gitea.home.keithjackson.dev` through prod's Gateway VIP with a publicly-trusted cert.
            - [ ] This repo is pushed/mirrored into that in-cluster Gitea.
            - [ ] The `clusters/prod` GitRepository url is changed to
                  `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`
                  and the TEMPORARY comment block is **deleted** (dev's stanza is what "done" looks like).
            - [ ] `flux get sources git -n flux-system` on prod reconciles a revision from the new URL.
            - [ ] **Only then** stop the workstation container — kept stopped-not-deleted as recovery,
                  exactly as dev retained its own bootstrap Gitea.
      - [x] **`roles/flux_bootstrap` could only bootstrap dev** — FIXED 2026-09-04. It invoked
            `bootstrap/flux/bootstrap.sh` without ever exporting `FLUX_ENV_PATH`, so the script fell
            through to its `clusters/dev` default and §8 step 4 (`--path=clusters/prod`) was
            unreachable without editing the role. Now parameterised on the standard group_vars seam:
            role default `flux_bootstrap_env_path: clusters/dev` (defaults/main.yml) is exported as
            `FLUX_ENV_PATH` by the role's task, and
            `inventory/group_vars/k3s_prod/vars.yml` overrides it to `clusters/prod` — the same
            mechanism as `kube_vip_address` / `k3s_data_dir` / `cluster_kubeconfig_name`. Dev has no
            override anywhere, so dev resolves `clusters/dev` exactly as before.
- [x] **The Kyverno CRDs had to be applied BY HAND, and nothing in the repo did it** — AUTHORED
      2026-09-04 (`bootstrap/flux/preapply-crds.sh`), **not yet run against prod**. This is a hard
      **prerequisite of a reconcilable cluster**, not a nice-to-have: `platform/kyverno/admission/
      helmrelease.yaml` sets `crds: {install: false}` — deliberately, because the CRD bundle is
      **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured 2026-09-04 with `helm template
      kyverno kyverno --repo https://kyverno.github.io/kyverno/ --version 3.9.x --include-crds`), far
      past kubectl's 256 KiB `last-applied-configuration` annotation limit, so Helm's client-side apply
      fails `metadata.annotations: Too long` and **stalls** the HelmRelease. The consequence nobody
      had automated: with CRD install off, **nothing installs those CRDs at all**. On a cold cluster
      Kyverno's Deployments come up with no API types, `platform` never goes Ready, and all 9
      downstream Kustomizations stall behind it. Dev only ever had them because the command was once
      run in somebody's shell — the exact undocumented manual step this project exists to delete.
      - **The fix, in git:** `bootstrap/flux/bootstrap.sh` now runs `./preapply-crds.sh` as step
        **[2/4]** — after `flux install`, before `kubectl apply -k ${FLUX_ENV_PATH}`. It renders the
        chart, keeps only `kind: CustomResourceDefinition` documents, applies them
        `--server-side --force-conflicts` (no annotation, so no size limit), then waits for
        `condition=Established`. The chart name, version constraint, repo URL and namespace are
        **derived from that same HelmRelease/HelmRepository** — there is no second copy of the version
        to drift, and a Renovate bump of the one pinned line carries this step with it.
      - **No environment fork (R-ENV-02):** the script names no environment and takes no overlay
        input; it reads repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step, so
        `roles/flux_bootstrap` needed **no change** — it already calls `bootstrap.sh`, the single
        entrypoint.
      - **It cannot silently skip.** Three guards, and deliberately no opt-out flag: a derived
        coordinate that is empty/`null` aborts; a render that yields **0** CRDs aborts (an empty
        stream would otherwise `kubectl apply` cleanly and exit 0 — the silent-success failure mode
        that has bitten this repo repeatedly); and every applied CRD must reach `Established`.
      - **Toolchain:** needs `helm` and `yq` on PATH in addition to `flux`/`kubectl` — both are baked
        into `infra/devcontainer` (`HELM_VERSION=3.16.0`, `YQ_VERSION=4.44.6`), and `bootstrap.sh`
        now fails at step 0 if either is missing. It also needs **outbound network to
        `kyverno.github.io`** from wherever the play runs (the Ansible role delegates to localhost).
      - **PM to verify in the container** (the authoring agent ran it only with a stubbed `kubectl`,
        never against a cluster): pinned `helm` 3.16.0 renders the same 22 CRDs, and the run is a
        clean no-op the second time.
      - **Unblocking the already-bootstrapped prod cluster** does not need a full re-bootstrap:
        ```bash
        # inside infra/devcontainer, KUBECONFIG=.cache/prod.kubeconfig
        bash bootstrap/flux/preapply-crds.sh
        ```
- [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
      the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).

## 8. Build sequence (operator decision, 2026-08-31: **wait for all three nodes**)

Node 1 is baselined now; the cluster is **not** initialized until nodes 2 and 3 are on the bench.
This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a
single-server-etcd window, and avoids a later member-add dance.

1. Per-node §1–§5 on each of the three boxes.
2. §7 repo work lands on `main` with the gate green.
3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`.
4. `flux bootstrap ... --path=clusters/prod` — i.e. `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod`,
   which runs `bootstrap/flux/bootstrap.sh` with `FLUX_ENV_PATH=clusters/prod`. That script's step
   **[2/4]** pre-applies the 22 Kyverno CRDs server-side; **without it `platform` cannot go Ready**
   and the 9 tiers below it never reconcile (§7). It needs `helm` + `yq` on PATH and outbound network
   to the chart repo.
5. Conformance against prod (§9).

## 9. Exit criteria (Phase 7)

- [ ] Prod runs the same config as dev via the prod overlay, **no base edits** (R-ENV-02).
- [ ] Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01).
- [ ] Full **automated** prod DR drill (R-STATE-03) — `definition-of-done.md` requires this for prod
      specifically.
- [ ] Monitoring and alerts live on prod.
- [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
      Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10a
      "The L2 tests hardcode the apex".
- [ ] **Prod serves the apex DNS/TLS identity** and dev's records are provably untouched (§10/§10a):
      distinct `EXTERNALDNS_TXT_OWNER_ID` (`prod` vs `dev`), prod wildcard cert
      `*.home.keithjackson.dev`, dev migrated to `*.dev.home.keithjackson.dev` with its old apex
      records retired (§10a step D8), and prod's own Keycloak realm + per-service client secrets.

## 9a. PROD BRING-UP STATE (2026-09-04) — what came up, and the TWO blockers that need the operator

Prod's `platform` tier was unsuspended after the Kyverno CRDs were pre-applied (see §7). Result:
**8 of 9 HelmReleases Ready, 76/76 pods Running** — cert-manager, CNPG, Cilium, Longhorn,
kube-prometheus-stack, Loki, Alloy, external-dns. `platform` itself is **not** Ready, and the 10
downstream tiers remain blocked behind it. Two distinct problems, both needing an operator decision.

### The CRD pre-apply itself: WORKED, and is now proven against a live API server

22 CRDs applied server-side and all reached `Established`; a second run was a clean no-op (`rc=0`),
so the idempotency claim is proven, not asserted. Reproduced under the **pinned** toolchain
(helm 3.16.0, yq 4.44.6) matching an independent helm 3.20.0 run byte-for-byte: `3.9.x` → 3.9.0,
22 CRDs, 5,636,155 bytes.

**The CRDs were the real cause of a 4½-hour crash loop.** Kyverno's four Deployments had **55
restarts each** and stabilised within minutes of the CRD apply — its controllers were coming up with
no API types to watch. That is the failure mode §7 predicted, observed.

### BLOCKER 1 — Kyverno's Helm release is stuck in `uninstalling`

Diagnosis chain, from the HelmRelease conditions:

```
Released:   InstallFailed   — Helm install failed ...: context deadline exceeded
Remediated: UninstallFailed — Helm uninstall remediation ... timed out waiting for the condition
Ready:      StateError      — unable to determine state for release with status 'uninstalling'
```

The install timed out **because** the pods were crash-looping without CRDs. `install.remediation.
retries: 3` then triggered an uninstall, and that uninstall itself timed out — leaving
`sh.helm.release.v1.kyverno.v1` with `status=uninstalling`. helm-controller cannot proceed from that
state, so `platform` (which has `wait: true`) never goes Ready and all 10 tiers below it stay blocked.

**The Deployments are healthy and at the right chart version** (kyverno-3.9.0), so this is purely a
stale Helm bookkeeping record, not a broken workload.

**Remediation requires a destructive command and was deliberately NOT run unattended:**

```
helm uninstall kyverno -n kyverno --no-hooks --wait=false   # clear the stuck release
flux reconcile helmrelease kyverno -n kyverno --force        # let Flux install cold, CRDs now present
```

This is safe on prod specifically because prod carries no real workloads yet, and a clean cold
install is the stronger outcome: it *proves* the bootstrap path this repo now automates. Note the
`failurePolicy: Fail` webhooks (`kyverno-policy-*`, `kyverno-exception-*`, `kyverno-cleanup-*`,
`kyverno-global-context-*`) gate only **Kyverno's own** resource types, so a brief absence does not
block ordinary workload admission. The pre-applied CRDs live **outside** the Helm release and
survive the uninstall, which is exactly why the pre-apply had to be a separate step.

### BLOCKER 2 — prod's external-dns is an ARMED footgun: the installed release predates the fix

**This is the more dangerous of the two.** Prod's live external-dns Deployment renders:

```
--policy=sync  --registry=txt  --domain-filter=keithjackson.dev  --provider=cloudflare
```

**and NO `--txt-owner-id`.** Absent that flag external-dns falls back to the chart default `default`
— which is precisely the owner under which **all 36 of dev's TXT registry entries are recorded**.
Scaling this Deployment to 1 would make prod claim dev's records and, under `policy: sync`,
**delete them: the 2026-09-04 incident, exactly.**

The repo is CORRECT and is not the problem. `platform/externaldns/helmrelease.yaml` sets
`txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"`, prod's applied `cluster-vars` resolves it to `prod`, and
the **live HelmRelease `spec.values` carries `txtOwnerId: "prod"`**. Verified against the real chart:
`helm template external-dns --version 1.21.1` with `txtOwnerId: prod` renders `--txt-owner-id=prod`,
so the values key is right and an upgrade *would* fix it.

**The fault is Helm/Flux state, not authoring.** `helm get values external-dns` shows the INSTALLED
release carries `policy` and `domainFilters` but **no `txtOwnerId`** — it predates `c8c1d29`. Yet
helm-controller reports `Ready=True`, `observedGeneration=3`, and a history entry whose
`configDigest` **equals** `lastAttemptedConfigDigest`, i.e. **it believes it has already deployed the
current config.** So it will never upgrade on its own, and the stale release persists indefinitely.

**The only thing preventing the incident was `replicas=0` — manual drift from the incident recovery,
which any Helm upgrade would reset to the chart default of 1.** That is not a guard; it is luck.

Containment applied (reversible, and it costs nothing because prod publishes no hostnames yet — the
app tiers are suspended): **`flux suspend helmrelease external-dns -n external-dns`**, with
`replicas` left at 0. Flux can now neither upgrade the release nor reset the replica count.

**Verified remediation, in this order — the ordering is the whole point:**

1. Force the upgrade so the release picks up `txtOwnerId`, but **do not let the pod run before the
   flag is confirmed present.** The naive `resume` + `reconcile --force` resets `replicas` to 1 and
   starts the pod in the same operation, so there is no window in which to check — a race against a
   process whose failure mode is deleting dev's DNS.
2. Confirm the live Deployment args contain **`--txt-owner-id=prod`**.
3. Only then allow it to scale to 1, and immediately diff the Cloudflare zone against a
   pre-recorded snapshot.

**Dev was verified unharmed throughout this work:** the Cloudflare zone was snapshotted before
unsuspending `platform` and re-queried after — **57 records, byte-for-byte identical, all 36 TXT
entries still `owner=default`.** Snapshots are taken against the **Cloudflare API**, never a local
resolver (a pfSense override produced a false healthy reading during the incident).

## 10. DNS/TLS environment split — **the 2026-09-04 incident, and the target architecture**

> **Current state (2026-09-04):** the target below is **AUTHORED, NOT APPLIED.** Prod's
> `platform` tier and its 10 downstream tiers are SUSPENDED. Dev is **live on the apex**. The
> ordered cutover is **§10a**, and merging the authored change to `main` is step **D3** of it,
> not step one — dev self-reconciles `main`, so a merge outside the runbook window executes
> the migration unattended.

### What happened

Prod's `external-dns` **deleted every one of dev's public DNS records.**

Both clusters ran `policy: sync` against the **same Cloudflare zone**, served the **same
hostnames**, and — because neither overlay set `txtOwnerId` — both inherited the external-dns
chart's default owner id **`default`**. ExternalDNS decides what it may delete by reading its own
owner id back out of the TXT registry. Prod read dev's TXT entries, saw its own owner id, found no
local resource claiming those names, and garbage-collected the records behind them. Dev recovered
once prod was stopped. Prod's `platform` tier and everything downstream were then **suspended**, and
remain suspended, so it cannot recur before this change is reviewed.

`policy: sync` was not the bug and has not been changed. What was missing was **identity**.

### The second bug, queued directly behind it

`platform-config` would have requested a `*.home.keithjackson.dev` wildcard `Certificate` from
**letsencrypt-prod** — for a cert **dev already holds** — and both clusters would then have raced
each other's `_acme-challenge` TXT records on that one zone. Neither would reliably solve DNS-01.
This never fired only because the tier was suspended first.

### Root cause, stated precisely

Not external-dns. **Every hostname in the repo was a hardcoded literal.** `BASE_DOMAIN` existed in
`clusters/base/cluster-vars.yaml` but was referenced by **no reconciled manifest** — the repo-derived
status report flagged it as a dead knob. R-ENV-02 promises dev and prod differ only by `cluster-vars`
values; for DNS and TLS that promise was false, so `clusters/prod` was *authored* to serve dev's
public identity. Two clusters that describe the same DNS name will fight over it no matter how the
DNS controller is configured.

### The architecture decision (operator) — **INVERTED 2026-09-04, this is the current target**

```
dev   ->  *.dev.home.keithjackson.dev      gitea.dev.home.keithjackson.dev
prod  ->  *.home.keithjackson.dev          gitea.home.keithjackson.dev     (THE APEX)
```

**Prod *is* the apex. There is no `prod.` label and no CNAME layer at all** —
`<svc>.home.keithjackson.dev` is prod's real name, not an alias for one. This **supersedes** the
interim split committed earlier the same day (`c8c1d29`), which put prod on
`*.prod.home.keithjackson.dev` behind apex CNAMEs pointing at it. That layer is deleted: fewer
moving parts, no CNAME chain to keep in step, and prod ends up on the names users already type.

The cost of the inversion is that **dev must move**, and dev is the live cluster. The interim split
was chosen precisely because it moved only the unbuilt environment; this one does not have that
property, so the whole of §10's remaining risk now sits on dev's side, and it is sequenced below.

### The values, per environment

| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` | Ansible `k3s_base_domain` |
|---|---|---|---|---|
| base default | `dev.home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` | `dev.home.keithjackson.dev` (role default) |
| **dev** | `dev.home.keithjackson.dev` | `dev` | `keithjackson.dev` | (inherits the role default) |
| **prod** | `home.keithjackson.dev` | `prod` | `keithjackson.dev` | `home.keithjackson.dev` (`group_vars/k3s_prod`) |

`[parity]`: `k3s_base_domain` must equal the same cluster's `BASE_DOMAIN`. The kube-apiserver's
`oidc-issuer-url` is a k3s **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux
post-build substitution can never reach it — hence one fact in two places. If they disagree the
apiserver validates tokens against a different Keycloak than the cluster runs: kubectl SSO fails
while the cert-based admin kubeconfig keeps working, so it presents as "SSO is broken", not as skew.

**`EXTERNALDNS_DOMAIN_FILTER` is the Cloudflare ZONE for BOTH, and must stay that way.** FINDING
F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for **zone selection** as
well as record matching, so a filter that is a strict subdomain of the hosted zone matches **no
zone** — external-dns generates the record and then skips it with *"no hosted zone matching record
DNS Name"*, publishing **nothing**. This has been "hardened" and reverted more than once. It is not
defence in depth; it is a total outage of DNS publication. Isolation comes from the two guards below,
which a domain filter cannot provide anyway: both environments live in the one zone regardless.

**The two guards that actually work:**

1. **`txtOwnerId`, per-environment identity** — a cluster only ever reconciles records its own owner
   id claims in the TXT registry. This is the direct fix for the incident.
2. **Disjoint hostname sets** — after the migration the two clusters never describe the same record,
   so there is nothing to fight over. *This guard does not exist during the migration window*, which
   is the entire reason the ordering below is mandatory.

#### Why dev's owner id changed from `default` to `dev`

Until this migration dev **had** to render `default`: its live apex records were already registered
in Cloudflare's TXT registry under that owner, and renaming would have orphaned every one of them.
**The migration dissolves that constraint** — on `*.dev.home.keithjackson.dev` dev publishes an
entirely new record set with no prior registry entries to recognise, so the id is free and is chosen
on its merits. `default` has none: it is the value external-dns uses when you **forget** the flag,
which is exactly how both clusters came to share it, and an environment whose identity is the
forget-value can never be told apart from a misconfigured newcomer. Consequences, both deliberate:

- Dev's **old apex records stay owned by `default`**, which no environment now claims, so nothing
  deletes them implicitly on some future reconcile. Retiring them is an explicit step (**D7**) and
  prod's first publish is gated on that retirement being *confirmed* — not on two reconcile loops
  racing each other.
- **Prod stays `prod` and must never be given `default`.** Prod is taking over the apex *names*, and
  dev's existing apex records are registered under `default` — so `default` is the one value that
  would let prod adopt them, and under `policy: sync` delete the ones no prod source claims, on a
  cluster dev is still serving. That is the 2026-09-04 incident re-armed with the roles swapped.
  Prod inherits the apex **names**; it must never inherit the apex **owner id**.

All three values differ, and the base default remains a sentinel so an environment that forgets the
knob owns nothing and can therefore delete nothing.

### PHASE 1 — prod tokenized and moved off dev's names (`c8c1d29`, done; suffix since inverted)

`BASE_DOMAIN` was a **dead knob** — declared in `clusters/base/cluster-vars.yaml` and referenced by
no reconciled manifest, because every hostname was a hardcoded literal. `c8c1d29` made it real (141
references across the reconciled set) and gave prod its own suffix. That work stands; only the
*value* prod holds has changed, from `prod.home.keithjackson.dev` to the apex.

### What could NOT be tokenized, and why

- **`OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` in `platform/headlamp/headlamp-oidc-credentials.sops.yaml`**
  — SOPS-encrypted (`encrypted_regex: ^(data|stringData)$`), so the two hostnames are ciphertext.
  Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and
  **cannot be verified by the offline render harness** (`kustomize` does not decrypt), so a mistake
  would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately.
  **Both environments now need action:** dev's copy must move to `keycloak.dev.home.keithjackson.dev`
  as part of step **D5**, and prod needs its own copy with `keycloak.home.keithjackson.dev` and a
  client secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client
  secret is realm-scoped.
- **`oidc-issuer-url` in the kube-apiserver flags** (`roles/k3s_server/tasks/main.yml`) — a k3s
  server flag, not a manifest. Handled on the **Ansible** seam instead: role default
  `k3s_base_domain` (dev's value) + derived `k3s_oidc_issuer_url`, overridden for prod in
  `inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`,
  so a fourth prod server inherits it from group membership. Verified per host by a read-only probe
  (`connection: local`, zero SSH, `changed=0`): all three dev servers resolve
  `https://keycloak.dev.home.keithjackson.dev/realms/homelab`, all three prod servers resolve
  `https://keycloak.home.keithjackson.dev/realms/homelab`.
- **`spec.url` of dev's `GitRepository`** (`clusters/dev/kustomization.yaml`) — see the strand below.
  It *cannot* be tokenized: the `flux-system` root Kustomization that renders `./clusters/dev`
  declares no `postBuild.substituteFrom`, so `gitea.${BASE_DOMAIN}` there would ship the **literal
  token** and break the source outright (the L0 `flux-postbuild-substitution` check asserts exactly
  this). It is a hand-maintained literal that must move in lockstep with `BASE_DOMAIN`.

---

## 10a. THE MIGRATION RUNBOOK — ordering is load-bearing

> **Prod's `platform` tier and its 10 downstream tiers are SUSPENDED and must stay suspended until
> step P1.** Nothing below is safe to run out of order, and nothing below should be run while the
> operator is away.

### The two ways this goes wrong

**(1) BOTH CLUSTERS ON THE APEX — the 2026-09-04 incident, recreated.** If prod publishes
`*.home.keithjackson.dev` before dev has vacated it, two clusters describe the same records in the
same zone again. Distinct `txtOwnerId`s stop prod from *deleting* dev's records — that guard holds —
but they do not stop the second failure mode: both clusters requesting a `letsencrypt-prod` wildcard
for the **same name set** and racing each other's `_acme-challenge` TXT records, so neither reliably
solves DNS-01. **Rule: dev vacates the apex first, and the vacate is verified against Cloudflare
before prod is unsuspended.**

**(2) THE STRAND — dev's GitOps source is the thing that moves.** `spec.url` of dev's
`GitRepository` is `https://gitea.home.keithjackson.dev/...`, which is **both** dev's GitOps source
**and** the PM's push target. Flipping dev's `BASE_DOMAIN` moves Gitea's HTTPRoute, its public DNS
record and the wildcard cert covering it to `gitea.dev.home.keithjackson.dev` all at once. If that
host does not already resolve and serve a valid, Ready cert at that instant, **dev stops reconciling
and cannot be fixed by pushing** — the broken thing *is* the path Flux uses to read pushes. Recovery
is then hands-on and needs LAN access to the cluster.

**(2b) AND THE MERGE ITSELF IS THE TRIGGER.** `clusters/dev/sync.yaml` is a self-managing root
Kustomization reconciling `./clusters/dev` from **`main`** every 10 minutes, and prod tracks `main`
too. **A merge to `main` therefore *executes* this migration on the live dev cluster, unattended.**
The authored change is a *target*, not a deployment. **Do not merge these files to `main` outside the
runbook window.** Park them on a branch until the operator is at the keyboard and step **D1** has
been done.

### D — dev vacates the apex (operator present, in this order)

- [ ] **D0. Snapshot the escape routes before touching anything.** Confirm a working break-glass
      **cert-based** admin kubeconfig for dev (not the OIDC one — OIDC breaks in D6). Confirm the
      retained bootstrap Gitea can be started (`bootstrap/gitea/bootstrap-gitea.sh`; dev's historical
      recovery host was `192.168.2.103:3000`, prod's current bootstrap is `192.168.2.132:3000`).
      Record dev's current Cloudflare record set (names + TXT registry owners) so D7 has a checklist
      and so an accidental deletion is detectable.
- [ ] **D1. Break the automatic trigger.** Either `flux suspend kustomization flux-system -n
      flux-system` on dev (stops the root from self-applying the merge), **or** re-point dev's
      `GitRepository` url at the bootstrap Gitea for the window. Do this **before** the merge, not
      after. This is what converts "merging strands dev" into "merging is inert until you say go".
- [ ] **D2. Publish and prove the new host set *while dev still serves the apex*.** The new names
      must exist before anything depends on them:
      - a `*.dev.home.keithjackson.dev` + `dev.home.keithjackson.dev` **wildcard Certificate issued
        and `Ready`** (LE prod, DNS-01) — see the ACME note below before ordering it;
      - public DNS for `gitea.dev.home.keithjackson.dev` resolving to dev's Gateway VIP
        (192.168.1.53), **checked against Cloudflare / a public resolver, never through the pfSense
        override** — that mistake was made on 2026-09-04 and produced a false healthy reading;
      - `curl -sS https://gitea.dev.home.keithjackson.dev/api/v1/version` succeeding with a
        publicly-trusted cert from a host on the dev LAN.
      Until D2 is green, D3 is not safe.
- [ ] **D3. Merge the authored change to `main`** (`clusters/base/cluster-vars.yaml`,
      `clusters/dev/`, `clusters/prod/`, the two Ansible files, `platform/externaldns`,
      `catalog/render.py`). With D1 in place nothing moves yet.
- [ ] **D4. Release dev's reconciliation deliberately.** Resume the root Kustomization (or restore
      the in-cluster url, now pointing at `gitea.dev.home.keithjackson.dev`) and watch
      `flux get kustomizations -A` converge. Dev's Gateway listeners, HTTPRoutes, coredns-custom zone
      and Homepage links all move here. Dev's external-dns publishes the new record set under owner
      `dev`.
- [ ] **D5. Keycloak: issuer + every client.** Dev's realm issuer becomes
      `https://keycloak.dev.home.keithjackson.dev/realms/homelab`. The issuer is baked into every
      token, and **~12 OIDC clients** need their redirect URIs moved in the same change: `grafana`,
      `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`, `qbittorrent`,
      `radarr`, `sonarr`, plus the `kubernetes` public client. Most are reconciled from
      `platform/keycloak/keycloak-realm-import.yaml` and move with D4; **headlamp's is the exception**
      — its issuer/callback live in a SOPS-encrypted Secret and must be re-encrypted by hand.
- [ ] **D6. Rolling k3s restart + kubectl SSO re-issue.** `k3s_base_domain` now renders a different
      `oidc-issuer-url` into `/etc/rancher/k3s/config.yaml`, so the `k3s_server` role is **no longer
      a no-op on dev**: it rewrites the file and fires the Restart handler on all three servers. Run
      it **one server at a time**, confirming `kubectl get nodes` between each. Every user's OIDC
      kubeconfig is invalidated and must be re-issued; the cert-based admin kubeconfig keeps working
      throughout, which is what makes this step recoverable. **Do not reboot nodes** — restart the
      `k3s` service only.
- [ ] **D7. Retire dev's old apex records.** They are owned by `default`, which no environment now
      claims, so nothing removes them automatically. Delete the `<svc>.home.keithjackson.dev` A
      records, the apex record, **and their TXT registry entries** (`external-dns-*` TXT), plus any
      stale `_acme-challenge.home.keithjackson.dev`. *Alternative, if you prefer external-dns to do
      its own cleanup:* set dev's `EXTERNALDNS_TXT_OWNER_ID` back to `default` for exactly one
      reconcile after D4 — `policy: sync` will then garbage-collect the apex set it no longer has
      sources for — and only then switch it to `dev`. That is two commits and needs watching; the
      direct deletion is preferred because it is verifiable in one place.
- [ ] **D8. Verify the apex is EMPTY, against Cloudflare.** `dig +short <svc>.home.keithjackson.dev
      @1.1.1.1` returns nothing for every service name, and no TXT registry entry for those names
      remains. **This is the gate on the whole P sequence.** A pfSense/PiHole override is not
      evidence about public DNS.

### D0/D1 AS EXECUTED (2026-09-04) — and the self-reference that would have undone it

D1 was taken via the **repoint** option, not the suspend option, because it is strictly stronger:
suspending only defers the strand, while repointing removes the dependency that causes it. Dev's
`GitRepository` now reads **`http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git`,
branch `dev-live`** — IP-addressed, plain HTTP, so it depends on **neither DNS nor TLS**, which is
exactly the property needed during a DNS/TLS migration. **A merge to `main` is now inert for dev.**

**THE TRAP, and it is not obvious: a bare copy of `main@4809199` on the bootstrap Gitea would NOT
have held.** The root Kustomization renders `./clusters/dev`, and `clusters/dev/kustomization.yaml`
*contains the patch that sets `GitRepository.spec.url`*. So Flux would have fetched from the
bootstrap Gitea, applied the manifests it found there, and those manifests would have set the url
straight back to `gitea.home.keithjackson.dev` — reverting the repoint on the next reconcile and
re-arming the strand. **The branch must declare its own source.** `dev-live` (`a4a3c2e`) therefore
carries exactly one content change on top of `4809199`: `clusters/dev/kustomization.yaml`'s url +
`ref.branch`. Proven by `kubectl kustomize clusters/dev` at both revisions differing in **exactly
two lines** out of 357. The same property makes the revert self-sustaining in reverse, since
`main@4809199` carries the `gitea.home…` url. Revert command: see `REVERT.txt` in the run scratchpad.

Verified after the cutover: **13/13 Kustomizations Ready** on `dev-live@a4a3c2e`, `cluster-vars`
byte-identical to the D0 snapshot (27/27 keys, `BASE_DOMAIN` still `home.keithjackson.dev`),
**18/18 Longhorn volumes attached+healthy**, all 16 HTTPRoute hostnames still on the apex,
`wildcard-home` still Ready, and the Cloudflare record set re-queried and diffed **byte-for-byte
against the D0 snapshot — identical** (57 records, 36 TXT all `owner=default`).

> **NEW DEPENDENCY, AND IT IS THE WEAK LINK.** `192.168.2.132` is a **DHCP lease on the
> workstation's wifi** (`wlp0s20f3`, NetworkManager `ipv4.method=auto` — confirmed, not assumed). A
> lease change, a wifi drop, or the workstation going offline **stalls dev's fetching**. The cluster
> keeps running and serving; only reconciliation pauses, and the fix is a `kubectl patch` of the url.
> Two consequences: (a) a **static reservation** for this host is worth having before any repeat of
> this procedure, and (b) **keep the window short** — the exit is D4, which returns dev to its own
> in-cluster Gitea at `gitea.dev.home.keithjackson.dev`.
>
> Pointing Flux at dev's Gitea **Service** instead (`gitea.gitea.svc.cluster.local:3000`) would drop
> the DNS, TLS *and* workstation dependencies all at once, and is the better long-term shape. It was
> **not** done here: namespace `gitea` carries a `default-deny` NetworkPolicy plus a
> `gitea-allow-gateway-ingress` CiliumNetworkPolicy admitting **only** the Gateway, so
> source-controller cannot reach the Service without a new authored policy. Worth doing deliberately,
> not mid-migration.

### P — prod takes the apex

- [ ] **P1. Unsuspend prod**, in dependency order: `flux resume kustomization platform`, then the
      10 downstream tiers.
      **CORRECTION (2026-09-04) — the D8 gate is on prod taking the APEX, not on prod coming up.**
      This step originally read "only now unsuspend prod", which silently assumed prod would be
      serving `*.home.keithjackson.dev`. It is not: prod runs revision `c8c1d29`, where
      `BASE_DOMAIN=prod.home.keithjackson.dev` and `EXTERNALDNS_TXT_OWNER_ID=prod`. Prod's names are
      therefore a **disjoint subtree** under a **distinct registry owner**, so neither incident mode
      of §10a applies: it cannot claim dev's `default`-owned records, and its wildcard
      `*.prod.home.keithjackson.dev` is a **different name set** from dev's, so it has its own
      Let's Encrypt duplicate-certificate bucket and races no `_acme-challenge` with dev.
      **Prod may therefore be brought up in full while dev still holds the apex** — and should be,
      because it proves the entire platform stack end-to-end *before* the apex flip, turning the
      cutover into a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
      once. What D8 gates is **only** the later move of prod's `BASE_DOMAIN` to the bare apex.
      Verify the two guard values live in prod's applied `cluster-vars` before unsuspending; if
      `EXTERNALDNS_TXT_OWNER_ID` is not `prod`, STOP — that was the root cause of the incident.
- [ ] **P2. Prod's external-dns publishes under owner `prod`.** Check the logs for a matching hosted
      zone (F-DNS1 fail-closed mode: no zone selected ⇒ nothing created — which is safe, but means
      the filter was narrowed against instruction). Confirm the new TXT registry entries read owner
      `prod` and that no record owned by `dev` was touched.
- [ ] **P3. Prod's wildcard cert `*.home.keithjackson.dev` + `home.keithjackson.dev` is `Ready`** and
      solved DNS-01 without colliding with any `_acme-challenge` dev still holds.
- [ ] **P4. Prod's kubectl SSO works without an `/etc/hosts` pin.** Prod nodes resolve
      `keycloak.home.keithjackson.dev` through **public** upstreams (§4), so this only becomes true
      once P2 has published that name at prod's Gateway VIP (192.168.2.20). Before then a prod node
      resolving its own issuer reaches **dev**. Use the break-glass cert kubeconfig until P2 is done.
- [ ] **P5. Cut prod's GitOps source over** from the workstation bootstrap Gitea
      (`http://192.168.2.132:3000/...`) to `https://gitea.home.keithjackson.dev/...` once prod runs
      its own Gitea — the block in `clusters/prod/kustomization.yaml` owns that procedure. Note the
      inversion **simplified** this: prod's apex Gitea name and dev's `gitea.dev.home...` are now
      different hosts, so prod pointing at the apex means "prod's own Gitea", never dev's.

### ACME: the duplicate-certificate limit

Let's Encrypt rate-limits **duplicate certificates — 5 per week for an identical set of names.**
Prod's wildcard is `*.home.keithjackson.dev` + `home.keithjackson.dev`, which is **exactly the name
set dev holds today**. Every prod order therefore counts against the same bucket as dev's historical
renewals of that pair, and a retry loop during P3 can exhaust it and leave prod with **no trusted
cert for up to a week**. Mitigations:

- Order prod's cert **after** D8, so dev is no longer renewing the same pair.
- Let dev's old `*.home.keithjackson.dev` `Certificate`/Secret be **deleted** in D4/D7 rather than
  left to renew in the background.
- Rehearse against **letsencrypt-staging** first (`ACME_CLUSTER_ISSUER`) — staging has its own,
  far looser limits — and only switch prod to `letsencrypt-prod` when the DNS-01 solve is proven.
- Do **not** delete-and-recreate prod's `Certificate` to "retry"; each recreation is a fresh order.

### The L2 tests hardcode the apex

Six behavioural (L2) chainsaw tests embed `home.keithjackson.dev` as a literal:

| test | file |
|---|---|
| `dns-exposure` | `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml` |
| `dns-internal-resolution` | `conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml` |
| `gateway-ingress-reachable` | `conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml` (+ `fixtures/app-and-probe.yaml`) |
| `oidc-sso-grafana` | `conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml` |
| `sso-forward-auth` | `conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml` |
| `trusted-tls` | `conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml` |

They are **test inputs, not reconciled manifests**, so they are deliberately left alone by the
authoring change: they are the only thing still asserting the apex, and while dev is live on the apex
they are *correct*. Under the new direction the literal happens to become right for **prod** and
wrong for **dev**, so they must be **parameterized per environment** (not merely edited) before
either §9's *"suite passes green against the prod cluster"* or a green L2 run against migrated dev
can be claimed. Until then, expect the dev L2 suite to fail between D4 and that parameterization —
a known, scheduled failure, not a regression.

The L1 *rendering* tests need no change: their goldens carry `${BASE_DOMAIN}` as a token, and the
literal hostnames in them come from each test's own fixture `ingressRoutes[].host`, which is
environment-independent by construction.

### Offline proof carried out for the authoring change (no cluster touched)

All 12 `substituteFrom` paths were rendered for **both** overlays through the real toolchain
(`kubectl kustomize --load-restrictor LoadRestrictionsNone <spec.path> | flux envsubst --strict`,
with each overlay's effective `cluster-vars` taken from `kubectl kustomize clusters/<env>`), before
and after the change.

- **Dev's diff is *only* the hostname move.** Proven by inverse substitution rather than by reading
  the diff: applying `s/dev.home.keithjackson.dev/home.keithjackson.dev/` to the new render restores
  the baseline **byte-for-byte** in all 12 paths, plus 2 declared `txtOwnerId: default → dev` lines.
  145 changed line pairs, all classified: 143 hostname-move, 2 declared owner-id, **0 unexplained**,
  and **0 bare-apex remnants**. The 2 paths with no change (`cilium`, `velero`) provably contain no
  hostname at all (`grep -c keithjackson.dev` = 0).
- **Prod lands on the bare apex:** **0** occurrences of `prod.home.keithjackson.dev` and **0** of
  `dev.home.keithjackson.dev` anywhere in prod's rendered output; 143 apex names.
- **Certs:** dev `*.dev.home.keithjackson.dev` + `dev.home.keithjackson.dev`; prod
  `*.home.keithjackson.dev` + `home.keithjackson.dev`.
- **Positive controls** (the harness is provably not blind): perturbing an unrelated knob
  (`GATEWAY_VIP`) surfaces 1 unexplained line; perturbing `BASE_DOMAIN` to a wrong value fails 10 of
  12 paths; injecting a single un-moved hostname is caught by the independent bare-apex scan (which
  the byte-identity check alone cannot see — which is why both exist).
- Gates: `ci/l0.sh`, `ci/l1.sh`, `ci/render-golden.sh`, `ci/lint.sh`,
  `flux-postbuild-substitution/check.sh` (254 assertions), `catalog/render.py --selftest`,
  `ci/spec-immutability.sh` — all **PASS**.