home-network-k3s-impl/infra/PROD-PROVISIONING.md

# Prod bare-metal provisioning checklist (T-7.1)

**Task:** T-7.1 · **Workstream:** WS-INFRA · **Serves:** R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03
**Status:** Phase 7 **opened by the operator 2026-08-31** — *prep only*. Nothing is installed on prod
hardware until the blockers in §7 are closed and `make -C ci gate` is green.

> **Entry-gate note (recorded, not waived).** `roadmap/phases.md` gates Phase 7 on *"Phases 1–6
> proven in dev/test; operator declares prod hardware ready."* The operator has declared hardware
> ready. The first condition is **not** met: conformance is 37/51, and the uncovered set includes
> **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work
> knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry).

Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)).
The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to
call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse
the *same* Ansible roles and the *same* Flux base as dev — that reuse is the R-ENV-02 obligation.

---

## 0. Node inventory

Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.

| | node 1 | node 2 | node 3 |
|---|---|---|---|
| Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
| LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | |
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | |
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | |
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | |
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | |
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | |
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | |
| `/dev/kvm` | present | present | |
| Serial | MXL1052NGN | MXL0073949 | |

**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a
healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica
placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's
NVMe is the drive node 1 should have had.

Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).

---

## 0b. SOLVED — a Linux kernel bug, NOT a hardware fault (2026-09-01)

> **This section previously read "OPEN HARDWARE FAULT" and blamed the DIMM, then BIOS memory
> remapping, then stale Fast Boot memory training. All three were wrong.** The hardware is fine.
> Keeping the correction visible because the wrong framing nearly cost money and a board RMA.

`k3s-prod-2` intermittently addresses only ~2.6 GiB of its 16 GB. **Root cause: HP's R21 firmware
publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12 fails to validate it before
reserving memory based on its contents.** Fixed upstream in **6.13**.

### The evidence that breaks it open

The same machine — same DIMM, same BIOS, same kernel, same cmdline — **booted with the full
15.9 GiB two boots earlier**:

| Boot | `Memory: … available` | reserved | memattr line |
|---|---|---|---|
| −2 | **15,917,744K** | 615,208K | `Unexpected EFI Memory Attributes table version 1463964749` |
| −1 | 2,613,912K | 13,913,856K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |
| 0 | 2,568,544K | 13,958,980K | `Failed to map EFI Memory Attributes table @ 0xbeeb0018` |

The `BIOS-e820` map is **byte-identical** across all three. No hardware fault is intermittent in
that shape, and none is fixed by a reboot.

### Mechanism (verified against kernel source)

`drivers/firmware/efi/memattr.c`, v6.12 and current `linux-6.12.y`:

```c
if (tbl->version > 2) { pr_warn("Unexpected EFI Memory Attributes table version %d\n", ...); goto unmap; }
tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
memblock_reserve(efi_mem_attr_table, tbl_size);
```

**`num_entries` and `desc_size` are never validated.** With a corrupt table:
- garbage `version` reads **> 2** → kernel warns and bails → memory intact *(the lucky path)*
- garbage `version` reads **≤ 2** → kernel multiplies two garbage fields and `memblock_reserve()`s
  the product — **gigabytes, from the table address to the top of RAM** *(the unlucky path)*

That accounts for every observation: `Normal` zone `present 3340288 / managed 0`, and `MemTotal` =
DMA (3840) + DMA32 (684481) pages = **2,753,284 kB exactly**, with Normal contributing zero.

### BOTH prod nodes are affected — node 1 has only been lucky

| Node | memattr version field | Outcome |
|---|---|---|
| `k3s-prod-1` | `1030` — garbage, but **> 2** so the kernel bails | full 32 GiB **by luck, not immunity** |
| `k3s-prod-2` | reads **≤ 2** so the kernel trusts it | 2.6 GiB |
| dev nodes | **no memattr table published** (Proxmox OVMF) | unaffected, full 24 GiB |

Node 1 is one unlucky boot away from the identical fault. **Fix both.**

### Mechanism, traced to the byte (second, independent confirmation)

A second research pass reached the same root cause by a different route and pinned the arithmetic:

- The garbage `version` node 2 read was `1463964749` = `0x5742544D` = the ASCII bytes **`MTBW`**. The
  kernel is reading unrelated data, not a table. HP publishes the MEMATTR pointer into memory that
  `/sys/firmware/memmap` calls plain `System RAM`, so the pre-Linux boot chain overwrites it.
- `tbl_size` is a **signed `int`**. `num_entries * desc_size` is `u32 * u32`, wraps mod 2^32, and
  when bit 31 lands the result is **negative**.
- `memblock_reserve(base, tbl_size)` converts that negative `int` to `phys_addr_t`, sign-extending to
  ~`0xFFFFFFFF_xxxxxxxx`; `memblock_cap_size()` clamps it to `PHYS_ADDR_MAX - base`. **Every physical
  address above `0xbeeb0018` is reserved.**
- `memblock_free_all()` then hands the buddy allocator nothing above that line, while `present_pages`
  (which comes from `memblock.memory`) is untouched — producing `present 3340288 / managed 0`.
- The `Failed to map EFI Memory Attributes table` message is emitted *later*, by
  `efi_memattr_apply_permissions()`, whose `memremap()` of the absurd size fails. **That message is
  therefore positive proof the reservation already happened** — it is a consequence, not the cause.

The cleanest A/B tell between the two nodes is one line of dmesg:

| | `NODE_DATA(0) allocated` | meaning |
|---|---|---|
| prod-2 (bad) | `[mem 0xbee85680-0xbeeaffff]` | ends at `0xbeeb0000` — pgdat forced immediately **below** the reservation |
| prod-1 (good) | `[mem 0x82f7d5680-0x82f7fffff]` | ends at `0x82f800000` — **top of RAM**, the normal placement |

Identical `0x2a980` allocation size, opposite placement. Node 2's DMA32 `managed` is also 684481 vs
node 1's 775777 on identical `present` — ~356 MiB of extra structures forced below 4 GiB.

### THERE IS NO KERNEL CMDLINE WORKAROUND

`efi_memattr_init()` is called from `efi_config_parse_tables()` under only
`!CONFIG_X86_32 && efi_enabled(EFI_MEMMAP)`, and x86's `efi_init()` calls `efi_config_init()`
**before** the `efi_runtime_disabled()` bail-out. So `noefi`, `efi=noruntime` and `efi=novamap`
**do not prevent the reservation.** Booting a fixed kernel is the only real fix.

### The fix

**Boot a kernel ≥ 6.13** — upstream commit `8fbe4c49c0cc` *"efi/memattr: Ignore table if the size is
clearly bogus"* (Ard Biesheuvel, 2024-10-31, first released in v6.13), whose commit message
describes this bug verbatim. Verified absent from v6.12 and from the current `linux-6.12.y` stable
branch. On Debian 13 that means `linux-image-amd64` from **trixie-backports**. A fixed kernel prints
`FW_BUG Corrupted EFI Memory Attributes Table detected!` and keeps all the memory.

The hardening landed exactly one release too late for Debian 13: the bug has existed since the
feature was introduced in 2016, `8fbe4c49c0cc` first shipped in **v6.13**, and a follow-up
`5d0faa8e8369` (2026-03-26) tightened it further to the current form, whose comment reads verbatim
*"This prevents a corrupted table from eating all system RAM."* Debian's 6.12.107-1 demonstrably
lacks it. **Worth a Debian BTS report against `src:linux`** — 6.12 is Debian 13's kernel for the
release lifetime, and the fix is small and already upstream.

Same signature reported elsewhere: [Ubuntu LP #2039382](https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-6.2/+bug/2039382)
(Dell, 16 GiB → 2.5 GiB, memtest clean, "fixed" by unplugging a USB mouse — which merely moved the
table) and [LP #1944019](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1944019) (**HP ZBook
Studio G5**, 16 GiB → 1.5 GiB).

### What NOT to do

- **Do not buy RAM, move the DIMM, RMA the board, or run memtest expecting failures.** The DIMM is
  a 16 GB 2Rx8 Samsung C/D-die, DDR4-2666, on HP's supported population (`DIMM1 must always be
  installed`). Coffee Lake supports 16 Gb x8 regardless. A density fault would yield 8 GiB, never 2.6.
- **BIOS 02.27.00** (sp174287) helps only by accident, by shifting where the table lands. The full
  R21 changelog 02.10.01 → 02.27.00 contains **zero** memory/MRC/DIMM entries.
- **Rebooting, disabling Fast Boot, clearing CMOS, unplugging USB** are all the same lottery ticket:
  they change the firmware's boot-services allocation footprint and hence where MEMATTR lands. They
  may appear to "fix" it. They are not fixes.

### Hypotheses killed, and why — so they are not re-run

- **Fast Boot / stale MRC training data** — killed as a *mechanism*. Intel MRC caches are guarded by
  an SPD-derived CRC plus a firmware-version tag; a DIMM swap is exactly what that guard catches. HP
  documents Fast Boot only as a POST-diagnostics shortcut, never as an MRC control.
- **Memory remap / above-4G decoding** — no such attribute exists among the 241 exposed by
  `hp-bioscfg` on this platform.
- **Integrated graphics** — `Graphics Stolen Memory` is exactly 64 MiB, below 4 GiB.
- **EFI_MEMORY_SP / "soft reserved" memory** — the strongest competing theory, **excluded by
  measurement**: `/sys/firmware/memmap/10/type` reads `System RAM` for `0x100000000-0x42f7fffff`.
  Soft-reserved ranges become `E820_TYPE_SOFT_RESERVED`, print as `Soft Reserved`, and are never
  `memblock_add()`ed — which would have left `present` at ~0, not 3340288.
- **SGX** — ruled out at runtime: `x86/cpu: SGX disabled or unsupported by BIOS`.
- **MTRR trim** — MTRRs near-identical to node 1, no trim warning.
- **A `last_pfn` cap** — a misreading; the second value is `max_low_pfn` and is normal.
- The 2.6 GiB figure was never a power of two, which fits no addressing theory. It is now exactly
  explained: everything from `0xbeeb0018` upward.

## 1. Firmware / BIOS (per node, before OS install)

> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes
> **241** BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/` (239 have a
> `current_value`; 215 returned a non-empty value — an earlier note here said "215 attributes",
> which was the non-empty subset, not the total). **Reads work fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye.
>
> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4,
> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is
> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means
> F10 on the box.
>
> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is
> achievable today with reads alone, and would have caught node 2's drift on its first health run.
> Converging it automatically additionally requires provisioning a BIOS Setup Password on every
> node, which is a real security decision (a stored credential that can rewrite firmware) and is
> the operator's call, not a default.


- [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
      boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
- [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
      human pressing the button. (`docs/02` calls out wake/boot behaviour explicitly.)
- [ ] **Wake-on-LAN enabled** on the onboard NIC (remote recovery when the box is off).
- [ ] **Boot order** — internal disk first; USB above it only while imaging. Disable network boot
      unless PXE is adopted.
- [ ] **Secure Boot** — record the setting. k3s itself is fine either way; note it so all three
      nodes match.
- [ ] **Firmware version recorded** in §0 and consistent across all three nodes.

### 1b. BIOS passwords and programmatic configuration — researched 2026-09-01

**There is no default BIOS password, and no factory default exists to look up.** HP business
desktops ship with **no** Setup password and **no** Power-On password set; the `is_enabled=0` we
observe on both is the factory state, not drift. HP's own Maintenance & Service Guide for the 800 G5
describes both purely as things the owner configures in Computer Setup.

**Owner recovery path, for the runbook:** if a Setup password is ever set and forgotten, HP documents
clearing it with the **PSWD jumper** on the system board (power off, open, remove jumper, boot once).
**Do NOT enable "stringent security"** — HP documents that it removes the physical bypass entirely,
turning a five-minute jumper fix into an HP service case requiring a signed, per-serial unlock. The
permissive default is the right setting for a homelab.

#### CORRECTION — error `0x4` is NOT an authorisation failure

I recorded that the firmware refused our write because BIOS-admin authentication was missing. **That
was wrong, and the kernel driver misled me.** `hp-bioscfg` passes the return value of HP's WMI
`SetBIOSSetting` method through `hp_wmi_error_and_message()` — which is **the error table for a
different interface** (the `"SECU"` commands under `HP_WMI_BIOS_GUID`). The two methods have
unrelated return-code spaces. Per HP's *Client Management Interface* white paper, `SetBIOSSetting`
returns:

| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Not Supported |
| **4** | **Failed** (generic) |
| 6 | Access Denied |

So the firmware said a generic **"Failed"**. An authorisation problem would have returned **6**, and
the driver has a *separate* `BIOS_ADMIN_NOT_SET = 0x38` → `-EPERM` for the no-password case — neither
of which we saw. The string "Invalid command type" printed in our logs is the driver looking `4` up
in the wrong table. This is a genuine upstream bug worth reporting to `platform-driver-x86`.

**Most likely actual cause (inferred, testable):** with an empty password the driver emits an
*untagged* empty UTF-16 string, whereas every HP-documented caller passes the encoding tag even when
blank (`Password = "<utf-16/>"`). The driver only prepends that prefix for a *non-empty* string. So
setting a Setup password may well fix writes — **but not because 0x4 means "unauthorised"**; it would
work by flipping the driver onto the tagged code path HP's own tools use. Corroborating smell: this
firmware already makes the driver log `Prerequisites size value exceeded the maximum number of
elements supported or data may be malformed` at boot.

**`hp-bioscfg` cannot set a password** — verified from source. Both `current_password` and
`new_password` funnel into `store_password_instance()`, which only `strscpy`s into driver RAM;
nothing ever transmits `new_password` to firmware. It is write-only into a buffer that is never read.

#### Recommendation

1. **Land the read-only drift check now** — encode this §1 checklist as expected values, diff
   `current_value` on every node in `roles/node_health`, fail on mismatch. That is the whole R-ENV
   win, at zero risk and with no new credential, and it is available today. Remediate by hand at F10.
2. **Do not provision a Setup password "to fix the auth error"** — there is no auth error. Provision
   one only as a deliberate experiment on **node 3** (not yet built): set it at F10, write it to
   `current_password`, retry one benign attribute. Return `0` confirms the `<utf-16/>` theory; `6`
   would mean a password mismatch; another `4` means the driver is incompatible with R21 and the
   fallback is `hp-repsetup`.
3. **Fallback if that fails:** HP's own Linux tool `hp-repsetup` (HP Flash and Replicated Setup
   Utilities, v3.26) can both dump/apply settings *and* set the Setup password from Linux — but it
   needs the out-of-tree `hpuefi.ko`, which requires **Secure Boot disabled or the module self-signed**.
   Secure Boot is currently enabled on these nodes, so this costs either a signing setup or a
   security downgrade.
4. **Skip HP Sure Admin / SPM and BCU.** Sure Admin's key-based model is genuinely better in
   principle — the private key never leaves the workstation, each change is individually signed, with
   anti-replay — but provisioning and every signature require Windows-only tooling. For three
   machines whose firmware settings change roughly never, the operational cost far exceeds the risk
   reduction. BCU is Windows/WinPE-only.
5. **Security trade-off if you do provision a password:** it is a *shared* credential granting full
   pre-OS control of all three nodes (disable Secure Boot, re-order boot, re-enable ports) —
   functionally equivalent to unsupervised physical access. Exposure is bounded in that using it via
   `hp-bioscfg` needs root on the node already (sysfs is `--w-------`), so the marginal loss is
   persistence and pre-boot control rather than initial compromise. Note `hp-repsetup -p` takes the
   password in cleartext on the command line — it would land in process listings and shell history
   unless wrapped. Store it in the Ansible vault **and** offline.

## 2. Disk layout — **OPEN DECISION, blocks the build**

`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
(`docs/plans/ssd-etcd-isolation.md`).

Node 1 as-found is a **single** 477 GB SATA SSD carrying everything:

```
sda  477G  C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
├─sda1 976M  /boot/efi
├─sda2  30G  /
├─sda3  12G  /var
├─sda4 977M  swap
└─sda5 433G  /srv          <- the only large free space
```

Consequences to settle before imaging nodes 2 and 3:

- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
      floor — see §5). Raw attributes, so the numbers are auditable:

      | | |
      |---|---|
      | Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) |
      | SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` |
      | Power-on hours | 6,814 (~284 days) · 147 power cycles |
      | **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) |
      | **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) |
      | **Erase failures** | **187** (`172`) |
      | **Unexpected power losses** | **32** (`174`) |
      | Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) |
      | UDMA CRC errors | 0 (`199`) — cabling is clean |

      **Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector
      count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its
      normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**,
      not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value;
      the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent
      uncorrectable counters instead.

      **Verdict: usable, but not for production state.** Nothing says imminent failure — no
      pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011
      consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad
      blocks, logged **187** erase failures, and taken **32** unclean power losses **with no
      power-loss protection**, in a single failure domain. etcd is the worst possible tenant for
      it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how
      a member corrupts — this drive has already logged 32 of them.
      (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
      `DECISIONS-LOG.md`. It is not part of this verdict.)
- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
      (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
      Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
      prod must override them in `group_vars`. On a single-disk node there is no second device at
      all, so one of:
      - **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
        SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
        spindle.
      - **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
        Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
      - **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
        physical failure domain.
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
- [ ] Record the chosen device paths here and set them in
      `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
      `/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
      `node-baseline` run against a node still carrying those placeholders now stops on an explicit
      assert naming this section, instead of failing obscurely at `mkfs` time.
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
      is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
      tasks this decision blocks — see §5's drive-health box for the exact command.

## 3. OS install (per node)

- [ ] **Debian 13 (trixie)**, matching dev exactly — same kernel series, same package baseline.
- [ ] Hostname `k3s-prod-N`.
- [ ] Partition per the §2 decision.
- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
      `roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
      so the same role covers bare metal. Nothing to port.

## 4. Network (per node)

- [ ] **Static IP** on the prod LAN, recorded in §0. Node 1 is `192.168.2.10/24`, gateway
      `192.168.2.1`.
- [x] **Prod address plan — DECIDED (operator, 2026-08-31).**

      | Role | Address | Owned by |
      |---|---|---|
      | `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory |
      | **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) |
      | **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) |
      | **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) |

      Two deliberate properties, both mirroring dev:
      - The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and
        Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way
        round (`.49` control-plane, pool `.53-.60`).
      - **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a
        defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool
        was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape
        (`GATEWAY_VIP` `.53` = first of `.53-.60`).
- [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before
      the build, and confirm nothing else on the network already answers on them.
- [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.**
      Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single
      upstream is a known SPOF that has already bitten this project — it broke containerd image
      pulls and CoreDNS forwarding on dev.

      The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in
      `roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes
      `/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed
      `/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager
      owns it), and configures the ordered upstream list either way. The earlier
      `fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan
      drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie
      nodes, so it would have broken `node-baseline.yml` on every host.

      What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets
      `node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid
      `BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert
      stops the play on it before writing anything. Choosing the second upstream is a **network
      decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would
      resolve reliably but would also bypass any local zone and any filtering the gateway applies.
      Decide between — and record the choice HERE, in the same change that edits the group_vars:
      - a second on-LAN resolver on `192.168.2.x`;
      - the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified —
        prod sits on the workstation side of the pfSense boundary);
      - an explicit, written-down acceptance of a public resolver as fallback-only.

      Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`)
      are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed
      servers, so keeping them means listing them explicitly.
      `node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's
      local domain is decided.
- [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT —
      prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind.

## 5. Access (per node) — **blocks all Ansible**

- [ ] **`k3s` user with the `home_network` SSH key.** Working on node 1.
- [ ] **Password-protected sudo, driven by Ansible `become` + a vaulted password.**
      **Prod does NOT use NOPASSWD sudo** (operator decision, 2026-08-31). Dev nodes do; prod
      deliberately keeps the sudo password in place and supplies it to Ansible as an
      **ansible-vault-encrypted** var, exactly like the existing `hypervisor` and `proxmox_hosts`
      groups:
      ```
      inventory/group_vars/k3s_prod/
        README.md     how to (re)create the vault — committed
        vault.yml     ansible_become_password: "..."  — committed, ENCRYPTED
      ```
      The passphrase that decrypts it lives **outside the repo** at
      `~/.config/home-k3s/ansible-vault-pass` (`ansible.cfg :: vault_password_file`), so plays still
      run unattended without a plaintext secret ever touching git.
      ```bash
      ansible-vault create inventory/group_vars/k3s_prod/vault.yml   # ansible_become_password: ...
      ```
      Prod nodes must therefore set `ansible_become: true` in their group_vars and must **not**
      inherit any passwordless-sudo assumption from `group_vars/all.yml`.
- [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is
      the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling,
      2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part
      of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline
      would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead
      (the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so
      the placeholder device paths are never even templated):
      ```bash
      cd infra/ansible
      ansible-playbook playbooks/node-baseline.yml \
        -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
      ```
      (Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading
      the drive is then:
      ```bash
      sudo smartctl -a /dev/sda   # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors
      ```
      Record the result in §2 and close the layout decision in the same change that replaces the
      two `node_baseline_*_device` placeholders.

## 6. Ansible baseline & cluster join

Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars
differ, plus a target-group override on the command line.

### How prod is targeted (done, 2026-08-31)

Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` /
`k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately
kept separate because Ansible resolves them at different times:

| Layer | Knob | Set via |
|---|---|---|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there |
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** |

So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and
tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with
no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after
`--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**.

- [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
      **directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping.
      (Nodes 2/3 stay commented out until they are physically on the bench.)
- [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`),
      `cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the
      §2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which
      are still BLOCKED placeholders.
- [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until
      it exists **no** prod play can escalate, and `--check` against prod cannot work either.
- [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read:
      ```bash
      ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
      ```
- [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel
      modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it
      asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare
      metal — the role already only *starts* it when the virtio channel exists, so it degrades
      cleanly.)
- [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`
      → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three
      nodes are present** — see §8.
- [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the
      prod LAN.
- [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts
      it afterwards. The bootstrap values and the HelmRelease **must stay in parity**
      (`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config
      (`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the
      dev one the dev container exports as `$KUBECONFIG`.
- [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`.
- [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**:
      `roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps
      `--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a
      target-group change. Tracked with the other repo blockers in §7.
- [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015).
      `playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=<group>`, but there is
      deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted
      PIA credentials — the existing vault is scoped to the dev group.

## 7. Repo blockers — **prod cannot be reconciled until these land**

`clusters/prod` is design-only today and **would build wrong**. R-ENV-02 promises dev and prod differ
*only* by `cluster-vars` values; in practice dev LAN addresses are hardcoded into base manifests:

| Hardcoded value | Where | Prod needs |
|---|---|---|
| `k8sServiceHost: 192.168.1.50` | `platform/cilium/helmrelease.yaml:50`, mirrored in `values.yaml:16` | prod primary node IP |
| LB pool `192.168.1.53–.60` | `platform-config/cilium/l2-ip-pool.yaml:15` | a prod-LAN pool |
| `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
| `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
| `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |

- [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
      overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
      header comment in `clusters/base/cluster-vars.yaml`). `platform`, `platform-config`, the app
      waves, `apps-vms` and `crafty` all already carry `postBuild.substituteFrom: cluster-vars`, so
      the seam exists.
- [ ] **`clusters/prod` must set `GATEWAY_VIP`.** It currently doesn't, so prod would inherit the
      base default `10.73.0.240` — an address from the **retired libvirt network**. The base file's
      own comment flags this as owed "when hardware is built."
- [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is
      byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`.
- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing
      the `dependsOn` fix. Only the first is now closed:
      - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became
            *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries
            `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block
            forever on a dependency that does not exist and nothing downstream would reconcile.
      - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key
            as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev
            exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster
            (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only.
      - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision
            2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and
            will be decommissioned once prod is stable. Recorded consequence: once prod is live,
            **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window.
            The URL is patched in but **cannot work on a cold prod cluster**, for two independent
            reasons found 2026-09-01:
            1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
               decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
               loses its source of truth. **This is the decommission plan's first task, not its last.**
            2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire
               `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
               prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
               `gitea.home.keithjackson.dev` to itself and never reach a repository.
            Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a
            cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own
            Phase-6 cutover.
- [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
      the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).

## 8. Build sequence (operator decision, 2026-08-31: **wait for all three nodes**)

Node 1 is baselined now; the cluster is **not** initialized until nodes 2 and 3 are on the bench.
This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a
single-server-etcd window, and avoids a later member-add dance.

1. Per-node §1–§5 on each of the three boxes.
2. §7 repo work lands on `main` with the gate green.
3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`.
4. `flux bootstrap ... --path=clusters/prod`.
5. Conformance against prod (§9).

## 9. Exit criteria (Phase 7)

- [ ] Prod runs the same config as dev via the prod overlay, **no base edits** (R-ENV-02).
- [ ] Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01).
- [ ] Full **automated** prod DR drill (R-STATE-03) — `definition-of-done.md` requires this for prod
      specifically.
- [ ] Monitoring and alerts live on prod.
- [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).