# Prod bare-metal provisioning checklist (T-7.1) **Task:** T-7.1 · **Workstream:** WS-INFRA · **Serves:** R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03 **Status:** Phase 7 **opened by the operator 2026-08-31** — *prep only*. Nothing is installed on prod hardware until the blockers in §7 are closed and `make -C ci gate` is green. > **Entry-gate note (recorded, not waived).** `roadmap/phases.md` gates Phase 7 on *"Phases 1–6 > proven in dev/test; operator declares prod hardware ready."* The operator has declared hardware > ready. The first condition is **not** met: conformance is 37/51, and the uncovered set includes > **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work > knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry). Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)). The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse the *same* Ansible roles and the *same* Flux base as dev — that reuse is the R-ENV-02 obligation. --- ## 0. Node inventory Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. | | node 1 | node 2 | node 3 | |---|---|---|---| | Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` | | LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` | | Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | | | CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | | | RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | | | Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | | | Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | | | OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | | | Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | | | `/dev/kvm` | present | present | | | Serial | MXL1052NGN | MXL0073949 | | **The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should have had. Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd). --- ## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install more RAM until this is understood** — see the warning at the end. | Evidence | Reading | |---|---| | DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty | | BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` | | Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` | | `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot | | Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** | | `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** | | Memory blocks | 127 online, 128 MiB block size | **The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping above 4 GB not being applied**, not of failing memory. Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00, identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it is per-machine state. **BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` kernel driver, which exposes **215 BIOS attributes** read/write at `/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords are set (`authentication/*/is_enabled` = 0). **They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was made on 2026-09-01 and the FIRMWARE refused it: `hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`. HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent — `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`. Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS Setup Password is deliberately provisioned (see §1). What the comparison ruled OUT as the cause: - **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and `Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting. - **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on either node, so it cannot be a mis-set option. - **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**, which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path. - **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory blocks report `online`. The memory reaches the kernel and is then reserved. Real BIOS differences found (node 1 = works, node 2 = broken): | Attribute | node 1 | node 2 | |---|---|---| | **After Power Loss** | Power On | **Power Off** ← violates §1 | | **Fast Boot** | Disable | **Enable** ← leading hypothesis | | Legacy Boot Options | Disable | Enable | | TPM Activation Policy | F1 to Boot | Allow user to reject | | Microsoft/Windows UEFI CA 2023 | Enable | Disable | | Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) | **Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data* rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale training data is a plausible route to memory being present, inventoried, handed over — and then reserved. It is also the one memory-relevant setting that differs from the working node. This is a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write and a reboot, which needs operator sanction. To investigate, cheapest first: - [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). **Must be done in F10** — the firmware refuses programmatic writes (above). - [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and node 2 currently violates it. - [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. - [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot vs board in one move. - [ ] `memtest86+` if the above do not resolve it. > ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS > setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted. > **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod` > reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share. ## 1. Firmware / BIOS (per node, before OS install) > **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes > 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work > fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found > at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye. > > **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4, > "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is > provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means > F10 on the box. > > **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is > achievable today with reads alone, and would have caught node 2's drift on its first health run. > Converging it automatically additionally requires provisioning a BIOS Setup Password on every > node, which is a real security decision (a stored credential that can rewrite firmware) and is > the operator's call, not a default. - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero. - [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a human pressing the button. (`docs/02` calls out wake/boot behaviour explicitly.) - [ ] **Wake-on-LAN enabled** on the onboard NIC (remote recovery when the box is off). - [ ] **Boot order** — internal disk first; USB above it only while imaging. Disable network boot unless PXE is adopted. - [ ] **Secure Boot** — record the setting. k3s itself is fine either way; note it so all three nodes match. - [ ] **Firmware version recorded** in §0 and consistent across all three nodes. ## 2. Disk layout — **OPEN DECISION, blocks the build** `docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went further and added a *third* device for etcd after HDD contention starved etcd WAL fsync (`docs/plans/ssd-etcd-isolation.md`). Node 1 as-found is a **single** 477 GB SATA SSD carrying everything: ``` sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial) ├─sda1 976M /boot/efi ├─sda2 30G / ├─sda3 12G /var ├─sda4 977M swap └─sda5 433G /srv <- the only large free space ``` Consequences to settle before imaging nodes 2 and 3: - [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package floor — see §5). Raw attributes, so the numbers are auditable: | | | |---|---| | Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) | | SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` | | Power-on hours | 6,814 (~284 days) · 147 power cycles | | **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) | | **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) | | **Erase failures** | **187** (`172`) | | **Unexpected power losses** | **32** (`174`) | | Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) | | UDMA CRC errors | 0 (`199`) — cabling is clean | **Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**, not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value; the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent uncorrectable counters instead. **Verdict: usable, but not for production state.** Nothing says imminent failure — no pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011 consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad blocks, logged **187** erase failures, and taken **32** unclean power losses **with no power-loss protection**, in a single failure domain. etcd is the worst possible tenant for it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how a member corrupts — this drive has already logged 32 of them. (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see `DECISIONS-LOG.md`. It is not part of this verdict.) - [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices* (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them. Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** — prod must override them in `group_vars`. On a single-disk node there is no second device at all, so one of: - **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own spindle. - **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`. Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by. - **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one physical failure domain. - [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry. - [ ] Record the chosen device paths here and set them in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately `/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full `node-baseline` run against a node still carrying those placeholders now stops on an explicit assert naming this section, instead of failing obscurely at `mkfs` time. - [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline` is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk tasks this decision blocks — see §5's drive-health box for the exact command. ## 3. OS install (per node) - [ ] **Debian 13 (trixie)**, matching dev exactly — same kernel series, same package baseline. - [ ] Hostname `k3s-prod-N`. - [ ] Partition per the §2 decision. - [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into `roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now), so the same role covers bare metal. Nothing to port. ## 4. Network (per node) - [ ] **Static IP** on the prod LAN, recorded in §0. Node 1 is `192.168.2.10/24`, gateway `192.168.2.1`. - [x] **Prod address plan — DECIDED (operator, 2026-08-31).** | Role | Address | Owned by | |---|---|---| | `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory | | **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) | | **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) | | **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) | Two deliberate properties, both mirroring dev: - The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way round (`.49` control-plane, pool `.53-.60`). - **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape (`GATEWAY_VIP` `.53` = first of `.53-.60`). - [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before the build, and confirm nothing else on the network already answers on them. - [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.** Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single upstream is a known SPOF that has already bitten this project — it broke containerd image pulls and CoreDNS forwarding on dev. The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in `roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes `/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed `/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager owns it), and configures the ordered upstream list either way. The earlier `fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie nodes, so it would have broken `node-baseline.yml` on every host. What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets `node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid `BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert stops the play on it before writing anything. Choosing the second upstream is a **network decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would resolve reliably but would also bypass any local zone and any filtering the gateway applies. Decide between — and record the choice HERE, in the same change that edits the group_vars: - a second on-LAN resolver on `192.168.2.x`; - the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified — prod sits on the workstation side of the pfSense boundary); - an explicit, written-down acceptance of a public resolver as fallback-only. Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`) are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed servers, so keeping them means listing them explicitly. `node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's local domain is decided. - [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT — prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind. ## 5. Access (per node) — **blocks all Ansible** - [ ] **`k3s` user with the `home_network` SSH key.** Working on node 1. - [ ] **Password-protected sudo, driven by Ansible `become` + a vaulted password.** **Prod does NOT use NOPASSWD sudo** (operator decision, 2026-08-31). Dev nodes do; prod deliberately keeps the sudo password in place and supplies it to Ansible as an **ansible-vault-encrypted** var, exactly like the existing `hypervisor` and `proxmox_hosts` groups: ``` inventory/group_vars/k3s_prod/ README.md how to (re)create the vault — committed vault.yml ansible_become_password: "..." — committed, ENCRYPTED ``` The passphrase that decrypts it lives **outside the repo** at `~/.config/home-k3s/ansible-vault-pass` (`ansible.cfg :: vault_password_file`), so plays still run unattended without a plaintext secret ever touching git. ```bash ansible-vault create inventory/group_vars/k3s_prod/vault.yml # ansible_become_password: ... ``` Prod nodes must therefore set `ansible_become: true` in their group_vars and must **not** inherit any passwordless-sudo assumption from `group_vars/all.yml`. - [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead (the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so the placeholder device paths are never even templated): ```bash cd infra/ansible ansible-playbook playbooks/node-baseline.yml \ -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1 ``` (Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading the drive is then: ```bash sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors ``` Record the result in §2 and close the layout decision in the same change that replaces the two `node_baseline_*_device` placeholders. ## 6. Ansible baseline & cluster join Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars differ, plus a target-group override on the command line. ### How prod is targeted (done, 2026-08-31) Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` / `k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately kept separate because Ansible resolves them at different times: | Layer | Knob | Set via | |---|---|---| | Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there | | Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** | So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after `--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**. - [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached **directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping. (Nodes 2/3 stay commented out until they are physically on the bench.) - [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`), `cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the §2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which are still BLOCKED placeholders. - [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until it exists **no** prod play can escalate, and `--check` against prod cannot work either. - [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read: ```bash ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages ``` - [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare metal — the role already only *starts* it when the virtio channel exists, so it degrades cleanly.) - [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary` → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three nodes are present** — see §8. - [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the prod LAN. - [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The bootstrap values and the HelmRelease **must stay in parity** (`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config (`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the dev one the dev container exports as `$KUBECONFIG`. - [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`. - [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**: `roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a target-group change. Tracked with the other repo blockers in §7. - [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015). `playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=`, but there is deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted PIA credentials — the existing vault is scoped to the dev group. ## 7. Repo blockers — **prod cannot be reconciled until these land** `clusters/prod` is design-only today and **would build wrong**. R-ENV-02 promises dev and prod differ *only* by `cluster-vars` values; in practice dev LAN addresses are hardcoded into base manifests: | Hardcoded value | Where | Prod needs | |---|---|---| | `k8sServiceHost: 192.168.1.50` | `platform/cilium/helmrelease.yaml:50`, mirrored in `values.yaml:16` | prod primary node IP | | LB pool `192.168.1.53–.60` | `platform-config/cilium/l2-ip-pool.yaml:15` | a prod-LAN pool | | `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address | | `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address | | `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target | - [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both* overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the header comment in `clusters/base/cluster-vars.yaml`). `platform`, `platform-config`, the app waves, `apps-vms` and `crafty` all already carry `postBuild.substituteFrom: cluster-vars`, so the seam exists. - [ ] **`clusters/prod` must set `GATEWAY_VIP`.** It currently doesn't, so prod would inherit the base default `10.73.0.240` — an address from the **retired libvirt network**. The base file's own comment flags this as owed "when hardware is built." - [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`. - [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing the `dependsOn` fix. Only the first is now closed: - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block forever on a dependency that does not exist and nothing downstream would reconcile. - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only. - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and will be decommissioned once prod is stable. Recorded consequence: once prod is live, **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window. The URL is patched in but **cannot work on a cold prod cluster**, for two independent reasons found 2026-09-01: 1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth. **This is the decommission plan's first task, not its last.** 2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`: prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve `gitea.home.keithjackson.dev` to itself and never reach a repository. Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own Phase-6 cutover. - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03). ## 8. Build sequence (operator decision, 2026-08-31: **wait for all three nodes**) Node 1 is baselined now; the cluster is **not** initialized until nodes 2 and 3 are on the bench. This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a single-server-etcd window, and avoids a later member-add dance. 1. Per-node §1–§5 on each of the three boxes. 2. §7 repo work lands on `main` with the gate green. 3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`. 4. `flux bootstrap ... --path=clusters/prod`. 5. Conformance against prod (§9). ## 9. Exit criteria (Phase 7) - [ ] Prod runs the same config as dev via the prod overlay, **no base edits** (R-ENV-02). - [ ] Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01). - [ ] Full **automated** prod DR drill (R-STATE-03) — `definition-of-done.md` requires this for prod specifically. - [ ] Monitoring and alerts live on prod. - [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).