home-network-k3s-impl/infra/PROD-PROVISIONING.md

Prod bare-metal provisioning checklist (T-7.1)

Task: T-7.1 · Workstream: WS-INFRA · Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03 Status: Phase 7 opened by the operator 2026-08-31 — prep only. Nothing is installed on prod hardware until the blockers in §7 are closed and make -C ci gate is green.

Entry-gate note (recorded, not waived). roadmap/phases.md gates Phase 7 on "Phases 1–6 proven in dev/test; operator declares prod hardware ready." The operator has declared hardware ready. The first condition is not met: conformance is 37/51, and the uncovered set includes R-ENV-01/02/03 — the very requirements this document serves. We proceed with prep work knowingly; see DECISIONS-LOG.md (2026-08-31 Phase-7 entry).

Prod is 3× mini-PC bare metal (spec docs/02). The dev path (OpenTofu → Proxmox VM → cloud-init) does not apply: there is no hypervisor API to call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse the same Ansible roles and the same Flux base as dev — that reuse is the R-ENV-02 obligation.

---

0. Node inventory

Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.

	node 1	node 2	node 3
Hostname	k3s-prod-1	k3s-prod-2	k3s-prod-3
LAN IP	192.168.2.10	—	—
Model	HP EliteDesk 800 G5 Desktop Mini
CPU	i7-9700T, 8c/8t, VT-x
RAM	31 GiB
Disks	1× 477 GB SATA SSD (C400-MTFDDAC512M)
OS	Debian 13 (trixie), kernel 6.12
Firmware	HP R21 Ver. 02.26.00
/dev/kvm	present

Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, two dedicated data disks (Longhorn + etcd).

---

1. Firmware / BIOS (per node, before OS install)

• Virtualization enabled — VT-x and VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with /dev/kvm present and grep -cE 'vmx|svm' /proc/cpuinfo non-zero.
• AC power recovery = Power On. A homelab node must return after a power cut without a human pressing the button. (docs/02 calls out wake/boot behaviour explicitly.)
• Wake-on-LAN enabled on the onboard NIC (remote recovery when the box is off).
• Boot order — internal disk first; USB above it only while imaging. Disable network boot unless PXE is adopted.
• Secure Boot — record the setting. k3s itself is fine either way; note it so all three nodes match.
• Firmware version recorded in §0 and consistent across all three nodes.

2. Disk layout — OPEN DECISION, blocks the build

docs/02 requires the OS-vs-Longhorn device split to be documented before prod build. Dev went further and added a third device for etcd after HDD contention starved etcd WAL fsync (docs/plans/ssd-etcd-isolation.md).

Node 1 as-found is a single 477 GB SATA SSD carrying everything:

sda  477G  C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
├─sda1 976M  /boot/efi
├─sda2  30G  /
├─sda3  12G  /var
├─sda4 977M  swap
└─sda5 433G  /srv          <- the only large free space

Consequences to settle before imaging nodes 2 and 3:

• Drive health checked first (operator decision, 2026-08-31). See §5 for the command — needs root. A ~15-year-old consumer SSD is not an obvious place for prod state.
• Decide the layout. roles/node_baseline expects block devices (node_baseline_longhorn_device, node_baseline_etcd_device) and makes a filesystem on them. Its defaults /dev/vdb and /dev/vdc are virtio names that do not exist on bare metal — prod must override them in group_vars. On a single-disk node there is no second device at all, so one of: - Add a second drive (M.2 NVMe if the chassis has a free slot; verify physically — only a SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own spindle. - Use /dev/sda5 (the 433 GB /srv partition) as the Longhorn device, etcd stays on /. Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by. - Repartition into dedicated OS / etcd / Longhorn partitions. Better isolation, still one physical failure domain.
• All three nodes get the identical layout. Longhorn replica placement assumes symmetry.
• Record the chosen device paths here and set them in inventory/group_vars/k3s_prod.yml.

3. OS install (per node)

• Debian 13 (trixie), matching dev exactly — same kernel series, same package baseline.
• Hostname k3s-prod-N.
• Partition per the §2 decision.
• No cloud-init needed. Dev's first-boot work was deliberately moved into roles/node_baseline when provisioning became API-only (infra/cloud-init/ is a README now), so the same role covers bare metal. Nothing to port.

4. Network (per node)

• Static IP on the prod LAN, recorded in §0. Node 1 is 192.168.2.10/24, gateway 192.168.2.1.
• Reserve the VIP addresses before the build — prod needs its own kube-vip control-plane VIP and its own Cilium L2 announcement pool for ingress. Neither exists yet (§7).
• Resolver: configure at least two upstreams. Node 1 currently has a single IPv4 nameserver (192.168.2.1). A single upstream is a known SPOF that has already bitten this project — it broke containerd image pulls and CoreDNS forwarding on dev. A fix exists but is stranded unmerged on fix/node-dns-resilient-upstream; land it, then apply the same pattern here.
• Confirm the prod LAN's relationship to the dev LAN (192.168.1.x) and to the pfSense NAT — prod on 192.168.2.x sits on the workstation side of the boundary dev lives behind.

5. Access (per node) — blocks all Ansible

• k3s user with the home_network SSH key. Working on node 1.
• Password-protected sudo, driven by Ansible become + a vaulted password. Prod does NOT use NOPASSWD sudo (operator decision, 2026-08-31). Dev nodes do; prod deliberately keeps the sudo password in place and supplies it to Ansible as an ansible-vault-encrypted var, exactly like the existing hypervisor and proxmox_hosts groups: inventory/group_vars/k3s_prod/ README.md how to (re)create the vault — committed vault.yml ansible_become_password: "..." — committed, ENCRYPTED The passphrase that decrypts it lives outside the repo at ~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays still run unattended without a plaintext secret ever touching git. bash ansible-vault create inventory/group_vars/k3s_prod/vault.yml # ansible_become_password: ... Prod nodes must therefore set ansible_become: true in their group_vars and must not inherit any passwordless-sudo assumption from group_vars/all.yml.
• Drive health baseline (§2), needs root: bash sudo apt-get update && sudo apt-get install -y smartmontools sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors

6. Ansible baseline & cluster join

Reuses the dev roles unchanged — that reuse is R-ENV-02. Only inventory and group_vars differ.

• Add a k3s_prod inventory group (k3s_prod_primary / k3s_prod_secondary). Prod is reached directly on the LAN — no NAT port-forward hop, unlike dev's 192.168.2.102:225x mapping.
• group_vars/k3s_prod.yml: kube_vip_address, cluster_controller_api_server, and the §2 device overrides for node_baseline_longhorn_device / node_baseline_etcd_device.
• Run playbooks/node-baseline.yml → packages, kernel modules, sysctls, iscsid, data-disk filesystems + mounts. (qemu-guest-agent is a no-op on bare metal — the role already only starts it when the virtio channel exists, so it degrades cleanly.)
• Run playbooks/k3s.yml → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. Only once all three nodes are present — see §8.
• Run playbooks/kube-vip.yml → control-plane VIP on the prod LAN.
• Run playbooks/cilium.yml → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The bootstrap values and the HelmRelease must stay in parity (kubeProxyReplacement, k8sServiceHost/Port) or the two layers fight over CNI config (docs/02).
• Decide whether prod runs a VPN egress gateway node (dev uses k3s-dev-3, ADR-0015).

7. Repo blockers — prod cannot be reconciled until these land

clusters/prod is design-only today and would build wrong. R-ENV-02 promises dev and prod differ only by cluster-vars values; in practice dev LAN addresses are hardcoded into base manifests:

Hardcoded value	Where	Prod needs
k8sServiceHost: 192.168.1.50	platform/cilium/helmrelease.yaml:50, mirrored in values.yaml:16	prod primary node IP
LB pool 192.168.1.53–.60	platform-config/cilium/l2-ip-pool.yaml:15	a prod-LAN pool
192.168.1.55 (crafty)	platform/crafty/service-l4.yaml	prod-LAN address
192.168.1.54 (UniFi VM)	apps/vms/unifi/50-service-l4.yaml	prod-LAN address
192.168.1.9:9000 (S3 backup)	platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,semaphore}-db.yaml	reachable prod backup target

• Tokenize each into a cluster-vars knob with a base default, then set the value in both overlays. Adding a knob before referencing ${KNOB} is the sanctioned mechanism (see the header comment in clusters/base/cluster-vars.yaml). platform, platform-config, the app waves, apps-vms and crafty all already carry postBuild.substituteFrom: cluster-vars, so the seam exists.
• clusters/prod must set GATEWAY_VIP. It currently doesn't, so prod would inherit the base default 10.73.0.240 — an address from the retired libvirt network. The base file's own comment flags this as owed "when hardware is built."
• Dev overlay sets every new knob to today's exact value so rendered dev output is byte-identical and the live cluster sees zero drift. Verify with make -C ci render-golden.
• Confirm prod's backup target: dev's S3 (versitygw on the Proxmox host, 192.168.1.9) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).

8. Build sequence (operator decision, 2026-08-31: wait for all three nodes)

Node 1 is baselined now; the cluster is not initialized until nodes 2 and 3 are on the bench. This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a single-server-etcd window, and avoids a later member-add dance.

1. Per-node §1–§5 on each of the three boxes.
2. §7 repo work lands on main with the gate green.
3. node-baseline → k3s (all three) → kube-vip → cilium.
4. flux bootstrap ... --path=clusters/prod.
5. Conformance against prod (§9).

9. Exit criteria (Phase 7)

• Prod runs the same config as dev via the prod overlay, no base edits (R-ENV-02).
• Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01).
• Full automated prod DR drill (R-STATE-03) — definition-of-done.md requires this for prod specifically.
• Monitoring and alerts live on prod.
• The conformance suite passes green against the prod cluster, not just dev (R-TEST-03).