home-network-k3s-impl/infra/PROD-PROVISIONING.md

Prod bare-metal provisioning checklist (T-7.1)

Task: T-7.1 · Workstream: WS-INFRA · Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03 Status: Phase 7 opened by the operator 2026-08-31 — prep only. Nothing is installed on prod hardware until the blockers in §7 are closed and make -C ci gate is green.

Entry-gate note (recorded, not waived). roadmap/phases.md gates Phase 7 on "Phases 1–6 proven in dev/test; operator declares prod hardware ready." The operator has declared hardware ready. The first condition is not met: conformance is 37/51, and the uncovered set includes R-ENV-01/02/03 — the very requirements this document serves. We proceed with prep work knowingly; see DECISIONS-LOG.md (2026-08-31 Phase-7 entry).

Prod is 3× mini-PC bare metal (spec docs/02). The dev path (OpenTofu → Proxmox VM → cloud-init) does not apply: there is no hypervisor API to call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse the same Ansible roles and the same Flux base as dev — that reuse is the R-ENV-02 obligation.

---

0. Node inventory

Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.

	node 1	node 2	node 3
Hostname	k3s-prod-1	k3s-prod-2	k3s-prod-3
LAN IP	192.168.2.10	192.168.2.11	192.168.2.12
Model	HP EliteDesk 800 G5 Desktop Mini
CPU	i7-9700T, 8c/8t, VT-x
RAM	31 GiB
Disks	1× 477 GB SATA SSD (C400-MTFDDAC512M)
OS	Debian 13 (trixie), kernel 6.12
Firmware	HP R21 Ver. 02.26.00
/dev/kvm	present

Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, two dedicated data disks (Longhorn + etcd).

---

1. Firmware / BIOS (per node, before OS install)

• Virtualization enabled — VT-x and VT-d. Required for KubeVirt (R-PLAT-03); verify after boot with /dev/kvm present and grep -cE 'vmx|svm' /proc/cpuinfo non-zero.
• AC power recovery = Power On. A homelab node must return after a power cut without a human pressing the button. (docs/02 calls out wake/boot behaviour explicitly.)
• Wake-on-LAN enabled on the onboard NIC (remote recovery when the box is off).
• Boot order — internal disk first; USB above it only while imaging. Disable network boot unless PXE is adopted.
• Secure Boot — record the setting. k3s itself is fine either way; note it so all three nodes match.
• Firmware version recorded in §0 and consistent across all three nodes.

2. Disk layout — OPEN DECISION, blocks the build

docs/02 requires the OS-vs-Longhorn device split to be documented before prod build. Dev went further and added a third device for etcd after HDD contention starved etcd WAL fsync (docs/plans/ssd-etcd-isolation.md).

Node 1 as-found is a single 477 GB SATA SSD carrying everything:

sda  477G  C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
├─sda1 976M  /boot/efi
├─sda2  30G  /
├─sda3  12G  /var
├─sda4 977M  swap
└─sda5 433G  /srv          <- the only large free space

Consequences to settle before imaging nodes 2 and 3:

• Drive health MEASURED 2026-09-01 (smartctl -a /dev/sda, via the baseline package floor — see §5). Raw attributes, so the numbers are auditable:

  | | |
  |---|---|
  | Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) |
  | SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` |
  | Power-on hours | 6,814 (~284 days) · 147 power cycles |
  | **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) |
  | **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) |
  | **Erase failures** | **187** (`172`) |
  | **Unexpected power losses** | **32** (`174`) |
  | Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) |
  | UDMA CRC errors | 0 (`199`) — cabling is clean |
  
  **Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector
  count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its
  normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**,
  not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value;
  the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent
  uncorrectable counters instead.
  
  **Verdict: usable, but not for production state.** Nothing says imminent failure — no
  pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011
  consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad
  blocks, logged **187** erase failures, and taken **32** unclean power losses **with no
  power-loss protection**, in a single failure domain. etcd is the worst possible tenant for
  it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how
  a member corrupts — this drive has already logged 32 of them.
  (Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
  `DECISIONS-LOG.md`. It is not part of this verdict.)
  

• Decide the layout.- [ ] Decide the layout. roles/node_baseline expects block devices (node_baseline_longhorn_device, node_baseline_etcd_device) and makes a filesystem on them. Its defaults /dev/vdb and /dev/vdc are virtio names that do not exist on bare metal — prod must override them in group_vars. On a single-disk node there is no second device at all, so one of: - Add a second drive (M.2 NVMe if the chassis has a free slot; verify physically — only a SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own spindle. - Use /dev/sda5 (the 433 GB /srv partition) as the Longhorn device, etcd stays on /. Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by. - Repartition into dedicated OS / etcd / Longhorn partitions. Better isolation, still one physical failure domain.

• All three nodes get the identical layout. Longhorn replica placement assumes symmetry.

• Record the chosen device paths here and set them in infra/ansible/inventory/group_vars/k3s_prod/vars.yml (they are deliberately /dev/BLOCKED-pending-disk-decision-… placeholders today — do not guess). A full node-baseline run against a node still carrying those placeholders now stops on an explicit assert naming this section, instead of failing obscurely at mkfs time.

• The SMART-vs-smartmontools circular dependency is broken (2026-08-31). roles/node_baseline is tagged packages / system / disks, so the package floor can land without the disk tasks this decision blocks — see §5's drive-health box for the exact command.

3. OS install (per node)

• Debian 13 (trixie), matching dev exactly — same kernel series, same package baseline.
• Hostname k3s-prod-N.
• Partition per the §2 decision.
• No cloud-init needed. Dev's first-boot work was deliberately moved into roles/node_baseline when provisioning became API-only (infra/cloud-init/ is a README now), so the same role covers bare metal. Nothing to port.

4. Network (per node)

• Static IP on the prod LAN, recorded in §0. Node 1 is 192.168.2.10/24, gateway 192.168.2.1.

• Prod address plan — DECIDED (operator, 2026-08-31).

  | Role | Address | Owned by |
  |---|---|---|
  | `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory |
  | **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) |
  | **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) |
  | **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) |
  
  Two deliberate properties, both mirroring dev:
  - The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and
    Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way
    round (`.49` control-plane, pool `.53-.60`).
  - **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a
    defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool
    was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape
    (`GATEWAY_VIP` `.53` = first of `.53-.60`).
  

• Reserve all of 192.168.2.9 and .20-.30 outside any DHCP range on the prod LAN before the build, and confirm nothing else on the network already answers on them.

• Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks node-baseline.yml --tags dns. Node 1 has a single IPv4 nameserver, 192.168.2.1, which is also the LAN gateway. A single upstream is a known SPOF that has already bitten this project — it broke containerd image pulls and CoreDNS forwarding on dev.

  The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in
  `roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes
  `/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed
  `/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager
  owns it), and configures the ordered upstream list either way. The earlier
  `fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan
  drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie
  nodes, so it would have broken `node-baseline.yml` on every host.
  
  What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets
  `node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid
  `BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert
  stops the play on it before writing anything. Choosing the second upstream is a **network
  decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would
  resolve reliably but would also bypass any local zone and any filtering the gateway applies.
  Decide between — and record the choice HERE, in the same change that edits the group_vars:
  - a second on-LAN resolver on `192.168.2.x`;
  - the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified —
    prod sits on the workstation side of the pfSense boundary);
  - an explicit, written-down acceptance of a public resolver as fallback-only.
  
  Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`)
  are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed
  servers, so keeping them means listing them explicitly.
  `node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's
  local domain is decided.
  

• Confirm the prod LAN's relationship to the dev LAN (192.168.1.x) and to the pfSense NAT — prod on 192.168.2.x sits on the workstation side of the boundary dev lives behind.

5. Access (per node) — blocks all Ansible

• k3s user with the home_network SSH key. Working on node 1.
• Password-protected sudo, driven by Ansible become + a vaulted password. Prod does NOT use NOPASSWD sudo (operator decision, 2026-08-31). Dev nodes do; prod deliberately keeps the sudo password in place and supplies it to Ansible as an ansible-vault-encrypted var, exactly like the existing hypervisor and proxmox_hosts groups: inventory/group_vars/k3s_prod/ README.md how to (re)create the vault — committed vault.yml ansible_become_password: "..." — committed, ENCRYPTED The passphrase that decrypts it lives outside the repo at ~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays still run unattended without a plaintext secret ever touching git. bash ansible-vault create inventory/group_vars/k3s_prod/vault.yml # ansible_become_password: ... Prod nodes must therefore set ansible_become: true in their group_vars and must not inherit any passwordless-sudo assumption from group_vars/all.yml.
• Drive health baseline (§2). Do not hand-install the tooling — package installation is the node_baseline role's job, never a manual apt-get on a node (operator ruling, 2026-08-31; same click-ops-never principle as the rest of the repo). smartmontools is part of node_baseline_packages, so it arrives with the baseline play — but a full baseline would hit the disk tasks §2 has not decided yet. Run the packages-only baseline instead (the disks-tagged tasks, including the BLOCKED-device assert, are not selected at all, so the placeholder device paths are never even templated): bash cd infra/ansible ansible-playbook playbooks/node-baseline.yml \ -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1 (Needs group_vars/k3s_prod/vault.yml to exist first — the become password above.) Reading the drive is then: bash sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors Record the result in §2 and close the layout decision in the same change that replaces the two node_baseline_*_device placeholders.

6. Ansible baseline & cluster join

Reuses the dev roles unchanged — that reuse is R-ENV-02. Only inventory and group_vars differ, plus a target-group override on the command line.

How prod is targeted (done, 2026-08-31)

Every node-host play used to hardcode the literal dev groups k3s_servers / k3s_primary / k3s_secondary, so nothing here could select prod at all. Two knobs now close that, deliberately kept separate because Ansible resolves them at different times:

Layer	Knob	Set via
Which hosts a play runs on	k3s_hosts, k3s_primary_hosts, k3s_secondary_hosts (each defaults to the dev group)	-e extra vars only — a play's hosts: is templated before any host exists, so group_vars is not readable there
Which group a role means by "my primary" / "my servers"	k3s_primary_group, k3s_servers_group, cluster_kubeconfig_name	group_vars — dev in all.yml, prod in k3s_prod/vars.yml; no per-run flag

So roles/k3s_server finds the prod --cluster-init founder, builds the prod join target and tls-san list, and roles/cilium fetches the prod kubeconfig, with no role fork. Running with no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after --list-hosts diff on every playbook). Full detail: infra/ansible/README.md → Targeting prod.

• Add a k3s_prod inventory group (k3s_prod_primary / k3s_prod_secondary). Prod is reached directly on the LAN — no NAT port-forward hop, unlike dev's 192.168.2.102:225x mapping. (Nodes 2/3 stay commented out until they are physically on the bench.)
• inventory/group_vars/k3s_prod/vars.yml: kube_vip_address (192.168.2.9), cluster_controller_api_server, k3s_api_host, the group-name overrides above — and the §2 device overrides for node_baseline_longhorn_device / node_baseline_etcd_device, which are still BLOCKED placeholders.
• inventory/group_vars/k3s_prod/vault.yml — the vaulted ansible_become_password (§5). Until it exists no prod play can escalate, and --check against prod cannot work either.
• Run the packages-only baseline first, to unblock the §2 drive-health read: bash ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
• Run the full playbooks/node-baseline.yml -e k3s_hosts=k3s_prod → packages, kernel modules, sysctls, iscsid, data-disk filesystems + mounts. Blocked until §2 is decided: it asserts on the placeholder device paths and stops. (qemu-guest-agent is a no-op on bare metal — the role already only starts it when the virtio channel exists, so it degrades cleanly.)
• Run playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. Only once all three nodes are present — see §8.
• Run playbooks/kube-vip.yml -e k3s_hosts=k3s_prod → control-plane VIP (192.168.2.9) on the prod LAN.
• Run playbooks/cilium.yml -e k3s_hosts=k3s_prod → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The bootstrap values and the HelmRelease must stay in parity (kubeProxyReplacement, k8sServiceHost/Port) or the two layers fight over CNI config (docs/02). Note the prod kubeconfig is fetched to .cache/prod.kubeconfig, never over the dev one the dev container exports as $KUBECONFIG.
• Run playbooks/verify.yml -e k3s_hosts=k3s_prod.
• playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod retargets but is not yet correct: roles/flux_bootstrap shells out to bootstrap/flux/bootstrap.sh, which bootstraps --path=clusters/dev. Prod needs --path=clusters/prod — a bootstrap/ change, not a target-group change. Tracked with the other repo blockers in §7.
• Decide whether prod runs a VPN egress gateway node (dev uses k3s-dev-3, ADR-0015). playbooks/vpn.egress.gateway.yml takes -e vpn_egress_gateway_hosts=<group>, but there is deliberately no prod egress group in the inventory yet, and it would need its own vaulted PIA credentials — the existing vault is scoped to the dev group.

7. Repo blockers — prod cannot be reconciled until these land

clusters/prod is design-only today and would build wrong. R-ENV-02 promises dev and prod differ only by cluster-vars values; in practice dev LAN addresses are hardcoded into base manifests:

Hardcoded value	Where	Prod needs
k8sServiceHost: 192.168.1.50	platform/cilium/helmrelease.yaml:50, mirrored in values.yaml:16	prod primary node IP
LB pool 192.168.1.53–.60	platform-config/cilium/l2-ip-pool.yaml:15	a prod-LAN pool
192.168.1.55 (crafty)	platform/crafty/service-l4.yaml	prod-LAN address
192.168.1.54 (UniFi VM)	apps/vms/unifi/50-service-l4.yaml	prod-LAN address
192.168.1.9:9000 (S3 backup)	platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,semaphore}-db.yaml	reachable prod backup target

• Tokenize each into a cluster-vars knob with a base default, then set the value in both overlays. Adding a knob before referencing ${KNOB} is the sanctioned mechanism (see the header comment in clusters/base/cluster-vars.yaml). platform, platform-config, the app waves, apps-vms and crafty all already carry postBuild.substituteFrom: cluster-vars, so the seam exists.
• clusters/prod must set GATEWAY_VIP. It currently doesn't, so prod would inherit the base default 10.73.0.240 — an address from the retired libvirt network. The base file's own comment flags this as owed "when hardware is built."
• Dev overlay sets every new knob to today's exact value so rendered dev output is byte-identical and the live cluster sees zero drift. Verify with make -C ci render-golden.
• clusters/prod is NOT bootstrappable as-is — three gaps, found 2026-09-01 while landing the dependsOn fix. Only the first is now closed: - [x] Root Kustomization — clusters/prod/sync.yaml authored, mirroring dev. This became load-bearing, not cosmetic: clusters/base/platform.yaml now carries dependsOn: [flux-system], so without a root object on prod, platform would block forever on a dependency that does not exist and nothing downstream would reconcile. - [x] SOPS decryption — DONE. Operator decision 2026-09-01: prod uses the SAME age key as dev. Eight decryption: {provider: sops, secretRef: sops-age} patches mirror dev exactly. The sops-age Secret is still created OUT OF BAND on the prod cluster (bootstrap/sops/install-age-key.sh) — the overlay references it by name only. - [~] GitRepository source — BRANCH settled, URL still blocked. Operator decision 2026-09-01: prod tracks main, the same branch as dev, because dev is temporary and will be decommissioned once prod is stable. Recorded consequence: once prod is live, every merge to main reconciles BOTH clusters — no promotion step, no soak window. The URL is patched in but cannot work on a cold prod cluster, for two independent reasons found 2026-09-01: 1. Self-hosting circularity — Gitea runs on the DEV cluster, the one slated for decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth. This is the decommission plan's first task, not its last. 2. Split-horizon sends it to the wrong place — coredns-custom answers the entire home.keithjackson.dev zone with ${GATEWAY_VIP}, which on prod is 192.168.2.20: prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository. Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a cutover once Gitea runs on prod — the same sequence docs/05 describes for dev's own Phase-6 cutover.
• Confirm prod's backup target: dev's S3 (versitygw on the Proxmox host, 192.168.1.9) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).

8. Build sequence (operator decision, 2026-08-31: wait for all three nodes)

Node 1 is baselined now; the cluster is not initialized until nodes 2 and 3 are on the bench. This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a single-server-etcd window, and avoids a later member-add dance.

1. Per-node §1–§5 on each of the three boxes.
2. §7 repo work lands on main with the gate green.
3. node-baseline → k3s (all three) → kube-vip → cilium.
4. flux bootstrap ... --path=clusters/prod.
5. Conformance against prod (§9).

9. Exit criteria (Phase 7)

• Prod runs the same config as dev via the prod overlay, no base edits (R-ENV-02).
• Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01).
• Full automated prod DR drill (R-STATE-03) — definition-of-done.md requires this for prod specifically.
• Monitoring and alerts live on prod.
• The conformance suite passes green against the prod cluster, not just dev (R-TEST-03).