Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity
Files
feat/prod-node-2
home-network-k3s-impl/infra/PROD-PROVISIONING.md
T
jakman85 9f7052ab2b docs(prod): fill in node-2 inventory; drop stale investigate bullets 
Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier
edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice).

Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a
~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2
is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition
layout — very different storage. §2 requires an identical disk layout across all three nodes
because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the
disk decision will be made.

Serves: R-ENV-03. Task: T-7.1.
2026-09-01 21:45:37 -05:00

467 lines
30 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Prod bare-metal provisioning checklist (T-7.1)
**Task:** T-7.1 · **Workstream:** WS-INFRA · **Serves:** R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03
**Status:** Phase 7 **opened by the operator 2026-08-31** — *prep only*. Nothing is installed on prod
hardware until the blockers in §7 are closed and `make -C ci gate` is green.
> **Entry-gate note (recorded, not waived).** `roadmap/phases.md` gates Phase 7 on *"Phases 1–6
> proven in dev/test; operator declares prod hardware ready."* The operator has declared hardware
> ready. The first condition is **not** met: conformance is 37/51, and the uncovered set includes
> **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work
> knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry).
Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)).
The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to
call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse
the *same* Ansible roles and the *same* Flux base as dev — that reuse is the R-ENV-02 obligation.
---
## 0. Node inventory
Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
| | node 1 | node 2 | node 3 |
|---|---|---|---|
| Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
| LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | |
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | |
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | |
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | |
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | |
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | |
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | |
| `/dev/kvm` | present | present | |
| Serial | MXL1052NGN | MXL0073949 | |
**The nodes are NOT hardware-symmetric.** Node 1 has a ~2011 SATA SSD with real wear; node 2 has a
healthy modern NVMe. §2 requires an identical disk layout across all three because Longhorn replica
placement assumes symmetry — so this is a live input to the disk decision, not a footnote. Node 2's
NVMe is the drive node 1 should have had.
Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).
---
## 0b. OPEN HARDWARE FAULT — `k3s-prod-2` can only address ~2.6 GiB of its 16 GB
Found 2026-09-01 by `playbooks/node-health.yml` on node 2's first health run. **Do not install
more RAM until this is understood** — see the warning at the end.
| Evidence | Reading |
|---|---|
| DMI (`dmidecode -t memory`) | **16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s**, DIMM3 empty |
| BIOS e820 map | reports ~16 GiB usable, incl. `0x100000000-0x42f7fffff` |
| Kernel summary | `Memory: 2568544K/16546088K available (13958980K reserved)` |
| `last_pfn` | revised **down** from `0x42f800` (~16.7 GiB) to `0xc3c10` (~3.06 GiB) in early boot |
| Zones | `Normal` zone `present 3340288` pages (~12.7 GiB) but **`managed 0`** |
| `MemTotal` | **2,753,284 kB ≈ 2.6 GiB** |
| Memory blocks | 127 online, 128 MiB block size |
**The DIMM is healthy.** It is present, correctly identified, and running at its rated speed. The
kernel *sees* all 16 GB and then reserves 13.3 GiB of it. Everything below ~3 GiB is usable and
everything above 4 GiB is present-but-unmanaged — the classic signature of **memory remapping
above 4 GB not being applied**, not of failing memory.
Node 1 is the control: **identical** HP EliteDesk 800 G5 Mini, **identical** BIOS R21 Ver. 02.26.00,
identical i7-9700T — and it addresses its full 31 GiB. So this is not a firmware-version defect; it
is per-machine state.
**BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg`
kernel driver, which exposes **215 BIOS attributes** read/write at
`/sys/class/firmware-attributes/hp-bioscfg/attributes/<name>/current_value`. No BIOS passwords
are set (`authentication/*/is_enabled` = 0).
**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was
made on 2026-09-01 and the FIRMWARE refused it:
`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`.
HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent
— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`.
Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS
Setup Password is deliberately provisioned (see §1).
What the comparison ruled OUT as the cause:
- **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and
`Processor 1 DIMM1=16 GB Samsung` / `DIMM3=Empty`. The firmware is not under-counting.
- **No memory-remap / above-4G-decoding attribute exists** on this platform — not present on
either node, so it cannot be a mis-set option.
- **MTRRs are effectively identical** between the nodes and there is **no MTRR trim warning**,
which rules out the classic "BIOS bug: MTRRs don't cover all of memory, losing N MB" path.
- **`/proc/iomem` registers `100000000-42f7fffff` as `System RAM`** and 127 × 128 MiB memory
blocks report `online`. The memory reaches the kernel and is then reserved.
Real BIOS differences found (node 1 = works, node 2 = broken):
| Attribute | node 1 | node 2 |
|---|---|---|
| **After Power Loss** | Power On | **Power Off** ← violates §1 |
| **Fast Boot** | Disable | **Enable** ← leading hypothesis |
| Legacy Boot Options | Disable | Enable |
| TPM Activation Policy | F1 to Boot | Allow user to reject |
| Microsoft/Windows UEFI CA 2023 | Enable | Disable |
| Memory Size | 32768 MB (2×16 GB Micron) | 16384 MB (1×16 GB Samsung) |
**Leading hypothesis: `Fast Boot=Enable`.** HP Fast Boot reuses *cached memory-training data*
rather than retraining at each boot. If the DIMM was fitted while Fast Boot was on, stale
training data is a plausible route to memory being present, inventoried, handed over — and then
reserved. It is also the one memory-relevant setting that differs from the working node. This is
a hypothesis, not a conclusion: it was NOT tested, because changing it requires a firmware write
and a reboot, which needs operator sanction.
To investigate, cheapest first:
- [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1).
**Must be done in F10** — the firmware refuses programmatic writes (above).
- [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and
node 2 currently violates it.
- [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check.
- [ ] Reseat the DIMM; try it in DIMM3; try it in node 1 (swap test) — that isolates DIMM vs slot
vs board in one move.
- [ ] `memtest86+` if the above do not resolve it.
> ⚠ **Do not assume the incoming 32 GB fixes this.** If the fault is the slot, the board, or a BIOS
> setting rather than the DIMM, 32 GB will present exactly the same way and the money is wasted.
> **Verify `MemTotal` after any RAM change** — `playbooks/node-health.yml -e node_health_hosts=k3s_prod`
> reports it. A node that can address 2.6 GiB cannot run k3s + Longhorn + a workload share.
## 1. Firmware / BIOS (per node, before OS install)
> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes
> 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work
> fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found
> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye.
>
> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4,
> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is
> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means
> F10 on the box.
>
> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is
> achievable today with reads alone, and would have caught node 2's drift on its first health run.
> Converging it automatically additionally requires provisioning a BIOS Setup Password on every
> node, which is a real security decision (a stored credential that can rewrite firmware) and is
> the operator's call, not a default.
- [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
- [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
human pressing the button. (`docs/02` calls out wake/boot behaviour explicitly.)
- [ ] **Wake-on-LAN enabled** on the onboard NIC (remote recovery when the box is off).
- [ ] **Boot order** — internal disk first; USB above it only while imaging. Disable network boot
unless PXE is adopted.
- [ ] **Secure Boot** — record the setting. k3s itself is fine either way; note it so all three
nodes match.
- [ ] **Firmware version recorded** in §0 and consistent across all three nodes.
## 2. Disk layout — **OPEN DECISION, blocks the build**
`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
(`docs/plans/ssd-etcd-isolation.md`).
Node 1 as-found is a **single** 477 GB SATA SSD carrying everything:
```
sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
├─sda1 976M /boot/efi
├─sda2 30G /
├─sda3 12G /var
├─sda4 977M swap
└─sda5 433G /srv <- the only large free space
```
Consequences to settle before imaging nodes 2 and 3:
- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
floor — see §5). Raw attributes, so the numbers are auditable:
| | |
|---|---|
| Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) |
| SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` |
| Power-on hours | 6,814 (~284 days) · 147 power cycles |
| **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) |
| **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) |
| **Erase failures** | **187** (`172`) |
| **Unexpected power losses** | **32** (`174`) |
| Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) |
| UDMA CRC errors | 0 (`199`) — cabling is clean |
**Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector
count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its
normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**,
not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value;
the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent
uncorrectable counters instead.
**Verdict: usable, but not for production state.** Nothing says imminent failure — no
pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011
consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad
blocks, logged **187** erase failures, and taken **32** unclean power losses **with no
power-loss protection**, in a single failure domain. etcd is the worst possible tenant for
it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how
a member corrupts — this drive has already logged 32 of them.
(Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
`DECISIONS-LOG.md`. It is not part of this verdict.)
- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
(`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
prod must override them in `group_vars`. On a single-disk node there is no second device at
all, so one of:
- **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
spindle.
- **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
- **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
physical failure domain.
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
- [ ] Record the chosen device paths here and set them in
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
`/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
`node-baseline` run against a node still carrying those placeholders now stops on an explicit
assert naming this section, instead of failing obscurely at `mkfs` time.
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
tasks this decision blocks — see §5's drive-health box for the exact command.
## 3. OS install (per node)
- [ ] **Debian 13 (trixie)**, matching dev exactly — same kernel series, same package baseline.
- [ ] Hostname `k3s-prod-N`.
- [ ] Partition per the §2 decision.
- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
`roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
so the same role covers bare metal. Nothing to port.
## 4. Network (per node)
- [ ] **Static IP** on the prod LAN, recorded in §0. Node 1 is `192.168.2.10/24`, gateway
`192.168.2.1`.
- [x] **Prod address plan — DECIDED (operator, 2026-08-31).**
| Role | Address | Owned by |
|---|---|---|
| `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory |
| **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) |
| **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) |
| **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) |
Two deliberate properties, both mirroring dev:
- The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and
Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way
round (`.49` control-plane, pool `.53-.60`).
- **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a
defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool
was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape
(`GATEWAY_VIP` `.53` = first of `.53-.60`).
- [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before
the build, and confirm nothing else on the network already answers on them.
- [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.**
Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single
upstream is a known SPOF that has already bitten this project — it broke containerd image
pulls and CoreDNS forwarding on dev.
The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in
`roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes
`/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed
`/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager
owns it), and configures the ordered upstream list either way. The earlier
`fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan
drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie
nodes, so it would have broken `node-baseline.yml` on every host.
What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets
`node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid
`BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert
stops the play on it before writing anything. Choosing the second upstream is a **network
decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would
resolve reliably but would also bypass any local zone and any filtering the gateway applies.
Decide between — and record the choice HERE, in the same change that edits the group_vars:
- a second on-LAN resolver on `192.168.2.x`;
- the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified —
prod sits on the workstation side of the pfSense boundary);
- an explicit, written-down acceptance of a public resolver as fallback-only.
Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`)
are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed
servers, so keeping them means listing them explicitly.
`node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's
local domain is decided.
- [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT —
prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind.
## 5. Access (per node) — **blocks all Ansible**
- [ ] **`k3s` user with the `home_network` SSH key.** Working on node 1.
- [ ] **Password-protected sudo, driven by Ansible `become` + a vaulted password.**
**Prod does NOT use NOPASSWD sudo** (operator decision, 2026-08-31). Dev nodes do; prod
deliberately keeps the sudo password in place and supplies it to Ansible as an
**ansible-vault-encrypted** var, exactly like the existing `hypervisor` and `proxmox_hosts`
groups:
```
inventory/group_vars/k3s_prod/
README.md how to (re)create the vault — committed
vault.yml ansible_become_password: "..." — committed, ENCRYPTED
```
The passphrase that decrypts it lives **outside the repo** at
`~/.config/home-k3s/ansible-vault-pass` (`ansible.cfg :: vault_password_file`), so plays still
run unattended without a plaintext secret ever touching git.
```bash
ansible-vault create inventory/group_vars/k3s_prod/vault.yml # ansible_become_password: ...
```
Prod nodes must therefore set `ansible_become: true` in their group_vars and must **not**
inherit any passwordless-sudo assumption from `group_vars/all.yml`.
- [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is
the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling,
2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part
of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline
would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead
(the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so
the placeholder device paths are never even templated):
```bash
cd infra/ansible
ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
```
(Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading
the drive is then:
```bash
sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors
```
Record the result in §2 and close the layout decision in the same change that replaces the
two `node_baseline_*_device` placeholders.
## 6. Ansible baseline & cluster join
Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars
differ, plus a target-group override on the command line.
### How prod is targeted (done, 2026-08-31)
Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` /
`k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately
kept separate because Ansible resolves them at different times:
| Layer | Knob | Set via |
|---|---|---|
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there |
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** |
So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and
tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with
no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after
`--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**.
- [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
**directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping.
(Nodes 2/3 stay commented out until they are physically on the bench.)
- [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`),
`cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the
§2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which
are still BLOCKED placeholders.
- [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until
it exists **no** prod play can escalate, and `--check` against prod cannot work either.
- [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read:
```bash
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
```
- [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel
modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it
asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare
metal — the role already only *starts* it when the virtio channel exists, so it degrades
cleanly.)
- [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`
→ HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three
nodes are present** — see §8.
- [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the
prod LAN.
- [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts
it afterwards. The bootstrap values and the HelmRelease **must stay in parity**
(`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config
(`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the
dev one the dev container exports as `$KUBECONFIG`.
- [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`.
- [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**:
`roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps
`--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a
target-group change. Tracked with the other repo blockers in §7.
- [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015).
`playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=<group>`, but there is
deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted
PIA credentials — the existing vault is scoped to the dev group.
## 7. Repo blockers — **prod cannot be reconciled until these land**
`clusters/prod` is design-only today and **would build wrong**. R-ENV-02 promises dev and prod differ
*only* by `cluster-vars` values; in practice dev LAN addresses are hardcoded into base manifests:
| Hardcoded value | Where | Prod needs |
|---|---|---|
| `k8sServiceHost: 192.168.1.50` | `platform/cilium/helmrelease.yaml:50`, mirrored in `values.yaml:16` | prod primary node IP |
| LB pool `192.168.1.53–.60` | `platform-config/cilium/l2-ip-pool.yaml:15` | a prod-LAN pool |
| `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
| `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
| `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
- [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
header comment in `clusters/base/cluster-vars.yaml`). `platform`, `platform-config`, the app
waves, `apps-vms` and `crafty` all already carry `postBuild.substituteFrom: cluster-vars`, so
the seam exists.
- [ ] **`clusters/prod` must set `GATEWAY_VIP`.** It currently doesn't, so prod would inherit the
base default `10.73.0.240` — an address from the **retired libvirt network**. The base file's
own comment flags this as owed "when hardware is built."
- [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is
byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`.
- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing
the `dependsOn` fix. Only the first is now closed:
- [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became
*load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries
`dependsOn: [flux-system]`, so without a root object on prod, `platform` would block
forever on a dependency that does not exist and nothing downstream would reconcile.
- [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key
as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev
exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster
(`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only.
- [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision
2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and
will be decommissioned once prod is stable. Recorded consequence: once prod is live,
**every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window.
The URL is patched in but **cannot work on a cold prod cluster**, for two independent
reasons found 2026-09-01:
1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
loses its source of truth. **This is the decommission plan's first task, not its last.**
2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire
`home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
`gitea.home.keithjackson.dev` to itself and never reach a repository.
Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a
cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own
Phase-6 cutover.
- [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
## 8. Build sequence (operator decision, 2026-08-31: **wait for all three nodes**)
Node 1 is baselined now; the cluster is **not** initialized until nodes 2 and 3 are on the bench.
This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a
single-server-etcd window, and avoids a later member-add dance.
1. Per-node §1–§5 on each of the three boxes.
2. §7 repo work lands on `main` with the gate green.
3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`.
4. `flux bootstrap ... --path=clusters/prod`.
5. Conformance against prod (§9).
## 9. Exit criteria (Phase 7)
- [ ] Prod runs the same config as dev via the prod overlay, **no base edits** (R-ENV-02).
- [ ] Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01).
- [ ] Full **automated** prod DR drill (R-STATE-03) — `definition-of-done.md` requires this for prod
specifically.
- [ ] Monitoring and alerts live on prod.
- [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
Reference in new issue View Git Blame Copy Permalink