diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index d203550..c70e513 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1250,3 +1250,102 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an 4. **Remaining Phase-6 onboarding** — see DESIGN-waves.md / `catalog/` (services rendered-but-not-onboarded). 5. **Vendor platform/kubevirt** remote bases proactively (same flake class as #8). 6. **Human action**: finish Headlamp login = complete TOTP enrollment in the browser. + +--- + +## 2026-08-31 — **PHASE 7 OPENED (prep only)**: first prod node on the bench + +The operator added the first **production** node and opened roadmap Phase 7. This entry records the +opening, the state of the hardware, the operator's three decisions, and the blockers that stand +between "a node exists" and "prod reconciles from `clusters/prod`". + +### Governance — Phase 7 entry gate, recorded not waived + +`roadmap/phases.md` gates Phase 7 on **two** conditions: *"Phases 1–6 proven in dev/test; operator +declares prod hardware ready."* The operator has declared hardware ready. **The first condition is +not met** — conformance is 37/51, and the uncovered set includes **R-ENV-01/02/03**, the exact +requirements prod provisioning serves. We proceed with *prep only* on that basis, with no prod +install until the §7 repo blockers close and the gate is green. This is an accepted, explicit risk, +not an oversight. + +### Node 1 as-found (`192.168.2.10`, verified by SSH 2026-08-31) + +Not blank — imaged and reachable. **HP EliteDesk 800 G5 Desktop Mini** (matches the spec's `3× mini-PC +bare metal`), i7-9700T 8c/8t, **31 GiB RAM**, Debian 13 trixie / kernel 6.12, hostname already +`k3s-prod-1`, `k3s` user reachable with the `home_network` key, VT-x + `/dev/kvm` present +(KubeVirt-capable), k3s not installed. Firmware HP R21 Ver. 02.26.00. + +**One 477 GB SATA SSD** (`C400-MTFDDAC512M` — a ~2011-era Micron/Crucial consumer drive) carrying +everything: `/` 30G, `/var` 12G, swap, and `/srv` 433G. No second device. Dev, by contrast, runs two +dedicated data disks per node (Longhorn + a dedicated SSD for etcd, added after HDD contention +starved etcd WAL fsync). + +### Operator decisions (2026-08-31) + +1. **Phase 7 opens, prep work only** — repo work and node baselining now; no k3s install until the + parameterization blockers land and the gate is green. +2. **Wait for all three nodes before `cluster-init`** — prod comes up as spec-conformant 3-node HA + embedded etcd on day one; no single-server-etcd window, no later member-add. +3. **Check drive health before choosing the disk layout** — a 15-year-old consumer SSD is not an + obvious home for prod state. + +### Findings — prod would build WRONG today + +1. **R-ENV-02 is violated in the base (the headline blocker).** The overlay model promises dev and + prod differ *only* by `cluster-vars` values. In fact dev LAN addresses are hardcoded into base + manifests: `k8sServiceHost: 192.168.1.50` (`platform/cilium/helmrelease.yaml:50` + the + `values.yaml` mirror), the Cilium LB pool `192.168.1.53-.60` + (`platform-config/cilium/l2-ip-pool.yaml`), crafty `.55`, the UniFi VM `.54`, and three S3 backup + endpoints at `192.168.1.9:9000`. **Prod is on `192.168.2.x` — none of these follow.** Fix is to + tokenize each into a `cluster-vars` knob and set it in both overlays; the `postBuild.substituteFrom` + seam already exists on every relevant Kustomization. Note `platform/cilium/values.yaml` is only a + *reference mirror* — the authoritative pair is the HelmRelease (Flux) and the Ansible j2 template + (already parameterized), so the Ansible side needs no change. +2. **`clusters/prod` never sets `GATEWAY_VIP`** → it would inherit the base default `10.73.0.240`, + an address on the **retired libvirt network**. `cluster-vars`' own comment flags this as owed + "when hardware is built". +3. **No NOPASSWD sudo for `k3s` on node 1** — `sudo -n` demands a password, so every Ansible + playbook fails at its first `become`. Dev nodes have it; prod must match. Operator action. +4. **`node_baseline` device defaults are virtio** (`/dev/vdb`, `/dev/vdc`) and do not exist on bare + metal; the role makes a filesystem on a *block device*, so a single-disk node needs either a + second drive, `/dev/sda5`, or a repartition. Prod `group_vars` must override. +5. **Single upstream resolver on node 1** (`192.168.2.1`) — the same SPOF class that broke + containerd pulls and CoreDNS forwarding on dev. The fix is **stranded unmerged** on + `fix/node-dns-resilient-upstream`. + +### ESCALATION — the task board is structurally unmaintainable + +`AGENTS.md` §11 and `coordination-protocol.md` §1 designate `agents/task-board.md` as *the one +surface agents may edit* — claim items, update status, raise `OQ-*`. But `ci/spec-immutability.sh` +fails the merge gate on **any** dirt in the spec repo, with **no path exception** for the board, and +committing the board there moves spec `HEAD` off `ci/SPEC_PIN` — which also fails. So editing the +board as instructed **blocks every merge**. + +This fully explains why the board still reads all-`todo` after six delivered phases, and why +`OQ-BKP` is still marked open although Velero + CNPG Barman → S3 landed on `main`. It is not +neglect — the gate made the board unusable, and coordination state migrated here instead. + +I did **not** work around it: adding a path exception would weaken an R-TEST-06 gate, which the +protocol forbids ("never weaken a test to get green"). Options for the operator/architect: + +- **(a) Preferred** — the human commits board updates in the spec repo, and the impl repo adopts each + via a reviewed `ci/SPEC_PIN` bump. That is exactly the flow `SPEC_PIN`'s own comment describes, and + it keeps both the board truthful and the gate honest. Cost: a pin bump per board update. +- **(b)** Amend `spec-immutability.sh` to exempt `agents/task-board.md` only. Lower friction, + but it narrows an immutability gate — a **human** decision, not an agent's. +- **(c)** Formally declare `DECISIONS-LOG.md` the board of record and retire the spec board. Matches + practice, but contradicts the spec as written, so it needs a spec amendment. + +Until this is resolved, Phase-7 coordination state lives in this log. + +### Next steps + +1. **Operator:** configure NOPASSWD sudo on `k3s-prod-1`, then capture `smartctl -a /dev/sda` + (commands in `infra/PROD-PROVISIONING.md` §5). The disk decision waits on that output. +2. **Land the R-ENV-02 tokenization** — the critical path for prod reconciling at all. Dev overlay + must set every new knob to today's exact value so live dev sees zero drift (`make -C ci render-golden`). +3. **Complete `clusters/prod`** — `GATEWAY_VIP`, the L2 pool, and the prod backup target. +4. **Land the two stranded fix branches** (`fix/node-dns-resilient-upstream`, + `fix/act-runner-dind-readiness`) — the DNS one now matters for prod too. +5. Resolve the task-board escalation above. +6. Nodes 2 and 3 through `infra/PROD-PROVISIONING.md` §1–§5 as they arrive. diff --git a/infra/PHASE1-STATUS.md b/infra/PHASE1-STATUS.md index eb18fe5..e3b7a79 100644 --- a/infra/PHASE1-STATUS.md +++ b/infra/PHASE1-STATUS.md @@ -8,6 +8,12 @@ reconnects to it and re-establishes the toolchain. > `DECISIONS-LOG.md` entry (2026-08-31)** — it is the current-state + gotchas + prioritized **Next > Steps** handoff — and the plans under `docs/plans/`. Start at Next Steps. +> **Phase 7 (prod) is OPEN as of 2026-08-31 — prep only.** The first prod node +> (`k3s-prod-1`, `192.168.2.10`) is on the bench. This document covers the **dev** cluster; +> for prod see [`PROD-PROVISIONING.md`](PROD-PROVISIONING.md) (T-7.1 checklist) and the +> Phase-7 entry in `DECISIONS-LOG.md`. Nothing is installed on prod hardware until the +> R-ENV-02 tokenization blockers close and the gate is green. + > **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation > (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster > now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`, diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md new file mode 100644 index 0000000..342cb9a --- /dev/null +++ b/infra/PROD-PROVISIONING.md @@ -0,0 +1,192 @@ +# Prod bare-metal provisioning checklist (T-7.1) + +**Task:** T-7.1 · **Workstream:** WS-INFRA · **Serves:** R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03 +**Status:** Phase 7 **opened by the operator 2026-08-31** — *prep only*. Nothing is installed on prod +hardware until the blockers in §7 are closed and `make -C ci gate` is green. + +> **Entry-gate note (recorded, not waived).** `roadmap/phases.md` gates Phase 7 on *"Phases 1–6 +> proven in dev/test; operator declares prod hardware ready."* The operator has declared hardware +> ready. The first condition is **not** met: conformance is 37/51, and the uncovered set includes +> **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work +> knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry). + +Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)). +The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to +call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse +the *same* Ansible roles and the *same* Flux base as dev — that reuse is the R-ENV-02 obligation. + +--- + +## 0. Node inventory + +Fill a row per node as it lands. Node 1 is as-found on 2026-08-31. + +| | node 1 | node 2 | node 3 | +|---|---|---|---| +| Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` | +| LAN IP | `192.168.2.10` | — | — | +| Model | HP EliteDesk 800 G5 Desktop Mini | | | +| CPU | i7-9700T, 8c/8t, VT-x | | | +| RAM | 31 GiB | | | +| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | | +| OS | Debian 13 (trixie), kernel 6.12 | | | +| Firmware | HP R21 Ver. 02.26.00 | | | +| `/dev/kvm` | present | | | + +Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd). + +--- + +## 1. Firmware / BIOS (per node, before OS install) + +- [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after + boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero. +- [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a + human pressing the button. (`docs/02` calls out wake/boot behaviour explicitly.) +- [ ] **Wake-on-LAN enabled** on the onboard NIC (remote recovery when the box is off). +- [ ] **Boot order** — internal disk first; USB above it only while imaging. Disable network boot + unless PXE is adopted. +- [ ] **Secure Boot** — record the setting. k3s itself is fine either way; note it so all three + nodes match. +- [ ] **Firmware version recorded** in §0 and consistent across all three nodes. + +## 2. Disk layout — **OPEN DECISION, blocks the build** + +`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went +further and added a *third* device for etcd after HDD contention starved etcd WAL fsync +(`docs/plans/ssd-etcd-isolation.md`). + +Node 1 as-found is a **single** 477 GB SATA SSD carrying everything: + +``` +sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial) +├─sda1 976M /boot/efi +├─sda2 30G / +├─sda3 12G /var +├─sda4 977M swap +└─sda5 433G /srv <- the only large free space +``` + +Consequences to settle before imaging nodes 2 and 3: + +- [ ] **Drive health checked first** (operator decision, 2026-08-31). See §5 for the command — + needs root. A ~15-year-old consumer SSD is not an obvious place for prod state. +- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices* + (`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them. + Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** — + prod must override them in `group_vars`. On a single-disk node there is no second device at + all, so one of: + - **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a + SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own + spindle. + - **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`. + Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by. + - **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one + physical failure domain. +- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry. +- [ ] Record the chosen device paths here and set them in `inventory/group_vars/k3s_prod.yml`. + +## 3. OS install (per node) + +- [ ] **Debian 13 (trixie)**, matching dev exactly — same kernel series, same package baseline. +- [ ] Hostname `k3s-prod-N`. +- [ ] Partition per the §2 decision. +- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into + `roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now), + so the same role covers bare metal. Nothing to port. + +## 4. Network (per node) + +- [ ] **Static IP** on the prod LAN, recorded in §0. Node 1 is `192.168.2.10/24`, gateway + `192.168.2.1`. +- [ ] **Reserve the VIP addresses** before the build — prod needs its own kube-vip control-plane VIP + *and* its own Cilium L2 announcement pool for ingress. Neither exists yet (§7). +- [ ] **Resolver: configure at least two upstreams.** Node 1 currently has a single IPv4 nameserver + (`192.168.2.1`). A single upstream is a known SPOF that has already bitten this project — + it broke containerd image pulls and CoreDNS forwarding on dev. A fix exists but is **stranded + unmerged** on `fix/node-dns-resilient-upstream`; land it, then apply the same pattern here. +- [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT — + prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind. + +## 5. Access (per node) — **blocks all Ansible** + +- [ ] **`k3s` user with the `home_network` SSH key.** Working on node 1. +- [ ] **NOPASSWD sudo.** *Not configured on node 1* — `sudo -n` demands a password, so every + playbook fails at its first `become` task. Dev nodes have this; prod must match: + ```bash + # on the node, as a user who can already sudo: + echo 'k3s ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/90-k3s + sudo chmod 0440 /etc/sudoers.d/90-k3s + sudo visudo -c + ``` +- [ ] **Drive health baseline** (§2), needs root: + ```bash + sudo apt-get update && sudo apt-get install -y smartmontools + sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors + ``` + +## 6. Ansible baseline & cluster join + +Reuses the dev roles unchanged — that reuse **is** R-ENV-02. Only inventory and group_vars differ. + +- [ ] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached + **directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping. +- [ ] `group_vars/k3s_prod.yml`: `kube_vip_address`, `cluster_controller_api_server`, and the §2 + device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`. +- [ ] Run `playbooks/node-baseline.yml` → packages, kernel modules, sysctls, iscsid, data-disk + filesystems + mounts. (`qemu-guest-agent` is a no-op on bare metal — the role already only + *starts* it when the virtio channel exists, so it degrades cleanly.) +- [ ] Run `playbooks/k3s.yml` → HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. + **Only once all three nodes are present** — see §8. +- [ ] Run `playbooks/kube-vip.yml` → control-plane VIP on the prod LAN. +- [ ] Run `playbooks/cilium.yml` → CNI at bootstrap; Flux's HelmRelease adopts it afterwards. The + bootstrap values and the HelmRelease **must stay in parity** (`kubeProxyReplacement`, + `k8sServiceHost`/`Port`) or the two layers fight over CNI config (`docs/02`). +- [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015). + +## 7. Repo blockers — **prod cannot be reconciled until these land** + +`clusters/prod` is design-only today and **would build wrong**. R-ENV-02 promises dev and prod differ +*only* by `cluster-vars` values; in practice dev LAN addresses are hardcoded into base manifests: + +| Hardcoded value | Where | Prod needs | +|---|---|---| +| `k8sServiceHost: 192.168.1.50` | `platform/cilium/helmrelease.yaml:50`, mirrored in `values.yaml:16` | prod primary node IP | +| LB pool `192.168.1.53–.60` | `platform-config/cilium/l2-ip-pool.yaml:15` | a prod-LAN pool | +| `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address | +| `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address | +| `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target | + +- [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both* + overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the + header comment in `clusters/base/cluster-vars.yaml`). `platform`, `platform-config`, the app + waves, `apps-vms` and `crafty` all already carry `postBuild.substituteFrom: cluster-vars`, so + the seam exists. +- [ ] **`clusters/prod` must set `GATEWAY_VIP`.** It currently doesn't, so prod would inherit the + base default `10.73.0.240` — an address from the **retired libvirt network**. The base file's + own comment flags this as owed "when hardware is built." +- [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is + byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`. +- [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on + the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03). + +## 8. Build sequence (operator decision, 2026-08-31: **wait for all three nodes**) + +Node 1 is baselined now; the cluster is **not** initialized until nodes 2 and 3 are on the bench. +This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a +single-server-etcd window, and avoids a later member-add dance. + +1. Per-node §1–§5 on each of the three boxes. +2. §7 repo work lands on `main` with the gate green. +3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`. +4. `flux bootstrap ... --path=clusters/prod`. +5. Conformance against prod (§9). + +## 9. Exit criteria (Phase 7) + +- [ ] Prod runs the same config as dev via the prod overlay, **no base edits** (R-ENV-02). +- [ ] Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01). +- [ ] Full **automated** prod DR drill (R-STATE-03) — `definition-of-done.md` requires this for prod + specifically. +- [ ] Monitoring and alerts live on prod. +- [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).