From 1eeb754bee57389e71a289c6f6e7575b55fc05e5 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 16 Aug 2026 19:35:01 -0500 Subject: [PATCH] docs(t-1.5): log destroy/rebuild proof outcome + IaC findings T-1.5 destructive IaC-validity test PASSED: dev cluster torn down and reconstituted from IaC + preserved disks + Flux git source with zero state loss (3/3 Ready, Cilium OK, Flux READY=True). Post-rebuild conformance L2 R-NET-01 + R-NET-02 both PASS. Two IaC-hardening findings for Phase 2: - F1: tofu destroy -target on a count-indexed resource over-destroyed all 3 domains; refactor libvirt_domain.node count -> for_each keyed by node name; never use destroy -target for single-node lifecycle. - F2: single-node ansible rejoin needs the primary in --limit to publish the k3s join-token fact; make the join self-contained via delegate_to the primary so --limit k3s-dev-N is hands-off. Co-Authored-By: Claude Opus 4.8 (1M context) --- DECISIONS-LOG.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index db1583f..fe6d910 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -38,6 +38,49 @@ need the operator are called out. - **NOT deployed:** `platform`/`apps` Kustomizations were deliberately NOT created — the running cluster (Cilium etc.) is untouched. Only Flux controllers + the GitRepository source exist. +## T-1.5 — Destroy/rebuild proof (destructive IaC-validity test) — 2026-08-16 + +**Goal (operator-scoped):** prove the IaC is *valid* — the dev cluster can be destroyed and rebuilt +from IaC + git with no data loss ("Approach A is acceptable"; "more about validity of the IaC than a +COOP/DR scenario"). Ran **after K2 conformance passed**, with the operator approving each destructive +step at the prompt. + +**Outcome: PASS (with one operator-visible scare and two concrete IaC findings).** The cluster was +torn down and reconstituted from IaC + preserved Longhorn/OS disks + the Flux git source. **End state: +3/3 Ready (k3s-dev-1/2/3, control-plane,etcd,master), Cilium status OK, Flux GitRepository READY=True, +zero state lost.** Behavioral health re-verified post-rebuild: conformance L2 **R-NET-01 default-deny + +R-NET-02 deny-undeclared both PASS** (exit 0, JUnit+coverage regenerated). The destroy/rebuild path is +demonstrably valid. + +### Finding F1 — `tofu destroy -target` on a `count`-indexed resource is unsafe (ROOT CAUSE) +`tofu destroy -target 'libvirt_domain.node[2]'` (+ that node's `os[2]`/`data[2]`/`seed[2]`) **destroyed +all three domains**, not just node3 — while the individually-targeted *volumes* correctly hit only +index 2. Root cause: the nodes are declared with **`count`** in `infra/tofu/main.tf`; `-target` against a +single `count` index is a documented Terraform/OpenTofu hazard (the docs explicitly warn `-target` is +"not for routine use"). count-index addresses are positional and brittle under targeted destroys. +- **Recommended fix (Phase-2 IaC hardening):** refactor `libvirt_domain.node` + its per-node volumes + from `count` to **`for_each`** keyed by node name (`k3s-dev-1/2/3`). `for_each` keys are stable and + individually addressable, so a single node can be `-replace`d/destroyed without touching siblings. +- **Interim rule:** **never** use `tofu destroy -target` on this module for single-node lifecycle. To + cycle one node, use `tofu apply -replace='libvirt_domain.node["k3s-dev-N"]'` (post-`for_each`), or + destroy the specific VM via `virsh` and let `tofu apply` recreate it. +- **Silver lining:** the over-destroy is exactly what *proved* IaC validity — disks survived, `tofu + apply` recreated the domains, nodes rejoined, Flux reconciled unchanged. The IaC round-trips. + +### Finding F2 — single-node Ansible rejoin needs the primary in scope for the join-token +Recovering node3 with `ansible-playbook ... --limit k3s-dev-3` **failed**: the `k3s_server` join task +reads `hostvars[groups['k3s_primary'][0]].k3s_node_token`, but that fact is only *published* when the +primary play runs, so a limit that excludes the primary yields +`'HostVarsVars' has no attribute 'k3s_node_token'`. Worked with `--limit 'k3s-dev-1,k3s-dev-3'`. +- **Recommended fix (Phase-2 IaC hardening):** make the join self-contained — have the secondary task + fetch the token on demand via `delegate_to: "{{ groups['k3s_primary'][0] }}"` (read + `/var/lib/rancher/k3s/server/node-token` at join time) instead of depending on a fact the primary + play must have populated in the same run. Then a true single-node rejoin (`--limit k3s-dev-N`) works + hands-off, which is the T-1.5 durability bar. + +Both fixes are **IaC-hardening items for Phase 2** (no cluster change now; the running cluster is +healthy). Logged here; not yet applied. + ## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed) | # | Issue | Impact | Fix | |---|---|---|---|