docs(t-1.5): log destroy/rebuild proof outcome + IaC findings
T-1.5 destructive IaC-validity test PASSED: dev cluster torn down and reconstituted from IaC + preserved disks + Flux git source with zero state loss (3/3 Ready, Cilium OK, Flux READY=True). Post-rebuild conformance L2 R-NET-01 + R-NET-02 both PASS. Two IaC-hardening findings for Phase 2: - F1: tofu destroy -target on a count-indexed resource over-destroyed all 3 domains; refactor libvirt_domain.node count -> for_each keyed by node name; never use destroy -target for single-node lifecycle. - F2: single-node ansible rejoin needs the primary in --limit to publish the k3s join-token fact; make the join self-contained via delegate_to the primary so --limit k3s-dev-N is hands-off. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
1 parent
6c3f9eddbd
commit
1eeb754bee
1 file changed
+43
@@ -38,6 +38,49 @@ need the operator are called out.
|
||||
- **NOT deployed:** `platform`/`apps` Kustomizations were deliberately NOT created — the running
|
||||
cluster (Cilium etc.) is untouched. Only Flux controllers + the GitRepository source exist.
|
||||
|
||||
## T-1.5 — Destroy/rebuild proof (destructive IaC-validity test) — 2026-08-16
|
||||
|
||||
**Goal (operator-scoped):** prove the IaC is *valid* — the dev cluster can be destroyed and rebuilt
|
||||
from IaC + git with no data loss ("Approach A is acceptable"; "more about validity of the IaC than a
|
||||
COOP/DR scenario"). Ran **after K2 conformance passed**, with the operator approving each destructive
|
||||
step at the prompt.
|
||||
|
||||
**Outcome: PASS (with one operator-visible scare and two concrete IaC findings).** The cluster was
|
||||
torn down and reconstituted from IaC + preserved Longhorn/OS disks + the Flux git source. **End state:
|
||||
3/3 Ready (k3s-dev-1/2/3, control-plane,etcd,master), Cilium status OK, Flux GitRepository READY=True,
|
||||
zero state lost.** Behavioral health re-verified post-rebuild: conformance L2 **R-NET-01 default-deny +
|
||||
R-NET-02 deny-undeclared both PASS** (exit 0, JUnit+coverage regenerated). The destroy/rebuild path is
|
||||
demonstrably valid.
|
||||
|
||||
### Finding F1 — `tofu destroy -target` on a `count`-indexed resource is unsafe (ROOT CAUSE)
|
||||
`tofu destroy -target 'libvirt_domain.node[2]'` (+ that node's `os[2]`/`data[2]`/`seed[2]`) **destroyed
|
||||
all three domains**, not just node3 — while the individually-targeted *volumes* correctly hit only
|
||||
index 2. Root cause: the nodes are declared with **`count`** in `infra/tofu/main.tf`; `-target` against a
|
||||
single `count` index is a documented Terraform/OpenTofu hazard (the docs explicitly warn `-target` is
|
||||
"not for routine use"). count-index addresses are positional and brittle under targeted destroys.
|
||||
- **Recommended fix (Phase-2 IaC hardening):** refactor `libvirt_domain.node` + its per-node volumes
|
||||
from `count` to **`for_each`** keyed by node name (`k3s-dev-1/2/3`). `for_each` keys are stable and
|
||||
individually addressable, so a single node can be `-replace`d/destroyed without touching siblings.
|
||||
- **Interim rule:** **never** use `tofu destroy -target` on this module for single-node lifecycle. To
|
||||
cycle one node, use `tofu apply -replace='libvirt_domain.node["k3s-dev-N"]'` (post-`for_each`), or
|
||||
destroy the specific VM via `virsh` and let `tofu apply` recreate it.
|
||||
- **Silver lining:** the over-destroy is exactly what *proved* IaC validity — disks survived, `tofu
|
||||
apply` recreated the domains, nodes rejoined, Flux reconciled unchanged. The IaC round-trips.
|
||||
|
||||
### Finding F2 — single-node Ansible rejoin needs the primary in scope for the join-token
|
||||
Recovering node3 with `ansible-playbook ... --limit k3s-dev-3` **failed**: the `k3s_server` join task
|
||||
reads `hostvars[groups['k3s_primary'][0]].k3s_node_token`, but that fact is only *published* when the
|
||||
primary play runs, so a limit that excludes the primary yields
|
||||
`'HostVarsVars' has no attribute 'k3s_node_token'`. Worked with `--limit 'k3s-dev-1,k3s-dev-3'`.
|
||||
- **Recommended fix (Phase-2 IaC hardening):** make the join self-contained — have the secondary task
|
||||
fetch the token on demand via `delegate_to: "{{ groups['k3s_primary'][0] }}"` (read
|
||||
`/var/lib/rancher/k3s/server/node-token` at join time) instead of depending on a fact the primary
|
||||
play must have populated in the same run. Then a true single-node rejoin (`--limit k3s-dev-N`) works
|
||||
hands-off, which is the T-1.5 durability bar.
|
||||
|
||||
Both fixes are **IaC-hardening items for Phase 2** (no cluster change now; the running cluster is
|
||||
healthy). Logged here; not yet applied.
|
||||
|
||||
## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
|
||||
| # | Issue | Impact | Fix |
|
||||
|---|---|---|---|
|
||||
|
||||
Reference in new issue
Block a user