home-network-k3s-impl/DECISIONS-LOG.md

Autonomous Session Decision Log (PM)

Program Manager working independently while the operator is AFK (started 2026-08-16). Every best-effort decision and assumption is recorded here for later review. Specs were NOT modified (PM cannot change ../home-network-k3s; that requires the System Architect hat). Blocked items that need the operator are called out.

Guardrails I'm holding to

• No spec changes. No fabricated secrets (Cloudflare token, PIA creds, SMTP — all await hand-off).
• Don't break the running cluster: platform/ stays suspended in Flux until each component is filled in + validated; platform/cilium must carry k8sServiceHost=10.73.0.11 before it ever reconciles.
• Everything author-first, make -C ci gate green before merge, feature-branch→main.

Decisions / assumptions (append-only)

#	Decision / assumption	Why	Review flag
A1	Complete Phase 1 T-1.4 via Option A (Flux bootstrapped, platform+apps suspended)	You picked Option A earlier as the safe path; "keep going" authorizes it	Confirm scope
A2	Advance Phase 2 only for components needing no operator secret: Longhorn, monitoring, Kyverno-admission, Cilium-parity	cert-manager/externaldns/velero/VPN are blocked on token/PIA/OQ-BKP	—
A3	Dev resource caps for monitoring (short retention, small PVCs, single replicas) from cluster-vars	22 GiB RAM ceiling	Tune later
A4	Host Gitea repo is public (NAT-only) so Flux needs no git creds	dev bootstrap simplicity	Fine for dev

Autonomous session results (2026-08-16)

• Phase-2 authoring (author-only, merged): platform/longhorn (3-replica SC + bulk indirection), platform/monitoring (kube-prometheus-stack + Loki + Alloy, dev-trimmed ~1.3 GiB, +8 alert rules), platform/cilium (parity fix: k8sServiceHost=10.73.0.11 + masquerade, safe to adopt in Phase 2), platform/kyverno (verified). Gate green.
• Flux INSTALLED on the cluster (4 controllers Running). Host Gitea working (named-volume fix), repo pushed. GitRepository created → BLOCKED: cluster pods can't reach 10.73.0.1:3000 (firewalld libvirt zone drops VM→host:3000).
  • RESOLVED (operator added the ansible-vault become password): ran host-prereqs.yml (firewall opened via vault→become), VM→Gitea = 200, and Flux GitRepository is READY=True (main@sha1:a2f3bdc…). Phase-1 T-1.4 core is DONE — Flux installed + reconciling clusters/dev from the host Gitea. platform/apps Kustomizations intentionally NOT created (Option A) — the running cluster is untouched.
  • Firewall + Gitea admin/repo/push + host become password are all IaC now (host-prereqs.yml, bootstrap-gitea.sh automation, vaulted group_vars/hypervisor) → destroy/rebuild is hands-off.
• NOT deployed: platform/apps Kustomizations were deliberately NOT created — the running cluster (Cilium etc.) is untouched. Only Flux controllers + the GitRepository source exist.

T-1.5 — Destroy/rebuild proof (destructive IaC-validity test) — 2026-08-16

Goal (operator-scoped): prove the IaC is valid — the dev cluster can be destroyed and rebuilt from IaC + git with no data loss ("Approach A is acceptable"; "more about validity of the IaC than a COOP/DR scenario"). Ran after K2 conformance passed, with the operator approving each destructive step at the prompt.

Outcome: PASS (with one operator-visible scare and two concrete IaC findings). The cluster was torn down and reconstituted from IaC + preserved Longhorn/OS disks + the Flux git source. End state: 3/3 Ready (k3s-dev-1/2/3, control-plane,etcd,master), Cilium status OK, Flux GitRepository READY=True, zero state lost. Behavioral health re-verified post-rebuild: conformance L2 R-NET-01 default-deny + R-NET-02 deny-undeclared both PASS (exit 0, JUnit+coverage regenerated). The destroy/rebuild path is demonstrably valid.

Finding F1 — tofu destroy -target on a count-indexed resource is unsafe (ROOT CAUSE)

tofu destroy -target 'libvirt_domain.node[2]' (+ that node's os[2]/data[2]/seed[2]) destroyed all three domains, not just node3 — while the individually-targeted volumes correctly hit only index 2. Root cause: the nodes are declared with count in infra/tofu/main.tf; -target against a single count index is a documented Terraform/OpenTofu hazard (the docs explicitly warn -target is "not for routine use"). count-index addresses are positional and brittle under targeted destroys.

• Recommended fix (Phase-2 IaC hardening): refactor libvirt_domain.node + its per-node volumes from count to for_each keyed by node name (k3s-dev-1/2/3). for_each keys are stable and individually addressable, so a single node can be -replaced/destroyed without touching siblings.
• Interim rule: never use tofu destroy -target on this module for single-node lifecycle. To cycle one node, use tofu apply -replace='libvirt_domain.node["k3s-dev-N"]' (post-for_each), or destroy the specific VM via virsh and let tofu apply recreate it.
• Silver lining: the over-destroy is exactly what proved IaC validity — disks survived, tofu apply recreated the domains, nodes rejoined, Flux reconciled unchanged. The IaC round-trips.

Finding F2 — single-node Ansible rejoin needs the primary in scope for the join-token

Recovering node3 with ansible-playbook ... --limit k3s-dev-3 failed: the k3s_server join task reads hostvars[groups['k3s_primary'][0]].k3s_node_token, but that fact is only published when the primary play runs, so a limit that excludes the primary yields 'HostVarsVars' has no attribute 'k3s_node_token'. Worked with --limit 'k3s-dev-1,k3s-dev-3'.

• Recommended fix (Phase-2 IaC hardening): make the join self-contained — have the secondary task fetch the token on demand via delegate_to: "{{ groups['k3s_primary'][0] }}" (read /var/lib/rancher/k3s/server/node-token at join time) instead of depending on a fact the primary play must have populated in the same run. Then a true single-node rejoin (--limit k3s-dev-N) works hands-off, which is the T-1.5 durability bar.

F1 / F2 — IMPLEMENTED (author-first, 2026-08-16)

• F1 done: infra/tofu/main.tf node resources refactored count → for_each (keyed by node name); ordered-list/primary helper locals added; outputs.tf updated off the map; moved.tf adds 12 moved blocks (os/data/seed/domain × 3). tofu validate OK and tofu plan = 0 to add, 0 to change, 0 to destroy with all 12 resources reported "has moved to" — a pure state-address rename, no VM churn. State-persist pending: a follow-up tofu apply (a proven 0-change apply that only rewrites state addresses) is operator-permission-gated; until then the migration re-shows harmlessly on each plan and lands automatically on the next real apply.
• F2 done: roles/k3s_server/tasks/main.yml — secondary now fetches the join token via delegate_to: primary (with the published fact as a fast path), so --limit k3s-dev-N rejoin is self-contained. ansible-playbook --syntax-check OK on all playbooks; role lint clean under the CI gate (make -C ci gate PASS). No cluster change; the running cluster is untouched.

KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)

#	Issue	Impact	Fix
K1	Substitution-var case mismatch (UPPERCASE refs vs lowercase cluster-vars keys)	—	✅ FIXED: uppercased cluster-vars keys + dev/prod overlay paths; all refs resolve; builds green
K2	Conformance L2 tests partial/unverified	—	✅ DONE: fixed 3 test bugs (see below), ran via run.sh against the live cluster → R-NET-01 default-deny + R-NET-02 deny-undeclared PASS, JUnit+coverage written, exit 0. chainsaw v0.2.12 installed (host ~/.local/bin)
K3	bootstrap/gitea/bootstrap-gitea.sh used a host bind-mount → rootless-podman perms broke app.ini	initial Gitea start failed	FIXED: switched to a named volume
K4	Kyverno/chainsaw findings during K2 (fixed in the tests): (a) chainsaw does STRICT (length-equal) list matching — asserting one condition against a pod's full conditions[] fails → used scalar phase: Running; (b) a default-deny baseline blocks the kubelet readiness probe (node→pod), so probe-bearing pods never go Ready → dropped probes; (c) chainsaw report path: ../_artifacts resolved to repo-root → exit 1 → fixed to _artifacts; (d) swapped agnhost→busybox httpd (cached, cache-timing-proof)	—	FIXED in the test files/config

FINDING for WS-NET (Phase 2, T-2.2 — default-deny baseline design)

The platform's real default-deny baseline must not break kubelet health/liveness/readiness probes (node→pod ingress). A naive ingress: [] NetworkPolicy in Cilium drops the kubelet TCP probe, so any pod with a probe never becomes Ready. The baseline must explicitly allow the health-check path (Cilium host/health-check allowance or an equivalent ingress exception). Surfaced empirically by the R-NET-02 conformance test; carry into T-2.2.

Toolchain note

chainsaw v0.2.12 is installed on the host (~/.local/bin) for local conformance runs; for full reproducibility it should also be baked into infra/devcontainer (pinned) — the in-cluster Chainsaw Job (ADR-0011) already uses the pinned image. Deferred (non-blocking).

Blocked on operator (hand-offs)

• T-2.4p Cloudflare API token → cert-manager DNS-01 + ExternalDNS (Phase 2 TLS/DNS).
• T-3.2p PIA username/password → VPN egress gateway (Phase 3).
• OQ-BKP off-cluster backup target → Velero (Phase 4).
• SMTP creds → Alertmanager email (Phase 4).