jakman85andClaude Opus 4.8
1eeb754bee
T-1.5 destructive IaC-validity test PASSED: dev cluster torn down and reconstituted from IaC + preserved disks + Flux git source with zero state loss (3/3 Ready, Cilium OK, Flux READY=True). Post-rebuild conformance L2 R-NET-01 + R-NET-02 both PASS. Two IaC-hardening findings for Phase 2: - F1: tofu destroy -target on a count-indexed resource over-destroyed all 3 domains; refactor libvirt_domain.node count -> for_each keyed by node name; never use destroy -target for single-node lifecycle. - F2: single-node ansible rejoin needs the primary in --limit to publish the k3s join-token fact; make the join self-contained via delegate_to the primary so --limit k3s-dev-N is hands-off. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
109 lines
8.7 KiB
Markdown
109 lines
8.7 KiB
Markdown
# Autonomous Session Decision Log (PM)
|
|
|
|
Program Manager working independently while the operator is AFK (started 2026-08-16). Every
|
|
best-effort decision and assumption is recorded here for later review. **Specs were NOT modified**
|
|
(PM cannot change `../home-network-k3s`; that requires the System Architect hat). Blocked items that
|
|
need the operator are called out.
|
|
|
|
## Guardrails I'm holding to
|
|
- No spec changes. No fabricated secrets (Cloudflare token, PIA creds, SMTP — all await hand-off).
|
|
- Don't break the running cluster: `platform/` stays **suspended** in Flux until each component is
|
|
filled in + validated; `platform/cilium` must carry `k8sServiceHost=10.73.0.11` before it ever
|
|
reconciles.
|
|
- Everything author-first, `make -C ci gate` green before merge, feature-branch→main.
|
|
|
|
## Decisions / assumptions (append-only)
|
|
| # | Decision / assumption | Why | Review flag |
|
|
|---|---|---|---|
|
|
| A1 | Complete Phase 1 T-1.4 via **Option A** (Flux bootstrapped, `platform`+`apps` suspended) | You picked Option A earlier as the safe path; "keep going" authorizes it | Confirm scope |
|
|
| A2 | Advance Phase 2 only for components needing **no operator secret**: Longhorn, monitoring, Kyverno-admission, Cilium-parity | cert-manager/externaldns/velero/VPN are blocked on token/PIA/OQ-BKP | — |
|
|
| A3 | Dev resource caps for monitoring (short retention, small PVCs, single replicas) from `cluster-vars` | 22 GiB RAM ceiling | Tune later |
|
|
| A4 | Host Gitea repo is **public** (NAT-only) so Flux needs no git creds | dev bootstrap simplicity | Fine for dev |
|
|
|
|
## Autonomous session results (2026-08-16)
|
|
- **Phase-2 authoring (author-only, merged):** `platform/longhorn` (3-replica SC + bulk indirection),
|
|
`platform/monitoring` (kube-prometheus-stack + Loki + Alloy, dev-trimmed ~1.3 GiB, +8 alert rules),
|
|
`platform/cilium` (parity fix: k8sServiceHost=10.73.0.11 + masquerade, safe to adopt in Phase 2),
|
|
`platform/kyverno` (verified). Gate green.
|
|
- **Flux INSTALLED** on the cluster (4 controllers Running). Host **Gitea working** (named-volume fix),
|
|
repo pushed. GitRepository created → **BLOCKED**: cluster pods can't reach `10.73.0.1:3000`
|
|
(firewalld `libvirt` zone drops VM→host:3000).
|
|
- **RESOLVED (operator added the ansible-vault become password):** ran `host-prereqs.yml` (firewall
|
|
opened via vault→become), VM→Gitea = 200, and **Flux GitRepository is READY=True**
|
|
(`main@sha1:a2f3bdc…`). **Phase-1 T-1.4 core is DONE** — Flux installed + reconciling clusters/dev
|
|
from the host Gitea. `platform`/`apps` Kustomizations intentionally NOT created (Option A) — the
|
|
running cluster is untouched.
|
|
- Firewall + Gitea admin/repo/push + host become password are all IaC now (host-prereqs.yml,
|
|
bootstrap-gitea.sh automation, vaulted group_vars/hypervisor) → destroy/rebuild is hands-off.
|
|
- **NOT deployed:** `platform`/`apps` Kustomizations were deliberately NOT created — the running
|
|
cluster (Cilium etc.) is untouched. Only Flux controllers + the GitRepository source exist.
|
|
|
|
## T-1.5 — Destroy/rebuild proof (destructive IaC-validity test) — 2026-08-16
|
|
|
|
**Goal (operator-scoped):** prove the IaC is *valid* — the dev cluster can be destroyed and rebuilt
|
|
from IaC + git with no data loss ("Approach A is acceptable"; "more about validity of the IaC than a
|
|
COOP/DR scenario"). Ran **after K2 conformance passed**, with the operator approving each destructive
|
|
step at the prompt.
|
|
|
|
**Outcome: PASS (with one operator-visible scare and two concrete IaC findings).** The cluster was
|
|
torn down and reconstituted from IaC + preserved Longhorn/OS disks + the Flux git source. **End state:
|
|
3/3 Ready (k3s-dev-1/2/3, control-plane,etcd,master), Cilium status OK, Flux GitRepository READY=True,
|
|
zero state lost.** Behavioral health re-verified post-rebuild: conformance L2 **R-NET-01 default-deny +
|
|
R-NET-02 deny-undeclared both PASS** (exit 0, JUnit+coverage regenerated). The destroy/rebuild path is
|
|
demonstrably valid.
|
|
|
|
### Finding F1 — `tofu destroy -target` on a `count`-indexed resource is unsafe (ROOT CAUSE)
|
|
`tofu destroy -target 'libvirt_domain.node[2]'` (+ that node's `os[2]`/`data[2]`/`seed[2]`) **destroyed
|
|
all three domains**, not just node3 — while the individually-targeted *volumes* correctly hit only
|
|
index 2. Root cause: the nodes are declared with **`count`** in `infra/tofu/main.tf`; `-target` against a
|
|
single `count` index is a documented Terraform/OpenTofu hazard (the docs explicitly warn `-target` is
|
|
"not for routine use"). count-index addresses are positional and brittle under targeted destroys.
|
|
- **Recommended fix (Phase-2 IaC hardening):** refactor `libvirt_domain.node` + its per-node volumes
|
|
from `count` to **`for_each`** keyed by node name (`k3s-dev-1/2/3`). `for_each` keys are stable and
|
|
individually addressable, so a single node can be `-replace`d/destroyed without touching siblings.
|
|
- **Interim rule:** **never** use `tofu destroy -target` on this module for single-node lifecycle. To
|
|
cycle one node, use `tofu apply -replace='libvirt_domain.node["k3s-dev-N"]'` (post-`for_each`), or
|
|
destroy the specific VM via `virsh` and let `tofu apply` recreate it.
|
|
- **Silver lining:** the over-destroy is exactly what *proved* IaC validity — disks survived, `tofu
|
|
apply` recreated the domains, nodes rejoined, Flux reconciled unchanged. The IaC round-trips.
|
|
|
|
### Finding F2 — single-node Ansible rejoin needs the primary in scope for the join-token
|
|
Recovering node3 with `ansible-playbook ... --limit k3s-dev-3` **failed**: the `k3s_server` join task
|
|
reads `hostvars[groups['k3s_primary'][0]].k3s_node_token`, but that fact is only *published* when the
|
|
primary play runs, so a limit that excludes the primary yields
|
|
`'HostVarsVars' has no attribute 'k3s_node_token'`. Worked with `--limit 'k3s-dev-1,k3s-dev-3'`.
|
|
- **Recommended fix (Phase-2 IaC hardening):** make the join self-contained — have the secondary task
|
|
fetch the token on demand via `delegate_to: "{{ groups['k3s_primary'][0] }}"` (read
|
|
`/var/lib/rancher/k3s/server/node-token` at join time) instead of depending on a fact the primary
|
|
play must have populated in the same run. Then a true single-node rejoin (`--limit k3s-dev-N`) works
|
|
hands-off, which is the T-1.5 durability bar.
|
|
|
|
Both fixes are **IaC-hardening items for Phase 2** (no cluster change now; the running cluster is
|
|
healthy). Logged here; not yet applied.
|
|
|
|
## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
|
|
| # | Issue | Impact | Fix |
|
|
|---|---|---|---|
|
|
| K1 | ~~Substitution-var case mismatch (UPPERCASE refs vs lowercase cluster-vars keys)~~ | — | ✅ **FIXED**: uppercased cluster-vars keys + dev/prod overlay paths; all refs resolve; builds green |
|
|
| K2 | ~~Conformance L2 tests partial/unverified~~ | — | ✅ **DONE**: fixed 3 test bugs (see below), ran via run.sh against the live cluster → **R-NET-01 default-deny + R-NET-02 deny-undeclared PASS**, JUnit+coverage written, exit 0. chainsaw v0.2.12 installed (host `~/.local/bin`) |
|
|
| K3 | `bootstrap/gitea/bootstrap-gitea.sh` used a host bind-mount → rootless-podman perms broke app.ini | initial Gitea start failed | FIXED: switched to a named volume |
|
|
| K4 | **Kyverno/chainsaw findings during K2 (fixed in the tests):** (a) chainsaw does STRICT (length-equal) list matching — asserting one condition against a pod's full `conditions[]` fails → used scalar `phase: Running`; (b) a default-deny baseline blocks the **kubelet readiness probe** (node→pod), so probe-bearing pods never go Ready → dropped probes; (c) chainsaw report `path: ../_artifacts` resolved to repo-root → exit 1 → fixed to `_artifacts`; (d) swapped agnhost→busybox httpd (cached, cache-timing-proof) | — | FIXED in the test files/config |
|
|
|
|
## FINDING for WS-NET (Phase 2, T-2.2 — default-deny baseline design)
|
|
The platform's real default-deny baseline **must not break kubelet health/liveness/readiness probes**
|
|
(node→pod ingress). A naive `ingress: []` NetworkPolicy in Cilium drops the kubelet TCP probe, so any
|
|
pod with a probe never becomes Ready. The baseline must explicitly allow the health-check path (Cilium
|
|
host/health-check allowance or an equivalent ingress exception). Surfaced empirically by the R-NET-02
|
|
conformance test; **carry into T-2.2**.
|
|
|
|
## Toolchain note
|
|
`chainsaw v0.2.12` is installed on the host (`~/.local/bin`) for local conformance runs; for full
|
|
reproducibility it should also be baked into `infra/devcontainer` (pinned) — the in-cluster Chainsaw
|
|
Job (ADR-0011) already uses the pinned image. Deferred (non-blocking).
|
|
|
|
## Blocked on operator (hand-offs)
|
|
- **T-2.4p** Cloudflare API token → cert-manager DNS-01 + ExternalDNS (Phase 2 TLS/DNS).
|
|
- **T-3.2p** PIA username/password → VPN egress gateway (Phase 3).
|
|
- **OQ-BKP** off-cluster backup target → Velero (Phase 4).
|
|
- SMTP creds → Alertmanager email (Phase 4).
|