home-network-k3s-impl/DECISIONS-LOG.md

# Autonomous Session Decision Log (PM)

Program Manager working independently while the operator is AFK (started 2026-08-16). Every
best-effort decision and assumption is recorded here for later review. **Specs were NOT modified**
(PM cannot change `../home-network-k3s`; that requires the System Architect hat). Blocked items that
need the operator are called out.

## Guardrails I'm holding to
- No spec changes. No fabricated secrets (Cloudflare token, PIA creds, SMTP — all await hand-off).
- Don't break the running cluster: `platform/` stays **suspended** in Flux until each component is
  filled in + validated; `platform/cilium` must carry `k8sServiceHost=10.73.0.11` before it ever
  reconciles.
- Everything author-first, `make -C ci gate` green before merge, feature-branch→main.

## Decisions / assumptions (append-only)
| # | Decision / assumption | Why | Review flag |
|---|---|---|---|
| A1 | Complete Phase 1 T-1.4 via **Option A** (Flux bootstrapped, `platform`+`apps` suspended) | You picked Option A earlier as the safe path; "keep going" authorizes it | Confirm scope |
| A2 | Advance Phase 2 only for components needing **no operator secret**: Longhorn, monitoring, Kyverno-admission, Cilium-parity | cert-manager/externaldns/velero/VPN are blocked on token/PIA/OQ-BKP | — |
| A3 | Dev resource caps for monitoring (short retention, small PVCs, single replicas) from `cluster-vars` | 22 GiB RAM ceiling | Tune later |
| A4 | Host Gitea repo is **public** (NAT-only) so Flux needs no git creds | dev bootstrap simplicity | Fine for dev |

## Autonomous session results (2026-08-16)
- **Phase-2 authoring (author-only, merged):** `platform/longhorn` (3-replica SC + bulk indirection),
  `platform/monitoring` (kube-prometheus-stack + Loki + Alloy, dev-trimmed ~1.3 GiB, +8 alert rules),
  `platform/cilium` (parity fix: k8sServiceHost=10.73.0.11 + masquerade, safe to adopt in Phase 2),
  `platform/kyverno` (verified). Gate green.
- **Flux INSTALLED** on the cluster (4 controllers Running). Host **Gitea working** (named-volume fix),
  repo pushed. GitRepository created → **BLOCKED**: cluster pods can't reach `10.73.0.1:3000`
  (firewalld `libvirt` zone drops VM→host:3000).
  - **RESOLVED (operator added the ansible-vault become password):** ran `host-prereqs.yml` (firewall
    opened via vault→become), VM→Gitea = 200, and **Flux GitRepository is READY=True**
    (`main@sha1:a2f3bdc…`). **Phase-1 T-1.4 core is DONE** — Flux installed + reconciling clusters/dev
    from the host Gitea. `platform`/`apps` Kustomizations intentionally NOT created (Option A) — the
    running cluster is untouched.
  - Firewall + Gitea admin/repo/push + host become password are all IaC now (host-prereqs.yml,
    bootstrap-gitea.sh automation, vaulted group_vars/hypervisor) → destroy/rebuild is hands-off.
- **NOT deployed:** `platform`/`apps` Kustomizations were deliberately NOT created — the running
  cluster (Cilium etc.) is untouched. Only Flux controllers + the GitRepository source exist.

## T-1.5 — Destroy/rebuild proof (destructive IaC-validity test) — 2026-08-16

**Goal (operator-scoped):** prove the IaC is *valid* — the dev cluster can be destroyed and rebuilt
from IaC + git with no data loss ("Approach A is acceptable"; "more about validity of the IaC than a
COOP/DR scenario"). Ran **after K2 conformance passed**, with the operator approving each destructive
step at the prompt.

**Outcome: PASS (with one operator-visible scare and two concrete IaC findings).** The cluster was
torn down and reconstituted from IaC + preserved Longhorn/OS disks + the Flux git source. **End state:
3/3 Ready (k3s-dev-1/2/3, control-plane,etcd,master), Cilium status OK, Flux GitRepository READY=True,
zero state lost.** Behavioral health re-verified post-rebuild: conformance L2 **R-NET-01 default-deny +
R-NET-02 deny-undeclared both PASS** (exit 0, JUnit+coverage regenerated). The destroy/rebuild path is
demonstrably valid.

### Finding F1 — `tofu destroy -target` on a `count`-indexed resource is unsafe (ROOT CAUSE)
`tofu destroy -target 'libvirt_domain.node[2]'` (+ that node's `os[2]`/`data[2]`/`seed[2]`) **destroyed
all three domains**, not just node3 — while the individually-targeted *volumes* correctly hit only
index 2. Root cause: the nodes are declared with **`count`** in `infra/tofu/main.tf`; `-target` against a
single `count` index is a documented Terraform/OpenTofu hazard (the docs explicitly warn `-target` is
"not for routine use"). count-index addresses are positional and brittle under targeted destroys.
- **Recommended fix (Phase-2 IaC hardening):** refactor `libvirt_domain.node` + its per-node volumes
  from `count` to **`for_each`** keyed by node name (`k3s-dev-1/2/3`). `for_each` keys are stable and
  individually addressable, so a single node can be `-replace`d/destroyed without touching siblings.
- **Interim rule:** **never** use `tofu destroy -target` on this module for single-node lifecycle. To
  cycle one node, use `tofu apply -replace='libvirt_domain.node["k3s-dev-N"]'` (post-`for_each`), or
  destroy the specific VM via `virsh` and let `tofu apply` recreate it.
- **Silver lining:** the over-destroy is exactly what *proved* IaC validity — disks survived, `tofu
  apply` recreated the domains, nodes rejoined, Flux reconciled unchanged. The IaC round-trips.

### Finding F2 — single-node Ansible rejoin needs the primary in scope for the join-token
Recovering node3 with `ansible-playbook ... --limit k3s-dev-3` **failed**: the `k3s_server` join task
reads `hostvars[groups['k3s_primary'][0]].k3s_node_token`, but that fact is only *published* when the
primary play runs, so a limit that excludes the primary yields
`'HostVarsVars' has no attribute 'k3s_node_token'`. Worked with `--limit 'k3s-dev-1,k3s-dev-3'`.
- **Recommended fix (Phase-2 IaC hardening):** make the join self-contained — have the secondary task
  fetch the token on demand via `delegate_to: "{{ groups['k3s_primary'][0] }}"` (read
  `/var/lib/rancher/k3s/server/node-token` at join time) instead of depending on a fact the primary
  play must have populated in the same run. Then a true single-node rejoin (`--limit k3s-dev-N`) works
  hands-off, which is the T-1.5 durability bar.

### F1 / F2 — IMPLEMENTED (author-first, 2026-08-16)
- **F1 done:** `infra/tofu/main.tf` node resources refactored `count` → `for_each` (keyed by node
  name); ordered-list/`primary` helper locals added; `outputs.tf` updated off the map; **`moved.tf`**
  adds 12 `moved` blocks (os/data/seed/domain × 3). `tofu validate` OK and **`tofu plan` = `0 to add,
  0 to change, 0 to destroy`** with all 12 resources reported "has moved to" — a pure state-address
  rename, no VM churn. **State-persist pending:** a follow-up `tofu apply` (a proven 0-change apply
  that only rewrites state addresses) is operator-permission-gated; until then the migration re-shows
  harmlessly on each plan and lands automatically on the next real apply.
- **F2 done:** `roles/k3s_server/tasks/main.yml` — secondary now fetches the join token via
  `delegate_to: primary` (with the published fact as a fast path), so `--limit k3s-dev-N` rejoin is
  self-contained. `ansible-playbook --syntax-check` OK on all playbooks; role lint clean under the CI
  gate (`make -C ci gate` PASS). No cluster change; the running cluster is untouched.

## Phase 2 un-suspend PREP — no-secret rollout (author-first, 2026-08-16)

Operator directive: "Phase 2 un-suspend prep … verify cilium parity + fill REPLACE-MEs for the
no-secret components … author-first. Does NOT un-suspend/deploy yet." **Nothing was deployed;** the
running cluster is untouched (still 0 Flux Kustomizations / 0 HelmReleases — only controllers + the
GitRepository).

### Findings from three parallel readiness audits (Longhorn / monitoring / Kyverno)
- **Cilium parity VERIFIED (safety-critical):** every `[parity]` field in `platform/cilium/helmrelease.yaml`
  equals the live CNI — `k8sServiceHost=10.73.0.11` (rendered by the chart as the pod env
  `KUBERNETES_SERVICE_HOST`, **not** a configmap key — the empty `k8s-service-host` cm value was a red
  herring), kube-proxy-replacement, gateway-api, egress+bpf masquerade, hubble, l2-announcements,
  cgroup, v1.16.5. **Flux adoption is a no-op reconcile.**
- **Topology:** "suspended" = Option A (**no root Kustomization was ever created**), and the single base
  `platform` Kustomization renders **all 11 components at once** — so a naive un-suspend would drag in
  the secret-blocked components (cert-manager/externaldns/velero/keycloak/oauth2-proxy/image-automation)
  and never go healthy (`apps` dependsOn platform, wait:true).
- **Longhorn:** manifests clean; node prereqs (open-iscsi, /var/lib/longhorn mount, iscsid) are in
  cloud-init IaC; the REPLACE-ME bulk-NFS SC is isolated/harmless. Ready.
- **Monitoring:** right-sized (~1.25 GiB) but blocked by (a) needing the `longhorn` SC first and (b) two
  **by-name secrets** (`grafana-admin-credentials`, `alertmanager-smtp`) that don't exist — the missing
  mounts (not the REPLACE-ME text) are what wedge Grafana/Alertmanager.
- **Kyverno:** zero policy violations today (policies scope to `app-namespace=true`, none exist), but the
  fail-closed webhook + 3 HA replicas + Enforce are unsafe/oversized for a cold cluster.

### DECISION D-P2A — per-component Flux Kustomizations for the dev no-secret rollout (operator-approved)
Chose **Option A (per-component Kustomizations)**. Authored `clusters/dev/rollout/` = 4 Flux
Kustomizations (`kyverno` → `cilium` → `longhorn` → `monitoring`, monitoring `dependsOn: longhorn`),
and the dev overlay now `$patch: delete`s the monolithic base `platform`/`apps` and adds `./rollout`.
The no-secret tier carries **no SOPS content**, so the missing `sops-age` key does not block it
(decryption dropped from these Kustomizations). Secret-blocked components are simply **not created**
until their hand-offs land.
- **cilium** uses `prune: false` (adopt the out-of-band CNI; never own-delete it).
- **R-ENV-02 divergence — REVIEW FLAG (architect):** the dev overlay now *adds resources* and diverges
  its reconciliation graph from prod (base doc says "dev adds NO resources … graph reused verbatim").
  This is a **transitional bootstrap scaffold**, consistent with the already-inlined dev bootstrap
  patches (SOPS/URL). **Convergence path is coded in:** delete `clusters/dev/rollout/` + the two
  `$patch: delete`s to re-adopt the unified base graph once all hand-offs arrive. **prod is untouched**
  (still builds the monolith; verified). Flag for architect ratification.

### Component hardening (env-gated via cluster-vars so prod keeps the strict posture)
- New base knobs (strict/prod defaults) + dev overrides: `KYVERNO_VALIDATION_ACTION` (Enforce→**Audit**),
  `KYVERNO_WEBHOOK_FAILURE_POLICY` (Fail→**Ignore**), `KYVERNO_ADMISSION_REPLICAS` (3→**1**),
  `MONITORING_STORAGE_CLASS` (longhorn), `GRAFANA_ADMIN_SECRET` (grafana-admin-credentials→**""** in dev
  → chart-default password, dev-only/NAT-isolated).
- Kyverno webhook also gains an explicit `namespaceSelector` excluding kube-system/flux-system/kyverno/
  longhorn-system/monitoring (belt-and-suspenders vs a fail-closed brick).
- Monitoring Alertmanager: the `alertmanager-smtp` mount + smtp_*/email_configs are **deferred** (commented
  with verbatim re-enable markers) so Alertmanager starts with NULL receivers — alerts fire/route/show in
  UI+Grafana, just no email until the Phase-4 SMTP hand-off.
- Validated: `kubectl kustomize` clean for clusters/dev (exactly the 4 Kustomizations, monolith removed),
  clusters/prod (monolith intact), and all 4 components; envsubst→YAML-parse of kyverno+monitoring valid;
  `make -C ci gate` PASS.

### Still required BEFORE an actual deploy (not part of this author-first prep)
1. **Push current `main` to the host Gitea** — Flux source is pinned at `6c3f9ed` (pre-dates these merges).
2. **Runtime node check:** `systemctl is-active iscsid` + `mountpoint /var/lib/longhorn` on all 3 nodes.
3. **Cilium adoption mechanism:** confirm helm-controller can adopt the Ansible-installed release (was it
   `helm`-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization.
4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here.

## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
| # | Issue | Impact | Fix |
|---|---|---|---|
| K1 | ~~Substitution-var case mismatch (UPPERCASE refs vs lowercase cluster-vars keys)~~ | — | ✅ **FIXED**: uppercased cluster-vars keys + dev/prod overlay paths; all refs resolve; builds green |
| K2 | ~~Conformance L2 tests partial/unverified~~ | — | ✅ **DONE**: fixed 3 test bugs (see below), ran via run.sh against the live cluster → **R-NET-01 default-deny + R-NET-02 deny-undeclared PASS**, JUnit+coverage written, exit 0. chainsaw v0.2.12 installed (host `~/.local/bin`) |
| K3 | `bootstrap/gitea/bootstrap-gitea.sh` used a host bind-mount → rootless-podman perms broke app.ini | initial Gitea start failed | FIXED: switched to a named volume |
| K4 | **Kyverno/chainsaw findings during K2 (fixed in the tests):** (a) chainsaw does STRICT (length-equal) list matching — asserting one condition against a pod's full `conditions[]` fails → used scalar `phase: Running`; (b) a default-deny baseline blocks the **kubelet readiness probe** (node→pod), so probe-bearing pods never go Ready → dropped probes; (c) chainsaw report `path: ../_artifacts` resolved to repo-root → exit 1 → fixed to `_artifacts`; (d) swapped agnhost→busybox httpd (cached, cache-timing-proof) | — | FIXED in the test files/config |

## FINDING for WS-NET (Phase 2, T-2.2 — default-deny baseline design)
The platform's real default-deny baseline **must not break kubelet health/liveness/readiness probes**
(node→pod ingress). A naive `ingress: []` NetworkPolicy in Cilium drops the kubelet TCP probe, so any
pod with a probe never becomes Ready. The baseline must explicitly allow the health-check path (Cilium
host/health-check allowance or an equivalent ingress exception). Surfaced empirically by the R-NET-02
conformance test; **carry into T-2.2**.

## Toolchain note
`chainsaw v0.2.12` is installed on the host (`~/.local/bin`) for local conformance runs; for full
reproducibility it should also be baked into `infra/devcontainer` (pinned) — the in-cluster Chainsaw
Job (ADR-0011) already uses the pinned image. Deferred (non-blocking).

## Blocked on operator (hand-offs)
- **T-2.4p** Cloudflare API token → cert-manager DNS-01 + ExternalDNS (Phase 2 TLS/DNS).
- **T-3.2p** PIA username/password → VPN egress gateway (Phase 3).
- **OQ-BKP** off-cluster backup target → Velero (Phase 4).
- SMTP creds → Alertmanager email (Phase 4).