home-network-k3s-impl/DECISIONS-LOG.md

Autonomous Session Decision Log (PM)

Program Manager working independently while the operator is AFK (started 2026-08-16). Every best-effort decision and assumption is recorded here for later review. Specs were NOT modified (PM cannot change ../home-network-k3s; that requires the System Architect hat). Blocked items that need the operator are called out.

Guardrails I'm holding to

• No spec changes. No fabricated secrets (Cloudflare token, PIA creds, SMTP — all await hand-off).
• Don't break the running cluster: platform/ stays suspended in Flux until each component is filled in + validated; platform/cilium must carry k8sServiceHost=10.73.0.11 before it ever reconciles.
• Everything author-first, make -C ci gate green before merge, feature-branch→main.

Decisions / assumptions (append-only)

#	Decision / assumption	Why	Review flag
A1	Complete Phase 1 T-1.4 via Option A (Flux bootstrapped, platform+apps suspended)	You picked Option A earlier as the safe path; "keep going" authorizes it	Confirm scope
A2	Advance Phase 2 only for components needing no operator secret: Longhorn, monitoring, Kyverno-admission, Cilium-parity	cert-manager/externaldns/velero/VPN are blocked on token/PIA/OQ-BKP	—
A3	Dev resource caps for monitoring (short retention, small PVCs, single replicas) from cluster-vars	22 GiB RAM ceiling	Tune later
A4	Host Gitea repo is public (NAT-only) so Flux needs no git creds	dev bootstrap simplicity	Fine for dev

Autonomous session results (2026-08-16)

• Phase-2 authoring (author-only, merged): platform/longhorn (3-replica SC + bulk indirection), platform/monitoring (kube-prometheus-stack + Loki + Alloy, dev-trimmed ~1.3 GiB, +8 alert rules), platform/cilium (parity fix: k8sServiceHost=10.73.0.11 + masquerade, safe to adopt in Phase 2), platform/kyverno (verified). Gate green.
• Flux INSTALLED on the cluster (4 controllers Running). Host Gitea working (named-volume fix), repo pushed. GitRepository created → BLOCKED: cluster pods can't reach 10.73.0.1:3000 (firewalld libvirt zone drops VM→host:3000).
  • RESOLVED (operator added the ansible-vault become password): ran host-prereqs.yml (firewall opened via vault→become), VM→Gitea = 200, and Flux GitRepository is READY=True (main@sha1:a2f3bdc…). Phase-1 T-1.4 core is DONE — Flux installed + reconciling clusters/dev from the host Gitea. platform/apps Kustomizations intentionally NOT created (Option A) — the running cluster is untouched.
  • Firewall + Gitea admin/repo/push + host become password are all IaC now (host-prereqs.yml, bootstrap-gitea.sh automation, vaulted group_vars/hypervisor) → destroy/rebuild is hands-off.
• NOT deployed: platform/apps Kustomizations were deliberately NOT created — the running cluster (Cilium etc.) is untouched. Only Flux controllers + the GitRepository source exist.

T-1.5 — Destroy/rebuild proof (destructive IaC-validity test) — 2026-08-16

Goal (operator-scoped): prove the IaC is valid — the dev cluster can be destroyed and rebuilt from IaC + git with no data loss ("Approach A is acceptable"; "more about validity of the IaC than a COOP/DR scenario"). Ran after K2 conformance passed, with the operator approving each destructive step at the prompt.

Outcome: PASS (with one operator-visible scare and two concrete IaC findings). The cluster was torn down and reconstituted from IaC + preserved Longhorn/OS disks + the Flux git source. End state: 3/3 Ready (k3s-dev-1/2/3, control-plane,etcd,master), Cilium status OK, Flux GitRepository READY=True, zero state lost. Behavioral health re-verified post-rebuild: conformance L2 R-NET-01 default-deny + R-NET-02 deny-undeclared both PASS (exit 0, JUnit+coverage regenerated). The destroy/rebuild path is demonstrably valid.

Finding F1 — tofu destroy -target on a count-indexed resource is unsafe (ROOT CAUSE)

tofu destroy -target 'libvirt_domain.node[2]' (+ that node's os[2]/data[2]/seed[2]) destroyed all three domains, not just node3 — while the individually-targeted volumes correctly hit only index 2. Root cause: the nodes are declared with count in infra/tofu/main.tf; -target against a single count index is a documented Terraform/OpenTofu hazard (the docs explicitly warn -target is "not for routine use"). count-index addresses are positional and brittle under targeted destroys.

• Recommended fix (Phase-2 IaC hardening): refactor libvirt_domain.node + its per-node volumes from count to for_each keyed by node name (k3s-dev-1/2/3). for_each keys are stable and individually addressable, so a single node can be -replaced/destroyed without touching siblings.
• Interim rule: never use tofu destroy -target on this module for single-node lifecycle. To cycle one node, use tofu apply -replace='libvirt_domain.node["k3s-dev-N"]' (post-for_each), or destroy the specific VM via virsh and let tofu apply recreate it.
• Silver lining: the over-destroy is exactly what proved IaC validity — disks survived, tofu apply recreated the domains, nodes rejoined, Flux reconciled unchanged. The IaC round-trips.

Finding F2 — single-node Ansible rejoin needs the primary in scope for the join-token

Recovering node3 with ansible-playbook ... --limit k3s-dev-3 failed: the k3s_server join task reads hostvars[groups['k3s_primary'][0]].k3s_node_token, but that fact is only published when the primary play runs, so a limit that excludes the primary yields 'HostVarsVars' has no attribute 'k3s_node_token'. Worked with --limit 'k3s-dev-1,k3s-dev-3'.

• Recommended fix (Phase-2 IaC hardening): make the join self-contained — have the secondary task fetch the token on demand via delegate_to: "{{ groups['k3s_primary'][0] }}" (read /var/lib/rancher/k3s/server/node-token at join time) instead of depending on a fact the primary play must have populated in the same run. Then a true single-node rejoin (--limit k3s-dev-N) works hands-off, which is the T-1.5 durability bar.

F1 / F2 — IMPLEMENTED (author-first, 2026-08-16)

• F1 done: infra/tofu/main.tf node resources refactored count → for_each (keyed by node name); ordered-list/primary helper locals added; outputs.tf updated off the map; moved.tf adds 12 moved blocks (os/data/seed/domain × 3). tofu validate OK and tofu plan = 0 to add, 0 to change, 0 to destroy with all 12 resources reported "has moved to" — a pure state-address rename, no VM churn. State-persist pending: a follow-up tofu apply (a proven 0-change apply that only rewrites state addresses) is operator-permission-gated; until then the migration re-shows harmlessly on each plan and lands automatically on the next real apply.
• F2 done: roles/k3s_server/tasks/main.yml — secondary now fetches the join token via delegate_to: primary (with the published fact as a fast path), so --limit k3s-dev-N rejoin is self-contained. ansible-playbook --syntax-check OK on all playbooks; role lint clean under the CI gate (make -C ci gate PASS). No cluster change; the running cluster is untouched.

Phase 2 un-suspend PREP — no-secret rollout (author-first, 2026-08-16)

Operator directive: "Phase 2 un-suspend prep … verify cilium parity + fill REPLACE-MEs for the no-secret components … author-first. Does NOT un-suspend/deploy yet." Nothing was deployed; the running cluster is untouched (still 0 Flux Kustomizations / 0 HelmReleases — only controllers + the GitRepository).

Findings from three parallel readiness audits (Longhorn / monitoring / Kyverno)

• Cilium parity VERIFIED (safety-critical): every [parity] field in platform/cilium/helmrelease.yaml equals the live CNI — k8sServiceHost=10.73.0.11 (rendered by the chart as the pod env KUBERNETES_SERVICE_HOST, not a configmap key — the empty k8s-service-host cm value was a red herring), kube-proxy-replacement, gateway-api, egress+bpf masquerade, hubble, l2-announcements, cgroup, v1.16.5. Flux adoption is a no-op reconcile.
• Topology: "suspended" = Option A (no root Kustomization was ever created), and the single base platform Kustomization renders all 11 components at once — so a naive un-suspend would drag in the secret-blocked components (cert-manager/externaldns/velero/keycloak/oauth2-proxy/image-automation) and never go healthy (apps dependsOn platform, wait:true).
• Longhorn: manifests clean; node prereqs (open-iscsi, /var/lib/longhorn mount, iscsid) are in cloud-init IaC; the REPLACE-ME bulk-NFS SC is isolated/harmless. Ready.
• Monitoring: right-sized (~1.25 GiB) but blocked by (a) needing the longhorn SC first and (b) two by-name secrets (grafana-admin-credentials, alertmanager-smtp) that don't exist — the missing mounts (not the REPLACE-ME text) are what wedge Grafana/Alertmanager.
• Kyverno: zero policy violations today (policies scope to app-namespace=true, none exist), but the fail-closed webhook + 3 HA replicas + Enforce are unsafe/oversized for a cold cluster.

DECISION D-P2A — per-component Flux Kustomizations for the dev no-secret rollout (operator-approved)

Chose Option A (per-component Kustomizations). Authored clusters/dev/rollout/ = 4 Flux Kustomizations (kyverno → cilium → longhorn → monitoring, monitoring dependsOn: longhorn), and the dev overlay now $patch: deletes the monolithic base platform/apps and adds ./rollout. The no-secret tier carries no SOPS content, so the missing sops-age key does not block it (decryption dropped from these Kustomizations). Secret-blocked components are simply not created until their hand-offs land.

• cilium uses prune: false (adopt the out-of-band CNI; never own-delete it).
• R-ENV-02 divergence — REVIEW FLAG (architect): the dev overlay now adds resources and diverges its reconciliation graph from prod (base doc says "dev adds NO resources … graph reused verbatim"). This is a transitional bootstrap scaffold, consistent with the already-inlined dev bootstrap patches (SOPS/URL). Convergence path is coded in: delete clusters/dev/rollout/ + the two $patch: deletes to re-adopt the unified base graph once all hand-offs arrive. prod is untouched (still builds the monolith; verified). Flag for architect ratification.

Component hardening (env-gated via cluster-vars so prod keeps the strict posture)

• New base knobs (strict/prod defaults) + dev overrides: KYVERNO_VALIDATION_ACTION (Enforce→Audit), KYVERNO_WEBHOOK_FAILURE_POLICY (Fail→Ignore), KYVERNO_ADMISSION_REPLICAS (3→1), MONITORING_STORAGE_CLASS (longhorn), GRAFANA_ADMIN_SECRET (grafana-admin-credentials→"" in dev → chart-default password, dev-only/NAT-isolated).
• Kyverno webhook also gains an explicit namespaceSelector excluding kube-system/flux-system/kyverno/ longhorn-system/monitoring (belt-and-suspenders vs a fail-closed brick).
• Monitoring Alertmanager: the alertmanager-smtp mount + smtp_*/email_configs are deferred (commented with verbatim re-enable markers) so Alertmanager starts with NULL receivers — alerts fire/route/show in UI+Grafana, just no email until the Phase-4 SMTP hand-off.
• Validated: kubectl kustomize clean for clusters/dev (exactly the 4 Kustomizations, monolith removed), clusters/prod (monolith intact), and all 4 components; envsubst→YAML-parse of kyverno+monitoring valid; make -C ci gate PASS.

Still required BEFORE an actual deploy (not part of this author-first prep)

1. Push current main to the host Gitea — Flux source is pinned at 6c3f9ed (pre-dates these merges).
2. Runtime node check: systemctl is-active iscsid + mountpoint /var/lib/longhorn on all 3 nodes.
3. Cilium adoption mechanism: confirm helm-controller can adopt the Ansible-installed release (was it helm-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization.
4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here.

CORRECTION — credential-forward, unified graph (operator directive, 2026-08-16)

Operator pushback: "Are we making implementation choices solely on the fact that you have to ask for credentials? It is better to get the credentials upfront (encrypted file) and keep the rollout in line with the existing architecture." Correct — and it exposed a scope distortion.

• A2 SUPERSEDED / D-P2A REVERTED. Choosing components by "needs no operator secret" inverted the roadmap: roadmap/phases.md Phase 2 = Network, Storage, TLS, DNS, Ingress → the credential-gated cert-manager/externaldns/gateway ARE the Phase-2 core, while monitoring is Phase 4 (I'd pulled it forward). The per-component dev divergence (clusters/dev/rollout/) existed only to dodge asking for the Cloudflare token.
• Reverted to the existing architecture: deleted clusters/dev/rollout/ + the $patch: deletes; restored the unified base graph (dev = ../base + cluster-vars patches + SOPS decryption patches, one platform Kustomization, R-ENV-02 intact — removes the earlier architect-review flag). Reverted the monitoring Grafana/SMTP dodges (it's Phase 4). Kept the non-credential-driven improvements: tofu for_each/rejoin (F1/F2), and Kyverno parametrization + webhook namespace-excludes (Kyverno IS Phase 2).
• DECISION D-P2B — phase the unified platform/kustomization.yaml (operator-approved). Components are enabled in the single shared list per phase (the same pattern that deferred kyverno): Phase-2 set now active = kyverno, cilium, longhorn, cert-manager, externaldns, gateway; monitoring/ image-automation (P4), velero (P4, OQ-BKP), kubevirt (P5), keycloak/oauth2-proxy (P6) are commented with phase markers. dev+prod render the same list. Verified: kubectl kustomize platform renders only the 6 Phase-2 namespaces; dev shows the unified platform+apps Kustomizations; prod builds; gate PASS.
• DECISION D-SEC1 — provision ALL credentials upfront via SOPS; operator encrypts (both operator-chosen). Activated SOPS: .sops.yaml installed at repo root (recipient still REPLACE-ME — operator fills after age-keygen). Authored bootstrap/sops/secret-templates/*.yaml.example (cloudflare-api-token ×2 ns, grafana-admin, alertmanager-smtp, keycloak-admin, oauth2-proxy, pia — all placeholder-only, .example so kustomize/SOPS ignore them) + bootstrap/sops/CREDENTIALS.md (end-to-end checklist). Pre-wired commented - cloudflare-api-token.sops.yaml lines in cert-manager + externaldns kustomizations (operator uncomments post-encrypt). Filled the non-secret ACME email ([email protected], confirm/change). Agents never touch real secret values (per .sops.yaml).

Operator TODO to reach Phase-2 deploy (see bootstrap/sops/CREDENTIALS.md)

1. age-keygen → fill root .sops.yaml recipient → install sops-age on cluster.
2. Provision cloudflare-api-token (Zone:DNS:Edit + Zone:Read on keithjackson.dev) in both cert-manager + external-dns; uncomment the two kustomization lines.
3. Confirm ACME email. 4. Push main to host Gitea. 5. Create the dev root Kustomization (the un-suspend).
4. (Upfront, optional now) encrypt the Phase-3/4/6 secrets so later phases are credential-ready.
5. Chart-key verification (flagged in templates): keycloak + oauth2-proxy existingSecret key names, Keycloak DB secret, and PIA/Gluetun protocol (≠ TCP) — verify at each component's phase.

Phase-2 UN-SUSPEND executed + fixes (autonomous session, 2026-08-16 overnight)

Operator said "go" for the un-suspend, then went AFK: "continue to work without me. Any major decision that requires a system architecture change VIA SPEC CHANGE will need to be escalated. Anything else is allowed. Keep track of assumptions and report on return." None of the below is a spec change (the spec is silent on Flux Kustomization granularity / L2 wiring), so all within PM remit. Nothing touched the read-only spec repo.

• Un-suspended via the sanctioned kubectl apply -k clusters/dev (bootstrap.sh step 3): controllers already installed, sops-age present. Pre-deploy checks all green first — Cilium adoption safe (an existing helm release cilium v3 exists → Flux adopts by name, no-op upgrade), Longhorn node prereqs green on all 3 nodes (iscsid active, /var/lib/longhorn→/dev/vdb), host Gitea pushed + Flux source Ready.

Fix 1 — Longhorn StorageClass numeric coercion (commit ~c04aea8)

Reconcile stalled: .parameters.numberOfReplicas: expected string, got 3. kustomize strips the quotes around a bare "${VAR}" placeholder, so Flux substitution yielded an unquoted numeric that SSA rejects (StorageClass.parameters is map[string]string). Hardcoded numberOfReplicas: "3" (quoted numeric literals ARE preserved by kustomize; topology LOCKED at 3). The replica knob still drives the global default via defaultSettings.defaultReplicaCount in the helmrelease (a Helm int value, unaffected).

Fix 2 — DECISION: controllers/configs split (commit ~458a867) — NOTABLE, please review

Reconcile then failed atomically: no matches for kind "ClusterIssuer". The single platform Kustomization applied HelmReleases (which register CRDs) AND CRs of those CRDs (ClusterIssuer, Certificate, Kyverno ClusterPolicies) in one pass → Flux SSA fails the WHOLE apply → the CRD-registering HelmReleases never run. Deadlock. Adopted the flux2 controllers/configs pattern: new platform-config Flux Kustomization (dependsOn: platform, wait) carries the CRs; platform keeps only controllers; apps now dependsOn platform-config. Applies to dev AND prod (shared base) — no R-ENV-02 divergence. platform-config/ references the CR files IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/ for the CI L0 kyverno test gate) — ASSUMPTION: Flux's default LoadRestrictionsNone permits the ../ refs (builds verified locally with --load-restrictor LoadRestrictionsNone; confirming at reconcile).

Fix 3 — Authored the missing Cilium L2 resources (in Fix 2 commit)

l2announcements.enabled: true was set but the IP pool + announcement policy were only a TODO, so a Gateway LoadBalancer would never get an IP. Authored platform-config/cilium/: CiliumLoadBalancerIPPool (dev NAT range 10.73.0.240-.250, matching the tofu reservation) + CiliumL2AnnouncementPolicy (interface regex ^en, node interface is ens3). ASSUMPTIONS: (a) VIP range hardcoded to the dev NAT range — a prod overlay must parameterize it (via cluster-vars) when prod is built; (b) ^en matches the node LAN NIC (ens3); (c) CRD apiVersion cilium.io/v2alpha1 (confirmed served on the cluster).

Fix 4 — Kyverno chart webhook schema (commit ~c6331c8)

Kyverno HelmRelease install FAILED at Helm templating: I'd written config.webhooks as a LIST [{failurePolicy, namespaceSelector}], but the kyverno 3.3.9 chart expects config.webhooks to be a MAP (webhooks.namespaceSelector), and the fail-open switch is a separate global config.forceFailurePolicyIgnore.enabled (bool), not a per-webhook field. Fixed both; renamed the cluster-var KYVERNO_WEBHOOK_FAILURE_POLICY(Fail/Ignore) → KYVERNO_FORCE_FAILURE_POLICY_IGNORE (false base / true dev). Verified with helm template before pushing. PROCESS SLIP: this fix landed directly on main (I forgot to branch first). Content is gate-green; noted for the record. Fixes 5+ went back through feature branches.

Fix 5 — cert-manager DNS-01 self-check via public resolvers (commit ~59dd3d9)

Wildcard cert stalled: challenge Presented=true (Cloudflare TXT created — the token WORKS) but the propagation self-check looped "DNS record not yet propagated". VERIFIED the record IS publicly resolvable (1.1.1.1/8.8.8.8); the NAT'd k3s-dev cluster's own DNS just can't see it. Set dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53" + dns01RecursiveNameserversOnly: true (realises a pre-existing TODO in the file). Challenge went pending→valid, cert Issued.

Fix 6 — cilium-operator restart to program the Gateway (MANUAL recovery, see note)

Gateway stuck Accepted=Unknown:Pending for ~20min. cilium-operator logs: no kind is registered for the type v1.Gateway in scheme → one-shot job errored ... leader-lifecycle — the operator's gateway-controller failed scheme registration during the Flux adoption restart and gave up (rest of the operator ran fine). kubectl -n kube-system rollout restart deploy/cilium-operator fixed it instantly: Gateway → Accepted+Programmed, address 10.73.0.240 from the L2 pool, cilium-gateway-shared LB service created. ASSUMPTION/NOTE: this was a manual (non-IaC) recovery of a transient adoption casualty; a fresh ansible bootstrap installs the Gateway API CRDs BEFORE Cilium, so a clean rebuild should not hit it — VERIFY on the next destroy/rebuild, and if it recurs, codify an operator-restart handler (or bump Cilium). Not codified now.

RESULT — Phase-2 platform FULLY UP + green (2026-08-17)

platform, platform-config, apps all Ready=True (revision 59dd3d9). 5/5 HelmReleases Ready; wildcard *.home.keithjackson.dev cert Issued (LE-staging per ADR-0006); Gateway Programmed @ 10.73.0.240; 6 Kyverno policies Ready (Audit in dev); Longhorn default SC live; no not-ready pods.

FOLLOW-UPS (non-blocking, for review)

1. Two default StorageClasses — k3s's built-in local-path AND longhorn are both marked default. Ambiguous. Fix: disable k3s local-storage in the ansible k3s_server args (--disable local-storage) OR strip the default annotation from local-path so longhorn is the sole default. (Non-blocking: platform components pin storageClassName: longhorn explicitly.)
2. Phase-2 EXIT not yet verified — the roadmap Phase-2 exit is passing conformance (R-NET-01 default-deny, R-TLS-01/03 cert served, R-DNS-02 auto record). Platform is UP; conformance run is the next step (dev cert is LE-STAGING = non-self-signed but not browser-trusted — expected in dev).
3. cilium-operator gateway restart (Fix 6) — verify on next rebuild / codify if it recurs.

Autonomous session 2 — Phase-2 verification + hardening (2026-08-17)

Operator away again, same SOP. All safe, no spec changes.

• R-NET conformance GREEN against the DEPLOYED platform: ran conformance/run.sh --layer l2 --requirements R-NET-01,R-NET-02 → default-deny (R-NET-01) + deny-undeclared (R-NET-02) both PASS (2/2). Confirms the live platform enforces the network isolation baseline, not just the K2 fixtures.
• TLS path functionally VERIFIED (not the formal test): openssl s_client to the gateway VIP 10.73.0.240 with SNI returns CN=*.home.keithjackson.dev, SAN {*.home, home}.keithjackson.dev, issuer Let's Encrypt (STAGING), 90-day validity. The full DNS-01 → issuance → gateway-termination path works. (Dev cert is LE-STAGING per ADR-0006 = non-self-signed but NOT system-trusted.)
• Fix: single default StorageClass (commit ~7a45eb1) — added local-storage to the k3s --disable list so longhorn is the sole default. Idempotent install task → effective on rebuild, no disruption to the running cluster (which harmlessly keeps both defaults; all platform PVCs pin longhorn). ASSUMPTION: disabling k3s local-path entirely is fine (Longhorn is the storage).
• Fix: conformance run.sh CWD-robustness (commit ~ee5ad90) — chainsaw's config report.path is relative, so run.sh exited non-zero (breaking the L2 gate) when invoked from repo root. Pass absolute --report-path ${ARTIFACTS} + mkdir. Verified exit 0 + report written from repo root.

REMAINING for Phase-2 EXIT (needs an operator call — did NOT do autonomously)

The roadmap Phase-2 exit is conformance-green for R-TLS + R-DNS too, but those tests are Phase-0 scaffolds (TODO placeholders) — they must be AUTHORED against the live platform. Two reasons I deferred rather than best-effort them:

1. R-TLS trusted-cert (trusted-tls test) — DECISION NEEDED: the test checks the SYSTEM trust store (no -k), but dev serves an LE-staging cert (not system-trusted). How should dev verify "trusted"? (a) verify the cert chains to the real LE staging root (embed the staging root in the probe) — most faithful; (b) verify LE-issued + valid + SAN + non-self-signed only in dev, full system-trust in prod; (c) use LE-prod for a designated test host. Recommend (a) or (b). Side-effect-free to implement once decided.
2. R-DNS auto-record (dns-exposure test) — EXTERNAL SIDE-EFFECT: verifying R-DNS-02 means letting ExternalDNS create/delete a real record in the live keithjackson.dev Cloudflare zone. Deferred to run with the operator present (or confirm it's fine to exercise autonomously with guaranteed cleanup).

Phase-2 CONFORMANCE EXIT ACHIEVED (2026-08-17, main @ 1762206 + cluster-vars re-apply)

Operator decided: TLS trust = A (chain to LE-staging root); DNS test = OK to run live. Authored both Phase-0 scaffolds into real tests against the deployed platform; all 4 Phase-2 L2 tests PASS: R-NET-01 (default-deny), R-NET-02 (deny-undeclared), R-TLS-01/03 (trusted-tls), R-DNS-02 (dns-exposure). This covers the roadmap Phase-2 exit demonstrables (default-deny holds, trusted cert served at *.home.keithjackson.dev, auto DNS record appears).

• trusted-tls (R-TLS-01/03): in-cluster curl probe dials the gateway Service (env-agnostic via --connect-to, no hardcoded VIP) with wildcard SNI and requires the served chain to verify against the embedded LE-staging root ("(STAGING) Pretend Pear X1", fixtures/le-staging-root-ca.yaml). Probes a public + an internal-only hostname (R-TLS-03). Empirically validated: correct root → rc 0; system trust → rc 60.
• dns-exposure (R-DNS-02): a LoadBalancer Service + external-dns hostname annotation; polls public DNS (1.1.1.1) until ExternalDNS publishes the A record into the cluster LB range; chainsaw teardown
  • policy=sync removes it. Verified the full CREATE→DELETE lifecycle in ExternalDNS logs; the live keithjackson.dev zone is left clean (back to the pre-existing wildcard).

FINDING F-DNS1 — ExternalDNS domain-filter must be the ZONE, not the subdomain (FIXED)

The DNS test first failed: ExternalDNS generated the record but skipped it — "no hosted zone matching record DNS Name". --domain-filter=home.keithjackson.dev filtered OUT the real Cloudflare zone keithjackson.dev (a subdomain filter drops the parent zone). Added a DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters: [${DNS_ZONE}]. Scope stays effectively home.keithjackson.dev (only suffix services request) and policy=sync+txt-registry never touch non-owned records.

FINDING F-OPS1 — cluster-vars changes need a manual kubectl apply -k — RESOLVED (main @ 4381a71)

Added clusters/dev/sync.yaml: a self-managing ROOT Kustomization (flux2 gotk-sync pattern, name flux-system) that reconciles ./clusters/dev from git — cluster-vars, the GitRepository, platform/ platform-config/apps, and itself (inventory = 6 objects, confirmed). Bootstrap unchanged (kubectl apply -k clusters/dev creates it once). Verified: root → Ready, adopts the graph without disruption, all Kustomizations settle green; a git push now auto-applies overlay/cluster-vars changes. prod needs its own sync.yaml (→ ./clusters/prod) when built. Original gap description below:

Adding DNS_ZONE didn't take effect until I re-ran kubectl apply -k clusters/dev (the on-cluster cluster-vars ConfigMap isn't reconciled by Flux — this bootstrap model has NO self-managing root Kustomization that watches clusters/dev). Then a platform Kustomization reconcile re-ran the postBuild substitution. RECOMMEND: add a self-managing root Flux Kustomization (path ./clusters/dev) so cluster-vars / graph changes auto-apply — otherwise every overlay change is a manual re-apply (and, transiently, ${DNS_ZONE} substituted empty → --domain-filter= = all-zones, before re-apply).

FINDING F-KY1 — kyverno chart's cleanup image (bitnami/kubectl) purged from Docker Hub (FIXED)

Surfaced when a platform reconcile triggered a kyverno helm upgrade: the chart's cleanup jobs (policyReportsCleanup, webhooksCleanup) default to bitnami/kubectl:1.30.2, which Bitnami REMOVED from Docker Hub (2024 catalog purge) → ImagePullBackOff. policyReportsCleanup runs as a Helm post-upgrade HOOK, so its failure stalled the upgrade (5m timeout) → kyverno HR not-Ready → platform Unknown → platform-config/apps cascade to False. Fixed: override both cleanup images to Bitnami's relocated bitnamilegacy/kubectl mirror (identical image — has /bin/bash + awk/wc/xargs + kubectl the jobs need; verified via helm template that no bitnami/kubectl ref remains). After the fix the hook job Completed, kyverno HR → True, and the whole graph returned to green (platform/platform-config/apps all Ready, 5/5 HelmReleases Ready, no unhealthy pods). NOTE: bitnamilegacy is itself a temporary mirror — a durable follow-up is to pin a maintained kubectl image (shell-capable, e.g. rancher or alpine/k8s) once tested, or a newer kyverno chart that drops the bitnami dependency.

R-STATE-01 GREEN + harness fix F-TEST1 (2026-08-17, main @ 76478f2)

• R-STATE-01 (data durability) PROVEN: the L3 data-survival test runs against live Longhorn — writes a canary to a longhorn PVC, DESTROYS the pod entirely, re-binds a fresh pod to the same PVC, and the data survives (DATA_SURVIVED_OK). This is the platform's headline stateful-durability requirement.
• F-TEST1 — every chainsaw check used an unsupported =~ operator (FIXED): all 10 tests wrote (($stdout =~ 'X')): true; chainsaw's expression engine rejects =~ ('SyntaxError: Unknown char: ~'), so each test FAILED on the assertion even when the behavior passed (caught because data-survival printed DATA_SURVIVED_OK yet FAILED). Converted all to JMESPath contains($stdout, 'X') in BLOCK style (flow { } breaks on the comma inside contains()). data-survival passes; the 9 later-phase tests (egress/vpn/alerting/dr/updates/drift/sso) are now syntactically correct for when their phases deploy.
• R-NET-01/02 FAIL-CLOSED (L3) PROVEN (main @ cde3335): implemented the default-deny-failclosed scaffold — fixtures = default-deny baseline + source + dest:8080 + the sole explicit allow. Verifies source→dest works, then REVOKES the allow (real fault: kubectl delete networkpolicy), and asserts connectivity fails CLOSED while DNS still resolves. Proves default-deny is datapath-enforced, not just declared. PASSES. (DNS check asserts the resolved answer — busybox nslookup exits non-zero on success.) Conformance green now: R-NET-01/02 (L2+L3), R-TLS-01/03, R-DNS-02, R-STATE-01. Remaining tests need later-phase components (VPN P3, monitoring/backup/updates P4, SSO P6) → operator secrets.

Phase 3 + 4 ("Both") + an infra recovery (2026-08-17, main @ 4f17c53)

Operator: WireGuard preferred (I confirmed PIA supports it on all regions + Gluetun derives the WG config from username/password, so the PIA secret shape is unchanged); keycloak creds added; oauth2-proxy deferred to Phase 6; "proceed to next phase." Chose Both (deploy Phase-4 monitoring + author Phase-3 VPN).

INFRA RECOVERY — cluster VMs + Gitea were found STOPPED (not my doing)

Mid-task, kubectl + git push failed: the 3 k3s VMs were shut off and the Gitea container stopped (host up 12.5h — not a reboot; something halted them). Recovered via the operator-provisioned ansible-become path: virsh -c qemu:///system start k3s-dev-{1,2,3} → cluster back, all Kustomizations + HelmReleases Ready from disk in ~80s; restarted Gitea (named volume gitea-bootstrap-data persisted) and pushed the queued commits. No data lost. (The VMs are SYSTEM-libvirt — user-session virsh shows nothing; recovery needs sudo, which only the ansible vault-become has.)

Phase 4 — MONITORING DEPLOYED + healthy

Enabled monitoring in platform/kustomization.yaml. All 3 HelmReleases Ready (kube-prometheus-stack 65.8.1, Loki 6.55.0, Alloy 0.12.6); Prometheus + Loki PVCs bound on Longhorn (5Gi each); 38 PrometheusRules applied; no unhealthy pods. Grafana admin + Alertmanager SMTP from the operator's SOPS secrets. SMTP caveats (flagged, not blocking): operator set smtp_smarthost=smtp.gmail.com:587, [email protected], [email protected]; I set the receiver to: to a valid inbox (was REPLACE-ME). Gmail needs an App Password and the From must be the auth user or a verified send-as alias, else delivery fails at alert time (stack health unaffected).

FINDING F-MON1 — monitoring PrometheusRule CRD ordering (FIXED)

Enabling monitoring first deadlocked the platform Kustomization: alerts.yaml (PrometheusRule CRs) needs the monitoring.coreos.com CRDs the kube-prometheus-stack HelmRelease installs in the SAME pass → atomic 'no matches for kind PrometheusRule' → NO monitoring HRs created. Same controllers/configs fix (F-DNS1/kyverno lineage): moved alerts.yaml to platform-config (dependsOn platform). This is now a RECURRING pattern — any component shipping CRs of a CRD it installs must split the CRs into platform-config.

Phase 3 — VPN egress gateway AUTHORED (author-first, not deployed)

platform/vpn-egress/: single shared Gluetun gateway (PIA/WireGuard, kill-switch FIREWALL=on, native PF, control-API Service), non-app namespace (NET_ADMIN + /dev/net/tun), pre-wired (commented) pia-credentials line. Defined the PIA secret location (platform/vpn-egress, keys username/password → OPENVPN_USER/OPENVPN_PASSWORD) and updated the SOPS template. NOT wired into the graph. Open design (README): egress steering (vpn-class pods → the one shared tunnel) — recommended Cilium Egress Gateway → the Gluetun node's wg0 (fail-closed when the tunnel drops), deliberately NOT committed as an untested CiliumEgressGatewayPolicy. Deploy needs PIA creds + PIA_REGION (PF-capable) + CLUSTER_POD_CIDR + steering validation + the qbittorrent-natmap port-sync helper. Cluster fully green (4/4 KS, 8/8 HR).

KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)

#	Issue	Impact	Fix
K1	Substitution-var case mismatch (UPPERCASE refs vs lowercase cluster-vars keys)	—	✅ FIXED: uppercased cluster-vars keys + dev/prod overlay paths; all refs resolve; builds green
K2	Conformance L2 tests partial/unverified	—	✅ DONE: fixed 3 test bugs (see below), ran via run.sh against the live cluster → R-NET-01 default-deny + R-NET-02 deny-undeclared PASS, JUnit+coverage written, exit 0. chainsaw v0.2.12 installed (host ~/.local/bin)
K3	bootstrap/gitea/bootstrap-gitea.sh used a host bind-mount → rootless-podman perms broke app.ini	initial Gitea start failed	FIXED: switched to a named volume
K4	Kyverno/chainsaw findings during K2 (fixed in the tests): (a) chainsaw does STRICT (length-equal) list matching — asserting one condition against a pod's full conditions[] fails → used scalar phase: Running; (b) a default-deny baseline blocks the kubelet readiness probe (node→pod), so probe-bearing pods never go Ready → dropped probes; (c) chainsaw report path: ../_artifacts resolved to repo-root → exit 1 → fixed to _artifacts; (d) swapped agnhost→busybox httpd (cached, cache-timing-proof)	—	FIXED in the test files/config

FINDING for WS-NET (Phase 2, T-2.2 — default-deny baseline design)

The platform's real default-deny baseline must not break kubelet health/liveness/readiness probes (node→pod ingress). A naive ingress: [] NetworkPolicy in Cilium drops the kubelet TCP probe, so any pod with a probe never becomes Ready. The baseline must explicitly allow the health-check path (Cilium host/health-check allowance or an equivalent ingress exception). Surfaced empirically by the R-NET-02 conformance test; carry into T-2.2.

Toolchain note

chainsaw v0.2.12 is installed on the host (~/.local/bin) for local conformance runs; for full reproducibility it should also be baked into infra/devcontainer (pinned) — the in-cluster Chainsaw Job (ADR-0011) already uses the pinned image. Deferred (non-blocking).

Blocked on operator (hand-offs)

• T-2.4p Cloudflare API token → cert-manager DNS-01 + ExternalDNS (Phase 2 TLS/DNS).
• T-3.2p PIA username/password → VPN egress gateway (Phase 3).
• OQ-BKP off-cluster backup target → Velero (Phase 4).
• SMTP creds → Alertmanager email (Phase 4).

Disk/drive incident + k3s relocated to NVMe (2026-08-22)

Trigger: host root fs (Kingston SSD /dev/sdb3, btrfs) hit ~99% (<1GB); the system paused the k3s VMs on I/O error. Operator asked to move the VM disks off root to a larger drive.

Why root filled: the k3s VM disks lived under the snapper-snapshotted root subvolume; high VM-disk churn multiplied across hourly snapshots. Freed by deleting the 11 Snapper timeline snapshots + virsh destroy-ing the paused VMs (which held the old, rsync-unlinked inodes open) → btrfs reclaim. Root 99% → 30% (326G free).

F-DISK1 — the Seagate 4TB HDD is FAILING. First relocated the VM disks to the HDD (/mnt/seagate); etcd then hit 48% iowait + constant "apply took too long", pods drifted to Unknown, never converged. SMART on /dev/sda (ST4000DM000): 29,408 reallocated, 448 current-pending + 448 offline-uncorrectable, ~25k power-on hrs. A dying drive, not "HDD is slow." Do NOT use for etcd/critical data.

Resolution: k3s disks moved to the WD_BLACK SN770 NVMe (/dev/nvme0n1p3, /mnt/games) in a dedicated btrfs subvol /mnt/games/libvirt-k3s (nodatacow), bind-mounted to /var/lib/libvirt/images so tofu/libvirt paths + the qcow2 backing chain are unchanged (no IaC edits). etcd iowait → 0%, cluster fully converged (64 Running). fstab updated; SELinux relabeled. Archival VMs (rocky10/silverblue/win11) repointed to their real HDD path and LEFT on the failing drive.

Follow-ups (assumptions logged): (1) the NVMe bind-mount is a runtime change — codify in an Ansible host-prereqs task for reproducibility; (2) redundant k3s copies remain at /mnt/seagate/libvirt-images/k3s-dev-* (safe to delete; NVMe authoritative) — left pending operator OK; (3) archival VMs on the failing HDD need migration + the disk replaced.

Phase 3 VPN — protocol resolved to OpenVPN-UDP; deploying (2026-08-22)

ADR-0013 said PIA/WireGuard, but Gluetun does not support PIA over WireGuard (upstream: native support "cannot be added… slow WIP"; live error "provider name is not valid for Wireguard"). Operator (as System Architect) approved OpenVPN-UDP and ADR-0013 was amended in the spec repo (protocol only; architecture unchanged; thrnz/docker-wireguard-pia recorded as the WireGuard fallback). Impl aligned: VPN_TYPE=openvpn, vpn-egress ENABLED in platform/kustomization.yaml, stale WireGuard comments cleaned (gluetun-gateway.yaml, README; steering iface wg0→tun0). Pushed to Gitea → Flux reconciles the OpenVPN gateway. Stale WireGuard comments remain inside the operator's encrypted pia-credentials.sops.yaml header — left untouched (editing breaks the SOPS MAC); cosmetic.

Next: confirm the OpenVPN tunnel comes up + correct egress IP, then egress-steering validation (README option 1, tun0) + kill-switch conformance (R-NET-06).

Phase 3 VPN — tunnel BLOCKED by home network (2026-08-22)

After deploying OpenVPN + fixing the region (SE Stockholm), gluetun goes Running and OpenVPN dials PIA Sweden servers over UDP/1197, but the handshake never completes — no TLS: Initial packet from the server, gluetun healthcheck restarts it in a loop. Not auth (no AUTH_FAILED), not a k8s NetworkPolicy (vpn-egress is app-namespace=false, no policies touch it), not the cluster/host path.

Traced with tcpdump end-to-end:

• On the node (k3s-dev-3): pod 10.0.2.161 → Cilium SNAT → ens3 Out 10.73.0.13 > 46.246.x:1197. Leaves fine.
• On the host: virbr1 In 10.73.0.13 > 46.246.x:1197 → wlp4s0 Out 192.168.2.103 > 46.246.x:1197. Host NATs it out the WiFi onto the 192.168.2.x home LAN. Leaves fine.
• On the host wlp4s0, filtering src net 46.246.0.0/16: 0 reply packets in 20s. Meanwhile host UDP/53 (dig @9.9.9.9) works. → OpenVPN UDP replies never return to the host.

Conclusion: the break is UPSTREAM of the host — the home router/ISP on the 192.168.2.x WiFi network is dropping/not-returning OpenVPN UDP/1197 (TCP egress + UDP/53 both work; only the VPN UDP is affected; this box is on 192.168.2.x via WiFi, not the main 192.168.1.x LAN). Environmental, outside the repo.

Action → operator: fix at the network (check the 192.168.2.x router for outbound UDP/1197 or VPN filtering; try a wired connection to the main 192.168.1.x LAN; or a different port/provider). No in-config workaround — the user requires non-TCP, and WireGuard (the fallback) is also UDP so likely blocked the same way. vpn-egress reverted to disabled meanwhile (config is complete + correct; one-line re-enable once the network passes OpenVPN UDP).

Phase 3 VPN — RESOLVED: it was a port-1197 collision, moved to UDP/53 (2026-08-22)

Operator clarified: the home network already runs its own OpenVPN server NAT'd on WAN:1197, which collided with the cluster's outbound PIA-on-1197 flow at the router (why UDP/53 worked but 1197 didn't — port-specific, not a general UDP block). Fix: set OPENVPN_ENDPOINT_PORT to a non-1197 PIA OpenVPN port. PIA accepts 53, 1194, 1197, 1198, 8080, 9201 (gluetun issue #2252). Chose 53 — empirically proven to traverse this network (the dig @9.9.9.9 test) and no conflict. Added cluster-var PIA_OPENVPN_PORT="53", wired to gluetun OPENVPN_ENDPOINT_PORT, and re-enabled vpn-egress. Port-forwarding is an over-tunnel PIA API call, unaffected by the transport port. (WireGuard fallback would also have worked — PIA WG is UDP/1337, not 1197 — but staying on Gluetun+OpenVPN is the smaller change.)