From 4260487af549d29ca42855a54faa3d2a7493c4a0 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 17 Aug 2026 01:08:35 -0500 Subject: [PATCH] docs: log Phase-2 un-suspend + fixes (longhorn coercion, controllers/configs split, cilium L2) Co-Authored-By: Claude Opus 4.8 (1M context) --- DECISIONS-LOG.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 84e8c10..e312290 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -192,6 +192,50 @@ line with the existing architecture." **Correct — and it exposed a scope disto 7. **Chart-key verification** (flagged in templates): keycloak + oauth2-proxy `existingSecret` key names, Keycloak DB secret, and PIA/Gluetun protocol (≠ TCP) — verify at each component's phase. +## Phase-2 UN-SUSPEND executed + fixes (autonomous session, 2026-08-16 overnight) + +Operator said "go" for the un-suspend, then went AFK: "continue to work without me. Any major decision +that requires a system architecture change VIA SPEC CHANGE will need to be escalated. Anything else is +allowed. Keep track of assumptions and report on return." None of the below is a spec change (the spec +is silent on Flux Kustomization granularity / L2 wiring), so all within PM remit. **Nothing touched the +read-only spec repo.** + +- **Un-suspended** via the sanctioned `kubectl apply -k clusters/dev` (bootstrap.sh step 3): controllers + already installed, `sops-age` present. Pre-deploy checks all green first — Cilium adoption safe (an + existing `helm` release `cilium` v3 exists → Flux adopts by name, no-op upgrade), Longhorn node prereqs + green on all 3 nodes (`iscsid` active, `/var/lib/longhorn`→`/dev/vdb`), host Gitea pushed + Flux source Ready. + +### Fix 1 — Longhorn StorageClass numeric coercion (commit ~c04aea8) +Reconcile stalled: `.parameters.numberOfReplicas: expected string, got 3`. kustomize strips the quotes +around a bare `"${VAR}"` placeholder, so Flux substitution yielded an unquoted numeric that SSA rejects +(StorageClass.parameters is map[string]string). **Hardcoded `numberOfReplicas: "3"`** (quoted numeric +literals ARE preserved by kustomize; topology LOCKED at 3). The replica knob still drives the global +default via `defaultSettings.defaultReplicaCount` in the helmrelease (a Helm int value, unaffected). + +### Fix 2 — DECISION: controllers/configs split (commit ~458a867) — NOTABLE, please review +Reconcile then failed atomically: `no matches for kind "ClusterIssuer"`. The single `platform` +Kustomization applied HelmReleases (which register CRDs) AND CRs of those CRDs (ClusterIssuer, +Certificate, Kyverno ClusterPolicies) in one pass → Flux SSA fails the WHOLE apply → the CRD-registering +HelmReleases never run. Deadlock. **Adopted the flux2 controllers/configs pattern:** new +`platform-config` Flux Kustomization (`dependsOn: platform`, wait) carries the CRs; `platform` keeps only +controllers; `apps` now `dependsOn platform-config`. Applies to dev AND prod (shared base) — no R-ENV-02 +divergence. `platform-config/` references the CR files IN PLACE via `../platform/...` (Flux builds with +LoadRestrictionsNone; Kyverno policies must stay under `platform/kyverno/` for the CI L0 `kyverno test` +gate) — **ASSUMPTION: Flux's default LoadRestrictionsNone permits the `../` refs** (builds verified +locally with `--load-restrictor LoadRestrictionsNone`; confirming at reconcile). + +### Fix 3 — Authored the missing Cilium L2 resources (in Fix 2 commit) +`l2announcements.enabled: true` was set but the IP pool + announcement policy were only a TODO, so a +Gateway LoadBalancer would never get an IP. Authored `platform-config/cilium/`: **CiliumLoadBalancerIPPool** +(dev NAT range **10.73.0.240-.250**, matching the tofu reservation) + **CiliumL2AnnouncementPolicy** +(interface regex `^en`, node interface is `ens3`). **ASSUMPTIONS:** (a) VIP range hardcoded to the dev +NAT range — a prod overlay must parameterize it (via cluster-vars) when prod is built; (b) `^en` matches +the node LAN NIC (`ens3`); (c) CRD apiVersion `cilium.io/v2alpha1` (confirmed served on the cluster). + +### Result +Un-suspend reconcile in progress at last checkpoint (controllers installing → platform-config → apps). +Outcome + any further fixes appended when the reconcile settles. + ## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed) | # | Issue | Impact | Fix | |---|---|---|---|