Merge Phase-2 deployed decision log

2026-08-17 01:53:13 -05:00
commit a07151ab7c
@@ -232,9 +232,48 @@ Gateway LoadBalancer would never get an IP. Authored `platform-config/cilium/`:
 NAT range — a prod overlay must parameterize it (via cluster-vars) when prod is built; (b) `^en` matches
 the node LAN NIC (`ens3`); (c) CRD apiVersion `cilium.io/v2alpha1` (confirmed served on the cluster).

-### Result
-Un-suspend reconcile in progress at last checkpoint (controllers installing → platform-config → apps).
-Outcome + any further fixes appended when the reconcile settles.
+### Fix 4 — Kyverno chart webhook schema (commit ~c6331c8)
+Kyverno HelmRelease install FAILED at Helm templating: I'd written `config.webhooks` as a LIST
+`[{failurePolicy, namespaceSelector}]`, but the kyverno 3.3.9 chart expects `config.webhooks` to be a
+MAP (`webhooks.namespaceSelector`), and the fail-open switch is a separate global
+`config.forceFailurePolicyIgnore.enabled` (bool), not a per-webhook field. Fixed both; renamed the
+cluster-var `KYVERNO_WEBHOOK_FAILURE_POLICY`(Fail/Ignore) → `KYVERNO_FORCE_FAILURE_POLICY_IGNORE`
+(false base / true dev). Verified with `helm template` before pushing.
+**PROCESS SLIP:** this fix landed directly on `main` (I forgot to branch first). Content is gate-green;
+noted for the record. Fixes 5+ went back through feature branches.
+
+### Fix 5 — cert-manager DNS-01 self-check via public resolvers (commit ~59dd3d9)
+Wildcard cert stalled: challenge `Presented=true` (Cloudflare TXT created — the token WORKS) but the
+propagation self-check looped "DNS record not yet propagated". VERIFIED the record IS publicly
+resolvable (1.1.1.1/8.8.8.8); the NAT'd k3s-dev cluster's own DNS just can't see it. Set
+`dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` + `dns01RecursiveNameserversOnly: true` (realises
+a pre-existing TODO in the file). Challenge went pending→valid, cert **Issued**.
+
+### Fix 6 — cilium-operator restart to program the Gateway (MANUAL recovery, see note)
+Gateway stuck `Accepted=Unknown:Pending` for ~20min. cilium-operator logs: `no kind is registered for
+the type v1.Gateway in scheme` → `one-shot job errored ... leader-lifecycle` — the operator's
+gateway-controller failed scheme registration during the Flux **adoption** restart and gave up (rest of
+the operator ran fine). `kubectl -n kube-system rollout restart deploy/cilium-operator` fixed it
+instantly: Gateway → Accepted+Programmed, **address 10.73.0.240** from the L2 pool, `cilium-gateway-shared`
+LB service created. **ASSUMPTION/NOTE:** this was a manual (non-IaC) recovery of a transient adoption
+casualty; a fresh ansible bootstrap installs the Gateway API CRDs BEFORE Cilium, so a clean rebuild
+should not hit it — VERIFY on the next destroy/rebuild, and if it recurs, codify an operator-restart
+handler (or bump Cilium). Not codified now.
+
+### RESULT — Phase-2 platform FULLY UP + green (2026-08-17)
+`platform`, `platform-config`, `apps` all Ready=True (revision 59dd3d9). 5/5 HelmReleases Ready;
+wildcard `*.home.keithjackson.dev` cert Issued (LE-staging per ADR-0006); Gateway Programmed @
+10.73.0.240; 6 Kyverno policies Ready (Audit in dev); Longhorn default SC live; no not-ready pods.
+
+### FOLLOW-UPS (non-blocking, for review)
+1. **Two default StorageClasses** — k3s's built-in `local-path` AND `longhorn` are both marked
+   default. Ambiguous. Fix: disable k3s local-storage in the ansible k3s_server args
+   (`--disable local-storage`) OR strip the default annotation from local-path so `longhorn` is the
+   sole default. (Non-blocking: platform components pin `storageClassName: longhorn` explicitly.)
+2. **Phase-2 EXIT not yet verified** — the roadmap Phase-2 exit is passing conformance (R-NET-01
+   default-deny, R-TLS-01/03 cert served, R-DNS-02 auto record). Platform is UP; conformance run is the
+   next step (dev cert is LE-STAGING = non-self-signed but not browser-trusted — expected in dev).
+3. **cilium-operator gateway restart** (Fix 6) — verify on next rebuild / codify if it recurs.

 ## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
 | # | Issue | Impact | Fix |