From 0811e2533bc6cec36afc049114f0c49bedba11ad Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 17 Aug 2026 01:53:13 -0500 Subject: [PATCH] docs: Phase-2 platform fully deployed + green (fixes 4-6, follow-ups) Co-Authored-By: Claude Opus 4.8 (1M context) --- DECISIONS-LOG.md | 45 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index e312290..3453547 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -232,9 +232,48 @@ Gateway LoadBalancer would never get an IP. Authored `platform-config/cilium/`: NAT range — a prod overlay must parameterize it (via cluster-vars) when prod is built; (b) `^en` matches the node LAN NIC (`ens3`); (c) CRD apiVersion `cilium.io/v2alpha1` (confirmed served on the cluster). -### Result -Un-suspend reconcile in progress at last checkpoint (controllers installing → platform-config → apps). -Outcome + any further fixes appended when the reconcile settles. +### Fix 4 — Kyverno chart webhook schema (commit ~c6331c8) +Kyverno HelmRelease install FAILED at Helm templating: I'd written `config.webhooks` as a LIST +`[{failurePolicy, namespaceSelector}]`, but the kyverno 3.3.9 chart expects `config.webhooks` to be a +MAP (`webhooks.namespaceSelector`), and the fail-open switch is a separate global +`config.forceFailurePolicyIgnore.enabled` (bool), not a per-webhook field. Fixed both; renamed the +cluster-var `KYVERNO_WEBHOOK_FAILURE_POLICY`(Fail/Ignore) → `KYVERNO_FORCE_FAILURE_POLICY_IGNORE` +(false base / true dev). Verified with `helm template` before pushing. +**PROCESS SLIP:** this fix landed directly on `main` (I forgot to branch first). Content is gate-green; +noted for the record. Fixes 5+ went back through feature branches. + +### Fix 5 — cert-manager DNS-01 self-check via public resolvers (commit ~59dd3d9) +Wildcard cert stalled: challenge `Presented=true` (Cloudflare TXT created — the token WORKS) but the +propagation self-check looped "DNS record not yet propagated". VERIFIED the record IS publicly +resolvable (1.1.1.1/8.8.8.8); the NAT'd k3s-dev cluster's own DNS just can't see it. Set +`dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` + `dns01RecursiveNameserversOnly: true` (realises +a pre-existing TODO in the file). Challenge went pending→valid, cert **Issued**. + +### Fix 6 — cilium-operator restart to program the Gateway (MANUAL recovery, see note) +Gateway stuck `Accepted=Unknown:Pending` for ~20min. cilium-operator logs: `no kind is registered for +the type v1.Gateway in scheme` → `one-shot job errored ... leader-lifecycle` — the operator's +gateway-controller failed scheme registration during the Flux **adoption** restart and gave up (rest of +the operator ran fine). `kubectl -n kube-system rollout restart deploy/cilium-operator` fixed it +instantly: Gateway → Accepted+Programmed, **address 10.73.0.240** from the L2 pool, `cilium-gateway-shared` +LB service created. **ASSUMPTION/NOTE:** this was a manual (non-IaC) recovery of a transient adoption +casualty; a fresh ansible bootstrap installs the Gateway API CRDs BEFORE Cilium, so a clean rebuild +should not hit it — VERIFY on the next destroy/rebuild, and if it recurs, codify an operator-restart +handler (or bump Cilium). Not codified now. + +### RESULT — Phase-2 platform FULLY UP + green (2026-08-17) +`platform`, `platform-config`, `apps` all Ready=True (revision 59dd3d9). 5/5 HelmReleases Ready; +wildcard `*.home.keithjackson.dev` cert Issued (LE-staging per ADR-0006); Gateway Programmed @ +10.73.0.240; 6 Kyverno policies Ready (Audit in dev); Longhorn default SC live; no not-ready pods. + +### FOLLOW-UPS (non-blocking, for review) +1. **Two default StorageClasses** — k3s's built-in `local-path` AND `longhorn` are both marked + default. Ambiguous. Fix: disable k3s local-storage in the ansible k3s_server args + (`--disable local-storage`) OR strip the default annotation from local-path so `longhorn` is the + sole default. (Non-blocking: platform components pin `storageClassName: longhorn` explicitly.) +2. **Phase-2 EXIT not yet verified** — the roadmap Phase-2 exit is passing conformance (R-NET-01 + default-deny, R-TLS-01/03 cert served, R-DNS-02 auto record). Platform is UP; conformance run is the + next step (dev cert is LE-STAGING = non-self-signed but not browser-trusted — expected in dev). +3. **cilium-operator gateway restart** (Fix 6) — verify on next rebuild / codify if it recurs. ## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed) | # | Issue | Impact | Fix |