diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 6e68972..76bbc7e 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -906,3 +906,19 @@ renderer skip-mode first; trusted-tls test → LE-prod root; oauth2-proxy tier m - **FOLLOW-UPS:** the Cilium fix; re-verify pihole reachability + trusted-tls; oauth2-proxy->post-core Flux tier; enable image-automation (Gitea write-cred + controllers); WS-APPS (gitea->...); C2 UI; Velero (OQ-BKP); prod sync.yaml. + +**RESOLVED: the "Cilium pod->Gateway-VIP bug" was TRANSIENT (crash + hard reboot cleared it) (2026-08-25 AM):** +After the dev host crashed and was hard-rebooted, I re-tested in-cluster gateway reachability. From pods on +BOTH non-announcer nodes (k3s-dev-2 AND k3s-dev-3): DNS resolves *.home.keithjackson.dev -> VIP 10.73.0.240, +`grafana.home` -> 302, `keycloak.home` OIDC discovery -> 200, and **`pihole.home` -> 403 (pihole's own +response - the gateway REACHED the app)**. So pod->Gateway-VIP works from all nodes now; the pre-reboot +timeouts were **transient Cilium BPF/state degradation** (from the heavy KubeVirt + leaked-TCG-VM load), NOT +the `bpf-lb-external-clusterip=false` config theory. **NO CNI CHANGE NEEDED - config is correct.** Consequences: +(1) the deferred/risky Cilium LB fix is UNNECESSARY; (2) **pihole is reachable through the Gateway** - the +gateway-ingress `fromEntities:[ingress]` fix works, WS-APPS golden path fully proven; (3) **trusted-tls +R-TLS-01/03 GREEN** (rewrote the probe for LE-prod system-trust + split-horizon DNS; passes live in 9s, main @ +1eb2cb7). Earlier IP-SNI probes returned curl exit 35 (TLS error, expected - the gateway needs a +*.home.keithjackson.dev SNI), which had confused the pre-reboot diagnosis alongside the real transient timeouts. +LESSON: heavy sustained node load (TCG-emulated VMs) can degrade Cilium datapath -> intermittent LB-VIP +reachability; a restart clears it. NOTE: after a host reboot, restart the bootstrap Gitea container +(`bash bootstrap/gitea/bootstrap-gitea.sh`) - it is Flux's source and does not auto-start; the k3s VMs auto-recover.