feat(dns): split-horizon in-cluster DNS via coredns-custom (R-DNS-01/R-DNS-03)

In-cluster clients resolving a platform hostname over its PUBLIC name got the
off-cluster answer, so pod->keycloak.home.keithjackson.dev hit a path that does not
serve the *.home.keithjackson.dev SNI -> `tls: unrecognized name` (oauth2-proxy OIDC
discovery crash; Grafana generic_oauth would hit the same wall).

Add a GitOps-managed coredns-custom ConfigMap (kube-system) that k3s CoreDNS imports as
a `*.server` block. The template plugin answers the whole home.keithjackson.dev zone with
${GATEWAY_VIP} (the shared Gateway VIP), so pod->public-hostname traffic hairpins through
the Gateway with the correct SNI -> wildcard cert validates. CoreDNS reloads the ConfigMap
live (no pod restart). Wired into platform/kustomization.yaml (Phase 2 network core).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

platform/coredns/coredns-custom.yaml

@@ -0,0 +1,45 @@
+---
+# Split-horizon DNS for cluster pods (R-DNS-01 / R-DNS-03).
+#
+# PROBLEM: an in-cluster client resolving a platform hostname over its PUBLIC name
+# (e.g. `keycloak.home.keithjackson.dev`) gets the off-cluster/public answer — it does NOT reach the
+# in-cluster Gateway. That path does not serve the `*.home.keithjackson.dev` SNI, so TLS fails with
+# `remote error: tls: unrecognized name` (oauth2-proxy OIDC discovery, Grafana generic_oauth, etc.).
+#
+# FIX (split-horizon, docs/09 "Split-horizon resolution"): inside the cluster, answer the whole
+# `home.keithjackson.dev` zone with the L2-announced ingress VIP of the `shared` Gateway. Pod ->
+# public-hostname traffic then hairpins through the Gateway with the correct SNI -> wildcard cert ->
+# works. External/LAN clients keep their own answers (PiHole split-horizon, docs/09) — this ConfigMap
+# only affects CLUSTER DNS (CoreDNS), not LAN DNS.
+#
+# MECHANISM (k3s-specific, F-DNS / docs/09): k3s CoreDNS ships a Corefile with
+# `import /etc/coredns/custom/*.server`, and mounts the `coredns-custom` ConfigMap (kube-system) at
+# /etc/coredns/custom. Each `*.server` key becomes a top-level server block imported into the
+# Corefile. CoreDNS `reload` picks up ConfigMap changes automatically — NO pod restart needed
+# (GitOps-friendly; the change reconciles live).
+#
+# ${GATEWAY_VIP} is substituted by Flux postBuild.substituteFrom cluster-vars (R-ENV-02): dev = the
+# dev L2 pool VIP, prod = the prod LAN VIP. The unified graph is identical across environments; only
+# the cluster-var differs. See clusters/base/cluster-vars.yaml.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: coredns-custom
+  namespace: kube-system
+data:
+  # Filename `home.server` -> imported as a standalone server block for the whole zone.
+  home.server: |
+    home.keithjackson.dev:53 {
+        errors
+        # Answer EVERY name in the zone (apex + all `<svc>.home.keithjackson.dev`) with the Gateway
+        # VIP. The template plugin's default match is `.*`, so this covers current and FUTURE app
+        # hostnames without editing this file per service. `{{ .Name }}` echoes the queried FQDN.
+        template IN A {
+            answer "{{ .Name }} 60 IN A ${GATEWAY_VIP}"
+        }
+        # AAAA -> NODATA (NOERROR, no answer): the name has an A but no AAAA, so IPv6-capable clients
+        # correctly fall back to the A record instead of getting NXDOMAIN/SERVFAIL.
+        template IN AAAA {
+            rcode NOERROR
+        }
+    }

platform/coredns/kustomization.yaml

@@ -0,0 +1,9 @@
+---
+# platform/coredns — in-cluster split-horizon DNS (R-DNS-01 / R-DNS-03).
+# A single `coredns-custom` ConfigMap in kube-system that k3s CoreDNS imports as an extra server
+# block (see coredns-custom.yaml). Rendered by the `platform` Flux Kustomization, which supplies
+# ${GATEWAY_VIP} via substituteFrom cluster-vars (R-ENV-02).
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - coredns-custom.yaml

platform/kustomization.yaml

@@ -17,6 +17,7 @@ resources:
   - cert-manager       # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token`
   - externaldns        # auto DNS records — needs SOPS `cloudflare-api-token`
   - gateway            # Gateway API listeners (ingress; replaces Traefik)
+  - coredns            # split-horizon in-cluster DNS: *.home.keithjackson.dev -> Gateway VIP (R-DNS-01/03)
   # ---- Phase 3 — Egress / VPN (ADR-0015): node-level host OpenVPN split-tunnel (Ansible role
   #      vpn_egress_gateway) steered by a Cilium Egress Gateway policy, with a host nftables kill-switch.
   #      No in-cluster gateway pod (Gluetun retired). See ADR-0015 + DECISIONS-LOG "Phase 3 VPN". ----