diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 5f8b345..5040f9f 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2423,3 +2423,89 @@ container dies with `PermissionError` on every run; my older base had `0400` and does not exist. - The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm follows git once the knob moves; it does not move it. + +## 2026-09-13 — **PROD IS THE APEX.** Dev decommissioned; and a GitOps deadlock found the hard way + +Prod serves `home.keithjackson.dev`. Dev's three VMs (Proxmox 101/102/105) are stopped with +`onboot=0`. pfSense (VM 100) and every legacy service (103 nextcloud, 104 arr, 112 +docker-services, 120 monitoring, 121 unifi, 122 auth) are untouched and still running. + +Verified from an EXTERNAL vantage, not from inside a pod: apex `200`, keycloak `302`, gitea `200`, +grafana `302`, headlamp `200`, all with `ssl_verify_result=0`. 3/3 nodes, 3 etcd voters, 14/14 +tiers, 24/24 Longhorn volumes healthy, 0 pods not Running. + +### The two namespaces were never in conflict — check this before panicking about the wildcard + +| | | +|---|---| +| legacy non-k3s prod | `*.keithjackson.dev` → `136.51.115.252` (public) | +| k3s prod (was dev's) | `*.home.keithjackson.dev` → `192.168.2.20` | + +The zone carries wildcards for BOTH `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at +the public IP. Specific records SHADOW them: dev's did, prod's do now. Releasing dev's 18 names did +not NXDOMAIN them, it let them fall through to the legacy wildcard — benign, and reversible. + +### The blocker nothing would have cleaned up + +external-dns v0.21.0 deleted only the `a-` prefixed TXT registry entries and then logged +*"All records are already up to date"*. The **older unprefixed** form — `home.keithjackson.dev`, +`keycloak.home.keithjackson.dev`, … — it no longer writes but still READS, so it neither removed +them nor would ever have. All 18 carried `owner=default`; prod runs `owner=prod`, so **prod could +not have claimed these names while they existed.** Deleted by hand via the Cloudflare API, matched +on `name ∈ dev's 18` AND `external-dns` AND `owner=default`, leaving prod's 18 `a-*.prod.home.*` +entries alone. + +**Verify DNS at the AUTHORITY (`dig @betty.ns.cloudflare.com`), never a resolver.** 1.1.1.1 served +deleted TXT for minutes afterwards and showed a false failure. Separately, `dig +short` with a +CR-contaminated hostname list produced a false "all 18 already gone" — the same CR class that +caused a false alarm on 2026-09-06. + +### The GitOps deadlock — the expensive lesson + +Prod's `GitRepository.spec.url` was a **hardcoded literal**, so the flip did not change it. That was +checked and was true. What was missed is the SECOND-ORDER effect: the flip renames prod's own +HTTPRoute hostnames, so external-dns deletes the old name and CoreDNS's split-horizon zone moves +with `${BASE_DOMAIN}`. The url pointed at a name about to stop serving. + +Moving it to the new public name created a WORSE deadlock: the HTTPRoute that would serve that name +lives in a **downstream tier**, which cannot reconcile while the source is down. Source down → +routes never rename → source stays down. Observed: `srcReady=False`, *"unable to list remote for +https://gitea.home.keithjackson.dev"*. + +**Durable fix: the source is now the in-cluster Service** `http://gitea.gitea.svc.cluster.local:3000`. +Prod's GitOps no longer depends on public DNS, Cloudflare, its own Gateway, or `${BASE_DOMAIN}`. No +future domain change can sever prod from its own source. A third wall appeared on the way: the +rendered `gitea-allow-gateway-ingress` admits only `fromEntities: [ingress]`, so `source-controller` +was denied (`wget` timed out). + +> **The dependency cycle should have been mapped before the first push, not discovered in three +> stages on a live cluster.** The repo already documents this class as "the self-reference rule". + +### Live-only debt — READ THIS + +`gitea-allow-flux-ingress` (ns gitea) is **applied live and NOT in git**, carrying a provenance +annotation. It is what stops the deadlock above recurring. It cannot simply be added: +`apps/wave-0/gitea.yaml` is a render golden and `catalog/render.py` generates gitea's CNPs from +`ingressRoutes:`, with no field for an in-cluster client. Being landed on +`fix/gitea-allow-flux-ingress-rendered` as a renderer-contract change. **Until then, one +`kubectl delete` or a cluster rebuild re-creates the deadlock.** + +### Other outcomes + +- **`origin` removed.** It pointed at dev's Gitea and was stale at `4809199` while real main was 68 + commits ahead — it cost one agent an entire run, which "found" three premises that were all + artifacts of reading the wrong ref. The sole remote is now `origin` → `gitea.home.keithjackson.dev`. +- **Only prod-1 needed the apiserver issuer change** — prod-2/prod-3 already carried the apex value. + One node restarted, not three. All three now agree; containerd kept pods through it, so no CNPG + switchover was even needed. +- **Redirect URIs followed automatically.** A scheduled run of the client-URL reconciler had already + converged all 13 clients before the manual trigger; confirmed against the admin API. +- **Workstation DNS still needs the operator.** `/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` + forces `*.home.keithjackson.dev` → `192.168.2.50` (dev's dead NAT address) with `local=` so it never + forwards. Public DNS is now complete and correct, including the L4 VIPs (unifi `.21`, crafty-mc + `.22`), so both that file and `20-prod.home.keithjackson.dev.conf` should simply be removed. Needs + sudo. Until then the workstation cannot reach prod by name and `git push` cannot resolve the remote. +- **Dev VM disks deliberately NOT destroyed.** Shutdown is reversible; deleting ~260 GB of disks is + not, and there is no urgency. +- Testing HTTPS from inside a pod is misleading: nextcloud's Cilium egress policy returned envoy + `403 Access denied` for other apps' FQDNs. Test from outside with `--resolve`.