docs: record the apex migration, the GitOps deadlock, and the live-only policy

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -2423,3 +2423,89 @@ container dies with `PermissionError` on every run; my older base had `0400` and
   does not exist.
 - The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm
   follows git once the knob moves; it does not move it.
+
+## 2026-09-13 — **PROD IS THE APEX.** Dev decommissioned; and a GitOps deadlock found the hard way
+
+Prod serves `home.keithjackson.dev`. Dev's three VMs (Proxmox 101/102/105) are stopped with
+`onboot=0`. pfSense (VM 100) and every legacy service (103 nextcloud, 104 arr, 112
+docker-services, 120 monitoring, 121 unifi, 122 auth) are untouched and still running.
+
+Verified from an EXTERNAL vantage, not from inside a pod: apex `200`, keycloak `302`, gitea `200`,
+grafana `302`, headlamp `200`, all with `ssl_verify_result=0`. 3/3 nodes, 3 etcd voters, 14/14
+tiers, 24/24 Longhorn volumes healthy, 0 pods not Running.
+
+### The two namespaces were never in conflict — check this before panicking about the wildcard
+
+| | |
+|---|---|
+| legacy non-k3s prod | `*.keithjackson.dev` → `136.51.115.252` (public) |
+| k3s prod (was dev's) | `*.home.keithjackson.dev` → `192.168.2.20` |
+
+The zone carries wildcards for BOTH `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at
+the public IP. Specific records SHADOW them: dev's did, prod's do now. Releasing dev's 18 names did
+not NXDOMAIN them, it let them fall through to the legacy wildcard — benign, and reversible.
+
+### The blocker nothing would have cleaned up
+
+external-dns v0.21.0 deleted only the `a-` prefixed TXT registry entries and then logged
+*"All records are already up to date"*. The **older unprefixed** form — `home.keithjackson.dev`,
+`keycloak.home.keithjackson.dev`, … — it no longer writes but still READS, so it neither removed
+them nor would ever have. All 18 carried `owner=default`; prod runs `owner=prod`, so **prod could
+not have claimed these names while they existed.** Deleted by hand via the Cloudflare API, matched
+on `name ∈ dev's 18` AND `external-dns` AND `owner=default`, leaving prod's 18 `a-*.prod.home.*`
+entries alone.
+
+**Verify DNS at the AUTHORITY (`dig @betty.ns.cloudflare.com`), never a resolver.** 1.1.1.1 served
+deleted TXT for minutes afterwards and showed a false failure. Separately, `dig +short` with a
+CR-contaminated hostname list produced a false "all 18 already gone" — the same CR class that
+caused a false alarm on 2026-09-06.
+
+### The GitOps deadlock — the expensive lesson
+
+Prod's `GitRepository.spec.url` was a **hardcoded literal**, so the flip did not change it. That was
+checked and was true. What was missed is the SECOND-ORDER effect: the flip renames prod's own
+HTTPRoute hostnames, so external-dns deletes the old name and CoreDNS's split-horizon zone moves
+with `${BASE_DOMAIN}`. The url pointed at a name about to stop serving.
+
+Moving it to the new public name created a WORSE deadlock: the HTTPRoute that would serve that name
+lives in a **downstream tier**, which cannot reconcile while the source is down. Source down →
+routes never rename → source stays down. Observed: `srcReady=False`, *"unable to list remote for
+https://gitea.home.keithjackson.dev"*.
+
+**Durable fix: the source is now the in-cluster Service** `http://gitea.gitea.svc.cluster.local:3000`.
+Prod's GitOps no longer depends on public DNS, Cloudflare, its own Gateway, or `${BASE_DOMAIN}`. No
+future domain change can sever prod from its own source. A third wall appeared on the way: the
+rendered `gitea-allow-gateway-ingress` admits only `fromEntities: [ingress]`, so `source-controller`
+was denied (`wget` timed out).
+
+> **The dependency cycle should have been mapped before the first push, not discovered in three
+> stages on a live cluster.** The repo already documents this class as "the self-reference rule".
+
+### Live-only debt — READ THIS
+
+`gitea-allow-flux-ingress` (ns gitea) is **applied live and NOT in git**, carrying a provenance
+annotation. It is what stops the deadlock above recurring. It cannot simply be added:
+`apps/wave-0/gitea.yaml` is a render golden and `catalog/render.py` generates gitea's CNPs from
+`ingressRoutes:`, with no field for an in-cluster client. Being landed on
+`fix/gitea-allow-flux-ingress-rendered` as a renderer-contract change. **Until then, one
+`kubectl delete` or a cluster rebuild re-creates the deadlock.**
+
+### Other outcomes
+
+- **`origin` removed.** It pointed at dev's Gitea and was stale at `4809199` while real main was 68
+  commits ahead — it cost one agent an entire run, which "found" three premises that were all
+  artifacts of reading the wrong ref. The sole remote is now `origin` → `gitea.home.keithjackson.dev`.
+- **Only prod-1 needed the apiserver issuer change** — prod-2/prod-3 already carried the apex value.
+  One node restarted, not three. All three now agree; containerd kept pods through it, so no CNPG
+  switchover was even needed.
+- **Redirect URIs followed automatically.** A scheduled run of the client-URL reconciler had already
+  converged all 13 clients before the manual trigger; confirmed against the admin API.
+- **Workstation DNS still needs the operator.** `/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf`
+  forces `*.home.keithjackson.dev` → `192.168.2.50` (dev's dead NAT address) with `local=` so it never
+  forwards. Public DNS is now complete and correct, including the L4 VIPs (unifi `.21`, crafty-mc
+  `.22`), so both that file and `20-prod.home.keithjackson.dev.conf` should simply be removed. Needs
+  sudo. Until then the workstation cannot reach prod by name and `git push` cannot resolve the remote.
+- **Dev VM disks deliberately NOT destroyed.** Shutdown is reversible; deleting ~260 GB of disks is
+  not, and there is no urgency.
+- Testing HTTPS from inside a pod is misleading: nextcloud's Cilium egress policy returned envoy
+  `403 Access denied` for other apps' FQDNs. Test from outside with `--resolve`.