docs: DECISIONS-LOG — overnight prod standup handoff

Prod is up: 12/13 tiers Ready, LE wildcard issued, 113 pods, dev's 57 DNS
records byte-identical throughout. Records both cold-start deadlocks (Kyverno
CRDs, CDI StorageProfile) as one root cause, the external-dns footgun and the
prove-before-acting pattern that disarmed it, three bugs in my own verification,
the pfSense wildcard now shadowing prod, and what remains for the operator.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -1758,3 +1758,105 @@ into a transient hard Kustomization failure.
 - **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
   with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
   Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
+
+## 2026-09-05 — **PROD IS UP.** Overnight standup: 12/13 tiers, LE wildcard, dev untouched
+
+Operator asleep; standing instruction was "finish out what is necessary and I want it working by
+the morning." **Prod is serving.** Dev was verified unharmed at every step.
+
+### Final state
+
+| | prod | dev |
+|---|---|---|
+| Kustomizations | **12/13 Ready** (`velero` suspended, deliberately) | **13/13 Ready** |
+| Source | bootstrap Gitea `main@c8c1d29` | bootstrap Gitea `dev-live@a4a3c2e` |
+| `BASE_DOMAIN` | `prod.home.keithjackson.dev` | `home.keithjackson.dev` (apex) |
+| Pods | **113, 0 not Running/Completed** | healthy, all HelmReleases Ready |
+| Wildcard cert | **Ready** — LE prod, `*.prod.home.keithjackson.dev` | unchanged |
+| DNS | **36 records**, every TXT `owner=prod` | **57 records, byte-identical to baseline** |
+
+End-to-end HTTPS with certificate validation, pinned to prod's gateway: apex `200`,
+keycloak `302` (login), gitea `200`, grafana `302` (SSO).
+
+### What was actually blocking, and what it taught
+
+**Both blockers were the same bug in different clothes: Flux server-side dry-runs every object in a
+Kustomization before applying any of them, so ONE unknown `kind` aborts the entire tier.**
+
+1. **Kyverno** — chart sets `crds.install: false` (5.6 MiB bundle > kubectl's 256 KiB annotation
+   limit) and nothing else installed them. Its pods had **55 restarts each over 4.5h**, controllers
+   with no API types to watch. The failed install then tripped `install.remediation.retries: 3`,
+   whose uninstall ALSO timed out, wedging the release in `uninstalling` — a state helm-controller
+   cannot advance from. Cleared with `helm uninstall` + a cold reinstall.
+2. **CDI — a true deadlock, and the one worth remembering.** `platform-config` applies the CDI CR
+   *and* `cdi-storageprofiles.yaml` in the SAME pass, but `StorageProfile`'s CRD does not exist until
+   `cdi-operator` reconciles that CR. Dry-run fails → CR never applied → CRD never created → the tier
+   can NEVER converge. `cdi-operator` had been Available for **12h** with no CR. **Invisible on dev**
+   (its CDI CRDs have existed for months) **and fatal on any cold cluster.** Broken by pre-applying
+   only the CR; the CRD appeared in 10s.
+
+> **DURABLE FIX STILL OWED:** move `cdi-storageprofiles.yaml` out of `platform-config` into its own
+> Kustomization with `dependsOn: platform-config`. Not done overnight — it edits the reconciliation
+> graph both clusters share, while prod reads a pinned revision and dev sits mid-migration on a
+> separate branch. Until it lands, `standup.sh crds` is a prerequisite on any cold cluster.
+
+### The armed footgun, disarmed
+
+Prod's external-dns Deployment was running `--policy=sync --registry=txt
+--domain-filter=keithjackson.dev` **with no `--txt-owner-id`** — it would have defaulted to `default`,
+the owner of **all 36 of dev's TXT registry entries**, and `policy: sync` would have deleted the
+records behind them. The 2026-09-04 incident, re-armed. **The only thing preventing it was
+`replicas=0` — manual drift that any Helm upgrade resets to the chart default of 1. That is not a
+guard, it is luck.**
+
+The repo was never at fault: the live HelmRelease `spec.values` carried `txtOwnerId: "prod"`. The
+fault was Helm/Flux STATE — the installed release predated `c8c1d29`, yet helm-controller reported
+`Ready=True` with `configDigest == lastAttemptedConfigDigest`, believing it had already deployed the
+current config, so it would never have upgraded on its own.
+
+**The fix pattern is the reusable part: PROVE THE OUTCOME BEFORE CAUSING IT.** A Helm upgrade sets
+replicas to 1 and applies new args in one operation — there is no window to check first. So
+`standup.sh externaldns` renders the chart with the LIVE values and asserts `--txt-owner-id=<owner>`
+appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into
+a precondition.
+
+### Three bugs in my own verification — recorded because a check you cannot trust is worse than none
+
+1. **`SNAP_DIR` defaulted to `$TMPDIR`.** Every `dev.sh run` is a fresh container, so the DNS baseline
+   was discarded between steps and the before/after comparison — the check that detects damage to the
+   other cluster — could never run across invocations. Now repo-relative and gitignored.
+2. **The first real comparison cried wolf**, reporting "RECORDS OUTSIDE prod.home CHANGED" and listing
+   pairs of IDENTICAL lines. Locale-dependent sort collation: the host ordered `*.keithjackson.dev`
+   differently from the container. Proved it was noise under `LC_ALL=C` (all 57 identical); snapshots
+   now strip CRs and sort under a fixed collation.
+3. **preflight `die`d on `EXTERNALDNS_TXT_OWNER_ID=default`** — dev's legitimate id — so a script
+   claiming "no dev/prod fork" (R-ENV-02) could never run against dev. A fork in disguise. preflight
+   is now a pure diagnostic; the refusal moved to the mutating path where acting on a bad value is
+   what actually causes harm.
+
+### I also fell into the pfSense trap — ACTION REQUIRED
+
+The local override is a wildcard on `*.home.keithjackson.dev` and now swallows the whole new
+`prod.home.*` subtree, resolving **every prod hostname to `192.168.2.50` (dev)**. Cloudflare
+correctly returns `192.168.2.20`.
+
+**Prod is reachable from anywhere using public DNS, but NOT from behind pfSense.** Narrow the
+override, or add one pointing `prod.home.*` at `192.168.2.20`.
+
+It is subtle because it **partially works**: dev's `*.home.keithjackson.dev` wildcard matches
+`prod.home.keithjackson.dev` (one label) so the apex returns 200 over valid TLS, while
+`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. I read that
+as a broken prod cert before checking the resolver. **This is the same trap that produced the false
+healthy reading during the incident. Never verify DNS or TLS here through a local resolver.**
+
+### Still open for the operator
+
+- **`S3_BACKUP_ENDPOINT` is `TBD-PROD-*`** — `velero` stays suspended, and note `apps-wave-0` consumes
+  it too and applies GREEN while anything writing there fails at runtime. **Prod has no backup
+  coverage.** Needs a real prod endpoint (prod cannot reach dev's `192.168.1.9:9000`).
+- **Dev's migration off the apex** (§10a sequence D) — Keycloak issuer, ~12 OIDC redirect URIs, a
+  rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
+- **Dev's GitOps source is this workstation** — `192.168.2.132` is a **DHCP lease on wifi**. A lease
+  change stalls dev's fetching (cluster keeps serving; fix is a `kubectl patch`). Static reservation.
+- **Six L2 tests hardcode the apex** and must be parameterized per environment.
+- Nothing merged to `main`; nothing pushed. `infra/PROD-STANDUP.md` is the command sequence.