feat(prod): STANDUP COMPLETE — 12/13 tiers Ready, LE wildcard issued, dev untouched

Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home
Certificate Ready (Let's Encrypt PROD, *.prod.home.keithjackson.dev). Gateway
VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under
prod.home.keithjackson.dev, every TXT registry entry owner=prod.

End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway:
apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO).

DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup
baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD
placeholder, and resuming it would create a BackupStorageLocation that can never
reach a bucket: a tier reporting healthy while backups silently do not exist.

TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check
worthless. Recording them because a check you cannot trust is worse than none —
it trains you to skip the one alarm that matters.

1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the
   DNS baseline was discarded between steps and `verify`'s before/after
   comparison — the check that detects damage to the other cluster's records —
   could never run across invocations. Now repo-relative (bind-mounted, so it
   persists on the host) and gitignored as a run artifact.

2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED",
   listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation —
   the host ordered `*.keithjackson.dev` differently from the container, so a
   diff of two identical sets reported changes. Proved it was noise by
   re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips
   CRs and sorts under LC_ALL=C so ordering is reproducible everywhere.

ACTION REQUIRED (operator, pfSense): the local override is a wildcard on
*.home.keithjackson.dev and now also swallows the new prod.home.* subtree,
resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly
returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT
from behind pfSense. Narrow the override, or add one pointing prod.home.* at
192.168.2.20.

That override is also the trap that produced a false healthy reading during the
2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's
*.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label)
so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels)
fails with a certificate-name error. I hit this myself during verification and
mistook it for a broken prod cert. Never verify DNS or TLS here through a local
resolver.

Refs R-ENV-02, R-GITOPS-01, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/PROD-STANDUP.md

@@ -220,6 +220,45 @@ timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
 
 ---
 
+## VERIFIED WORKING — 2026-09-05, standup complete
+
+| | |
+|---|---|
+| Kustomizations | **12/13 Ready** (`velero` deliberately suspended — see §Blockers) |
+| Pods | **113 total, 0 not Running/Completed** |
+| Wildcard cert | `wildcard-home` **Ready**, Let's Encrypt **prod**, `*.prod.home.keithjackson.dev` |
+| Gateway VIP | `192.168.2.20` · crafty L4 `192.168.2.22` · unifi L4 `192.168.2.21` |
+| Published DNS | **36 records** under `prod.home.keithjackson.dev`, every TXT registry entry `owner=prod` |
+| Dev's records | **57, byte-identical to the pre-standup baseline — nothing added, changed or removed** |
+
+End-to-end HTTPS, certificate validated (no `-k`), pinned to prod's gateway:
+
+```
+prod.home.keithjackson.dev            200     keycloak.…  302 (login redirect)
+gitea.prod.home.keithjackson.dev      200     grafana.…   302 (SSO redirect)
+```
+
+### ACTION REQUIRED — the pfSense override shadows prod from the LAN
+
+`getent hosts` on the workstation returns **192.168.2.50** (dev) for **every** `*.home.keithjackson.dev`
+name — and the new `prod.home.…` subtree falls inside that wildcard. Cloudflare correctly returns
+`192.168.2.20`. So **prod is reachable from anywhere using public DNS, but from this workstation and
+anything else behind pfSense, prod hostnames resolve to dev.**
+
+Narrow the override so it does not cover `prod.home.keithjackson.dev`, or add a matching override
+pointing that subtree at `192.168.2.20`. Until then, test prod with an explicit resolve:
+
+```bash
+curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
+  https://gitea.prod.home.keithjackson.dev/
+```
+
+**This is also the trap that produced a false healthy reading during the 2026-09-04 incident.** It is
+subtle because the *apex* appears to work: dev's `*.home.keithjackson.dev` wildcard matches
+`prod.home.keithjackson.dev` (one label) so TLS succeeds, while `keycloak.prod.home.keithjackson.dev`
+(two labels) fails with a certificate-name error. A partial success is what makes it convincing.
+**Never verify DNS or TLS through a local resolver here.**
+
 ## Blockers
 
 ### 1. `S3_BACKUP_ENDPOINT` is an intentional placeholder — operator decision required