jakman85andClaude Opus 5
4f37fc375e
Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home Certificate Ready (Let's Encrypt PROD, *.prod.home.keithjackson.dev). Gateway VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under prod.home.keithjackson.dev, every TXT registry entry owner=prod. End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway: apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO). DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD placeholder, and resuming it would create a BackupStorageLocation that can never reach a bucket: a tier reporting healthy while backups silently do not exist. TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check worthless. Recording them because a check you cannot trust is worse than none — it trains you to skip the one alarm that matters. 1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the DNS baseline was discarded between steps and `verify`'s before/after comparison — the check that detects damage to the other cluster's records — could never run across invocations. Now repo-relative (bind-mounted, so it persists on the host) and gitignored as a run artifact. 2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED", listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation — the host ordered `*.keithjackson.dev` differently from the container, so a diff of two identical sets reported changes. Proved it was noise by re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips CRs and sorts under LC_ALL=C so ordering is reproducible everywhere. ACTION REQUIRED (operator, pfSense): the local override is a wildcard on *.home.keithjackson.dev and now also swallows the new prod.home.* subtree, resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT from behind pfSense. Narrow the override, or add one pointing prod.home.* at 192.168.2.20. That override is also the trap that produced a false healthy reading during the 2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's *.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label) so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels) fails with a certificate-name error. I hit this myself during verification and mistook it for a broken prod cert. Never verify DNS or TLS here through a local resolver. Refs R-ENV-02, R-GITOPS-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>
bootstrap/ — Flux + SOPS age-key bootstrap (Phase 1 T-1.4)
sops/: age key generation + Flux SOPS decryption wiring (private key NEVER committed).
flux/: flux bootstrap config pointing at clusters/dev. Spec: ADR-0004, ADR-0007.