From e6bcecc78cb9fb85203e598122c6ac3d9fc84e70 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 5 Sep 2026 10:06:42 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20DECISIONS-LOG=20=E2=80=94=20overnight?= =?UTF-8?q?=20prod=20standup=20handoff?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prod is up: 12/13 tiers Ready, LE wildcard issued, 113 pods, dev's 57 DNS records byte-identical throughout. Records both cold-start deadlocks (Kyverno CRDs, CDI StorageProfile) as one root cause, the external-dns footgun and the prove-before-acting pattern that disarmed it, three bugs in my own verification, the pfSense wildcard now shadowing prod, and what remains for the operator. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 102 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index cf00eac..ccc77cf 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1758,3 +1758,105 @@ into a transient hard Kustomization failure. - **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway. Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6. + +## 2026-09-05 — **PROD IS UP.** Overnight standup: 12/13 tiers, LE wildcard, dev untouched + +Operator asleep; standing instruction was "finish out what is necessary and I want it working by +the morning." **Prod is serving.** Dev was verified unharmed at every step. + +### Final state + +| | prod | dev | +|---|---|---| +| Kustomizations | **12/13 Ready** (`velero` suspended, deliberately) | **13/13 Ready** | +| Source | bootstrap Gitea `main@c8c1d29` | bootstrap Gitea `dev-live@a4a3c2e` | +| `BASE_DOMAIN` | `prod.home.keithjackson.dev` | `home.keithjackson.dev` (apex) | +| Pods | **113, 0 not Running/Completed** | healthy, all HelmReleases Ready | +| Wildcard cert | **Ready** — LE prod, `*.prod.home.keithjackson.dev` | unchanged | +| DNS | **36 records**, every TXT `owner=prod` | **57 records, byte-identical to baseline** | + +End-to-end HTTPS with certificate validation, pinned to prod's gateway: apex `200`, +keycloak `302` (login), gitea `200`, grafana `302` (SSO). + +### What was actually blocking, and what it taught + +**Both blockers were the same bug in different clothes: Flux server-side dry-runs every object in a +Kustomization before applying any of them, so ONE unknown `kind` aborts the entire tier.** + +1. **Kyverno** — chart sets `crds.install: false` (5.6 MiB bundle > kubectl's 256 KiB annotation + limit) and nothing else installed them. Its pods had **55 restarts each over 4.5h**, controllers + with no API types to watch. The failed install then tripped `install.remediation.retries: 3`, + whose uninstall ALSO timed out, wedging the release in `uninstalling` — a state helm-controller + cannot advance from. Cleared with `helm uninstall` + a cold reinstall. +2. **CDI — a true deadlock, and the one worth remembering.** `platform-config` applies the CDI CR + *and* `cdi-storageprofiles.yaml` in the SAME pass, but `StorageProfile`'s CRD does not exist until + `cdi-operator` reconciles that CR. Dry-run fails → CR never applied → CRD never created → the tier + can NEVER converge. `cdi-operator` had been Available for **12h** with no CR. **Invisible on dev** + (its CDI CRDs have existed for months) **and fatal on any cold cluster.** Broken by pre-applying + only the CR; the CRD appeared in 10s. + +> **DURABLE FIX STILL OWED:** move `cdi-storageprofiles.yaml` out of `platform-config` into its own +> Kustomization with `dependsOn: platform-config`. Not done overnight — it edits the reconciliation +> graph both clusters share, while prod reads a pinned revision and dev sits mid-migration on a +> separate branch. Until it lands, `standup.sh crds` is a prerequisite on any cold cluster. + +### The armed footgun, disarmed + +Prod's external-dns Deployment was running `--policy=sync --registry=txt +--domain-filter=keithjackson.dev` **with no `--txt-owner-id`** — it would have defaulted to `default`, +the owner of **all 36 of dev's TXT registry entries**, and `policy: sync` would have deleted the +records behind them. The 2026-09-04 incident, re-armed. **The only thing preventing it was +`replicas=0` — manual drift that any Helm upgrade resets to the chart default of 1. That is not a +guard, it is luck.** + +The repo was never at fault: the live HelmRelease `spec.values` carried `txtOwnerId: "prod"`. The +fault was Helm/Flux STATE — the installed release predated `c8c1d29`, yet helm-controller reported +`Ready=True` with `configDigest == lastAttemptedConfigDigest`, believing it had already deployed the +current config, so it would never have upgraded on its own. + +**The fix pattern is the reusable part: PROVE THE OUTCOME BEFORE CAUSING IT.** A Helm upgrade sets +replicas to 1 and applies new args in one operation — there is no window to check first. So +`standup.sh externaldns` renders the chart with the LIVE values and asserts `--txt-owner-id=` +appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into +a precondition. + +### Three bugs in my own verification — recorded because a check you cannot trust is worse than none + +1. **`SNAP_DIR` defaulted to `$TMPDIR`.** Every `dev.sh run` is a fresh container, so the DNS baseline + was discarded between steps and the before/after comparison — the check that detects damage to the + other cluster — could never run across invocations. Now repo-relative and gitignored. +2. **The first real comparison cried wolf**, reporting "RECORDS OUTSIDE prod.home CHANGED" and listing + pairs of IDENTICAL lines. Locale-dependent sort collation: the host ordered `*.keithjackson.dev` + differently from the container. Proved it was noise under `LC_ALL=C` (all 57 identical); snapshots + now strip CRs and sort under a fixed collation. +3. **preflight `die`d on `EXTERNALDNS_TXT_OWNER_ID=default`** — dev's legitimate id — so a script + claiming "no dev/prod fork" (R-ENV-02) could never run against dev. A fork in disguise. preflight + is now a pure diagnostic; the refusal moved to the mutating path where acting on a bad value is + what actually causes harm. + +### I also fell into the pfSense trap — ACTION REQUIRED + +The local override is a wildcard on `*.home.keithjackson.dev` and now swallows the whole new +`prod.home.*` subtree, resolving **every prod hostname to `192.168.2.50` (dev)**. Cloudflare +correctly returns `192.168.2.20`. + +**Prod is reachable from anywhere using public DNS, but NOT from behind pfSense.** Narrow the +override, or add one pointing `prod.home.*` at `192.168.2.20`. + +It is subtle because it **partially works**: dev's `*.home.keithjackson.dev` wildcard matches +`prod.home.keithjackson.dev` (one label) so the apex returns 200 over valid TLS, while +`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. I read that +as a broken prod cert before checking the resolver. **This is the same trap that produced the false +healthy reading during the incident. Never verify DNS or TLS here through a local resolver.** + +### Still open for the operator + +- **`S3_BACKUP_ENDPOINT` is `TBD-PROD-*`** — `velero` stays suspended, and note `apps-wave-0` consumes + it too and applies GREEN while anything writing there fails at runtime. **Prod has no backup + coverage.** Needs a real prod endpoint (prod cannot reach dev's `192.168.1.9:9000`). +- **Dev's migration off the apex** (§10a sequence D) — Keycloak issuer, ~12 OIDC redirect URIs, a + rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present. +- **Dev's GitOps source is this workstation** — `192.168.2.132` is a **DHCP lease on wifi**. A lease + change stalls dev's fetching (cluster keeps serving; fix is a `kubectl patch`). Static reservation. +- **Six L2 tests hardcode the apex** and must be parameterized per environment. +- Nothing merged to `main`; nothing pushed. `infra/PROD-STANDUP.md` is the command sequence.