docs: DECISIONS-LOG — overnight prod standup handoff
Prod is up: 12/13 tiers Ready, LE wildcard issued, 113 pods, dev's 57 DNS records byte-identical throughout. Records both cold-start deadlocks (Kyverno CRDs, CDI StorageProfile) as one root cause, the external-dns footgun and the prove-before-acting pattern that disarmed it, three bugs in my own verification, the pfSense wildcard now shadowing prod, and what remains for the operator. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
acb5cabdad
commit
e6bcecc78c
1 file changed
+102
@@ -1758,3 +1758,105 @@ into a transient hard Kustomization failure.
|
|||||||
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
||||||
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
||||||
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
||||||
|
|
||||||
|
## 2026-09-05 — **PROD IS UP.** Overnight standup: 12/13 tiers, LE wildcard, dev untouched
|
||||||
|
|
||||||
|
Operator asleep; standing instruction was "finish out what is necessary and I want it working by
|
||||||
|
the morning." **Prod is serving.** Dev was verified unharmed at every step.
|
||||||
|
|
||||||
|
### Final state
|
||||||
|
|
||||||
|
| | prod | dev |
|
||||||
|
|---|---|---|
|
||||||
|
| Kustomizations | **12/13 Ready** (`velero` suspended, deliberately) | **13/13 Ready** |
|
||||||
|
| Source | bootstrap Gitea `main@c8c1d29` | bootstrap Gitea `dev-live@a4a3c2e` |
|
||||||
|
| `BASE_DOMAIN` | `prod.home.keithjackson.dev` | `home.keithjackson.dev` (apex) |
|
||||||
|
| Pods | **113, 0 not Running/Completed** | healthy, all HelmReleases Ready |
|
||||||
|
| Wildcard cert | **Ready** — LE prod, `*.prod.home.keithjackson.dev` | unchanged |
|
||||||
|
| DNS | **36 records**, every TXT `owner=prod` | **57 records, byte-identical to baseline** |
|
||||||
|
|
||||||
|
End-to-end HTTPS with certificate validation, pinned to prod's gateway: apex `200`,
|
||||||
|
keycloak `302` (login), gitea `200`, grafana `302` (SSO).
|
||||||
|
|
||||||
|
### What was actually blocking, and what it taught
|
||||||
|
|
||||||
|
**Both blockers were the same bug in different clothes: Flux server-side dry-runs every object in a
|
||||||
|
Kustomization before applying any of them, so ONE unknown `kind` aborts the entire tier.**
|
||||||
|
|
||||||
|
1. **Kyverno** — chart sets `crds.install: false` (5.6 MiB bundle > kubectl's 256 KiB annotation
|
||||||
|
limit) and nothing else installed them. Its pods had **55 restarts each over 4.5h**, controllers
|
||||||
|
with no API types to watch. The failed install then tripped `install.remediation.retries: 3`,
|
||||||
|
whose uninstall ALSO timed out, wedging the release in `uninstalling` — a state helm-controller
|
||||||
|
cannot advance from. Cleared with `helm uninstall` + a cold reinstall.
|
||||||
|
2. **CDI — a true deadlock, and the one worth remembering.** `platform-config` applies the CDI CR
|
||||||
|
*and* `cdi-storageprofiles.yaml` in the SAME pass, but `StorageProfile`'s CRD does not exist until
|
||||||
|
`cdi-operator` reconciles that CR. Dry-run fails → CR never applied → CRD never created → the tier
|
||||||
|
can NEVER converge. `cdi-operator` had been Available for **12h** with no CR. **Invisible on dev**
|
||||||
|
(its CDI CRDs have existed for months) **and fatal on any cold cluster.** Broken by pre-applying
|
||||||
|
only the CR; the CRD appeared in 10s.
|
||||||
|
|
||||||
|
> **DURABLE FIX STILL OWED:** move `cdi-storageprofiles.yaml` out of `platform-config` into its own
|
||||||
|
> Kustomization with `dependsOn: platform-config`. Not done overnight — it edits the reconciliation
|
||||||
|
> graph both clusters share, while prod reads a pinned revision and dev sits mid-migration on a
|
||||||
|
> separate branch. Until it lands, `standup.sh crds` is a prerequisite on any cold cluster.
|
||||||
|
|
||||||
|
### The armed footgun, disarmed
|
||||||
|
|
||||||
|
Prod's external-dns Deployment was running `--policy=sync --registry=txt
|
||||||
|
--domain-filter=keithjackson.dev` **with no `--txt-owner-id`** — it would have defaulted to `default`,
|
||||||
|
the owner of **all 36 of dev's TXT registry entries**, and `policy: sync` would have deleted the
|
||||||
|
records behind them. The 2026-09-04 incident, re-armed. **The only thing preventing it was
|
||||||
|
`replicas=0` — manual drift that any Helm upgrade resets to the chart default of 1. That is not a
|
||||||
|
guard, it is luck.**
|
||||||
|
|
||||||
|
The repo was never at fault: the live HelmRelease `spec.values` carried `txtOwnerId: "prod"`. The
|
||||||
|
fault was Helm/Flux STATE — the installed release predated `c8c1d29`, yet helm-controller reported
|
||||||
|
`Ready=True` with `configDigest == lastAttemptedConfigDigest`, believing it had already deployed the
|
||||||
|
current config, so it would never have upgraded on its own.
|
||||||
|
|
||||||
|
**The fix pattern is the reusable part: PROVE THE OUTCOME BEFORE CAUSING IT.** A Helm upgrade sets
|
||||||
|
replicas to 1 and applies new args in one operation — there is no window to check first. So
|
||||||
|
`standup.sh externaldns` renders the chart with the LIVE values and asserts `--txt-owner-id=<owner>`
|
||||||
|
appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into
|
||||||
|
a precondition.
|
||||||
|
|
||||||
|
### Three bugs in my own verification — recorded because a check you cannot trust is worse than none
|
||||||
|
|
||||||
|
1. **`SNAP_DIR` defaulted to `$TMPDIR`.** Every `dev.sh run` is a fresh container, so the DNS baseline
|
||||||
|
was discarded between steps and the before/after comparison — the check that detects damage to the
|
||||||
|
other cluster — could never run across invocations. Now repo-relative and gitignored.
|
||||||
|
2. **The first real comparison cried wolf**, reporting "RECORDS OUTSIDE prod.home CHANGED" and listing
|
||||||
|
pairs of IDENTICAL lines. Locale-dependent sort collation: the host ordered `*.keithjackson.dev`
|
||||||
|
differently from the container. Proved it was noise under `LC_ALL=C` (all 57 identical); snapshots
|
||||||
|
now strip CRs and sort under a fixed collation.
|
||||||
|
3. **preflight `die`d on `EXTERNALDNS_TXT_OWNER_ID=default`** — dev's legitimate id — so a script
|
||||||
|
claiming "no dev/prod fork" (R-ENV-02) could never run against dev. A fork in disguise. preflight
|
||||||
|
is now a pure diagnostic; the refusal moved to the mutating path where acting on a bad value is
|
||||||
|
what actually causes harm.
|
||||||
|
|
||||||
|
### I also fell into the pfSense trap — ACTION REQUIRED
|
||||||
|
|
||||||
|
The local override is a wildcard on `*.home.keithjackson.dev` and now swallows the whole new
|
||||||
|
`prod.home.*` subtree, resolving **every prod hostname to `192.168.2.50` (dev)**. Cloudflare
|
||||||
|
correctly returns `192.168.2.20`.
|
||||||
|
|
||||||
|
**Prod is reachable from anywhere using public DNS, but NOT from behind pfSense.** Narrow the
|
||||||
|
override, or add one pointing `prod.home.*` at `192.168.2.20`.
|
||||||
|
|
||||||
|
It is subtle because it **partially works**: dev's `*.home.keithjackson.dev` wildcard matches
|
||||||
|
`prod.home.keithjackson.dev` (one label) so the apex returns 200 over valid TLS, while
|
||||||
|
`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. I read that
|
||||||
|
as a broken prod cert before checking the resolver. **This is the same trap that produced the false
|
||||||
|
healthy reading during the incident. Never verify DNS or TLS here through a local resolver.**
|
||||||
|
|
||||||
|
### Still open for the operator
|
||||||
|
|
||||||
|
- **`S3_BACKUP_ENDPOINT` is `TBD-PROD-*`** — `velero` stays suspended, and note `apps-wave-0` consumes
|
||||||
|
it too and applies GREEN while anything writing there fails at runtime. **Prod has no backup
|
||||||
|
coverage.** Needs a real prod endpoint (prod cannot reach dev's `192.168.1.9:9000`).
|
||||||
|
- **Dev's migration off the apex** (§10a sequence D) — Keycloak issuer, ~12 OIDC redirect URIs, a
|
||||||
|
rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
|
||||||
|
- **Dev's GitOps source is this workstation** — `192.168.2.132` is a **DHCP lease on wifi**. A lease
|
||||||
|
change stalls dev's fetching (cluster keeps serving; fix is a `kubectl patch`). Static reservation.
|
||||||
|
- **Six L2 tests hardcode the apex** and must be parameterized per environment.
|
||||||
|
- Nothing merged to `main`; nothing pushed. `infra/PROD-STANDUP.md` is the command sequence.
|
||||||
Reference in new issue
Block a user