diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index f043169..dfe2512 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -1333,6 +1333,103 @@ single-server-etcd window, and avoids a later member-add dance. `*.home.keithjackson.dev`, dev migrated to `*.dev.home.keithjackson.dev` with its old apex records retired (§10a step D8), and prod's own Keycloak realm + per-service client secrets. +## 9a. PROD BRING-UP STATE (2026-09-04) — what came up, and the TWO blockers that need the operator + +Prod's `platform` tier was unsuspended after the Kyverno CRDs were pre-applied (see §7). Result: +**8 of 9 HelmReleases Ready, 76/76 pods Running** — cert-manager, CNPG, Cilium, Longhorn, +kube-prometheus-stack, Loki, Alloy, external-dns. `platform` itself is **not** Ready, and the 10 +downstream tiers remain blocked behind it. Two distinct problems, both needing an operator decision. + +### The CRD pre-apply itself: WORKED, and is now proven against a live API server + +22 CRDs applied server-side and all reached `Established`; a second run was a clean no-op (`rc=0`), +so the idempotency claim is proven, not asserted. Reproduced under the **pinned** toolchain +(helm 3.16.0, yq 4.44.6) matching an independent helm 3.20.0 run byte-for-byte: `3.9.x` → 3.9.0, +22 CRDs, 5,636,155 bytes. + +**The CRDs were the real cause of a 4½-hour crash loop.** Kyverno's four Deployments had **55 +restarts each** and stabilised within minutes of the CRD apply — its controllers were coming up with +no API types to watch. That is the failure mode §7 predicted, observed. + +### BLOCKER 1 — Kyverno's Helm release is stuck in `uninstalling` + +Diagnosis chain, from the HelmRelease conditions: + +``` +Released: InstallFailed — Helm install failed ...: context deadline exceeded +Remediated: UninstallFailed — Helm uninstall remediation ... timed out waiting for the condition +Ready: StateError — unable to determine state for release with status 'uninstalling' +``` + +The install timed out **because** the pods were crash-looping without CRDs. `install.remediation. +retries: 3` then triggered an uninstall, and that uninstall itself timed out — leaving +`sh.helm.release.v1.kyverno.v1` with `status=uninstalling`. helm-controller cannot proceed from that +state, so `platform` (which has `wait: true`) never goes Ready and all 10 tiers below it stay blocked. + +**The Deployments are healthy and at the right chart version** (kyverno-3.9.0), so this is purely a +stale Helm bookkeeping record, not a broken workload. + +**Remediation requires a destructive command and was deliberately NOT run unattended:** + +``` +helm uninstall kyverno -n kyverno --no-hooks --wait=false # clear the stuck release +flux reconcile helmrelease kyverno -n kyverno --force # let Flux install cold, CRDs now present +``` + +This is safe on prod specifically because prod carries no real workloads yet, and a clean cold +install is the stronger outcome: it *proves* the bootstrap path this repo now automates. Note the +`failurePolicy: Fail` webhooks (`kyverno-policy-*`, `kyverno-exception-*`, `kyverno-cleanup-*`, +`kyverno-global-context-*`) gate only **Kyverno's own** resource types, so a brief absence does not +block ordinary workload admission. The pre-applied CRDs live **outside** the Helm release and +survive the uninstall, which is exactly why the pre-apply had to be a separate step. + +### BLOCKER 2 — prod's external-dns is an ARMED footgun: the installed release predates the fix + +**This is the more dangerous of the two.** Prod's live external-dns Deployment renders: + +``` +--policy=sync --registry=txt --domain-filter=keithjackson.dev --provider=cloudflare +``` + +**and NO `--txt-owner-id`.** Absent that flag external-dns falls back to the chart default `default` +— which is precisely the owner under which **all 36 of dev's TXT registry entries are recorded**. +Scaling this Deployment to 1 would make prod claim dev's records and, under `policy: sync`, +**delete them: the 2026-09-04 incident, exactly.** + +The repo is CORRECT and is not the problem. `platform/externaldns/helmrelease.yaml` sets +`txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"`, prod's applied `cluster-vars` resolves it to `prod`, and +the **live HelmRelease `spec.values` carries `txtOwnerId: "prod"`**. Verified against the real chart: +`helm template external-dns --version 1.21.1` with `txtOwnerId: prod` renders `--txt-owner-id=prod`, +so the values key is right and an upgrade *would* fix it. + +**The fault is Helm/Flux state, not authoring.** `helm get values external-dns` shows the INSTALLED +release carries `policy` and `domainFilters` but **no `txtOwnerId`** — it predates `c8c1d29`. Yet +helm-controller reports `Ready=True`, `observedGeneration=3`, and a history entry whose +`configDigest` **equals** `lastAttemptedConfigDigest`, i.e. **it believes it has already deployed the +current config.** So it will never upgrade on its own, and the stale release persists indefinitely. + +**The only thing preventing the incident was `replicas=0` — manual drift from the incident recovery, +which any Helm upgrade would reset to the chart default of 1.** That is not a guard; it is luck. + +Containment applied (reversible, and it costs nothing because prod publishes no hostnames yet — the +app tiers are suspended): **`flux suspend helmrelease external-dns -n external-dns`**, with +`replicas` left at 0. Flux can now neither upgrade the release nor reset the replica count. + +**Verified remediation, in this order — the ordering is the whole point:** + +1. Force the upgrade so the release picks up `txtOwnerId`, but **do not let the pod run before the + flag is confirmed present.** The naive `resume` + `reconcile --force` resets `replicas` to 1 and + starts the pod in the same operation, so there is no window in which to check — a race against a + process whose failure mode is deleting dev's DNS. +2. Confirm the live Deployment args contain **`--txt-owner-id=prod`**. +3. Only then allow it to scale to 1, and immediately diff the Cloudflare zone against a + pre-recorded snapshot. + +**Dev was verified unharmed throughout this work:** the Cloudflare zone was snapshotted before +unsuspending `platform` and re-queried after — **57 records, byte-for-byte identical, all 36 TXT +entries still `owner=default`.** Snapshots are taken against the **Cloudflare API**, never a local +resolver (a pfSense override produced a false healthy reading during the incident). + ## 10. DNS/TLS environment split — **the 2026-09-04 incident, and the target architecture** > **Current state (2026-09-04):** the target below is **AUTHORED, NOT APPLIED.** Prod's