From 74320127131b4a0dabd63c6149c312e25b7792ad Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 4 Sep 2026 23:54:33 -0500 Subject: [PATCH] =?UTF-8?q?docs(prod):=20platform=20tier=20up=208/9=20?= =?UTF-8?q?=E2=80=94=20and=20TWO=20blockers,=20one=20of=20them=20an=20arme?= =?UTF-8?q?d=20footgun?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Unsuspended prod's `platform` after pre-applying the Kyverno CRDs. 8/9 HelmReleases Ready, 76/76 pods Running (cert-manager, CNPG, Cilium, Longhorn, kube-prometheus-stack, Loki, Alloy, external-dns). `platform` itself is NOT Ready and the 10 downstream tiers stay blocked. THE CRD PRE-APPLY WORKED and is now proven against a live API server: 22 CRDs Established, a second run a clean no-op (rc=0), reproduced under the PINNED toolchain (helm 3.16.0) matching the 3.20.0 run byte-for-byte. It also confirmed the predicted failure mode: Kyverno's Deployments had 55 RESTARTS EACH over 4.5h and stabilised within minutes of the apply — controllers with no API types. BLOCKER 1 — Kyverno's Helm release is wedged in `uninstalling`. The install timed out *because* of that crash loop; install.remediation.retries:3 then fired an uninstall which ALSO timed out, leaving sh.helm.release.v1.kyverno.v1 at status=uninstalling. helm-controller cannot advance from there, so `platform` (wait: true) never goes Ready. The Deployments themselves are healthy at the right chart version — this is stale Helm bookkeeping, not a broken workload. Fix needs `helm uninstall` + `flux reconcile --force`; destructive, so NOT run unattended. Safe here because prod carries no workloads and a cold install proves the bootstrap path. The pre-applied CRDs live OUTSIDE the release and survive the uninstall — precisely why the pre-apply is a separate step. BLOCKER 2 — AND THIS ONE IS ARMED. Prod's external-dns Deployment runs --policy=sync --registry=txt --domain-filter=keithjackson.dev WITH NO --txt-owner-id. Absent that flag it defaults to `default` — the owner of ALL 36 of dev's TXT registry entries. Scaling it to 1 would claim dev's records and, under policy: sync, DELETE THEM: the 2026-09-04 incident, exactly. The repo is not at fault. helmrelease.yaml sets txtOwnerId from cluster-vars, prod resolves it to `prod`, and the LIVE HelmRelease spec.values carries txtOwnerId: "prod". Verified against the real chart: 1.21.1 with txtOwnerId: prod renders --txt-owner-id=prod, so the key is right and an upgrade would fix it. The fault is Helm/Flux STATE: `helm get values` shows the installed release has policy and domainFilters but NO txtOwnerId — it predates c8c1d29 — while helm-controller reports Ready=True, observedGeneration=3, and a history entry whose configDigest EQUALS lastAttemptedConfigDigest. It believes it already deployed the current config, so it will never upgrade on its own. The only thing that prevented the incident was replicas=0 — manual drift from the incident recovery, which any Helm upgrade resets to the chart default of 1. That is not a guard, it is luck. CONTAINED: `flux suspend helmrelease external-dns`, replicas left at 0. Costs nothing (prod publishes no hostnames yet; app tiers suspended) and is reversible. Remediation ordering matters and is documented: the naive resume + reconcile --force adds the flag and starts the pod in ONE operation, leaving no window to verify — a race whose failure mode is deleting dev's DNS. Pin replicas at 0, confirm --txt-owner-id=prod in the live args, THEN scale. Dev verified unharmed throughout: Cloudflare snapshotted before the unsuspend and re-queried after — 57 records byte-for-byte identical, 36 TXT still owner=default. Queried against the Cloudflare API, never a local resolver. Refs R-GITOPS-01, R-ENV-02, R-SEC-01. Co-Authored-By: Claude Opus 5 --- infra/PROD-PROVISIONING.md | 97 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index f043169..dfe2512 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -1333,6 +1333,103 @@ single-server-etcd window, and avoids a later member-add dance. `*.home.keithjackson.dev`, dev migrated to `*.dev.home.keithjackson.dev` with its old apex records retired (§10a step D8), and prod's own Keycloak realm + per-service client secrets. +## 9a. PROD BRING-UP STATE (2026-09-04) — what came up, and the TWO blockers that need the operator + +Prod's `platform` tier was unsuspended after the Kyverno CRDs were pre-applied (see §7). Result: +**8 of 9 HelmReleases Ready, 76/76 pods Running** — cert-manager, CNPG, Cilium, Longhorn, +kube-prometheus-stack, Loki, Alloy, external-dns. `platform` itself is **not** Ready, and the 10 +downstream tiers remain blocked behind it. Two distinct problems, both needing an operator decision. + +### The CRD pre-apply itself: WORKED, and is now proven against a live API server + +22 CRDs applied server-side and all reached `Established`; a second run was a clean no-op (`rc=0`), +so the idempotency claim is proven, not asserted. Reproduced under the **pinned** toolchain +(helm 3.16.0, yq 4.44.6) matching an independent helm 3.20.0 run byte-for-byte: `3.9.x` → 3.9.0, +22 CRDs, 5,636,155 bytes. + +**The CRDs were the real cause of a 4½-hour crash loop.** Kyverno's four Deployments had **55 +restarts each** and stabilised within minutes of the CRD apply — its controllers were coming up with +no API types to watch. That is the failure mode §7 predicted, observed. + +### BLOCKER 1 — Kyverno's Helm release is stuck in `uninstalling` + +Diagnosis chain, from the HelmRelease conditions: + +``` +Released: InstallFailed — Helm install failed ...: context deadline exceeded +Remediated: UninstallFailed — Helm uninstall remediation ... timed out waiting for the condition +Ready: StateError — unable to determine state for release with status 'uninstalling' +``` + +The install timed out **because** the pods were crash-looping without CRDs. `install.remediation. +retries: 3` then triggered an uninstall, and that uninstall itself timed out — leaving +`sh.helm.release.v1.kyverno.v1` with `status=uninstalling`. helm-controller cannot proceed from that +state, so `platform` (which has `wait: true`) never goes Ready and all 10 tiers below it stay blocked. + +**The Deployments are healthy and at the right chart version** (kyverno-3.9.0), so this is purely a +stale Helm bookkeeping record, not a broken workload. + +**Remediation requires a destructive command and was deliberately NOT run unattended:** + +``` +helm uninstall kyverno -n kyverno --no-hooks --wait=false # clear the stuck release +flux reconcile helmrelease kyverno -n kyverno --force # let Flux install cold, CRDs now present +``` + +This is safe on prod specifically because prod carries no real workloads yet, and a clean cold +install is the stronger outcome: it *proves* the bootstrap path this repo now automates. Note the +`failurePolicy: Fail` webhooks (`kyverno-policy-*`, `kyverno-exception-*`, `kyverno-cleanup-*`, +`kyverno-global-context-*`) gate only **Kyverno's own** resource types, so a brief absence does not +block ordinary workload admission. The pre-applied CRDs live **outside** the Helm release and +survive the uninstall, which is exactly why the pre-apply had to be a separate step. + +### BLOCKER 2 — prod's external-dns is an ARMED footgun: the installed release predates the fix + +**This is the more dangerous of the two.** Prod's live external-dns Deployment renders: + +``` +--policy=sync --registry=txt --domain-filter=keithjackson.dev --provider=cloudflare +``` + +**and NO `--txt-owner-id`.** Absent that flag external-dns falls back to the chart default `default` +— which is precisely the owner under which **all 36 of dev's TXT registry entries are recorded**. +Scaling this Deployment to 1 would make prod claim dev's records and, under `policy: sync`, +**delete them: the 2026-09-04 incident, exactly.** + +The repo is CORRECT and is not the problem. `platform/externaldns/helmrelease.yaml` sets +`txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"`, prod's applied `cluster-vars` resolves it to `prod`, and +the **live HelmRelease `spec.values` carries `txtOwnerId: "prod"`**. Verified against the real chart: +`helm template external-dns --version 1.21.1` with `txtOwnerId: prod` renders `--txt-owner-id=prod`, +so the values key is right and an upgrade *would* fix it. + +**The fault is Helm/Flux state, not authoring.** `helm get values external-dns` shows the INSTALLED +release carries `policy` and `domainFilters` but **no `txtOwnerId`** — it predates `c8c1d29`. Yet +helm-controller reports `Ready=True`, `observedGeneration=3`, and a history entry whose +`configDigest` **equals** `lastAttemptedConfigDigest`, i.e. **it believes it has already deployed the +current config.** So it will never upgrade on its own, and the stale release persists indefinitely. + +**The only thing preventing the incident was `replicas=0` — manual drift from the incident recovery, +which any Helm upgrade would reset to the chart default of 1.** That is not a guard; it is luck. + +Containment applied (reversible, and it costs nothing because prod publishes no hostnames yet — the +app tiers are suspended): **`flux suspend helmrelease external-dns -n external-dns`**, with +`replicas` left at 0. Flux can now neither upgrade the release nor reset the replica count. + +**Verified remediation, in this order — the ordering is the whole point:** + +1. Force the upgrade so the release picks up `txtOwnerId`, but **do not let the pod run before the + flag is confirmed present.** The naive `resume` + `reconcile --force` resets `replicas` to 1 and + starts the pod in the same operation, so there is no window in which to check — a race against a + process whose failure mode is deleting dev's DNS. +2. Confirm the live Deployment args contain **`--txt-owner-id=prod`**. +3. Only then allow it to scale to 1, and immediately diff the Cloudflare zone against a + pre-recorded snapshot. + +**Dev was verified unharmed throughout this work:** the Cloudflare zone was snapshotted before +unsuspending `platform` and re-queried after — **57 records, byte-for-byte identical, all 36 TXT +entries still `owner=default`.** Snapshots are taken against the **Cloudflare API**, never a local +resolver (a pfSense override produced a false healthy reading during the incident). + ## 10. DNS/TLS environment split — **the 2026-09-04 incident, and the target architecture** > **Current state (2026-09-04):** the target below is **AUTHORED, NOT APPLIED.** Prod's