docs(prod): platform tier up 8/9 — and TWO blockers, one of them an armed footgun

Unsuspended prod's `platform` after pre-applying the Kyverno CRDs.
8/9 HelmReleases Ready, 76/76 pods Running (cert-manager, CNPG, Cilium,
Longhorn, kube-prometheus-stack, Loki, Alloy, external-dns). `platform` itself
is NOT Ready and the 10 downstream tiers stay blocked.

THE CRD PRE-APPLY WORKED and is now proven against a live API server: 22 CRDs
Established, a second run a clean no-op (rc=0), reproduced under the PINNED
toolchain (helm 3.16.0) matching the 3.20.0 run byte-for-byte. It also confirmed
the predicted failure mode: Kyverno's Deployments had 55 RESTARTS EACH over 4.5h
and stabilised within minutes of the apply — controllers with no API types.

BLOCKER 1 — Kyverno's Helm release is wedged in `uninstalling`. The install
timed out *because* of that crash loop; install.remediation.retries:3 then fired
an uninstall which ALSO timed out, leaving sh.helm.release.v1.kyverno.v1 at
status=uninstalling. helm-controller cannot advance from there, so `platform`
(wait: true) never goes Ready. The Deployments themselves are healthy at the
right chart version — this is stale Helm bookkeeping, not a broken workload.
Fix needs `helm uninstall` + `flux reconcile --force`; destructive, so NOT run
unattended. Safe here because prod carries no workloads and a cold install
proves the bootstrap path. The pre-applied CRDs live OUTSIDE the release and
survive the uninstall — precisely why the pre-apply is a separate step.

BLOCKER 2 — AND THIS ONE IS ARMED. Prod's external-dns Deployment runs
--policy=sync --registry=txt --domain-filter=keithjackson.dev WITH NO
--txt-owner-id. Absent that flag it defaults to `default` — the owner of ALL 36
of dev's TXT registry entries. Scaling it to 1 would claim dev's records and,
under policy: sync, DELETE THEM: the 2026-09-04 incident, exactly.

The repo is not at fault. helmrelease.yaml sets txtOwnerId from cluster-vars,
prod resolves it to `prod`, and the LIVE HelmRelease spec.values carries
txtOwnerId: "prod". Verified against the real chart: 1.21.1 with txtOwnerId:
prod renders --txt-owner-id=prod, so the key is right and an upgrade would fix
it. The fault is Helm/Flux STATE: `helm get values` shows the installed release
has policy and domainFilters but NO txtOwnerId — it predates c8c1d29 — while
helm-controller reports Ready=True, observedGeneration=3, and a history entry
whose configDigest EQUALS lastAttemptedConfigDigest. It believes it already
deployed the current config, so it will never upgrade on its own.

The only thing that prevented the incident was replicas=0 — manual drift from
the incident recovery, which any Helm upgrade resets to the chart default of 1.
That is not a guard, it is luck.

CONTAINED: `flux suspend helmrelease external-dns`, replicas left at 0. Costs
nothing (prod publishes no hostnames yet; app tiers suspended) and is reversible.
Remediation ordering matters and is documented: the naive resume + reconcile
--force adds the flag and starts the pod in ONE operation, leaving no window to
verify — a race whose failure mode is deleting dev's DNS. Pin replicas at 0,
confirm --txt-owner-id=prod in the live args, THEN scale.

Dev verified unharmed throughout: Cloudflare snapshotted before the unsuspend
and re-queried after — 57 records byte-for-byte identical, 36 TXT still
owner=default. Queried against the Cloudflare API, never a local resolver.

Refs R-GITOPS-01, R-ENV-02, R-SEC-01.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/PROD-PROVISIONING.md

@@ -1333,6 +1333,103 @@ single-server-etcd window, and avoids a later member-add dance.
       `*.home.keithjackson.dev`, dev migrated to `*.dev.home.keithjackson.dev` with its old apex
       records retired (§10a step D8), and prod's own Keycloak realm + per-service client secrets.
 
+## 9a. PROD BRING-UP STATE (2026-09-04) — what came up, and the TWO blockers that need the operator
+
+Prod's `platform` tier was unsuspended after the Kyverno CRDs were pre-applied (see §7). Result:
+**8 of 9 HelmReleases Ready, 76/76 pods Running** — cert-manager, CNPG, Cilium, Longhorn,
+kube-prometheus-stack, Loki, Alloy, external-dns. `platform` itself is **not** Ready, and the 10
+downstream tiers remain blocked behind it. Two distinct problems, both needing an operator decision.
+
+### The CRD pre-apply itself: WORKED, and is now proven against a live API server
+
+22 CRDs applied server-side and all reached `Established`; a second run was a clean no-op (`rc=0`),
+so the idempotency claim is proven, not asserted. Reproduced under the **pinned** toolchain
+(helm 3.16.0, yq 4.44.6) matching an independent helm 3.20.0 run byte-for-byte: `3.9.x` → 3.9.0,
+22 CRDs, 5,636,155 bytes.
+
+**The CRDs were the real cause of a 4½-hour crash loop.** Kyverno's four Deployments had **55
+restarts each** and stabilised within minutes of the CRD apply — its controllers were coming up with
+no API types to watch. That is the failure mode §7 predicted, observed.
+
+### BLOCKER 1 — Kyverno's Helm release is stuck in `uninstalling`
+
+Diagnosis chain, from the HelmRelease conditions:
+
+```
+Released:   InstallFailed   — Helm install failed ...: context deadline exceeded
+Remediated: UninstallFailed — Helm uninstall remediation ... timed out waiting for the condition
+Ready:      StateError      — unable to determine state for release with status 'uninstalling'
+```
+
+The install timed out **because** the pods were crash-looping without CRDs. `install.remediation.
+retries: 3` then triggered an uninstall, and that uninstall itself timed out — leaving
+`sh.helm.release.v1.kyverno.v1` with `status=uninstalling`. helm-controller cannot proceed from that
+state, so `platform` (which has `wait: true`) never goes Ready and all 10 tiers below it stay blocked.
+
+**The Deployments are healthy and at the right chart version** (kyverno-3.9.0), so this is purely a
+stale Helm bookkeeping record, not a broken workload.
+
+**Remediation requires a destructive command and was deliberately NOT run unattended:**
+
+```
+helm uninstall kyverno -n kyverno --no-hooks --wait=false   # clear the stuck release
+flux reconcile helmrelease kyverno -n kyverno --force        # let Flux install cold, CRDs now present
+```
+
+This is safe on prod specifically because prod carries no real workloads yet, and a clean cold
+install is the stronger outcome: it *proves* the bootstrap path this repo now automates. Note the
+`failurePolicy: Fail` webhooks (`kyverno-policy-*`, `kyverno-exception-*`, `kyverno-cleanup-*`,
+`kyverno-global-context-*`) gate only **Kyverno's own** resource types, so a brief absence does not
+block ordinary workload admission. The pre-applied CRDs live **outside** the Helm release and
+survive the uninstall, which is exactly why the pre-apply had to be a separate step.
+
+### BLOCKER 2 — prod's external-dns is an ARMED footgun: the installed release predates the fix
+
+**This is the more dangerous of the two.** Prod's live external-dns Deployment renders:
+
+```
+--policy=sync  --registry=txt  --domain-filter=keithjackson.dev  --provider=cloudflare
+```
+
+**and NO `--txt-owner-id`.** Absent that flag external-dns falls back to the chart default `default`
+— which is precisely the owner under which **all 36 of dev's TXT registry entries are recorded**.
+Scaling this Deployment to 1 would make prod claim dev's records and, under `policy: sync`,
+**delete them: the 2026-09-04 incident, exactly.**
+
+The repo is CORRECT and is not the problem. `platform/externaldns/helmrelease.yaml` sets
+`txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"`, prod's applied `cluster-vars` resolves it to `prod`, and
+the **live HelmRelease `spec.values` carries `txtOwnerId: "prod"`**. Verified against the real chart:
+`helm template external-dns --version 1.21.1` with `txtOwnerId: prod` renders `--txt-owner-id=prod`,
+so the values key is right and an upgrade *would* fix it.
+
+**The fault is Helm/Flux state, not authoring.** `helm get values external-dns` shows the INSTALLED
+release carries `policy` and `domainFilters` but **no `txtOwnerId`** — it predates `c8c1d29`. Yet
+helm-controller reports `Ready=True`, `observedGeneration=3`, and a history entry whose
+`configDigest` **equals** `lastAttemptedConfigDigest`, i.e. **it believes it has already deployed the
+current config.** So it will never upgrade on its own, and the stale release persists indefinitely.
+
+**The only thing preventing the incident was `replicas=0` — manual drift from the incident recovery,
+which any Helm upgrade would reset to the chart default of 1.** That is not a guard; it is luck.
+
+Containment applied (reversible, and it costs nothing because prod publishes no hostnames yet — the
+app tiers are suspended): **`flux suspend helmrelease external-dns -n external-dns`**, with
+`replicas` left at 0. Flux can now neither upgrade the release nor reset the replica count.
+
+**Verified remediation, in this order — the ordering is the whole point:**
+
+1. Force the upgrade so the release picks up `txtOwnerId`, but **do not let the pod run before the
+   flag is confirmed present.** The naive `resume` + `reconcile --force` resets `replicas` to 1 and
+   starts the pod in the same operation, so there is no window in which to check — a race against a
+   process whose failure mode is deleting dev's DNS.
+2. Confirm the live Deployment args contain **`--txt-owner-id=prod`**.
+3. Only then allow it to scale to 1, and immediately diff the Cloudflare zone against a
+   pre-recorded snapshot.
+
+**Dev was verified unharmed throughout this work:** the Cloudflare zone was snapshotted before
+unsuspending `platform` and re-queried after — **57 records, byte-for-byte identical, all 36 TXT
+entries still `owner=default`.** Snapshots are taken against the **Cloudflare API**, never a local
+resolver (a pfSense override produced a false healthy reading during the incident).
+
 ## 10. DNS/TLS environment split — **the 2026-09-04 incident, and the target architecture**
 
 > **Current state (2026-09-04):** the target below is **AUTHORED, NOT APPLIED.** Prod's