feat(bootstrap): automate the Kyverno CRD pre-apply — the last cold-start blocker

`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`
because the CRD bundle is 5,636,155 bytes, far past kubectl's 256 KiB
last-applied-configuration limit — a client-side apply fails with
"metadata.annotations: Too long" and STALLS the HelmRelease.

The consequence nobody automated: with CRD install off, NOTHING installed
them. `platform` has `wait: true` and every other tier chains transitively
through it, so on a cold cluster Kyverno's API types never exist, `platform`
never goes Ready, and all 10 downstream tiers stall. Dev only ever had these
CRDs because they were applied BY HAND — exactly the undocumented manual step
this repo exists to eliminate.

New `bootstrap/flux/preapply-crds.sh`, wired into bootstrap.sh as step [2/4]
(after `flux install`, before the graph is applied) and standalone-safe so an
already-bootstrapped cluster can be unblocked without re-running bootstrap.

NO SECOND PIN: all four chart coordinates are read with yq out of the same
HelmRelease/HelmRepository pair Flux reconciles. The version is passed through
as the CONSTRAINT ("3.9.x"), not a resolved tag — helm and helm-controller both
use Masterminds/semver, so the render is the version Flux installs. Renovate
bumps one line and this step follows; there is nothing here to drift from.

Only `kind: CustomResourceDefinition` documents are applied. The by-hand
command recorded in the HelmRelease comment piped the WHOLE template into
kubectl, which would create Deployments/webhooks outside Helm's ownership and
then fight the HelmRelease over field managers.

THREE GUARDS, because a silent skip here reproduces a failure mode this repo
has been bitten by repeatedly:
  1. any derived coordinate that is empty/null/multi-valued aborts;
  2. rendered CRD count must be > 0 — this caught a real bug in the first
     draft: on an empty render `yq .metadata.name` emits one BLANK line, which
     counted as 1 CRD and sailed past the guard into an apply of an empty file
     returning 0, i.e. the exact silent-success trap. Blank/null lines are now
     dropped and a chart with no CRDs exits 1;
  3. every applied CRD is waited on for condition=Established, so "kubectl
     returned 0" is not the only evidence.
No opt-out flag, deliberately: a skipped CRD apply yields a cluster that looks
bootstrapped and reconciles nothing.

`roles/flux_bootstrap` is untouched — it already shells to bootstrap.sh, the
documented single entrypoint, so it inherits the step with no dev/prod fork
(R-ENV-02). A second role task would double-apply.

MEASURED, not assumed — reproduced under the PINNED toolchain (helm 3.16.0,
yq 4.44.6) in infra/devcontainer, matching an independent run on helm 3.20.0
byte-for-byte: 3.9.x resolves to 3.9.0, 22 CRDs, 5,636,155 bytes CRD-only
(5,870,019 full render). Largest single CRD document is 1,378,544 bytes —
under etcd's 1.5 MiB default, but not by much; worth watching on chart bumps.

NOT YET EXERCISED against a live API server: the server-side apply and the
Established wait were tested with a stub kubectl. velero and
kube-prometheus-stack also say "CRDs pre-applied" but both still get theirs
from Helm's native crds/ dir on FIRST install, so neither is a cold-start
blocker; CRD_HELMRELEASE makes extending this a one-liner if that changes.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/PROD-PROVISIONING.md

@@ -1260,6 +1260,45 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
             `inventory/group_vars/k3s_prod/vars.yml` overrides it to `clusters/prod` — the same
             mechanism as `kube_vip_address` / `k3s_data_dir` / `cluster_kubeconfig_name`. Dev has no
             override anywhere, so dev resolves `clusters/dev` exactly as before.
+- [x] **The Kyverno CRDs had to be applied BY HAND, and nothing in the repo did it** — AUTHORED
+      2026-09-04 (`bootstrap/flux/preapply-crds.sh`), **not yet run against prod**. This is a hard
+      **prerequisite of a reconcilable cluster**, not a nice-to-have: `platform/kyverno/admission/
+      helmrelease.yaml` sets `crds: {install: false}` — deliberately, because the CRD bundle is
+      **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured 2026-09-04 with `helm template
+      kyverno kyverno --repo https://kyverno.github.io/kyverno/ --version 3.9.x --include-crds`), far
+      past kubectl's 256 KiB `last-applied-configuration` annotation limit, so Helm's client-side apply
+      fails `metadata.annotations: Too long` and **stalls** the HelmRelease. The consequence nobody
+      had automated: with CRD install off, **nothing installs those CRDs at all**. On a cold cluster
+      Kyverno's Deployments come up with no API types, `platform` never goes Ready, and all 9
+      downstream Kustomizations stall behind it. Dev only ever had them because the command was once
+      run in somebody's shell — the exact undocumented manual step this project exists to delete.
+      - **The fix, in git:** `bootstrap/flux/bootstrap.sh` now runs `./preapply-crds.sh` as step
+        **[2/4]** — after `flux install`, before `kubectl apply -k ${FLUX_ENV_PATH}`. It renders the
+        chart, keeps only `kind: CustomResourceDefinition` documents, applies them
+        `--server-side --force-conflicts` (no annotation, so no size limit), then waits for
+        `condition=Established`. The chart name, version constraint, repo URL and namespace are
+        **derived from that same HelmRelease/HelmRepository** — there is no second copy of the version
+        to drift, and a Renovate bump of the one pinned line carries this step with it.
+      - **No environment fork (R-ENV-02):** the script names no environment and takes no overlay
+        input; it reads repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step, so
+        `roles/flux_bootstrap` needed **no change** — it already calls `bootstrap.sh`, the single
+        entrypoint.
+      - **It cannot silently skip.** Three guards, and deliberately no opt-out flag: a derived
+        coordinate that is empty/`null` aborts; a render that yields **0** CRDs aborts (an empty
+        stream would otherwise `kubectl apply` cleanly and exit 0 — the silent-success failure mode
+        that has bitten this repo repeatedly); and every applied CRD must reach `Established`.
+      - **Toolchain:** needs `helm` and `yq` on PATH in addition to `flux`/`kubectl` — both are baked
+        into `infra/devcontainer` (`HELM_VERSION=3.16.0`, `YQ_VERSION=4.44.6`), and `bootstrap.sh`
+        now fails at step 0 if either is missing. It also needs **outbound network to
+        `kyverno.github.io`** from wherever the play runs (the Ansible role delegates to localhost).
+      - **PM to verify in the container** (the authoring agent ran it only with a stubbed `kubectl`,
+        never against a cluster): pinned `helm` 3.16.0 renders the same 22 CRDs, and the run is a
+        clean no-op the second time.
+      - **Unblocking the already-bootstrapped prod cluster** does not need a full re-bootstrap:
+        ```bash
+        # inside infra/devcontainer, KUBECONFIG=.cache/prod.kubeconfig
+        bash bootstrap/flux/preapply-crds.sh
+        ```
 - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
       the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
 
@@ -1272,7 +1311,11 @@ single-server-etcd window, and avoids a later member-add dance.
 1. Per-node §1–§5 on each of the three boxes.
 2. §7 repo work lands on `main` with the gate green.
 3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`.
-4. `flux bootstrap ... --path=clusters/prod`.
+4. `flux bootstrap ... --path=clusters/prod` — i.e. `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod`,
+   which runs `bootstrap/flux/bootstrap.sh` with `FLUX_ENV_PATH=clusters/prod`. That script's step
+   **[2/4]** pre-applies the 22 Kyverno CRDs server-side; **without it `platform` cannot go Ready**
+   and the 9 tiers below it never reconcile (§7). It needs `helm` + `yq` on PATH and outbound network
+   to the chart repo.
 5. Conformance against prod (§9).
 
 ## 9. Exit criteria (Phase 7)