diff --git a/bootstrap/flux/README.md b/bootstrap/flux/README.md index 1824ad0..2d0fe1e 100644 --- a/bootstrap/flux/README.md +++ b/bootstrap/flux/README.md @@ -11,7 +11,8 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux ## Files | File | Purpose | |---|---| -| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. | +| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → **pre-apply chart CRDs (`preapply-crds.sh`)** → verify `sops-age` Secret → `kubectl apply -k ${FLUX_ENV_PATH}`. | +| `preapply-crds.sh` | **Server-side pre-apply of the Kyverno CRD bundle** — the CRDs `platform/kyverno/admission/helmrelease.yaml` deliberately tells Helm *not* to install (`crds: {install: false}`, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step `platform` never goes Ready on a cold cluster. Chart name/version/repo are **derived from that HelmRelease** — no second pin. Idempotent; runnable standalone. | | `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. | | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). | | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). | @@ -74,6 +75,49 @@ flux get kustomizations -A # all 11 must return **Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off). +## The CRD pre-apply step (`preapply-crds.sh`) — why bootstrap has to do this + +`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`. That is correct and must +stay: the Kyverno CRD bundle is **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured with +`helm template … --include-crds`, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB +`last-applied-configuration` annotation limit — `metadata.annotations: Too long` — which stalls the +HelmRelease and blocks `platform` plus every tier that `dependsOn` it. + +The part that was never automated: **with CRD install off, nothing installs the CRDs.** Kyverno's +Deployments start, its API types do not exist, `platform` never reaches Ready. Dev has them only +because somebody once ran the command in a shell. `preapply-crds.sh` is that command, in git: + +``` +helm template --repo --version --include-crds --set crds.install=true + | yq 'select(.kind == "CustomResourceDefinition")' + | kubectl apply --server-side --force-conflicts +``` + +- **Version derivation, not a second pin.** `chart`, `version`, `namespace` and the repo `url` are + read out of the HelmRelease/HelmRepository pair with `yq`. `spec.chart.spec.version` is a semver + **constraint** (`3.9.x`), and `helm template --version` accepts the same constraint syntax + helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump + of that one line carries this step with it. There is nothing here that can drift. +- **CRDs only.** Piping the *whole* `helm template` output into kubectl (the by-hand command in the + HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the + HelmRelease over field managers. The `yq` filter is load-bearing. +- **Idempotent.** Server-side apply is an upsert; `--force-conflicts` adopts the fields dev's + hand-applied CRDs left owned by `kubectl-client-side-apply`. Re-running prints + `serverside-applied` and exits 0. +- **Cannot degrade to a no-op.** Three guards: every derived coordinate must be non-empty and not + `null`; the rendered CRD count must be `> 0` (an empty stream would otherwise `apply` cleanly and + exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is + waited on for `condition=Established`. There is **no skip flag** on purpose. +- **Environment-agnostic (R-ENV-02).** It names no environment and takes no overlay input — it reads + repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step. + +Standalone use (an already-bootstrapped cluster whose `platform` is blocked on the missing CRDs): + +```bash +# inside infra/devcontainer, KUBECONFIG pointed at the target cluster +bash bootstrap/flux/preapply-crds.sh +``` + ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1) The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker): @@ -98,7 +142,7 @@ tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1 ▼ (final Ansible play calls the entrypoints below) 1. bootstrap/sops/age-keygen.sh # operator generates age key OUTSIDE repo 2. bootstrap/sops/age-key-secret.md # operator installs private key as sops-age Secret -3. bootstrap/flux/bootstrap.sh # flux install → verify sops-age → apply -k clusters/dev +3. bootstrap/flux/bootstrap.sh # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH} │ ▼ Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config) @@ -118,8 +162,10 @@ live in `infra/devcontainer/versions.env`). template). `bootstrap.sh` **fails fast** if this Secret is absent. 3. `bootstrap/flux/bootstrap.sh` (this entrypoint). -`bootstrap.sh` is idempotent-friendly (`flux install` and `kubectl apply -k` re-run safely) and -never handles key material — it only *verifies* the Secret exists. +`bootstrap.sh` is idempotent-friendly (`flux install`, `preapply-crds.sh`'s server-side apply, and +`kubectl apply -k` all re-run safely) and never handles key material — it only *verifies* the Secret +exists. Beyond `flux`/`kubectl` it now also needs **`helm` + `yq`** on PATH (step 2's CRD render; +both are in `infra/devcontainer`) and outbound network to the Kyverno chart repo. ## Handoff — clusters owner / PM (files this worker may not edit) diff --git a/bootstrap/flux/bootstrap.sh b/bootstrap/flux/bootstrap.sh index 7617361..74b5f45 100755 --- a/bootstrap/flux/bootstrap.sh +++ b/bootstrap/flux/bootstrap.sh @@ -33,10 +33,19 @@ # the single apps.yaml was retired). Letting # bootstrap regenerate them would DUPLICATE/conflict. `flux install` + # applying the authored clusters/dev overlay is the reconciling path. -# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so +# 2. CRD pre-apply — ./preapply-crds.sh applies the Kyverno CRD bundle SERVER-SIDE, because +# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}` +# (the ~5.6 MiB bundle blows kubectl's 256 KiB last-applied annotation limit, +# which STALLS the HelmRelease and blocks the `platform` Kustomization and +# every tier below it). With CRD install off, NOTHING else installs them, so +# a cold cluster can never reconcile `platform` without this step. The chart +# version is DERIVED from that same HelmRelease — no second pin. Dev only +# ever had these CRDs because they were applied BY HAND; this is that step, +# in git, idempotent and version-derived. See ./preapply-crds.sh. +# 3. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so # decryption works on first pass (../sops/age-key-secret.md). This script # only VERIFIES it exists; it never handles key material. -# 3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph: +# 4. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph: # GitRepository `flux-system` (source) -> Kustomization `platform` -> # `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2` # (chained dependsOn) + `apps-vms`, with cluster-vars overlay @@ -71,7 +80,10 @@ repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}" # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------ -for bin in flux kubectl; do +# helm + yq are needed by step 2 (./preapply-crds.sh renders the CRD bundle from the authored +# HelmRelease). Checked HERE as well as in that script so a missing binary stops the run BEFORE the +# Flux controllers are installed, instead of half-way through. +for bin in flux kubectl helm yq; do command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; } done @@ -99,7 +111,7 @@ done # # ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO # strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so. -echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}" +echo "==> [1/4] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}" flux_install_args=(--namespace="${FLUX_NAMESPACE}") [ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}") @@ -130,18 +142,29 @@ else kubectl apply -f "${render_dir}/rendered.yaml" fi -# --- 2. verify the out-of-band age key Secret exists ------------------------- +# --- 2. pre-apply the CRDs Helm is not allowed to install --------------------- +# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`, so the 22 Kyverno CRDs +# must exist BEFORE the platform Kustomization reconciles that HelmRelease — otherwise Kyverno's +# controllers come up with no API types, `platform` never goes Ready, and all of its dependents stall. +# ./preapply-crds.sh derives the chart version from the HelmRelease itself (no second pin), applies +# only the CRD documents SERVER-SIDE (no 256 KiB annotation limit), and is idempotent. It FAILS HARD +# rather than skipping — there is deliberately NO opt-out flag: a skipped CRD apply produces a +# cluster that looks bootstrapped and reconciles nothing. +echo "==> [2/4] pre-applying chart CRDs (server-side)" +bash "${script_dir}/preapply-crds.sh" + +# --- 3. verify the out-of-band age key Secret exists ------------------------- # We DO NOT create it here (it holds the private key). The operator installs it per # ../sops/age-key-secret.md. Fail fast if missing so decryption can't silently no-op. -echo "==> [2/3] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}" +echo "==> [3/4] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}" if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null 2>&1; then echo "ERROR: Secret ${FLUX_NAMESPACE}/${SOPS_SECRET_NAME} not found." >&2 echo " Install the age private key first: bootstrap/sops/age-key-secret.md" >&2 exit 1 fi -# --- 3. apply the authored environment entrypoint ---------------------------- -echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}" +# --- 4. apply the authored environment entrypoint ---------------------------- +echo "==> [4/4] kubectl apply -k ${FLUX_ENV_PATH}" kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}" echo diff --git a/bootstrap/flux/preapply-crds.sh b/bootstrap/flux/preapply-crds.sh new file mode 100755 index 0000000..a97338f --- /dev/null +++ b/bootstrap/flux/preapply-crds.sh @@ -0,0 +1,169 @@ +#!/usr/bin/env bash +# +# preapply-crds.sh — server-side pre-apply of the CRDs that Helm is NOT allowed to install. +# ENVIRONMENT-AGNOSTIC: it reads the chart coordinates out of the authored HelmRelease and talks to +# whatever cluster $KUBECONFIG points at, so dev and prod run the SAME step with no fork (R-ENV-02). +# Called by ./bootstrap.sh (step 2/4); also safe to run STANDALONE against an already-bootstrapped +# cluster (that is how prod gets unblocked without re-running the whole bootstrap). +# +# R-GITOPS-01 (the graph must reconcile from git, not from someone's shell history) · R-ENV-02. +# ============================================================================= +# *** AUTHORED, NOT RUN. Agents must not execute it — it MUTATES a cluster. *** +# ============================================================================= +# +# WHY THIS EXISTS +# +# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`. That is deliberate: +# the Kyverno CRD bundle is ~5.6 MiB, far past kubectl's 256 KiB last-applied-configuration +# annotation limit, so Helm's client-side apply fails with "metadata.annotations: Too long" and +# STALLS the HelmRelease — which blocks the `platform` Kustomization and every tier that dependsOn +# it (the kps-cascade failure mode; see the HelmRelease's own comment). +# +# The consequence nobody automated: with CRD install off, NOTHING installs those CRDs. Kyverno's +# Deployments come up but its API types do not exist, so `platform` never goes Ready on a cold +# cluster. The dev cluster only has them because they were once applied BY HAND — exactly the +# undocumented manual step this repo exists to eliminate. This script is that step, in git, +# version-derived and idempotent. +# +# HOW THE VERSION IS DERIVED (there is deliberately NO second copy of it) +# +# Everything comes out of the HelmRelease/HelmRepository pair that Flux itself reconciles: +# +# chart name spec.chart.spec.chart (HelmRelease) +# chart version spec.chart.spec.version (HelmRelease) e.g. "3.9.x" +# chart repo url spec.url (HelmRepository) +# namespace metadata.namespace (HelmRelease) +# +# The version is a semver CONSTRAINT, not a fixed tag. `helm template --version` accepts the same +# constraint syntax Flux's helm-controller resolves (both use Masterminds/semver), so passing it +# through UNCHANGED renders the very chart version Flux will install. Renovate bumps one line in +# one file and this step follows automatically — it cannot drift, because there is nothing here to +# drift from. +# +# WHY ONLY THE CRD DOCUMENTS ARE APPLIED +# +# The by-hand command recorded in the HelmRelease comment piped the WHOLE `helm template` output +# into kubectl. Do not do that: it would create Deployments/Services/webhooks outside Helm's +# ownership, which then fight the HelmRelease over field managers and leave orphans behind. We +# filter to `kind: CustomResourceDefinition` and apply nothing else. Values are the chart DEFAULTS +# (plus an explicit crds.install=true) — NOT the repo's values, which switch CRD rendering off. +# +# IDEMPOTENCY +# +# `kubectl apply --server-side --force-conflicts` is a declarative upsert: the first run creates, +# every later run reports "serverside-applied" and exits 0, and field ownership is taken from +# whatever previously wrote those objects (that is what --force-conflicts is for — it resolves the +# hand-applied dev CRDs' `kubectl-client-side-apply` manager without a human deciding anything). +# Server-side apply does not write the last-applied annotation, which is the whole reason the +# 5.6 MiB bundle can be applied at all. +# +# LOUD FAILURE (this script must never degrade to a successful no-op) +# +# `set -euo pipefail`, plus three explicit guards, because a silent skip here reproduces the exact +# class of bug this repo has been bitten by more than once: +# 1. every derived coordinate must be non-empty and not "null" — a moved YAML path fails here; +# 2. the rendered CRD count must be > 0 — if a future chart moves/renames its CRD templates the +# filter would match nothing and `kubectl apply` on an empty stream EXITS 0. Guarded; +# 3. after applying, every CRD is waited on for condition=Established — so "applied" is proven +# to mean "the API type exists", not merely "kubectl returned 0". +# +set -euo pipefail + +# --- config (env-overridable) ------------------------------------------------ +# The single source of truth for the chart coordinates. Repo-relative on purpose: this is the file +# Flux reconciles, so it is the file this step must agree with. +CRD_HELMRELEASE="${CRD_HELMRELEASE:-platform/kyverno/admission/helmrelease.yaml}" +# How long to wait for the applied CRDs to become Established (apiextensions publishes the type). +CRD_ESTABLISH_TIMEOUT="${CRD_ESTABLISH_TIMEOUT:-120s}" + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))" + +echo "==> CRD pre-apply — source of truth: ${CRD_HELMRELEASE}" + +# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------ +# helm renders the bundle, yq selects the CRD documents, kubectl applies them server-side. All three +# are baked into infra/devcontainer (HELM_VERSION / YQ_VERSION / KUBECTL_VERSION in versions.env). +for bin in helm yq kubectl; do + command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; } +done + +helmrelease="${repo_root}/${CRD_HELMRELEASE}" +[ -f "${helmrelease}" ] || { + echo "ERROR: ${helmrelease} not found — CRD_HELMRELEASE points nowhere." >&2 + exit 1 +} + +# --- 1. derive the chart coordinates from the authored manifests -------------- +chart_name="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.chart' "${helmrelease}")" +chart_version="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "${helmrelease}")" +chart_ns="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .metadata.namespace' "${helmrelease}")" +chart_repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' "${helmrelease}")" + +# Guard 1: a moved/renamed YAML path must STOP the bootstrap, not render a bogus chart. yq prints +# "null" for a missing path and an empty string for a missing document, so both are rejected. +for pair in "chart:${chart_name}" "version:${chart_version}" "namespace:${chart_ns}" "repo:${chart_repo}"; do + key="${pair%%:*}"; val="${pair#*:}" + case "${val}" in + ""|null|*$'\n'*) + echo "ERROR: could not derive a single '${key}' from ${CRD_HELMRELEASE} (got: '${val}')." >&2 + echo " The HelmRelease/HelmRepository shape changed — fix the yq paths in $(basename "${BASH_SOURCE[0]}")." >&2 + exit 1 + ;; + esac +done + +echo " chart=${chart_name} version=${chart_version} repo=${chart_repo} namespace=${chart_ns}" +echo " (version is the constraint Flux resolves — NOT a second copy pinned here)" + +# --- 2. render the chart and keep ONLY the CRD documents ---------------------- +render_dir="$(mktemp -d)" +trap 'rm -rf "${render_dir}"' EXIT +crd_file="${render_dir}/crds.yaml" + +# --include-crds covers charts that ship a native crds/ dir; crds.install=true covers Kyverno, whose +# CRDs are subchart TEMPLATES that the repo's own values switch off. Chart defaults otherwise. +echo "==> rendering ${chart_name} ${chart_version} (CRD documents only)" +helm template "${chart_name}" "${chart_name}" \ + --repo "${chart_repo}" \ + --version "${chart_version}" \ + --namespace "${chart_ns}" \ + --include-crds \ + --set crds.install=true \ + | yq eval-all 'select(.kind == "CustomResourceDefinition")' - > "${crd_file}" + +# Blank/"null" lines are dropped on purpose: an EMPTY render makes `yq .metadata.name` emit one +# empty line, which would otherwise count as a CRD and sail straight past guard 2 below. +mapfile -t crd_names < <(yq eval-all --no-doc '.metadata.name' "${crd_file}" | grep -vx -e '' -e 'null' || true) +crd_count="${#crd_names[@]}" + +# Guard 2: an empty stream applies cleanly and exits 0 — the silent-no-op trap. Refuse it. +[ "${crd_count}" -gt 0 ] || { + echo "ERROR: rendered 0 CustomResourceDefinitions from ${chart_name} ${chart_version}." >&2 + echo " The chart's CRD layout or the crds.install value key changed. Refusing to continue:" >&2 + echo " applying nothing would look like success and leave 'platform' blocked forever." >&2 + exit 1 +} +echo " ${crd_count} CRDs, $(wc -c < "${crd_file}") bytes (>256 KiB — this is why it must be server-side)" + +# --- 3. server-side apply (idempotent upsert) --------------------------------- +# --server-side: no last-applied annotation, so the size limit does not apply. +# --force-conflicts: adopt fields owned by an earlier manager (e.g. dev's hand-applied CRDs, or +# Helm's own manager if a past chart version did install them). Makes a re-run a plain no-op. +echo "==> kubectl apply --server-side --force-conflicts (${crd_count} CRDs)" +kubectl apply --server-side --force-conflicts -f "${crd_file}" + +# --- 4. prove the API types actually exist ----------------------------------- +# Without this, "kubectl returned 0" is the only evidence. Establishment is what Kyverno's +# controllers and the platform Kustomization actually need. +echo "==> waiting for condition=Established (timeout ${CRD_ESTABLISH_TIMEOUT})" +crd_refs=() +for n in "${crd_names[@]}"; do crd_refs+=("crd/${n}"); done +kubectl wait --for=condition=Established --timeout="${CRD_ESTABLISH_TIMEOUT}" \ + "${crd_refs[@]}" >/dev/null || { + echo "ERROR: not all ${crd_count} CRDs reached Established within ${CRD_ESTABLISH_TIMEOUT}." >&2 + echo " Inspect: kubectl get crd | grep ${chart_name}" >&2 + exit 1 +} + +echo "==> ${crd_count} ${chart_name} CRDs applied and Established." diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 681bf88..f91ab5d 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -1260,6 +1260,45 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be `inventory/group_vars/k3s_prod/vars.yml` overrides it to `clusters/prod` — the same mechanism as `kube_vip_address` / `k3s_data_dir` / `cluster_kubeconfig_name`. Dev has no override anywhere, so dev resolves `clusters/dev` exactly as before. +- [x] **The Kyverno CRDs had to be applied BY HAND, and nothing in the repo did it** — AUTHORED + 2026-09-04 (`bootstrap/flux/preapply-crds.sh`), **not yet run against prod**. This is a hard + **prerequisite of a reconcilable cluster**, not a nice-to-have: `platform/kyverno/admission/ + helmrelease.yaml` sets `crds: {install: false}` — deliberately, because the CRD bundle is + **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured 2026-09-04 with `helm template + kyverno kyverno --repo https://kyverno.github.io/kyverno/ --version 3.9.x --include-crds`), far + past kubectl's 256 KiB `last-applied-configuration` annotation limit, so Helm's client-side apply + fails `metadata.annotations: Too long` and **stalls** the HelmRelease. The consequence nobody + had automated: with CRD install off, **nothing installs those CRDs at all**. On a cold cluster + Kyverno's Deployments come up with no API types, `platform` never goes Ready, and all 9 + downstream Kustomizations stall behind it. Dev only ever had them because the command was once + run in somebody's shell — the exact undocumented manual step this project exists to delete. + - **The fix, in git:** `bootstrap/flux/bootstrap.sh` now runs `./preapply-crds.sh` as step + **[2/4]** — after `flux install`, before `kubectl apply -k ${FLUX_ENV_PATH}`. It renders the + chart, keeps only `kind: CustomResourceDefinition` documents, applies them + `--server-side --force-conflicts` (no annotation, so no size limit), then waits for + `condition=Established`. The chart name, version constraint, repo URL and namespace are + **derived from that same HelmRelease/HelmRepository** — there is no second copy of the version + to drift, and a Renovate bump of the one pinned line carries this step with it. + - **No environment fork (R-ENV-02):** the script names no environment and takes no overlay + input; it reads repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step, so + `roles/flux_bootstrap` needed **no change** — it already calls `bootstrap.sh`, the single + entrypoint. + - **It cannot silently skip.** Three guards, and deliberately no opt-out flag: a derived + coordinate that is empty/`null` aborts; a render that yields **0** CRDs aborts (an empty + stream would otherwise `kubectl apply` cleanly and exit 0 — the silent-success failure mode + that has bitten this repo repeatedly); and every applied CRD must reach `Established`. + - **Toolchain:** needs `helm` and `yq` on PATH in addition to `flux`/`kubectl` — both are baked + into `infra/devcontainer` (`HELM_VERSION=3.16.0`, `YQ_VERSION=4.44.6`), and `bootstrap.sh` + now fails at step 0 if either is missing. It also needs **outbound network to + `kyverno.github.io`** from wherever the play runs (the Ansible role delegates to localhost). + - **PM to verify in the container** (the authoring agent ran it only with a stubbed `kubectl`, + never against a cluster): pinned `helm` 3.16.0 renders the same 22 CRDs, and the run is a + clean no-op the second time. + - **Unblocking the already-bootstrapped prod cluster** does not need a full re-bootstrap: + ```bash + # inside infra/devcontainer, KUBECONFIG=.cache/prod.kubeconfig + bash bootstrap/flux/preapply-crds.sh + ``` - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03). @@ -1272,7 +1311,11 @@ single-server-etcd window, and avoids a later member-add dance. 1. Per-node §1–§5 on each of the three boxes. 2. §7 repo work lands on `main` with the gate green. 3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`. -4. `flux bootstrap ... --path=clusters/prod`. +4. `flux bootstrap ... --path=clusters/prod` — i.e. `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod`, + which runs `bootstrap/flux/bootstrap.sh` with `FLUX_ENV_PATH=clusters/prod`. That script's step + **[2/4]** pre-applies the 22 Kyverno CRDs server-side; **without it `platform` cannot go Ready** + and the 9 tiers below it never reconcile (§7). It needs `helm` + `yq` on PATH and outbound network + to the chart repo. 5. Conformance against prod (§9). ## 9. Exit criteria (Phase 7)