feat(bootstrap): automate the Kyverno CRD pre-apply — the last cold-start blocker
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`
because the CRD bundle is 5,636,155 bytes, far past kubectl's 256 KiB
last-applied-configuration limit — a client-side apply fails with
"metadata.annotations: Too long" and STALLS the HelmRelease.
The consequence nobody automated: with CRD install off, NOTHING installed
them. `platform` has `wait: true` and every other tier chains transitively
through it, so on a cold cluster Kyverno's API types never exist, `platform`
never goes Ready, and all 10 downstream tiers stall. Dev only ever had these
CRDs because they were applied BY HAND — exactly the undocumented manual step
this repo exists to eliminate.
New `bootstrap/flux/preapply-crds.sh`, wired into bootstrap.sh as step [2/4]
(after `flux install`, before the graph is applied) and standalone-safe so an
already-bootstrapped cluster can be unblocked without re-running bootstrap.
NO SECOND PIN: all four chart coordinates are read with yq out of the same
HelmRelease/HelmRepository pair Flux reconciles. The version is passed through
as the CONSTRAINT ("3.9.x"), not a resolved tag — helm and helm-controller both
use Masterminds/semver, so the render is the version Flux installs. Renovate
bumps one line and this step follows; there is nothing here to drift from.
Only `kind: CustomResourceDefinition` documents are applied. The by-hand
command recorded in the HelmRelease comment piped the WHOLE template into
kubectl, which would create Deployments/webhooks outside Helm's ownership and
then fight the HelmRelease over field managers.
THREE GUARDS, because a silent skip here reproduces a failure mode this repo
has been bitten by repeatedly:
1. any derived coordinate that is empty/null/multi-valued aborts;
2. rendered CRD count must be > 0 — this caught a real bug in the first
draft: on an empty render `yq .metadata.name` emits one BLANK line, which
counted as 1 CRD and sailed past the guard into an apply of an empty file
returning 0, i.e. the exact silent-success trap. Blank/null lines are now
dropped and a chart with no CRDs exits 1;
3. every applied CRD is waited on for condition=Established, so "kubectl
returned 0" is not the only evidence.
No opt-out flag, deliberately: a skipped CRD apply yields a cluster that looks
bootstrapped and reconciles nothing.
`roles/flux_bootstrap` is untouched — it already shells to bootstrap.sh, the
documented single entrypoint, so it inherits the step with no dev/prod fork
(R-ENV-02). A second role task would double-apply.
MEASURED, not assumed — reproduced under the PINNED toolchain (helm 3.16.0,
yq 4.44.6) in infra/devcontainer, matching an independent run on helm 3.20.0
byte-for-byte: 3.9.x resolves to 3.9.0, 22 CRDs, 5,636,155 bytes CRD-only
(5,870,019 full render). Largest single CRD document is 1,378,544 bytes —
under etcd's 1.5 MiB default, but not by much; worth watching on chart bumps.
NOT YET EXERCISED against a live API server: the server-side apply and the
Established wait were tested with a stub kubectl. velero and
kube-prometheus-stack also say "CRDs pre-applied" but both still get theirs
from Helm's native crds/ dir on FIRST install, so neither is a cold-start
blocker; CRD_HELMRELEASE makes extending this a one-liner if that changes.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
9137bbd0c3
commit
13081a863f
4 files changed
+294
-13
No files matched your search
@@ -11,7 +11,8 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
|
||||
## Files
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
|
||||
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → **pre-apply chart CRDs (`preapply-crds.sh`)** → verify `sops-age` Secret → `kubectl apply -k ${FLUX_ENV_PATH}`. |
|
||||
| `preapply-crds.sh` | **Server-side pre-apply of the Kyverno CRD bundle** — the CRDs `platform/kyverno/admission/helmrelease.yaml` deliberately tells Helm *not* to install (`crds: {install: false}`, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step `platform` never goes Ready on a cold cluster. Chart name/version/repo are **derived from that HelmRelease** — no second pin. Idempotent; runnable standalone. |
|
||||
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
|
||||
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
|
||||
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
|
||||
@@ -74,6 +75,49 @@ flux get kustomizations -A # all 11 must return
|
||||
**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
|
||||
simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
|
||||
|
||||
## The CRD pre-apply step (`preapply-crds.sh`) — why bootstrap has to do this
|
||||
|
||||
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`. That is correct and must
|
||||
stay: the Kyverno CRD bundle is **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured with
|
||||
`helm template … --include-crds`, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB
|
||||
`last-applied-configuration` annotation limit — `metadata.annotations: Too long` — which stalls the
|
||||
HelmRelease and blocks `platform` plus every tier that `dependsOn` it.
|
||||
|
||||
The part that was never automated: **with CRD install off, nothing installs the CRDs.** Kyverno's
|
||||
Deployments start, its API types do not exist, `platform` never reaches Ready. Dev has them only
|
||||
because somebody once ran the command in a shell. `preapply-crds.sh` is that command, in git:
|
||||
|
||||
```
|
||||
helm template <chart> --repo <url> --version <constraint> --include-crds --set crds.install=true
|
||||
| yq 'select(.kind == "CustomResourceDefinition")'
|
||||
| kubectl apply --server-side --force-conflicts
|
||||
```
|
||||
|
||||
- **Version derivation, not a second pin.** `chart`, `version`, `namespace` and the repo `url` are
|
||||
read out of the HelmRelease/HelmRepository pair with `yq`. `spec.chart.spec.version` is a semver
|
||||
**constraint** (`3.9.x`), and `helm template --version` accepts the same constraint syntax
|
||||
helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump
|
||||
of that one line carries this step with it. There is nothing here that can drift.
|
||||
- **CRDs only.** Piping the *whole* `helm template` output into kubectl (the by-hand command in the
|
||||
HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the
|
||||
HelmRelease over field managers. The `yq` filter is load-bearing.
|
||||
- **Idempotent.** Server-side apply is an upsert; `--force-conflicts` adopts the fields dev's
|
||||
hand-applied CRDs left owned by `kubectl-client-side-apply`. Re-running prints
|
||||
`serverside-applied` and exits 0.
|
||||
- **Cannot degrade to a no-op.** Three guards: every derived coordinate must be non-empty and not
|
||||
`null`; the rendered CRD count must be `> 0` (an empty stream would otherwise `apply` cleanly and
|
||||
exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is
|
||||
waited on for `condition=Established`. There is **no skip flag** on purpose.
|
||||
- **Environment-agnostic (R-ENV-02).** It names no environment and takes no overlay input — it reads
|
||||
repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step.
|
||||
|
||||
Standalone use (an already-bootstrapped cluster whose `platform` is blocked on the missing CRDs):
|
||||
|
||||
```bash
|
||||
# inside infra/devcontainer, KUBECONFIG pointed at the target cluster
|
||||
bash bootstrap/flux/preapply-crds.sh
|
||||
```
|
||||
|
||||
## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
|
||||
|
||||
The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
|
||||
@@ -98,7 +142,7 @@ tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1
|
||||
▼ (final Ansible play calls the entrypoints below)
|
||||
1. bootstrap/sops/age-keygen.sh # operator generates age key OUTSIDE repo
|
||||
2. bootstrap/sops/age-key-secret.md # operator installs private key as sops-age Secret
|
||||
3. bootstrap/flux/bootstrap.sh # flux install → verify sops-age → apply -k clusters/dev
|
||||
3. bootstrap/flux/bootstrap.sh # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH}
|
||||
│
|
||||
▼
|
||||
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
|
||||
@@ -118,8 +162,10 @@ live in `infra/devcontainer/versions.env`).
|
||||
template). `bootstrap.sh` **fails fast** if this Secret is absent.
|
||||
3. `bootstrap/flux/bootstrap.sh` (this entrypoint).
|
||||
|
||||
`bootstrap.sh` is idempotent-friendly (`flux install` and `kubectl apply -k` re-run safely) and
|
||||
never handles key material — it only *verifies* the Secret exists.
|
||||
`bootstrap.sh` is idempotent-friendly (`flux install`, `preapply-crds.sh`'s server-side apply, and
|
||||
`kubectl apply -k` all re-run safely) and never handles key material — it only *verifies* the Secret
|
||||
exists. Beyond `flux`/`kubectl` it now also needs **`helm` + `yq`** on PATH (step 2's CRD render;
|
||||
both are in `infra/devcontainer`) and outbound network to the Kyverno chart repo.
|
||||
|
||||
## Handoff — clusters owner / PM (files this worker may not edit)
|
||||
|
||||
|
||||
@@ -33,10 +33,19 @@
|
||||
# the single apps.yaml was retired). Letting
|
||||
# bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
|
||||
# applying the authored clusters/dev overlay is the reconciling path.
|
||||
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
# 2. CRD pre-apply — ./preapply-crds.sh applies the Kyverno CRD bundle SERVER-SIDE, because
|
||||
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`
|
||||
# (the ~5.6 MiB bundle blows kubectl's 256 KiB last-applied annotation limit,
|
||||
# which STALLS the HelmRelease and blocks the `platform` Kustomization and
|
||||
# every tier below it). With CRD install off, NOTHING else installs them, so
|
||||
# a cold cluster can never reconcile `platform` without this step. The chart
|
||||
# version is DERIVED from that same HelmRelease — no second pin. Dev only
|
||||
# ever had these CRDs because they were applied BY HAND; this is that step,
|
||||
# in git, idempotent and version-derived. See ./preapply-crds.sh.
|
||||
# 3. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
# decryption works on first pass (../sops/age-key-secret.md). This script
|
||||
# only VERIFIES it exists; it never handles key material.
|
||||
# 3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
|
||||
# 4. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
|
||||
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
|
||||
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
|
||||
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
|
||||
@@ -71,7 +80,10 @@ repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd
|
||||
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
|
||||
|
||||
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
||||
for bin in flux kubectl; do
|
||||
# helm + yq are needed by step 2 (./preapply-crds.sh renders the CRD bundle from the authored
|
||||
# HelmRelease). Checked HERE as well as in that script so a missing binary stops the run BEFORE the
|
||||
# Flux controllers are installed, instead of half-way through.
|
||||
for bin in flux kubectl helm yq; do
|
||||
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
||||
done
|
||||
|
||||
@@ -99,7 +111,7 @@ done
|
||||
#
|
||||
# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
|
||||
# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
|
||||
echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
|
||||
echo "==> [1/4] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
|
||||
flux_install_args=(--namespace="${FLUX_NAMESPACE}")
|
||||
[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
|
||||
|
||||
@@ -130,18 +142,29 @@ else
|
||||
kubectl apply -f "${render_dir}/rendered.yaml"
|
||||
fi
|
||||
|
||||
# --- 2. verify the out-of-band age key Secret exists -------------------------
|
||||
# --- 2. pre-apply the CRDs Helm is not allowed to install ---------------------
|
||||
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`, so the 22 Kyverno CRDs
|
||||
# must exist BEFORE the platform Kustomization reconciles that HelmRelease — otherwise Kyverno's
|
||||
# controllers come up with no API types, `platform` never goes Ready, and all of its dependents stall.
|
||||
# ./preapply-crds.sh derives the chart version from the HelmRelease itself (no second pin), applies
|
||||
# only the CRD documents SERVER-SIDE (no 256 KiB annotation limit), and is idempotent. It FAILS HARD
|
||||
# rather than skipping — there is deliberately NO opt-out flag: a skipped CRD apply produces a
|
||||
# cluster that looks bootstrapped and reconciles nothing.
|
||||
echo "==> [2/4] pre-applying chart CRDs (server-side)"
|
||||
bash "${script_dir}/preapply-crds.sh"
|
||||
|
||||
# --- 3. verify the out-of-band age key Secret exists -------------------------
|
||||
# We DO NOT create it here (it holds the private key). The operator installs it per
|
||||
# ../sops/age-key-secret.md. Fail fast if missing so decryption can't silently no-op.
|
||||
echo "==> [2/3] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
|
||||
echo "==> [3/4] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
|
||||
if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null 2>&1; then
|
||||
echo "ERROR: Secret ${FLUX_NAMESPACE}/${SOPS_SECRET_NAME} not found." >&2
|
||||
echo " Install the age private key first: bootstrap/sops/age-key-secret.md" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- 3. apply the authored environment entrypoint ----------------------------
|
||||
echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
|
||||
# --- 4. apply the authored environment entrypoint ----------------------------
|
||||
echo "==> [4/4] kubectl apply -k ${FLUX_ENV_PATH}"
|
||||
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
|
||||
|
||||
echo
|
||||
|
||||
Executable
+169
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# preapply-crds.sh — server-side pre-apply of the CRDs that Helm is NOT allowed to install.
|
||||
# ENVIRONMENT-AGNOSTIC: it reads the chart coordinates out of the authored HelmRelease and talks to
|
||||
# whatever cluster $KUBECONFIG points at, so dev and prod run the SAME step with no fork (R-ENV-02).
|
||||
# Called by ./bootstrap.sh (step 2/4); also safe to run STANDALONE against an already-bootstrapped
|
||||
# cluster (that is how prod gets unblocked without re-running the whole bootstrap).
|
||||
#
|
||||
# R-GITOPS-01 (the graph must reconcile from git, not from someone's shell history) · R-ENV-02.
|
||||
# =============================================================================
|
||||
# *** AUTHORED, NOT RUN. Agents must not execute it — it MUTATES a cluster. ***
|
||||
# =============================================================================
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
#
|
||||
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`. That is deliberate:
|
||||
# the Kyverno CRD bundle is ~5.6 MiB, far past kubectl's 256 KiB last-applied-configuration
|
||||
# annotation limit, so Helm's client-side apply fails with "metadata.annotations: Too long" and
|
||||
# STALLS the HelmRelease — which blocks the `platform` Kustomization and every tier that dependsOn
|
||||
# it (the kps-cascade failure mode; see the HelmRelease's own comment).
|
||||
#
|
||||
# The consequence nobody automated: with CRD install off, NOTHING installs those CRDs. Kyverno's
|
||||
# Deployments come up but its API types do not exist, so `platform` never goes Ready on a cold
|
||||
# cluster. The dev cluster only has them because they were once applied BY HAND — exactly the
|
||||
# undocumented manual step this repo exists to eliminate. This script is that step, in git,
|
||||
# version-derived and idempotent.
|
||||
#
|
||||
# HOW THE VERSION IS DERIVED (there is deliberately NO second copy of it)
|
||||
#
|
||||
# Everything comes out of the HelmRelease/HelmRepository pair that Flux itself reconciles:
|
||||
#
|
||||
# chart name spec.chart.spec.chart (HelmRelease)
|
||||
# chart version spec.chart.spec.version (HelmRelease) e.g. "3.9.x"
|
||||
# chart repo url spec.url (HelmRepository)
|
||||
# namespace metadata.namespace (HelmRelease)
|
||||
#
|
||||
# The version is a semver CONSTRAINT, not a fixed tag. `helm template --version` accepts the same
|
||||
# constraint syntax Flux's helm-controller resolves (both use Masterminds/semver), so passing it
|
||||
# through UNCHANGED renders the very chart version Flux will install. Renovate bumps one line in
|
||||
# one file and this step follows automatically — it cannot drift, because there is nothing here to
|
||||
# drift from.
|
||||
#
|
||||
# WHY ONLY THE CRD DOCUMENTS ARE APPLIED
|
||||
#
|
||||
# The by-hand command recorded in the HelmRelease comment piped the WHOLE `helm template` output
|
||||
# into kubectl. Do not do that: it would create Deployments/Services/webhooks outside Helm's
|
||||
# ownership, which then fight the HelmRelease over field managers and leave orphans behind. We
|
||||
# filter to `kind: CustomResourceDefinition` and apply nothing else. Values are the chart DEFAULTS
|
||||
# (plus an explicit crds.install=true) — NOT the repo's values, which switch CRD rendering off.
|
||||
#
|
||||
# IDEMPOTENCY
|
||||
#
|
||||
# `kubectl apply --server-side --force-conflicts` is a declarative upsert: the first run creates,
|
||||
# every later run reports "serverside-applied" and exits 0, and field ownership is taken from
|
||||
# whatever previously wrote those objects (that is what --force-conflicts is for — it resolves the
|
||||
# hand-applied dev CRDs' `kubectl-client-side-apply` manager without a human deciding anything).
|
||||
# Server-side apply does not write the last-applied annotation, which is the whole reason the
|
||||
# 5.6 MiB bundle can be applied at all.
|
||||
#
|
||||
# LOUD FAILURE (this script must never degrade to a successful no-op)
|
||||
#
|
||||
# `set -euo pipefail`, plus three explicit guards, because a silent skip here reproduces the exact
|
||||
# class of bug this repo has been bitten by more than once:
|
||||
# 1. every derived coordinate must be non-empty and not "null" — a moved YAML path fails here;
|
||||
# 2. the rendered CRD count must be > 0 — if a future chart moves/renames its CRD templates the
|
||||
# filter would match nothing and `kubectl apply` on an empty stream EXITS 0. Guarded;
|
||||
# 3. after applying, every CRD is waited on for condition=Established — so "applied" is proven
|
||||
# to mean "the API type exists", not merely "kubectl returned 0".
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
# --- config (env-overridable) ------------------------------------------------
|
||||
# The single source of truth for the chart coordinates. Repo-relative on purpose: this is the file
|
||||
# Flux reconciles, so it is the file this step must agree with.
|
||||
CRD_HELMRELEASE="${CRD_HELMRELEASE:-platform/kyverno/admission/helmrelease.yaml}"
|
||||
# How long to wait for the applied CRDs to become Established (apiextensions publishes the type).
|
||||
CRD_ESTABLISH_TIMEOUT="${CRD_ESTABLISH_TIMEOUT:-120s}"
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
|
||||
|
||||
echo "==> CRD pre-apply — source of truth: ${CRD_HELMRELEASE}"
|
||||
|
||||
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
||||
# helm renders the bundle, yq selects the CRD documents, kubectl applies them server-side. All three
|
||||
# are baked into infra/devcontainer (HELM_VERSION / YQ_VERSION / KUBECTL_VERSION in versions.env).
|
||||
for bin in helm yq kubectl; do
|
||||
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
||||
done
|
||||
|
||||
helmrelease="${repo_root}/${CRD_HELMRELEASE}"
|
||||
[ -f "${helmrelease}" ] || {
|
||||
echo "ERROR: ${helmrelease} not found — CRD_HELMRELEASE points nowhere." >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# --- 1. derive the chart coordinates from the authored manifests --------------
|
||||
chart_name="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.chart' "${helmrelease}")"
|
||||
chart_version="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "${helmrelease}")"
|
||||
chart_ns="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .metadata.namespace' "${helmrelease}")"
|
||||
chart_repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' "${helmrelease}")"
|
||||
|
||||
# Guard 1: a moved/renamed YAML path must STOP the bootstrap, not render a bogus chart. yq prints
|
||||
# "null" for a missing path and an empty string for a missing document, so both are rejected.
|
||||
for pair in "chart:${chart_name}" "version:${chart_version}" "namespace:${chart_ns}" "repo:${chart_repo}"; do
|
||||
key="${pair%%:*}"; val="${pair#*:}"
|
||||
case "${val}" in
|
||||
""|null|*$'\n'*)
|
||||
echo "ERROR: could not derive a single '${key}' from ${CRD_HELMRELEASE} (got: '${val}')." >&2
|
||||
echo " The HelmRelease/HelmRepository shape changed — fix the yq paths in $(basename "${BASH_SOURCE[0]}")." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo " chart=${chart_name} version=${chart_version} repo=${chart_repo} namespace=${chart_ns}"
|
||||
echo " (version is the constraint Flux resolves — NOT a second copy pinned here)"
|
||||
|
||||
# --- 2. render the chart and keep ONLY the CRD documents ----------------------
|
||||
render_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "${render_dir}"' EXIT
|
||||
crd_file="${render_dir}/crds.yaml"
|
||||
|
||||
# --include-crds covers charts that ship a native crds/ dir; crds.install=true covers Kyverno, whose
|
||||
# CRDs are subchart TEMPLATES that the repo's own values switch off. Chart defaults otherwise.
|
||||
echo "==> rendering ${chart_name} ${chart_version} (CRD documents only)"
|
||||
helm template "${chart_name}" "${chart_name}" \
|
||||
--repo "${chart_repo}" \
|
||||
--version "${chart_version}" \
|
||||
--namespace "${chart_ns}" \
|
||||
--include-crds \
|
||||
--set crds.install=true \
|
||||
| yq eval-all 'select(.kind == "CustomResourceDefinition")' - > "${crd_file}"
|
||||
|
||||
# Blank/"null" lines are dropped on purpose: an EMPTY render makes `yq .metadata.name` emit one
|
||||
# empty line, which would otherwise count as a CRD and sail straight past guard 2 below.
|
||||
mapfile -t crd_names < <(yq eval-all --no-doc '.metadata.name' "${crd_file}" | grep -vx -e '' -e 'null' || true)
|
||||
crd_count="${#crd_names[@]}"
|
||||
|
||||
# Guard 2: an empty stream applies cleanly and exits 0 — the silent-no-op trap. Refuse it.
|
||||
[ "${crd_count}" -gt 0 ] || {
|
||||
echo "ERROR: rendered 0 CustomResourceDefinitions from ${chart_name} ${chart_version}." >&2
|
||||
echo " The chart's CRD layout or the crds.install value key changed. Refusing to continue:" >&2
|
||||
echo " applying nothing would look like success and leave 'platform' blocked forever." >&2
|
||||
exit 1
|
||||
}
|
||||
echo " ${crd_count} CRDs, $(wc -c < "${crd_file}") bytes (>256 KiB — this is why it must be server-side)"
|
||||
|
||||
# --- 3. server-side apply (idempotent upsert) ---------------------------------
|
||||
# --server-side: no last-applied annotation, so the size limit does not apply.
|
||||
# --force-conflicts: adopt fields owned by an earlier manager (e.g. dev's hand-applied CRDs, or
|
||||
# Helm's own manager if a past chart version did install them). Makes a re-run a plain no-op.
|
||||
echo "==> kubectl apply --server-side --force-conflicts (${crd_count} CRDs)"
|
||||
kubectl apply --server-side --force-conflicts -f "${crd_file}"
|
||||
|
||||
# --- 4. prove the API types actually exist -----------------------------------
|
||||
# Without this, "kubectl returned 0" is the only evidence. Establishment is what Kyverno's
|
||||
# controllers and the platform Kustomization actually need.
|
||||
echo "==> waiting for condition=Established (timeout ${CRD_ESTABLISH_TIMEOUT})"
|
||||
crd_refs=()
|
||||
for n in "${crd_names[@]}"; do crd_refs+=("crd/${n}"); done
|
||||
kubectl wait --for=condition=Established --timeout="${CRD_ESTABLISH_TIMEOUT}" \
|
||||
"${crd_refs[@]}" >/dev/null || {
|
||||
echo "ERROR: not all ${crd_count} CRDs reached Established within ${CRD_ESTABLISH_TIMEOUT}." >&2
|
||||
echo " Inspect: kubectl get crd | grep ${chart_name}" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "==> ${crd_count} ${chart_name} CRDs applied and Established."
|
||||
@@ -1260,6 +1260,45 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
|
||||
`inventory/group_vars/k3s_prod/vars.yml` overrides it to `clusters/prod` — the same
|
||||
mechanism as `kube_vip_address` / `k3s_data_dir` / `cluster_kubeconfig_name`. Dev has no
|
||||
override anywhere, so dev resolves `clusters/dev` exactly as before.
|
||||
- [x] **The Kyverno CRDs had to be applied BY HAND, and nothing in the repo did it** — AUTHORED
|
||||
2026-09-04 (`bootstrap/flux/preapply-crds.sh`), **not yet run against prod**. This is a hard
|
||||
**prerequisite of a reconcilable cluster**, not a nice-to-have: `platform/kyverno/admission/
|
||||
helmrelease.yaml` sets `crds: {install: false}` — deliberately, because the CRD bundle is
|
||||
**5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured 2026-09-04 with `helm template
|
||||
kyverno kyverno --repo https://kyverno.github.io/kyverno/ --version 3.9.x --include-crds`), far
|
||||
past kubectl's 256 KiB `last-applied-configuration` annotation limit, so Helm's client-side apply
|
||||
fails `metadata.annotations: Too long` and **stalls** the HelmRelease. The consequence nobody
|
||||
had automated: with CRD install off, **nothing installs those CRDs at all**. On a cold cluster
|
||||
Kyverno's Deployments come up with no API types, `platform` never goes Ready, and all 9
|
||||
downstream Kustomizations stall behind it. Dev only ever had them because the command was once
|
||||
run in somebody's shell — the exact undocumented manual step this project exists to delete.
|
||||
- **The fix, in git:** `bootstrap/flux/bootstrap.sh` now runs `./preapply-crds.sh` as step
|
||||
**[2/4]** — after `flux install`, before `kubectl apply -k ${FLUX_ENV_PATH}`. It renders the
|
||||
chart, keeps only `kind: CustomResourceDefinition` documents, applies them
|
||||
`--server-side --force-conflicts` (no annotation, so no size limit), then waits for
|
||||
`condition=Established`. The chart name, version constraint, repo URL and namespace are
|
||||
**derived from that same HelmRelease/HelmRepository** — there is no second copy of the version
|
||||
to drift, and a Renovate bump of the one pinned line carries this step with it.
|
||||
- **No environment fork (R-ENV-02):** the script names no environment and takes no overlay
|
||||
input; it reads repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step, so
|
||||
`roles/flux_bootstrap` needed **no change** — it already calls `bootstrap.sh`, the single
|
||||
entrypoint.
|
||||
- **It cannot silently skip.** Three guards, and deliberately no opt-out flag: a derived
|
||||
coordinate that is empty/`null` aborts; a render that yields **0** CRDs aborts (an empty
|
||||
stream would otherwise `kubectl apply` cleanly and exit 0 — the silent-success failure mode
|
||||
that has bitten this repo repeatedly); and every applied CRD must reach `Established`.
|
||||
- **Toolchain:** needs `helm` and `yq` on PATH in addition to `flux`/`kubectl` — both are baked
|
||||
into `infra/devcontainer` (`HELM_VERSION=3.16.0`, `YQ_VERSION=4.44.6`), and `bootstrap.sh`
|
||||
now fails at step 0 if either is missing. It also needs **outbound network to
|
||||
`kyverno.github.io`** from wherever the play runs (the Ansible role delegates to localhost).
|
||||
- **PM to verify in the container** (the authoring agent ran it only with a stubbed `kubectl`,
|
||||
never against a cluster): pinned `helm` 3.16.0 renders the same 22 CRDs, and the run is a
|
||||
clean no-op the second time.
|
||||
- **Unblocking the already-bootstrapped prod cluster** does not need a full re-bootstrap:
|
||||
```bash
|
||||
# inside infra/devcontainer, KUBECONFIG=.cache/prod.kubeconfig
|
||||
bash bootstrap/flux/preapply-crds.sh
|
||||
```
|
||||
- [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
|
||||
the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
|
||||
|
||||
@@ -1272,7 +1311,11 @@ single-server-etcd window, and avoids a later member-add dance.
|
||||
1. Per-node §1–§5 on each of the three boxes.
|
||||
2. §7 repo work lands on `main` with the gate green.
|
||||
3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`.
|
||||
4. `flux bootstrap ... --path=clusters/prod`.
|
||||
4. `flux bootstrap ... --path=clusters/prod` — i.e. `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod`,
|
||||
which runs `bootstrap/flux/bootstrap.sh` with `FLUX_ENV_PATH=clusters/prod`. That script's step
|
||||
**[2/4]** pre-applies the 22 Kyverno CRDs server-side; **without it `platform` cannot go Ready**
|
||||
and the 9 tiers below it never reconcile (§7). It needs `helm` + `yq` on PATH and outbound network
|
||||
to the chart repo.
|
||||
5. Conformance against prod (§9).
|
||||
|
||||
## 9. Exit criteria (Phase 7)
|
||||
|
||||
Reference in new issue
Block a user