feat(bootstrap): automate the Kyverno CRD pre-apply — the last cold-start blocker
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`
because the CRD bundle is 5,636,155 bytes, far past kubectl's 256 KiB
last-applied-configuration limit — a client-side apply fails with
"metadata.annotations: Too long" and STALLS the HelmRelease.
The consequence nobody automated: with CRD install off, NOTHING installed
them. `platform` has `wait: true` and every other tier chains transitively
through it, so on a cold cluster Kyverno's API types never exist, `platform`
never goes Ready, and all 10 downstream tiers stall. Dev only ever had these
CRDs because they were applied BY HAND — exactly the undocumented manual step
this repo exists to eliminate.
New `bootstrap/flux/preapply-crds.sh`, wired into bootstrap.sh as step [2/4]
(after `flux install`, before the graph is applied) and standalone-safe so an
already-bootstrapped cluster can be unblocked without re-running bootstrap.
NO SECOND PIN: all four chart coordinates are read with yq out of the same
HelmRelease/HelmRepository pair Flux reconciles. The version is passed through
as the CONSTRAINT ("3.9.x"), not a resolved tag — helm and helm-controller both
use Masterminds/semver, so the render is the version Flux installs. Renovate
bumps one line and this step follows; there is nothing here to drift from.
Only `kind: CustomResourceDefinition` documents are applied. The by-hand
command recorded in the HelmRelease comment piped the WHOLE template into
kubectl, which would create Deployments/webhooks outside Helm's ownership and
then fight the HelmRelease over field managers.
THREE GUARDS, because a silent skip here reproduces a failure mode this repo
has been bitten by repeatedly:
1. any derived coordinate that is empty/null/multi-valued aborts;
2. rendered CRD count must be > 0 — this caught a real bug in the first
draft: on an empty render `yq .metadata.name` emits one BLANK line, which
counted as 1 CRD and sailed past the guard into an apply of an empty file
returning 0, i.e. the exact silent-success trap. Blank/null lines are now
dropped and a chart with no CRDs exits 1;
3. every applied CRD is waited on for condition=Established, so "kubectl
returned 0" is not the only evidence.
No opt-out flag, deliberately: a skipped CRD apply yields a cluster that looks
bootstrapped and reconciles nothing.
`roles/flux_bootstrap` is untouched — it already shells to bootstrap.sh, the
documented single entrypoint, so it inherits the step with no dev/prod fork
(R-ENV-02). A second role task would double-apply.
MEASURED, not assumed — reproduced under the PINNED toolchain (helm 3.16.0,
yq 4.44.6) in infra/devcontainer, matching an independent run on helm 3.20.0
byte-for-byte: 3.9.x resolves to 3.9.0, 22 CRDs, 5,636,155 bytes CRD-only
(5,870,019 full render). Largest single CRD document is 1,378,544 bytes —
under etcd's 1.5 MiB default, but not by much; worth watching on chart bumps.
NOT YET EXERCISED against a live API server: the server-side apply and the
Established wait were tested with a stub kubectl. velero and
kube-prometheus-stack also say "CRDs pre-applied" but both still get theirs
from Helm's native crds/ dir on FIRST install, so neither is a cold-start
blocker; CRD_HELMRELEASE makes extending this a one-liner if that changes.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
9137bbd0c3
commit
13081a863f
4 files changed
+294
-13
No files matched your search
@@ -33,10 +33,19 @@
|
||||
# the single apps.yaml was retired). Letting
|
||||
# bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
|
||||
# applying the authored clusters/dev overlay is the reconciling path.
|
||||
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
# 2. CRD pre-apply — ./preapply-crds.sh applies the Kyverno CRD bundle SERVER-SIDE, because
|
||||
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`
|
||||
# (the ~5.6 MiB bundle blows kubectl's 256 KiB last-applied annotation limit,
|
||||
# which STALLS the HelmRelease and blocks the `platform` Kustomization and
|
||||
# every tier below it). With CRD install off, NOTHING else installs them, so
|
||||
# a cold cluster can never reconcile `platform` without this step. The chart
|
||||
# version is DERIVED from that same HelmRelease — no second pin. Dev only
|
||||
# ever had these CRDs because they were applied BY HAND; this is that step,
|
||||
# in git, idempotent and version-derived. See ./preapply-crds.sh.
|
||||
# 3. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
# decryption works on first pass (../sops/age-key-secret.md). This script
|
||||
# only VERIFIES it exists; it never handles key material.
|
||||
# 3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
|
||||
# 4. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
|
||||
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
|
||||
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
|
||||
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
|
||||
@@ -71,7 +80,10 @@ repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd
|
||||
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
|
||||
|
||||
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
||||
for bin in flux kubectl; do
|
||||
# helm + yq are needed by step 2 (./preapply-crds.sh renders the CRD bundle from the authored
|
||||
# HelmRelease). Checked HERE as well as in that script so a missing binary stops the run BEFORE the
|
||||
# Flux controllers are installed, instead of half-way through.
|
||||
for bin in flux kubectl helm yq; do
|
||||
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
||||
done
|
||||
|
||||
@@ -99,7 +111,7 @@ done
|
||||
#
|
||||
# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
|
||||
# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
|
||||
echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
|
||||
echo "==> [1/4] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
|
||||
flux_install_args=(--namespace="${FLUX_NAMESPACE}")
|
||||
[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
|
||||
|
||||
@@ -130,18 +142,29 @@ else
|
||||
kubectl apply -f "${render_dir}/rendered.yaml"
|
||||
fi
|
||||
|
||||
# --- 2. verify the out-of-band age key Secret exists -------------------------
|
||||
# --- 2. pre-apply the CRDs Helm is not allowed to install ---------------------
|
||||
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`, so the 22 Kyverno CRDs
|
||||
# must exist BEFORE the platform Kustomization reconciles that HelmRelease — otherwise Kyverno's
|
||||
# controllers come up with no API types, `platform` never goes Ready, and all of its dependents stall.
|
||||
# ./preapply-crds.sh derives the chart version from the HelmRelease itself (no second pin), applies
|
||||
# only the CRD documents SERVER-SIDE (no 256 KiB annotation limit), and is idempotent. It FAILS HARD
|
||||
# rather than skipping — there is deliberately NO opt-out flag: a skipped CRD apply produces a
|
||||
# cluster that looks bootstrapped and reconciles nothing.
|
||||
echo "==> [2/4] pre-applying chart CRDs (server-side)"
|
||||
bash "${script_dir}/preapply-crds.sh"
|
||||
|
||||
# --- 3. verify the out-of-band age key Secret exists -------------------------
|
||||
# We DO NOT create it here (it holds the private key). The operator installs it per
|
||||
# ../sops/age-key-secret.md. Fail fast if missing so decryption can't silently no-op.
|
||||
echo "==> [2/3] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
|
||||
echo "==> [3/4] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
|
||||
if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null 2>&1; then
|
||||
echo "ERROR: Secret ${FLUX_NAMESPACE}/${SOPS_SECRET_NAME} not found." >&2
|
||||
echo " Install the age private key first: bootstrap/sops/age-key-secret.md" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- 3. apply the authored environment entrypoint ----------------------------
|
||||
echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
|
||||
# --- 4. apply the authored environment entrypoint ----------------------------
|
||||
echo "==> [4/4] kubectl apply -k ${FLUX_ENV_PATH}"
|
||||
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
|
||||
|
||||
echo
|
||||
|
||||
Reference in new issue
Block a user