jakman85andClaude Opus 5
13081a863f
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`
because the CRD bundle is 5,636,155 bytes, far past kubectl's 256 KiB
last-applied-configuration limit — a client-side apply fails with
"metadata.annotations: Too long" and STALLS the HelmRelease.
The consequence nobody automated: with CRD install off, NOTHING installed
them. `platform` has `wait: true` and every other tier chains transitively
through it, so on a cold cluster Kyverno's API types never exist, `platform`
never goes Ready, and all 10 downstream tiers stall. Dev only ever had these
CRDs because they were applied BY HAND — exactly the undocumented manual step
this repo exists to eliminate.
New `bootstrap/flux/preapply-crds.sh`, wired into bootstrap.sh as step [2/4]
(after `flux install`, before the graph is applied) and standalone-safe so an
already-bootstrapped cluster can be unblocked without re-running bootstrap.
NO SECOND PIN: all four chart coordinates are read with yq out of the same
HelmRelease/HelmRepository pair Flux reconciles. The version is passed through
as the CONSTRAINT ("3.9.x"), not a resolved tag — helm and helm-controller both
use Masterminds/semver, so the render is the version Flux installs. Renovate
bumps one line and this step follows; there is nothing here to drift from.
Only `kind: CustomResourceDefinition` documents are applied. The by-hand
command recorded in the HelmRelease comment piped the WHOLE template into
kubectl, which would create Deployments/webhooks outside Helm's ownership and
then fight the HelmRelease over field managers.
THREE GUARDS, because a silent skip here reproduces a failure mode this repo
has been bitten by repeatedly:
1. any derived coordinate that is empty/null/multi-valued aborts;
2. rendered CRD count must be > 0 — this caught a real bug in the first
draft: on an empty render `yq .metadata.name` emits one BLANK line, which
counted as 1 CRD and sailed past the guard into an apply of an empty file
returning 0, i.e. the exact silent-success trap. Blank/null lines are now
dropped and a chart with no CRDs exits 1;
3. every applied CRD is waited on for condition=Established, so "kubectl
returned 0" is not the only evidence.
No opt-out flag, deliberately: a skipped CRD apply yields a cluster that looks
bootstrapped and reconciles nothing.
`roles/flux_bootstrap` is untouched — it already shells to bootstrap.sh, the
documented single entrypoint, so it inherits the step with no dev/prod fork
(R-ENV-02). A second role task would double-apply.
MEASURED, not assumed — reproduced under the PINNED toolchain (helm 3.16.0,
yq 4.44.6) in infra/devcontainer, matching an independent run on helm 3.20.0
byte-for-byte: 3.9.x resolves to 3.9.0, 22 CRDs, 5,636,155 bytes CRD-only
(5,870,019 full render). Largest single CRD document is 1,378,544 bytes —
under etcd's 1.5 MiB default, but not by much; worth watching on chart bumps.
NOT YET EXERCISED against a live API server: the server-side apply and the
Established wait were tested with a stub kubectl. velero and
kube-prometheus-stack also say "CRDs pre-applied" but both still get theirs
from Helm's native crds/ dir on FIRST install, so neither is a cold-start
blocker; CRD_HELMRELEASE makes extending this a one-liner if that changes.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
170 lines
9.7 KiB
Bash
Executable File
170 lines
9.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# preapply-crds.sh — server-side pre-apply of the CRDs that Helm is NOT allowed to install.
|
|
# ENVIRONMENT-AGNOSTIC: it reads the chart coordinates out of the authored HelmRelease and talks to
|
|
# whatever cluster $KUBECONFIG points at, so dev and prod run the SAME step with no fork (R-ENV-02).
|
|
# Called by ./bootstrap.sh (step 2/4); also safe to run STANDALONE against an already-bootstrapped
|
|
# cluster (that is how prod gets unblocked without re-running the whole bootstrap).
|
|
#
|
|
# R-GITOPS-01 (the graph must reconcile from git, not from someone's shell history) · R-ENV-02.
|
|
# =============================================================================
|
|
# *** AUTHORED, NOT RUN. Agents must not execute it — it MUTATES a cluster. ***
|
|
# =============================================================================
|
|
#
|
|
# WHY THIS EXISTS
|
|
#
|
|
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`. That is deliberate:
|
|
# the Kyverno CRD bundle is ~5.6 MiB, far past kubectl's 256 KiB last-applied-configuration
|
|
# annotation limit, so Helm's client-side apply fails with "metadata.annotations: Too long" and
|
|
# STALLS the HelmRelease — which blocks the `platform` Kustomization and every tier that dependsOn
|
|
# it (the kps-cascade failure mode; see the HelmRelease's own comment).
|
|
#
|
|
# The consequence nobody automated: with CRD install off, NOTHING installs those CRDs. Kyverno's
|
|
# Deployments come up but its API types do not exist, so `platform` never goes Ready on a cold
|
|
# cluster. The dev cluster only has them because they were once applied BY HAND — exactly the
|
|
# undocumented manual step this repo exists to eliminate. This script is that step, in git,
|
|
# version-derived and idempotent.
|
|
#
|
|
# HOW THE VERSION IS DERIVED (there is deliberately NO second copy of it)
|
|
#
|
|
# Everything comes out of the HelmRelease/HelmRepository pair that Flux itself reconciles:
|
|
#
|
|
# chart name spec.chart.spec.chart (HelmRelease)
|
|
# chart version spec.chart.spec.version (HelmRelease) e.g. "3.9.x"
|
|
# chart repo url spec.url (HelmRepository)
|
|
# namespace metadata.namespace (HelmRelease)
|
|
#
|
|
# The version is a semver CONSTRAINT, not a fixed tag. `helm template --version` accepts the same
|
|
# constraint syntax Flux's helm-controller resolves (both use Masterminds/semver), so passing it
|
|
# through UNCHANGED renders the very chart version Flux will install. Renovate bumps one line in
|
|
# one file and this step follows automatically — it cannot drift, because there is nothing here to
|
|
# drift from.
|
|
#
|
|
# WHY ONLY THE CRD DOCUMENTS ARE APPLIED
|
|
#
|
|
# The by-hand command recorded in the HelmRelease comment piped the WHOLE `helm template` output
|
|
# into kubectl. Do not do that: it would create Deployments/Services/webhooks outside Helm's
|
|
# ownership, which then fight the HelmRelease over field managers and leave orphans behind. We
|
|
# filter to `kind: CustomResourceDefinition` and apply nothing else. Values are the chart DEFAULTS
|
|
# (plus an explicit crds.install=true) — NOT the repo's values, which switch CRD rendering off.
|
|
#
|
|
# IDEMPOTENCY
|
|
#
|
|
# `kubectl apply --server-side --force-conflicts` is a declarative upsert: the first run creates,
|
|
# every later run reports "serverside-applied" and exits 0, and field ownership is taken from
|
|
# whatever previously wrote those objects (that is what --force-conflicts is for — it resolves the
|
|
# hand-applied dev CRDs' `kubectl-client-side-apply` manager without a human deciding anything).
|
|
# Server-side apply does not write the last-applied annotation, which is the whole reason the
|
|
# 5.6 MiB bundle can be applied at all.
|
|
#
|
|
# LOUD FAILURE (this script must never degrade to a successful no-op)
|
|
#
|
|
# `set -euo pipefail`, plus three explicit guards, because a silent skip here reproduces the exact
|
|
# class of bug this repo has been bitten by more than once:
|
|
# 1. every derived coordinate must be non-empty and not "null" — a moved YAML path fails here;
|
|
# 2. the rendered CRD count must be > 0 — if a future chart moves/renames its CRD templates the
|
|
# filter would match nothing and `kubectl apply` on an empty stream EXITS 0. Guarded;
|
|
# 3. after applying, every CRD is waited on for condition=Established — so "applied" is proven
|
|
# to mean "the API type exists", not merely "kubectl returned 0".
|
|
#
|
|
set -euo pipefail
|
|
|
|
# --- config (env-overridable) ------------------------------------------------
|
|
# The single source of truth for the chart coordinates. Repo-relative on purpose: this is the file
|
|
# Flux reconciles, so it is the file this step must agree with.
|
|
CRD_HELMRELEASE="${CRD_HELMRELEASE:-platform/kyverno/admission/helmrelease.yaml}"
|
|
# How long to wait for the applied CRDs to become Established (apiextensions publishes the type).
|
|
CRD_ESTABLISH_TIMEOUT="${CRD_ESTABLISH_TIMEOUT:-120s}"
|
|
|
|
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
|
|
|
|
echo "==> CRD pre-apply — source of truth: ${CRD_HELMRELEASE}"
|
|
|
|
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
|
# helm renders the bundle, yq selects the CRD documents, kubectl applies them server-side. All three
|
|
# are baked into infra/devcontainer (HELM_VERSION / YQ_VERSION / KUBECTL_VERSION in versions.env).
|
|
for bin in helm yq kubectl; do
|
|
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
|
done
|
|
|
|
helmrelease="${repo_root}/${CRD_HELMRELEASE}"
|
|
[ -f "${helmrelease}" ] || {
|
|
echo "ERROR: ${helmrelease} not found — CRD_HELMRELEASE points nowhere." >&2
|
|
exit 1
|
|
}
|
|
|
|
# --- 1. derive the chart coordinates from the authored manifests --------------
|
|
chart_name="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.chart' "${helmrelease}")"
|
|
chart_version="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "${helmrelease}")"
|
|
chart_ns="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .metadata.namespace' "${helmrelease}")"
|
|
chart_repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' "${helmrelease}")"
|
|
|
|
# Guard 1: a moved/renamed YAML path must STOP the bootstrap, not render a bogus chart. yq prints
|
|
# "null" for a missing path and an empty string for a missing document, so both are rejected.
|
|
for pair in "chart:${chart_name}" "version:${chart_version}" "namespace:${chart_ns}" "repo:${chart_repo}"; do
|
|
key="${pair%%:*}"; val="${pair#*:}"
|
|
case "${val}" in
|
|
""|null|*$'\n'*)
|
|
echo "ERROR: could not derive a single '${key}' from ${CRD_HELMRELEASE} (got: '${val}')." >&2
|
|
echo " The HelmRelease/HelmRepository shape changed — fix the yq paths in $(basename "${BASH_SOURCE[0]}")." >&2
|
|
exit 1
|
|
;;
|
|
esac
|
|
done
|
|
|
|
echo " chart=${chart_name} version=${chart_version} repo=${chart_repo} namespace=${chart_ns}"
|
|
echo " (version is the constraint Flux resolves — NOT a second copy pinned here)"
|
|
|
|
# --- 2. render the chart and keep ONLY the CRD documents ----------------------
|
|
render_dir="$(mktemp -d)"
|
|
trap 'rm -rf "${render_dir}"' EXIT
|
|
crd_file="${render_dir}/crds.yaml"
|
|
|
|
# --include-crds covers charts that ship a native crds/ dir; crds.install=true covers Kyverno, whose
|
|
# CRDs are subchart TEMPLATES that the repo's own values switch off. Chart defaults otherwise.
|
|
echo "==> rendering ${chart_name} ${chart_version} (CRD documents only)"
|
|
helm template "${chart_name}" "${chart_name}" \
|
|
--repo "${chart_repo}" \
|
|
--version "${chart_version}" \
|
|
--namespace "${chart_ns}" \
|
|
--include-crds \
|
|
--set crds.install=true \
|
|
| yq eval-all 'select(.kind == "CustomResourceDefinition")' - > "${crd_file}"
|
|
|
|
# Blank/"null" lines are dropped on purpose: an EMPTY render makes `yq .metadata.name` emit one
|
|
# empty line, which would otherwise count as a CRD and sail straight past guard 2 below.
|
|
mapfile -t crd_names < <(yq eval-all --no-doc '.metadata.name' "${crd_file}" | grep -vx -e '' -e 'null' || true)
|
|
crd_count="${#crd_names[@]}"
|
|
|
|
# Guard 2: an empty stream applies cleanly and exits 0 — the silent-no-op trap. Refuse it.
|
|
[ "${crd_count}" -gt 0 ] || {
|
|
echo "ERROR: rendered 0 CustomResourceDefinitions from ${chart_name} ${chart_version}." >&2
|
|
echo " The chart's CRD layout or the crds.install value key changed. Refusing to continue:" >&2
|
|
echo " applying nothing would look like success and leave 'platform' blocked forever." >&2
|
|
exit 1
|
|
}
|
|
echo " ${crd_count} CRDs, $(wc -c < "${crd_file}") bytes (>256 KiB — this is why it must be server-side)"
|
|
|
|
# --- 3. server-side apply (idempotent upsert) ---------------------------------
|
|
# --server-side: no last-applied annotation, so the size limit does not apply.
|
|
# --force-conflicts: adopt fields owned by an earlier manager (e.g. dev's hand-applied CRDs, or
|
|
# Helm's own manager if a past chart version did install them). Makes a re-run a plain no-op.
|
|
echo "==> kubectl apply --server-side --force-conflicts (${crd_count} CRDs)"
|
|
kubectl apply --server-side --force-conflicts -f "${crd_file}"
|
|
|
|
# --- 4. prove the API types actually exist -----------------------------------
|
|
# Without this, "kubectl returned 0" is the only evidence. Establishment is what Kyverno's
|
|
# controllers and the platform Kustomization actually need.
|
|
echo "==> waiting for condition=Established (timeout ${CRD_ESTABLISH_TIMEOUT})"
|
|
crd_refs=()
|
|
for n in "${crd_names[@]}"; do crd_refs+=("crd/${n}"); done
|
|
kubectl wait --for=condition=Established --timeout="${CRD_ESTABLISH_TIMEOUT}" \
|
|
"${crd_refs[@]}" >/dev/null || {
|
|
echo "ERROR: not all ${crd_count} CRDs reached Established within ${CRD_ESTABLISH_TIMEOUT}." >&2
|
|
echo " Inspect: kubectl get crd | grep ${chart_name}" >&2
|
|
exit 1
|
|
}
|
|
|
|
echo "==> ${crd_count} ${chart_name} CRDs applied and Established."
|