jakman85andClaude Opus 5
13081a863f
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`
because the CRD bundle is 5,636,155 bytes, far past kubectl's 256 KiB
last-applied-configuration limit — a client-side apply fails with
"metadata.annotations: Too long" and STALLS the HelmRelease.
The consequence nobody automated: with CRD install off, NOTHING installed
them. `platform` has `wait: true` and every other tier chains transitively
through it, so on a cold cluster Kyverno's API types never exist, `platform`
never goes Ready, and all 10 downstream tiers stall. Dev only ever had these
CRDs because they were applied BY HAND — exactly the undocumented manual step
this repo exists to eliminate.
New `bootstrap/flux/preapply-crds.sh`, wired into bootstrap.sh as step [2/4]
(after `flux install`, before the graph is applied) and standalone-safe so an
already-bootstrapped cluster can be unblocked without re-running bootstrap.
NO SECOND PIN: all four chart coordinates are read with yq out of the same
HelmRelease/HelmRepository pair Flux reconciles. The version is passed through
as the CONSTRAINT ("3.9.x"), not a resolved tag — helm and helm-controller both
use Masterminds/semver, so the render is the version Flux installs. Renovate
bumps one line and this step follows; there is nothing here to drift from.
Only `kind: CustomResourceDefinition` documents are applied. The by-hand
command recorded in the HelmRelease comment piped the WHOLE template into
kubectl, which would create Deployments/webhooks outside Helm's ownership and
then fight the HelmRelease over field managers.
THREE GUARDS, because a silent skip here reproduces a failure mode this repo
has been bitten by repeatedly:
1. any derived coordinate that is empty/null/multi-valued aborts;
2. rendered CRD count must be > 0 — this caught a real bug in the first
draft: on an empty render `yq .metadata.name` emits one BLANK line, which
counted as 1 CRD and sailed past the guard into an apply of an empty file
returning 0, i.e. the exact silent-success trap. Blank/null lines are now
dropped and a chart with no CRDs exits 1;
3. every applied CRD is waited on for condition=Established, so "kubectl
returned 0" is not the only evidence.
No opt-out flag, deliberately: a skipped CRD apply yields a cluster that looks
bootstrapped and reconciles nothing.
`roles/flux_bootstrap` is untouched — it already shells to bootstrap.sh, the
documented single entrypoint, so it inherits the step with no dev/prod fork
(R-ENV-02). A second role task would double-apply.
MEASURED, not assumed — reproduced under the PINNED toolchain (helm 3.16.0,
yq 4.44.6) in infra/devcontainer, matching an independent run on helm 3.20.0
byte-for-byte: 3.9.x resolves to 3.9.0, 22 CRDs, 5,636,155 bytes CRD-only
(5,870,019 full render). Largest single CRD document is 1,378,544 bytes —
under etcd's 1.5 MiB default, but not by much; worth watching on chart bumps.
NOT YET EXERCISED against a live API server: the server-side apply and the
Established wait were tested with a stub kubectl. velero and
kube-prometheus-stack also say "CRDs pre-applied" but both still get theirs
from Helm's native crds/ dir on FIRST install, so neither is a cold-start
blocker; CRD_HELMRELEASE makes extending this a one-liner if that changes.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
175 lines
12 KiB
Bash
Executable File
175 lines
12 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
|
|
# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
|
|
# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
|
|
# same script bootstraps dev and prod with no fork (R-ENV-02).
|
|
#
|
|
# Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
|
|
# =============================================================================
|
|
# *** AUTHORED, NOT RUN. The Infra worker's FINAL Ansible play invokes this ***
|
|
# *** (interface contract, see ../README.md). Agents must not execute it. ***
|
|
# =============================================================================
|
|
#
|
|
# WHAT THIS DOES (Phase 1, bootstrap-source method — reconciles from the LOCAL repo, no remote
|
|
# git host exists yet per spec docs/05 + CONTRIBUTING.md):
|
|
#
|
|
# 1. `flux install` — deploy the Flux controllers. Besides the default set (source, kustomize,
|
|
# helm, notification) we ALSO install the two IMAGE controllers via
|
|
# `--components-extra` (see FLUX_COMPONENTS_EXTRA below): image-reflector-
|
|
# controller + image-automation-controller. These register the
|
|
# image.toolkit.fluxcd.io CRDs (ImageRepository/ImagePolicy/ImageUpdateAutomation)
|
|
# that Phase-4 hands-off updates need (ADR-0009, R-UPD-02). They are part of the
|
|
# flux-system install exactly like the other 4 controllers — NOT a reconciled
|
|
# HelmRelease — so their CRDs pre-exist before the first reconcile (this is why
|
|
# platform/image-automation can carry the ImageUpdateAutomation CR without the
|
|
# F-MON1 controllers/configs split: the CRD is present at bootstrap, like
|
|
# GitRepository/Kustomization are). WS-UPDATE (T-4.5).
|
|
# We do NOT use `flux bootstrap git` in Phase 1 because that command commits
|
|
# and OWNS its own gotk-sync GitRepository+Kustomization named `flux-system`.
|
|
# Our entrypoints are ALREADY authored in clusters/base (flux-system.yaml =
|
|
# the GitRepository; platform.yaml + platform-config.yaml + the app-wave
|
|
# Kustomizations apps-wave-0/1/2.yaml + apps-vms.yaml = the Kustomizations;
|
|
# the single apps.yaml was retired). Letting
|
|
# bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
|
|
# applying the authored clusters/dev overlay is the reconciling path.
|
|
# 2. CRD pre-apply — ./preapply-crds.sh applies the Kyverno CRD bundle SERVER-SIDE, because
|
|
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`
|
|
# (the ~5.6 MiB bundle blows kubectl's 256 KiB last-applied annotation limit,
|
|
# which STALLS the HelmRelease and blocks the `platform` Kustomization and
|
|
# every tier below it). With CRD install off, NOTHING else installs them, so
|
|
# a cold cluster can never reconcile `platform` without this step. The chart
|
|
# version is DERIVED from that same HelmRelease — no second pin. Dev only
|
|
# ever had these CRDs because they were applied BY HAND; this is that step,
|
|
# in git, idempotent and version-derived. See ./preapply-crds.sh.
|
|
# 3. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
|
# decryption works on first pass (../sops/age-key-secret.md). This script
|
|
# only VERIFIES it exists; it never handles key material.
|
|
# 4. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
|
|
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
|
|
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
|
|
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
|
|
# (R-ENV-02) and per-Kustomization SOPS decryption enabled via ./patches/.
|
|
#
|
|
# PREREQUISITES the operator/PM resolves before this runs (see ../README.md "Handoff"):
|
|
# - clusters/base/flux-system.yaml url placeholder is overridden — wire ./patches/flux-system-
|
|
# source-url.yaml into clusters/dev/kustomization.yaml (or set the URL) so the source is real.
|
|
# - ./patches/platform-sops-decryption.yaml + apps-sops-decryption.yaml are wired into
|
|
# clusters/dev/kustomization.yaml `patches:` so decryption is enabled at reconcile.
|
|
# - the repo-root .sops.yaml carries the real age recipient (single canonical file; no copy under ../sops).
|
|
#
|
|
# PHASE 6 (WS-GITOPS, later — NOT Phase 1): once Gitea is up, cut the source over with
|
|
# `flux bootstrap git` against gitea.home.keithjackson.dev — see ../README.md "Phase-6 cutover".
|
|
#
|
|
set -euo pipefail
|
|
|
|
# --- config (env-overridable) ------------------------------------------------
|
|
# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
|
|
# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
|
|
# every existing caller — override it, never edit it.
|
|
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
|
|
FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
|
|
SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
|
|
# The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
|
|
# exist before reconcile (ADR-0009, R-UPD-02). Override to "" to build a cluster without them.
|
|
FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image-automation-controller}"
|
|
|
|
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
|
|
|
|
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
|
|
|
|
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
|
# helm + yq are needed by step 2 (./preapply-crds.sh renders the CRD bundle from the authored
|
|
# HelmRelease). Checked HERE as well as in that script so a missing binary stops the run BEFORE the
|
|
# Flux controllers are installed, instead of half-way through.
|
|
for bin in flux kubectl helm yq; do
|
|
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
|
done
|
|
|
|
# --- 1. install Flux controllers (rendered through ./components) --------------
|
|
# Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
|
|
# ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
|
|
# image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
|
|
#
|
|
# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
|
|
# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
|
|
# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
|
|
# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
|
|
#
|
|
# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
|
|
# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
|
|
# documents for hand-installed clusters ("Customize Flux manifests"):
|
|
#
|
|
# flux install --export > gotk-components.yaml # exact manifests `flux install` would apply
|
|
# kustomize overlay = ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
|
|
# kubectl apply -k
|
|
#
|
|
# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
|
|
# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
|
|
# a temp dir — the repo tree is never written to.
|
|
#
|
|
# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
|
|
# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
|
|
echo "==> [1/4] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
|
|
flux_install_args=(--namespace="${FLUX_NAMESPACE}")
|
|
[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
|
|
|
|
if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
|
|
echo " WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
|
|
echo " StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
|
|
flux install "${flux_install_args[@]}"
|
|
else
|
|
components_dir="${script_dir}/components"
|
|
[ -f "${components_dir}/kustomization.yaml" ] || {
|
|
echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
|
|
echo " (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
|
|
exit 1
|
|
}
|
|
render_dir="$(mktemp -d)"
|
|
trap 'rm -rf "${render_dir}"' EXIT
|
|
echo " rendering flux install --export through ${components_dir}"
|
|
flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
|
|
cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
|
|
# Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
|
|
# version bump) — better a hard stop than silently installing without the feature gate.
|
|
kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
|
|
grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
|
|
echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
|
|
echo " Check ./components/kustomization.yaml against this flux CLI version." >&2
|
|
exit 1
|
|
}
|
|
kubectl apply -f "${render_dir}/rendered.yaml"
|
|
fi
|
|
|
|
# --- 2. pre-apply the CRDs Helm is not allowed to install ---------------------
|
|
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`, so the 22 Kyverno CRDs
|
|
# must exist BEFORE the platform Kustomization reconciles that HelmRelease — otherwise Kyverno's
|
|
# controllers come up with no API types, `platform` never goes Ready, and all of its dependents stall.
|
|
# ./preapply-crds.sh derives the chart version from the HelmRelease itself (no second pin), applies
|
|
# only the CRD documents SERVER-SIDE (no 256 KiB annotation limit), and is idempotent. It FAILS HARD
|
|
# rather than skipping — there is deliberately NO opt-out flag: a skipped CRD apply produces a
|
|
# cluster that looks bootstrapped and reconciles nothing.
|
|
echo "==> [2/4] pre-applying chart CRDs (server-side)"
|
|
bash "${script_dir}/preapply-crds.sh"
|
|
|
|
# --- 3. verify the out-of-band age key Secret exists -------------------------
|
|
# We DO NOT create it here (it holds the private key). The operator installs it per
|
|
# ../sops/age-key-secret.md. Fail fast if missing so decryption can't silently no-op.
|
|
echo "==> [3/4] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
|
|
if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null 2>&1; then
|
|
echo "ERROR: Secret ${FLUX_NAMESPACE}/${SOPS_SECRET_NAME} not found." >&2
|
|
echo " Install the age private key first: bootstrap/sops/age-key-secret.md" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# --- 4. apply the authored environment entrypoint ----------------------------
|
|
echo "==> [4/4] kubectl apply -k ${FLUX_ENV_PATH}"
|
|
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
|
|
|
|
echo
|
|
echo "==> Bootstrap applied. Watch reconciliation with (read-only):"
|
|
echo " flux get kustomizations --watch"
|
|
echo " flux get sources git -n ${FLUX_NAMESPACE}"
|
|
echo " 'platform' should go Ready, then 'apps' (dependsOn platform)."
|