jakman85andClaude Opus 5
13081a863f
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`
because the CRD bundle is 5,636,155 bytes, far past kubectl's 256 KiB
last-applied-configuration limit — a client-side apply fails with
"metadata.annotations: Too long" and STALLS the HelmRelease.
The consequence nobody automated: with CRD install off, NOTHING installed
them. `platform` has `wait: true` and every other tier chains transitively
through it, so on a cold cluster Kyverno's API types never exist, `platform`
never goes Ready, and all 10 downstream tiers stall. Dev only ever had these
CRDs because they were applied BY HAND — exactly the undocumented manual step
this repo exists to eliminate.
New `bootstrap/flux/preapply-crds.sh`, wired into bootstrap.sh as step [2/4]
(after `flux install`, before the graph is applied) and standalone-safe so an
already-bootstrapped cluster can be unblocked without re-running bootstrap.
NO SECOND PIN: all four chart coordinates are read with yq out of the same
HelmRelease/HelmRepository pair Flux reconciles. The version is passed through
as the CONSTRAINT ("3.9.x"), not a resolved tag — helm and helm-controller both
use Masterminds/semver, so the render is the version Flux installs. Renovate
bumps one line and this step follows; there is nothing here to drift from.
Only `kind: CustomResourceDefinition` documents are applied. The by-hand
command recorded in the HelmRelease comment piped the WHOLE template into
kubectl, which would create Deployments/webhooks outside Helm's ownership and
then fight the HelmRelease over field managers.
THREE GUARDS, because a silent skip here reproduces a failure mode this repo
has been bitten by repeatedly:
1. any derived coordinate that is empty/null/multi-valued aborts;
2. rendered CRD count must be > 0 — this caught a real bug in the first
draft: on an empty render `yq .metadata.name` emits one BLANK line, which
counted as 1 CRD and sailed past the guard into an apply of an empty file
returning 0, i.e. the exact silent-success trap. Blank/null lines are now
dropped and a chart with no CRDs exits 1;
3. every applied CRD is waited on for condition=Established, so "kubectl
returned 0" is not the only evidence.
No opt-out flag, deliberately: a skipped CRD apply yields a cluster that looks
bootstrapped and reconciles nothing.
`roles/flux_bootstrap` is untouched — it already shells to bootstrap.sh, the
documented single entrypoint, so it inherits the step with no dev/prod fork
(R-ENV-02). A second role task would double-apply.
MEASURED, not assumed — reproduced under the PINNED toolchain (helm 3.16.0,
yq 4.44.6) in infra/devcontainer, matching an independent run on helm 3.20.0
byte-for-byte: 3.9.x resolves to 3.9.0, 22 CRDs, 5,636,155 bytes CRD-only
(5,870,019 full render). Largest single CRD document is 1,378,544 bytes —
under etcd's 1.5 MiB default, but not by much; worth watching on chart bumps.
NOT YET EXERCISED against a live API server: the server-side apply and the
Established wait were tested with a stub kubectl. velero and
kube-prometheus-stack also say "CRDs pre-applied" but both still get theirs
from Helm's native crds/ dir on FIRST install, so neither is a cold-start
blocker; CRD_HELMRELEASE makes extending this a one-liner if that changes.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
201 lines
12 KiB
Markdown
201 lines
12 KiB
Markdown
# bootstrap/flux — Flux bootstrap → `clusters/dev` (Phase 1, T-1.4)
|
|
|
|
Points Flux at the authored **`clusters/dev`** entrypoint and turns on SOPS decryption.
|
|
Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux.md),
|
|
[docs/05](../../../home-network-k3s/docs/05-gitops-and-delivery.md) · Requirements: **R-GITOPS-01**
|
|
(pull-based reconcile), **R-GITOPS-03** (drift correction), **R-SEC-01** (SOPS decryption).
|
|
|
|
> **AUTHOR-ONLY. Nothing in this directory has been executed.** No `flux`, `kubectl`, key-gen, or
|
|
> cluster contact was run. These are files + runbooks the PM/operator validates and runs.
|
|
|
|
## Files
|
|
| File | Purpose |
|
|
|---|---|
|
|
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → **pre-apply chart CRDs (`preapply-crds.sh`)** → verify `sops-age` Secret → `kubectl apply -k ${FLUX_ENV_PATH}`. |
|
|
| `preapply-crds.sh` | **Server-side pre-apply of the Kyverno CRD bundle** — the CRDs `platform/kyverno/admission/helmrelease.yaml` deliberately tells Helm *not* to install (`crds: {install: false}`, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step `platform` never goes Ready on a cold cluster. Chart name/version/repo are **derived from that HelmRelease** — no second pin. Idempotent; runnable standalone. |
|
|
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
|
|
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
|
|
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
|
|
| `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
|
|
|
|
## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
|
|
|
|
Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
|
|
and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
|
|
|
|
```
|
|
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
|
|
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
|
|
"app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
|
|
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
|
|
```
|
|
|
|
No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
|
|
`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
|
|
documents for hand-installed clusters ([Customize Flux
|
|
manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
|
|
|
|
```
|
|
flux install --export … -> <tmp>/gotk-components.yaml # regenerated from the pinned flux CLI
|
|
components/kustomization.yaml -> <tmp>/kustomization.yaml # the only hand-authored part (in git)
|
|
kubectl kustomize <tmp> | kubectl apply -f -
|
|
```
|
|
|
|
Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
|
|
(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
|
|
more feature gates) is another entry in `components/kustomization.yaml`.
|
|
|
|
### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
|
|
|
|
> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
|
|
> controller Deployment is outside Flux's inventory. Someone must run step 1.
|
|
|
|
Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
|
|
|
|
```bash
|
|
# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
|
|
d=$(mktemp -d)
|
|
flux install --export --namespace=flux-system \
|
|
--components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
|
|
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
|
|
kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict
|
|
kubectl -n flux-system rollout status deploy/kustomize-controller
|
|
```
|
|
|
|
**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
|
|
reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
|
|
is lost. Confirm afterwards:
|
|
|
|
```bash
|
|
kubectl -n flux-system get deploy kustomize-controller \
|
|
-o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true
|
|
flux get kustomizations -A # all 11 must return to Ready
|
|
```
|
|
|
|
**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
|
|
simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
|
|
|
|
## The CRD pre-apply step (`preapply-crds.sh`) — why bootstrap has to do this
|
|
|
|
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`. That is correct and must
|
|
stay: the Kyverno CRD bundle is **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured with
|
|
`helm template … --include-crds`, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB
|
|
`last-applied-configuration` annotation limit — `metadata.annotations: Too long` — which stalls the
|
|
HelmRelease and blocks `platform` plus every tier that `dependsOn` it.
|
|
|
|
The part that was never automated: **with CRD install off, nothing installs the CRDs.** Kyverno's
|
|
Deployments start, its API types do not exist, `platform` never reaches Ready. Dev has them only
|
|
because somebody once ran the command in a shell. `preapply-crds.sh` is that command, in git:
|
|
|
|
```
|
|
helm template <chart> --repo <url> --version <constraint> --include-crds --set crds.install=true
|
|
| yq 'select(.kind == "CustomResourceDefinition")'
|
|
| kubectl apply --server-side --force-conflicts
|
|
```
|
|
|
|
- **Version derivation, not a second pin.** `chart`, `version`, `namespace` and the repo `url` are
|
|
read out of the HelmRelease/HelmRepository pair with `yq`. `spec.chart.spec.version` is a semver
|
|
**constraint** (`3.9.x`), and `helm template --version` accepts the same constraint syntax
|
|
helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump
|
|
of that one line carries this step with it. There is nothing here that can drift.
|
|
- **CRDs only.** Piping the *whole* `helm template` output into kubectl (the by-hand command in the
|
|
HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the
|
|
HelmRelease over field managers. The `yq` filter is load-bearing.
|
|
- **Idempotent.** Server-side apply is an upsert; `--force-conflicts` adopts the fields dev's
|
|
hand-applied CRDs left owned by `kubectl-client-side-apply`. Re-running prints
|
|
`serverside-applied` and exits 0.
|
|
- **Cannot degrade to a no-op.** Three guards: every derived coordinate must be non-empty and not
|
|
`null`; the rendered CRD count must be `> 0` (an empty stream would otherwise `apply` cleanly and
|
|
exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is
|
|
waited on for `condition=Established`. There is **no skip flag** on purpose.
|
|
- **Environment-agnostic (R-ENV-02).** It names no environment and takes no overlay input — it reads
|
|
repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step.
|
|
|
|
Standalone use (an already-bootstrapped cluster whose `platform` is blocked on the missing CRDs):
|
|
|
|
```bash
|
|
# inside infra/devcontainer, KUBECONFIG pointed at the target cluster
|
|
bash bootstrap/flux/preapply-crds.sh
|
|
```
|
|
|
|
## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
|
|
|
|
The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
|
|
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`, `platform-config.yaml`
|
|
and the app-wave Kustomizations (`apps-wave-0/1/2.yaml` + `apps-vms.yaml` — the single `apps.yaml`
|
|
was retired) are the Flux **Kustomizations**; `clusters/dev` is the overlay that patches
|
|
`cluster-vars` (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.
|
|
|
|
`flux bootstrap git` **generates and owns its own** `gotk-sync` GitRepository + Kustomization named
|
|
`flux-system` and commits them into the repo — which would **duplicate/conflict** with the authored
|
|
base. It also requires a **writable remote git host**, which does not exist in Phase 1 (spec docs/05
|
|
"Source-of-truth", CONTRIBUTING.md: bring-up is from a local/out-of-band bootstrap source; Gitea
|
|
arrives in Phase 6). So Phase 1 uses `flux install` (controllers only) + `kubectl apply -k
|
|
clusters/dev` (materialise the authored graph). This **reconciles with** `clusters/base` instead of
|
|
regenerating it.
|
|
|
|
## Bootstrap order (what runs, in sequence)
|
|
|
|
```
|
|
tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1.1..1.3]
|
|
│
|
|
▼ (final Ansible play calls the entrypoints below)
|
|
1. bootstrap/sops/age-keygen.sh # operator generates age key OUTSIDE repo
|
|
2. bootstrap/sops/age-key-secret.md # operator installs private key as sops-age Secret
|
|
3. bootstrap/flux/bootstrap.sh # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH}
|
|
│
|
|
▼
|
|
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
|
|
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
|
|
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
|
|
```
|
|
|
|
## Interface note — for the Infra worker (T-1.3 Ansible)
|
|
|
|
**Single entrypoint your final play calls:** `bootstrap/flux/bootstrap.sh` (run inside the
|
|
`infra/devcontainer` toolchain so pinned `flux`/`kubectl` are on PATH; `FLUX_VERSION`/`SOPS_VERSION`
|
|
live in `infra/devcontainer/versions.env`).
|
|
|
|
**Order your play must enforce** (the age key is a hard dependency of a working decrypt):
|
|
1. `bootstrap/sops/age-keygen.sh` (operator step — generates key outside repo).
|
|
2. Install the `sops-age` Secret per `bootstrap/sops/age-key-secret.md` (the `kubectl create secret`
|
|
template). `bootstrap.sh` **fails fast** if this Secret is absent.
|
|
3. `bootstrap/flux/bootstrap.sh` (this entrypoint).
|
|
|
|
`bootstrap.sh` is idempotent-friendly (`flux install`, `preapply-crds.sh`'s server-side apply, and
|
|
`kubectl apply -k` all re-run safely) and never handles key material — it only *verifies* the Secret
|
|
exists. Beyond `flux`/`kubectl` it now also needs **`helm` + `yq`** on PATH (step 2's CRD render;
|
|
both are in `infra/devcontainer`) and outbound network to the Kyverno chart repo.
|
|
|
|
## Handoff — clusters owner / PM (files this worker may not edit)
|
|
|
|
`clusters/` is out of this worker's write scope, so three wirings are delivered as patches here and
|
|
must be applied into the `clusters/dev` overlay (`clusters/dev/kustomization.yaml` `patches:`):
|
|
|
|
```yaml
|
|
patches:
|
|
- path: ../../bootstrap/flux/patches/flux-system-source-url.yaml # real source URL
|
|
- path: ../../bootstrap/flux/patches/platform-sops-decryption.yaml # SOPS on platform
|
|
- path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml # SOPS on apps
|
|
```
|
|
|
|
Alternatively the base TODOs in `clusters/base/platform.yaml` and each app-wave Kustomization
|
|
(`apps-wave-0/1/2.yaml`, `apps-vms.yaml`) and the `url` in
|
|
`clusters/base/flux-system.yaml` can be filled in directly (same effect) — the clusters owner's call.
|
|
Either way the `sops-age` `secretRef` name and `flux-system` source name are kept in lockstep with
|
|
the authored base.
|
|
|
|
## Phase-6 cutover (WS-GITOPS — NOT now)
|
|
|
|
Once Gitea is stable, migrate the source to Gitea (docs/05, CONTRIBUTING.md). At that point
|
|
`flux bootstrap git` becomes appropriate (there is a writable remote to own):
|
|
|
|
```bash
|
|
# ILLUSTRATIVE — Phase 6, not run in Phase 1.
|
|
flux bootstrap git \
|
|
--url=https://gitea.home.keithjackson.dev/<org>/home-network-k3s-impl.git \
|
|
--branch=main \
|
|
--path=clusters/dev
|
|
```
|
|
Keep an off-cluster mirror so a source-of-truth outlives the cluster (docs/05 "Circularity & DR").
|