home-network-k3s-impl/bootstrap/flux/README.md

bootstrap/flux — Flux bootstrap → clusters/dev (Phase 1, T-1.4)

Points Flux at the authored clusters/dev entrypoint and turns on SOPS decryption. Spec: ADR-0004 (Flux), docs/05 · Requirements: R-GITOPS-01 (pull-based reconcile), R-GITOPS-03 (drift correction), R-SEC-01 (SOPS decryption).

AUTHOR-ONLY. Nothing in this directory has been executed. No flux, kubectl, key-gen, or cluster contact was run. These are files + runbooks the PM/operator validates and runs.

Files

File	Purpose
bootstrap.sh	The single bootstrap entrypoint the Infra worker's final Ansible play calls. render+apply the Flux controllers (components/) → pre-apply chart CRDs (preapply-crds.sh) → verify sops-age Secret → kubectl apply -k ${FLUX_ENV_PATH}.
preapply-crds.sh	Server-side pre-apply of the Kyverno CRD bundle — the CRDs platform/kyverno/admission/helmrelease.yaml deliberately tells Helm not to install (crds: {install: false}, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step platform never goes Ready on a cold cluster. Chart name/version/repo are derived from that HelmRelease — no second pin. Idempotent; runnable standalone.
components/kustomization.yaml	Customization of the Flux CONTROLLERS themselves. Overlays flux install --export output; currently enables StrictPostBuildSubstitutions on kustomize-controller. See "Customizing the controllers" below.
patches/platform-sops-decryption.yaml	Strategic-merge patch enabling decryption.provider: sops + secretRef: sops-age on the platform Kustomization (realises the base TODO).
patches/apps-sops-decryption.yaml	Same, for the app-wave Kustomizations. There is no single apps Kustomization any more — the apps tier is now apps-wave-0/1/2 + apps-vms, and SOPS decryption is layered on per wave (each wave Kustomization is patched by name in clusters/dev).
patches/flux-system-source-url.yaml	Overrides the flux-system GitRepository url placeholder with the real bootstrap source (template — fill at bootstrap).

Customizing the controllers (components/) — and how it reaches a RUNNING cluster

Because this repo does not use flux bootstrap, the Flux controller Deployments are not in git and not reconciled by Flux. Verified on the live dev cluster (2026-09-01):

$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
 "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1

No kustomize.toolkit.fluxcd.io/* label => no Flux Kustomization owns it. Nothing under clusters/** can change a controller flag. So bootstrap.sh step 1 uses the mechanism Flux documents for hand-installed clusters (Customize Flux manifests):

flux install --export …  ->  <tmp>/gotk-components.yaml     # regenerated from the pinned flux CLI
components/kustomization.yaml -> <tmp>/kustomization.yaml   # the only hand-authored part (in git)
kubectl kustomize <tmp> | kubectl apply -f -

Nothing is vendored into the repo, so the component manifests can never drift from FLUX_VERSION (infra/devcontainer/versions.env). Every future controller flag (vertical scaling, tolerations, more feature gates) is another entry in components/kustomization.yaml.

Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)

A git push does not deliver this. Merging the change reconciles clusters/** only; the controller Deployment is outside Flux's inventory. Someone must run step 1.

Re-running bootstrap.sh is safe and idempotent, but if you want only step 1:

# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
d=$(mktemp -d)
flux install --export --namespace=flux-system \
  --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
kubectl kustomize "$d" | kubectl apply -f -      # add --server-side --force-conflicts on a field-manager conflict
kubectl -n flux-system rollout status deploy/kustomize-controller

Expect kustomize-controller to restart (new arg => new pod template => new ReplicaSet). Any reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing is lost. Confirm afterwards:

kubectl -n flux-system get deploy kustomize-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}'   # must contain --feature-gates=StrictPostBuildSubstitutions=true
flux get kustomizations -A                                  # all 11 must return to Ready

Rollback is the same command with FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh, or simply flux install --namespace=flux-system --components-extra=… (drops the arg back off).

The CRD pre-apply step (preapply-crds.sh) — why bootstrap has to do this

platform/kyverno/admission/helmrelease.yaml sets crds: {install: false}. That is correct and must stay: the Kyverno CRD bundle is 5,636,155 bytes across 22 CRDs at chart 3.9.0 (measured with helm template … --include-crds, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB last-applied-configuration annotation limit — metadata.annotations: Too long — which stalls the HelmRelease and blocks platform plus every tier that dependsOn it.

The part that was never automated: with CRD install off, nothing installs the CRDs. Kyverno's Deployments start, its API types do not exist, platform never reaches Ready. Dev has them only because somebody once ran the command in a shell. preapply-crds.sh is that command, in git:

helm template <chart> --repo <url> --version <constraint> --include-crds --set crds.install=true
  | yq 'select(.kind == "CustomResourceDefinition")'
  | kubectl apply --server-side --force-conflicts

• Version derivation, not a second pin. chart, version, namespace and the repo url are read out of the HelmRelease/HelmRepository pair with yq. spec.chart.spec.version is a semver constraint (3.9.x), and helm template --version accepts the same constraint syntax helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump of that one line carries this step with it. There is nothing here that can drift.
• CRDs only. Piping the whole helm template output into kubectl (the by-hand command in the HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the HelmRelease over field managers. The yq filter is load-bearing.
• Idempotent. Server-side apply is an upsert; --force-conflicts adopts the fields dev's hand-applied CRDs left owned by kubectl-client-side-apply. Re-running prints serverside-applied and exits 0.
• Cannot degrade to a no-op. Three guards: every derived coordinate must be non-empty and not null; the rendered CRD count must be > 0 (an empty stream would otherwise apply cleanly and exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is waited on for condition=Established. There is no skip flag on purpose.
• Environment-agnostic (R-ENV-02). It names no environment and takes no overlay input — it reads repo files and acts on $KUBECONFIG. Dev and prod run the identical step.

Standalone use (an already-bootstrapped cluster whose platform is blocked on the missing CRDs):

# inside infra/devcontainer, KUBECONFIG pointed at the target cluster
bash bootstrap/flux/preapply-crds.sh

Why flux install + apply, not flux bootstrap git (Phase 1)

The reconciliation graph is already authored in clusters/base (read-only to this worker): flux-system.yaml is the flux-system GitRepository; platform.yaml, platform-config.yaml and the app-wave Kustomizations (apps-wave-0/1/2.yaml + apps-vms.yaml — the single apps.yaml was retired) are the Flux Kustomizations; clusters/dev is the overlay that patches cluster-vars (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.

flux bootstrap git generates and owns its own gotk-sync GitRepository + Kustomization named flux-system and commits them into the repo — which would duplicate/conflict with the authored base. It also requires a writable remote git host, which does not exist in Phase 1 (spec docs/05 "Source-of-truth", CONTRIBUTING.md: bring-up is from a local/out-of-band bootstrap source; Gitea arrives in Phase 6). So Phase 1 uses flux install (controllers only) + kubectl apply -k clusters/dev (materialise the authored graph). This reconciles with clusters/base instead of regenerating it.

Bootstrap order (what runs, in sequence)

tofu apply → cloud-init → Ansible (k3s + Cilium)          [Infra worker, T-1.1..1.3]
        │
        ▼  (final Ansible play calls the entrypoints below)
1. bootstrap/sops/age-keygen.sh          # operator generates age key OUTSIDE repo
2. bootstrap/sops/age-key-secret.md      # operator installs private key as sops-age Secret
3. bootstrap/flux/bootstrap.sh           # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH}
        │
        ▼
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).

Interface note — for the Infra worker (T-1.3 Ansible)

Single entrypoint your final play calls: bootstrap/flux/bootstrap.sh (run inside the infra/devcontainer toolchain so pinned flux/kubectl are on PATH; FLUX_VERSION/SOPS_VERSION live in infra/devcontainer/versions.env).

Order your play must enforce (the age key is a hard dependency of a working decrypt):

1. bootstrap/sops/age-keygen.sh (operator step — generates key outside repo).
2. Install the sops-age Secret per bootstrap/sops/age-key-secret.md (the kubectl create secret template). bootstrap.sh fails fast if this Secret is absent.
3. bootstrap/flux/bootstrap.sh (this entrypoint).

bootstrap.sh is idempotent-friendly (flux install, preapply-crds.sh's server-side apply, and kubectl apply -k all re-run safely) and never handles key material — it only verifies the Secret exists. Beyond flux/kubectl it now also needs helm + yq on PATH (step 2's CRD render; both are in infra/devcontainer) and outbound network to the Kyverno chart repo.

Handoff — clusters owner / PM (files this worker may not edit)

clusters/ is out of this worker's write scope, so three wirings are delivered as patches here and must be applied into the clusters/dev overlay (clusters/dev/kustomization.yaml patches:):

patches:
  - path: ../../bootstrap/flux/patches/flux-system-source-url.yaml    # real source URL
  - path: ../../bootstrap/flux/patches/platform-sops-decryption.yaml  # SOPS on platform
  - path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml      # SOPS on apps

Alternatively the base TODOs in clusters/base/platform.yaml and each app-wave Kustomization (apps-wave-0/1/2.yaml, apps-vms.yaml) and the url in clusters/base/flux-system.yaml can be filled in directly (same effect) — the clusters owner's call. Either way the sops-age secretRef name and flux-system source name are kept in lockstep with the authored base.

Phase-6 cutover (WS-GITOPS — NOT now)

Once Gitea is stable, migrate the source to Gitea (docs/05, CONTRIBUTING.md). At that point flux bootstrap git becomes appropriate (there is a writable remote to own):

# ILLUSTRATIVE — Phase 6, not run in Phase 1.
flux bootstrap git \
  --url=https://gitea.home.keithjackson.dev/<org>/home-network-k3s-impl.git \
  --branch=main \
  --path=clusters/dev

Keep an off-cluster mirror so a source-of-truth outlives the cluster (docs/05 "Circularity & DR").