# bootstrap/flux — Flux bootstrap → `clusters/dev` (Phase 1, T-1.4)

Points Flux at the authored **`clusters/dev`** entrypoint and turns on SOPS decryption.
Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux.md),
[docs/05](../../../home-network-k3s/docs/05-gitops-and-delivery.md) · Requirements: **R-GITOPS-01**
(pull-based reconcile), **R-GITOPS-03** (drift correction), **R-SEC-01** (SOPS decryption).

> **AUTHOR-ONLY. Nothing in this directory has been executed.** No `flux`, `kubectl`, key-gen, or
> cluster contact was run. These are files + runbooks the PM/operator validates and runs.

## Files
| File | Purpose |
|---|---|
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → **pre-apply chart CRDs (`preapply-crds.sh`)** → verify `sops-age` Secret → `kubectl apply -k ${FLUX_ENV_PATH}`. |
| `preapply-crds.sh` | **Server-side pre-apply of the Kyverno CRD bundle** — the CRDs `platform/kyverno/admission/helmrelease.yaml` deliberately tells Helm *not* to install (`crds: {install: false}`, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step `platform` never goes Ready on a cold cluster. Chart name/version/repo are **derived from that HelmRelease** — no second pin. Idempotent; runnable standalone. |
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
| `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |

## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster

Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):

```
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
 "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
```

No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
documents for hand-installed clusters ([Customize Flux
manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):

```
flux install --export …  ->  <tmp>/gotk-components.yaml     # regenerated from the pinned flux CLI
components/kustomization.yaml -> <tmp>/kustomization.yaml   # the only hand-authored part (in git)
kubectl kustomize <tmp> | kubectl apply -f -
```

Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
more feature gates) is another entry in `components/kustomization.yaml`.

### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)

> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
> controller Deployment is outside Flux's inventory. Someone must run step 1.

Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:

```bash
# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
d=$(mktemp -d)
flux install --export --namespace=flux-system \
  --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
kubectl kustomize "$d" | kubectl apply -f -      # add --server-side --force-conflicts on a field-manager conflict
kubectl -n flux-system rollout status deploy/kustomize-controller
```

**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
is lost. Confirm afterwards:

```bash
kubectl -n flux-system get deploy kustomize-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}'   # must contain --feature-gates=StrictPostBuildSubstitutions=true
flux get kustomizations -A                                  # all 11 must return to Ready
```

**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).

## The CRD pre-apply step (`preapply-crds.sh`) — why bootstrap has to do this

`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`. That is correct and must
stay: the Kyverno CRD bundle is **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured with
`helm template … --include-crds`, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB
`last-applied-configuration` annotation limit — `metadata.annotations: Too long` — which stalls the
HelmRelease and blocks `platform` plus every tier that `dependsOn` it.

The part that was never automated: **with CRD install off, nothing installs the CRDs.** Kyverno's
Deployments start, its API types do not exist, `platform` never reaches Ready. Dev has them only
because somebody once ran the command in a shell. `preapply-crds.sh` is that command, in git:

```
helm template <chart> --repo <url> --version <constraint> --include-crds --set crds.install=true
  | yq 'select(.kind == "CustomResourceDefinition")'
  | kubectl apply --server-side --force-conflicts
```

- **Version derivation, not a second pin.** `chart`, `version`, `namespace` and the repo `url` are
  read out of the HelmRelease/HelmRepository pair with `yq`. `spec.chart.spec.version` is a semver
  **constraint** (`3.9.x`), and `helm template --version` accepts the same constraint syntax
  helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump
  of that one line carries this step with it. There is nothing here that can drift.
- **CRDs only.** Piping the *whole* `helm template` output into kubectl (the by-hand command in the
  HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the
  HelmRelease over field managers. The `yq` filter is load-bearing.
- **Idempotent.** Server-side apply is an upsert; `--force-conflicts` adopts the fields dev's
  hand-applied CRDs left owned by `kubectl-client-side-apply`. Re-running prints
  `serverside-applied` and exits 0.
- **Cannot degrade to a no-op.** Three guards: every derived coordinate must be non-empty and not
  `null`; the rendered CRD count must be `> 0` (an empty stream would otherwise `apply` cleanly and
  exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is
  waited on for `condition=Established`. There is **no skip flag** on purpose.
- **Environment-agnostic (R-ENV-02).** It names no environment and takes no overlay input — it reads
  repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step.

Standalone use (an already-bootstrapped cluster whose `platform` is blocked on the missing CRDs):

```bash
# inside infra/devcontainer, KUBECONFIG pointed at the target cluster
bash bootstrap/flux/preapply-crds.sh
```

## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)

The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`, `platform-config.yaml`
and the app-wave Kustomizations (`apps-wave-0/1/2.yaml` + `apps-vms.yaml` — the single `apps.yaml`
was retired) are the Flux **Kustomizations**; `clusters/dev` is the overlay that patches
`cluster-vars` (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.

`flux bootstrap git` **generates and owns its own** `gotk-sync` GitRepository + Kustomization named
`flux-system` and commits them into the repo — which would **duplicate/conflict** with the authored
base. It also requires a **writable remote git host**, which does not exist in Phase 1 (spec docs/05
"Source-of-truth", CONTRIBUTING.md: bring-up is from a local/out-of-band bootstrap source; Gitea
arrives in Phase 6). So Phase 1 uses `flux install` (controllers only) + `kubectl apply -k
clusters/dev` (materialise the authored graph). This **reconciles with** `clusters/base` instead of
regenerating it.

## Bootstrap order (what runs, in sequence)

```
tofu apply → cloud-init → Ansible (k3s + Cilium)          [Infra worker, T-1.1..1.3]
        │
        ▼  (final Ansible play calls the entrypoints below)
1. bootstrap/sops/age-keygen.sh          # operator generates age key OUTSIDE repo
2. bootstrap/sops/age-key-secret.md      # operator installs private key as sops-age Secret
3. bootstrap/flux/bootstrap.sh           # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH}
        │
        ▼
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
```

## Interface note — for the Infra worker (T-1.3 Ansible)

**Single entrypoint your final play calls:** `bootstrap/flux/bootstrap.sh` (run inside the
`infra/devcontainer` toolchain so pinned `flux`/`kubectl` are on PATH; `FLUX_VERSION`/`SOPS_VERSION`
live in `infra/devcontainer/versions.env`).

**Order your play must enforce** (the age key is a hard dependency of a working decrypt):
1. `bootstrap/sops/age-keygen.sh` (operator step — generates key outside repo).
2. Install the `sops-age` Secret per `bootstrap/sops/age-key-secret.md` (the `kubectl create secret`
   template). `bootstrap.sh` **fails fast** if this Secret is absent.
3. `bootstrap/flux/bootstrap.sh` (this entrypoint).

`bootstrap.sh` is idempotent-friendly (`flux install`, `preapply-crds.sh`'s server-side apply, and
`kubectl apply -k` all re-run safely) and never handles key material — it only *verifies* the Secret
exists. Beyond `flux`/`kubectl` it now also needs **`helm` + `yq`** on PATH (step 2's CRD render;
both are in `infra/devcontainer`) and outbound network to the Kyverno chart repo.

## Handoff — clusters owner / PM (files this worker may not edit)

`clusters/` is out of this worker's write scope, so three wirings are delivered as patches here and
must be applied into the `clusters/dev` overlay (`clusters/dev/kustomization.yaml` `patches:`):

```yaml
patches:
  - path: ../../bootstrap/flux/patches/flux-system-source-url.yaml    # real source URL
  - path: ../../bootstrap/flux/patches/platform-sops-decryption.yaml  # SOPS on platform
  - path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml      # SOPS on apps
```

Alternatively the base TODOs in `clusters/base/platform.yaml` and each app-wave Kustomization
(`apps-wave-0/1/2.yaml`, `apps-vms.yaml`) and the `url` in
`clusters/base/flux-system.yaml` can be filled in directly (same effect) — the clusters owner's call.
Either way the `sops-age` `secretRef` name and `flux-system` source name are kept in lockstep with
the authored base.

## Phase-6 cutover (WS-GITOPS — NOT now)

Once Gitea is stable, migrate the source to Gitea (docs/05, CONTRIBUTING.md). At that point
`flux bootstrap git` becomes appropriate (there is a writable remote to own):

```bash
# ILLUSTRATIVE — Phase 6, not run in Phase 1.
flux bootstrap git \
  --url=https://gitea.home.keithjackson.dev/<org>/home-network-k3s-impl.git \
  --branch=main \
  --path=clusters/dev
```
Keep an off-cluster mirror so a source-of-truth outlives the cluster (docs/05 "Circularity & DR").