Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
2f2e98aa19db712a22696a61591fa38885e1d43b
home-network-k3s-impl/bootstrap/flux/README.md
T
jakman85andClaude Opus 5 666a60776a fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1) 
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.

Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.

Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.

P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.

So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.

Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).

ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.

ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.

Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).

Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.

Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 01:45:04 -05:00

155 lines
8.9 KiB
Markdown
Raw Blame History

# bootstrap/flux — Flux bootstrap → `clusters/dev` (Phase 1, T-1.4)
Points Flux at the authored **`clusters/dev`** entrypoint and turns on SOPS decryption.
Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux.md),
[docs/05](../../../home-network-k3s/docs/05-gitops-and-delivery.md) · Requirements: **R-GITOPS-01**
(pull-based reconcile), **R-GITOPS-03** (drift correction), **R-SEC-01** (SOPS decryption).
> **AUTHOR-ONLY. Nothing in this directory has been executed.** No `flux`, `kubectl`, key-gen, or
> cluster contact was run. These are files + runbooks the PM/operator validates and runs.
## Files
| File | Purpose |
|---|---|
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
| `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
```
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
"app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
```
No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
documents for hand-installed clusters ([Customize Flux
manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
```
flux install --export … -> <tmp>/gotk-components.yaml # regenerated from the pinned flux CLI
components/kustomization.yaml -> <tmp>/kustomization.yaml # the only hand-authored part (in git)
kubectl kustomize <tmp> | kubectl apply -f -
```
Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
more feature gates) is another entry in `components/kustomization.yaml`.
### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
> controller Deployment is outside Flux's inventory. Someone must run step 1.
Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
```bash
# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
d=$(mktemp -d)
flux install --export --namespace=flux-system \
--components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict
kubectl -n flux-system rollout status deploy/kustomize-controller
```
**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
is lost. Confirm afterwards:
```bash
kubectl -n flux-system get deploy kustomize-controller \
-o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true
flux get kustomizations -A # all 11 must return to Ready
```
**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`, `platform-config.yaml`
and the app-wave Kustomizations (`apps-wave-0/1/2.yaml` + `apps-vms.yaml` — the single `apps.yaml`
was retired) are the Flux **Kustomizations**; `clusters/dev` is the overlay that patches
`cluster-vars` (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.
`flux bootstrap git` **generates and owns its own** `gotk-sync` GitRepository + Kustomization named
`flux-system` and commits them into the repo — which would **duplicate/conflict** with the authored
base. It also requires a **writable remote git host**, which does not exist in Phase 1 (spec docs/05
"Source-of-truth", CONTRIBUTING.md: bring-up is from a local/out-of-band bootstrap source; Gitea
arrives in Phase 6). So Phase 1 uses `flux install` (controllers only) + `kubectl apply -k
clusters/dev` (materialise the authored graph). This **reconciles with** `clusters/base` instead of
regenerating it.
## Bootstrap order (what runs, in sequence)
```
tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1.1..1.3]
│
▼ (final Ansible play calls the entrypoints below)
1. bootstrap/sops/age-keygen.sh # operator generates age key OUTSIDE repo
2. bootstrap/sops/age-key-secret.md # operator installs private key as sops-age Secret
3. bootstrap/flux/bootstrap.sh # flux install → verify sops-age → apply -k clusters/dev
│
▼
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
```
## Interface note — for the Infra worker (T-1.3 Ansible)
**Single entrypoint your final play calls:** `bootstrap/flux/bootstrap.sh` (run inside the
`infra/devcontainer` toolchain so pinned `flux`/`kubectl` are on PATH; `FLUX_VERSION`/`SOPS_VERSION`
live in `infra/devcontainer/versions.env`).
**Order your play must enforce** (the age key is a hard dependency of a working decrypt):
1. `bootstrap/sops/age-keygen.sh` (operator step — generates key outside repo).
2. Install the `sops-age` Secret per `bootstrap/sops/age-key-secret.md` (the `kubectl create secret`
template). `bootstrap.sh` **fails fast** if this Secret is absent.
3. `bootstrap/flux/bootstrap.sh` (this entrypoint).
`bootstrap.sh` is idempotent-friendly (`flux install` and `kubectl apply -k` re-run safely) and
never handles key material — it only *verifies* the Secret exists.
## Handoff — clusters owner / PM (files this worker may not edit)
`clusters/` is out of this worker's write scope, so three wirings are delivered as patches here and
must be applied into the `clusters/dev` overlay (`clusters/dev/kustomization.yaml` `patches:`):
```yaml
patches:
- path: ../../bootstrap/flux/patches/flux-system-source-url.yaml # real source URL
- path: ../../bootstrap/flux/patches/platform-sops-decryption.yaml # SOPS on platform
- path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml # SOPS on apps
```
Alternatively the base TODOs in `clusters/base/platform.yaml` and each app-wave Kustomization
(`apps-wave-0/1/2.yaml`, `apps-vms.yaml`) and the `url` in
`clusters/base/flux-system.yaml` can be filled in directly (same effect) — the clusters owner's call.
Either way the `sops-age` `secretRef` name and `flux-system` source name are kept in lockstep with
the authored base.
## Phase-6 cutover (WS-GITOPS — NOT now)
Once Gitea is stable, migrate the source to Gitea (docs/05, CONTRIBUTING.md). At that point
`flux bootstrap git` becomes appropriate (there is a writable remote to own):
```bash
# ILLUSTRATIVE — Phase 6, not run in Phase 1.
flux bootstrap git \
--url=https://gitea.home.keithjackson.dev/<org>/home-network-k3s-impl.git \
--branch=main \
--path=clusters/dev
```
Keep an off-cluster mirror so a source-of-truth outlives the cluster (docs/05 "Circularity & DR").
Reference in new issue View Git Blame Copy Permalink