0 Fork 0

Files

T

jakman85andClaude Opus 5 666a60776a fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1)

P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.

Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.

Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.

P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.

So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.

Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).

ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.

ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.

Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).

Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.

Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-01 01:45:04 -05:00

8.9 KiB

Raw Blame History

bootstrap/flux — Flux bootstrap → `clusters/dev` (Phase 1, T-1.4)

Points Flux at the authored clusters/dev entrypoint and turns on SOPS decryption. Spec: ADR-0004 (Flux), docs/05 · Requirements: R-GITOPS-01 (pull-based reconcile), R-GITOPS-03 (drift correction), R-SEC-01 (SOPS decryption).

AUTHOR-ONLY. Nothing in this directory has been executed. No flux, kubectl, key-gen, or cluster contact was run. These are files + runbooks the PM/operator validates and runs.

Files

File	Purpose
`bootstrap.sh`	The single bootstrap entrypoint the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`.
`components/kustomization.yaml`	Customization of the Flux CONTROLLERS themselves. Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below.
`patches/platform-sops-decryption.yaml`	Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO).
`patches/apps-sops-decryption.yaml`	Same, for the app-wave Kustomizations. There is no single `apps` Kustomization any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on per wave (each wave Kustomization is patched by name in `clusters/dev`).
`patches/flux-system-source-url.yaml`	Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap).

Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster

Because this repo does not use flux bootstrap, the Flux controller Deployments are not in git and not reconciled by Flux. Verified on the live dev cluster (2026-09-01):

$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
 "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1

No kustomize.toolkit.fluxcd.io/* label => no Flux Kustomization owns it. Nothing under clusters/** can change a controller flag. So bootstrap.sh step 1 uses the mechanism Flux documents for hand-installed clusters (Customize Flux manifests):

flux install --export …  ->  <tmp>/gotk-components.yaml     # regenerated from the pinned flux CLI
components/kustomization.yaml -> <tmp>/kustomization.yaml   # the only hand-authored part (in git)
kubectl kustomize <tmp> | kubectl apply -f -

Nothing is vendored into the repo, so the component manifests can never drift from FLUX_VERSION (infra/devcontainer/versions.env). Every future controller flag (vertical scaling, tolerations, more feature gates) is another entry in components/kustomization.yaml.

Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)

A git push does not deliver this. Merging the change reconciles clusters/** only; the controller Deployment is outside Flux's inventory. Someone must run step 1.

Re-running bootstrap.sh is safe and idempotent, but if you want only step 1:

# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
d=$(mktemp -d)
flux install --export --namespace=flux-system \
  --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
kubectl kustomize "$d" | kubectl apply -f -      # add --server-side --force-conflicts on a field-manager conflict
kubectl -n flux-system rollout status deploy/kustomize-controller

Expect kustomize-controller to restart (new arg => new pod template => new ReplicaSet). Any reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing is lost. Confirm afterwards:

kubectl -n flux-system get deploy kustomize-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}'   # must contain --feature-gates=StrictPostBuildSubstitutions=true
flux get kustomizations -A                                  # all 11 must return to Ready

Rollback is the same command with FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh, or simply flux install --namespace=flux-system --components-extra=… (drops the arg back off).

Why `flux install` + apply, not `flux bootstrap git` (Phase 1)

The reconciliation graph is already authored in clusters/base (read-only to this worker): flux-system.yaml is the flux-system GitRepository; platform.yaml, platform-config.yaml and the app-wave Kustomizations (apps-wave-0/1/2.yaml + apps-vms.yaml — the single apps.yaml was retired) are the Flux Kustomizations; clusters/dev is the overlay that patches cluster-vars (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.

flux bootstrap git generates and owns its own gotk-sync GitRepository + Kustomization named flux-system and commits them into the repo — which would duplicate/conflict with the authored base. It also requires a writable remote git host, which does not exist in Phase 1 (spec docs/05 "Source-of-truth", CONTRIBUTING.md: bring-up is from a local/out-of-band bootstrap source; Gitea arrives in Phase 6). So Phase 1 uses flux install (controllers only) + kubectl apply -k clusters/dev (materialise the authored graph). This reconciles with clusters/base instead of regenerating it.

Bootstrap order (what runs, in sequence)

tofu apply → cloud-init → Ansible (k3s + Cilium)          [Infra worker, T-1.1..1.3]
        │
        ▼  (final Ansible play calls the entrypoints below)
1. bootstrap/sops/age-keygen.sh          # operator generates age key OUTSIDE repo
2. bootstrap/sops/age-key-secret.md      # operator installs private key as sops-age Secret
3. bootstrap/flux/bootstrap.sh           # flux install → verify sops-age → apply -k clusters/dev
        │
        ▼
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).

Interface note — for the Infra worker (T-1.3 Ansible)

Single entrypoint your final play calls: bootstrap/flux/bootstrap.sh (run inside the infra/devcontainer toolchain so pinned flux/kubectl are on PATH; FLUX_VERSION/SOPS_VERSION live in infra/devcontainer/versions.env).

Order your play must enforce (the age key is a hard dependency of a working decrypt):

bootstrap/sops/age-keygen.sh (operator step — generates key outside repo).
Install the sops-age Secret per bootstrap/sops/age-key-secret.md (the kubectl create secret template). bootstrap.sh fails fast if this Secret is absent.
bootstrap/flux/bootstrap.sh (this entrypoint).

bootstrap.sh is idempotent-friendly (flux install and kubectl apply -k re-run safely) and never handles key material — it only verifies the Secret exists.

Handoff — clusters owner / PM (files this worker may not edit)

clusters/ is out of this worker's write scope, so three wirings are delivered as patches here and must be applied into the clusters/dev overlay (clusters/dev/kustomization.yaml patches:):

patches:
  - path: ../../bootstrap/flux/patches/flux-system-source-url.yaml    # real source URL
  - path: ../../bootstrap/flux/patches/platform-sops-decryption.yaml  # SOPS on platform
  - path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml      # SOPS on apps

Alternatively the base TODOs in clusters/base/platform.yaml and each app-wave Kustomization (apps-wave-0/1/2.yaml, apps-vms.yaml) and the url in clusters/base/flux-system.yaml can be filled in directly (same effect) — the clusters owner's call. Either way the sops-age secretRef name and flux-system source name are kept in lockstep with the authored base.

Phase-6 cutover (WS-GITOPS — NOT now)

Once Gitea is stable, migrate the source to Gitea (docs/05, CONTRIBUTING.md). At that point flux bootstrap git becomes appropriate (there is a writable remote to own):

# ILLUSTRATIVE — Phase 6, not run in Phase 1.
flux bootstrap git \
  --url=https://gitea.home.keithjackson.dev/<org>/home-network-k3s-impl.git \
  --branch=main \
  --path=clusters/dev

Keep an off-cluster mirror so a source-of-truth outlives the cluster (docs/05 "Circularity & DR").

8.9 KiB Raw Blame History

bootstrap/flux — Flux bootstrap → clusters/dev (Phase 1, T-1.4)