jakman85andClaude Opus 5
666a60776a
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix for the incident, not the feature gate. Verified against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION- gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision N is exactly what writes a newly-added cluster-vars knob. Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all 11 substituting Kustomizations chain transitively through it (platform-config -> platform; the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config). The "not chaining through platform" set is EMPTY, so this one edge protects the whole graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/ sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready on apply success without health-gating its children. Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest anywhere under clusters/<env> — including clusters/base — now stalls platform and everything downstream, where today it would not. Fail-closed, per doctrine. P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README). There is no gotk-components.yaml in git, and the live kustomize-controller Deployment carries NO Flux ownership labels — it was installed imperatively by `flux install` from bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT the controllers. Nothing under clusters/** can reach a controller flag. So it uses Flux's documented mechanism for hand-installed clusters: a new bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a container's args is atomic and a merge patch would replace all five install-generated args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the patch, ASSERT the gate string is present so a future Flux bump that breaks the patch target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is vendored, so components can never drift from the pinned FLUX_VERSION. Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial Deployment as a Flux-managed resource (a future prune would delete kustomize-controller). ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive on a git push — the controller Deployment is outside Flux's inventory, so an operator must re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race turns a transient empty render into a transient hard Kustomization failure. ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left platform blocked forever on a flux-system Kustomization that does not exist, and nothing downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev. Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7): prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs the prod age-key decision, a security choice not a copy), and no GitRepository source URL patch (needs a decision on whether prod tracks the same branch as dev, which would make a dev-facing merge reconcile prod). Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
65 lines
3.7 KiB
YAML
65 lines
3.7 KiB
YAML
---
|
|
# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments).
|
|
#
|
|
# WHY THIS DIRECTORY EXISTS
|
|
# =========================
|
|
# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the
|
|
# reconciliation graph is authored by hand in clusters/base, and the controllers are installed
|
|
# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01:
|
|
#
|
|
# kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
|
|
# -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1
|
|
#
|
|
# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no
|
|
# existing mechanism to customize them, and no amount of editing under clusters/** can reach them.
|
|
#
|
|
# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" —
|
|
# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests
|
|
# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result.
|
|
# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported
|
|
# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the
|
|
# repo, so the manifests can never drift from the pinned FLUX_VERSION
|
|
# (infra/devcontainer/versions.env) the way a vendored copy would.
|
|
#
|
|
# Every future controller customization (vertical scaling, tolerations, extra feature gates,
|
|
# --concurrent) belongs HERE, as another entry under `patches:`.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
# Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in.
|
|
- gotk-components.yaml
|
|
patches:
|
|
# -------------------------------------------------------------------------------------------
|
|
# StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED.
|
|
#
|
|
# Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the
|
|
# EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium
|
|
# HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this
|
|
# gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state.
|
|
#
|
|
# PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with
|
|
# `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` and
|
|
# exited 0, with a negative control reproducing the incident. The permanent guard that keeps it
|
|
# true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/.
|
|
#
|
|
# NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) —
|
|
# Flux exempts them by design. That class is what the L0 gate covers; the two are complementary.
|
|
#
|
|
# This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in
|
|
# kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in.
|
|
#
|
|
# JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic
|
|
# list, so a strategic-merge patch would REPLACE every arg `flux install` generates
|
|
# (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election)
|
|
# and silently drop any the CLI adds in a future version.
|
|
- target:
|
|
group: apps
|
|
version: v1
|
|
kind: Deployment
|
|
name: kustomize-controller
|
|
namespace: flux-system
|
|
patch: |-
|
|
- op: add
|
|
path: /spec/template/spec/containers/0/args/-
|
|
value: --feature-gates=StrictPostBuildSubstitutions=true
|