From 666a60776a981c21d895e1fd079e286135939640 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 01:45:04 -0500 Subject: [PATCH] =?UTF-8?q?fix(flux):=20close=20the=20substitution=20race?= =?UTF-8?q?=20=E2=80=94=20platform=20dependsOn=20root=20+=20strict=20subst?= =?UTF-8?q?itution=20(P0/P1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix for the incident, not the feature gate. Verified against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION- gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision N is exactly what writes a newly-added cluster-vars knob. Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all 11 substituting Kustomizations chain transitively through it (platform-config -> platform; the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config). The "not chaining through platform" set is EMPTY, so this one edge protects the whole graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/ sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready on apply success without health-gating its children. Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest anywhere under clusters/ — including clusters/base — now stalls platform and everything downstream, where today it would not. Fail-closed, per doctrine. P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README). There is no gotk-components.yaml in git, and the live kustomize-controller Deployment carries NO Flux ownership labels — it was installed imperatively by `flux install` from bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT the controllers. Nothing under clusters/** can reach a controller flag. So it uses Flux's documented mechanism for hand-installed clusters: a new bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a container's args is atomic and a merge patch would replace all five install-generated args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the patch, ASSERT the gate string is present so a future Flux bump that breaks the patch target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is vendored, so components can never drift from the pinned FLUX_VERSION. Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial Deployment as a Flux-managed resource (a future prune would delete kustomize-controller). ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive on a git push — the controller Deployment is outside Flux's inventory, so an operator must re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race turns a transient empty render into a transient hard Kustomization failure. ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left platform blocked forever on a flux-system Kustomization that does not exist, and nothing downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev. Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7): prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs the prod age-key decision, a security choice not a copy), and no GitRepository source URL patch (needs a decision on whether prod tracks the same branch as dev, which would make a dev-facing merge reconcile prod). Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) --- bootstrap/flux/README.md | 60 +++++++++++++++++- bootstrap/flux/bootstrap.sh | 55 +++++++++++++++-- bootstrap/flux/components/kustomization.yaml | 64 ++++++++++++++++++++ clusters/base/platform.yaml | 27 +++++++++ clusters/prod/kustomization.yaml | 10 +++ clusters/prod/sync.yaml | 32 ++++++++++ infra/PROD-PROVISIONING.md | 13 ++++ 7 files changed, 254 insertions(+), 7 deletions(-) create mode 100644 bootstrap/flux/components/kustomization.yaml create mode 100644 clusters/prod/sync.yaml diff --git a/bootstrap/flux/README.md b/bootstrap/flux/README.md index 90465ce..1824ad0 100644 --- a/bootstrap/flux/README.md +++ b/bootstrap/flux/README.md @@ -11,11 +11,69 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux ## Files | File | Purpose | |---|---| -| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. | +| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. | +| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. | | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). | | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). | | `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). | +## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster + +Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git +and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01): + +``` +$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}' +{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system", + "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"} +# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1 +``` + +No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under +`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux +documents for hand-installed clusters ([Customize Flux +manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)): + +``` +flux install --export … -> /gotk-components.yaml # regenerated from the pinned flux CLI +components/kustomization.yaml -> /kustomization.yaml # the only hand-authored part (in git) +kubectl kustomize | kubectl apply -f - +``` + +Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION` +(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations, +more feature gates) is another entry in `components/kustomization.yaml`. + +### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic) + +> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the +> controller Deployment is outside Flux's inventory. Someone must run step 1. + +Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1: + +```bash +# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root +d=$(mktemp -d) +flux install --export --namespace=flux-system \ + --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml" +cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml" +kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict +kubectl -n flux-system rollout status deploy/kustomize-controller +``` + +**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any +reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing +is lost. Confirm afterwards: + +```bash +kubectl -n flux-system get deploy kustomize-controller \ + -o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true +flux get kustomizations -A # all 11 must return to Ready +``` + +**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or +simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off). + ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1) The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker): diff --git a/bootstrap/flux/bootstrap.sh b/bootstrap/flux/bootstrap.sh index 893b19c..33e35c7 100755 --- a/bootstrap/flux/bootstrap.sh +++ b/bootstrap/flux/bootstrap.sh @@ -69,16 +69,59 @@ for bin in flux kubectl; do command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; } done -# --- 1. install Flux controllers --------------------------------------------- +# --- 1. install Flux controllers (rendered through ./components) -------------- # Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates, # ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the # image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly. -if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then - echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})" - flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}" +# +# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently: +# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined +# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see +# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md). +# +# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the +# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux +# documents for hand-installed clusters ("Customize Flux manifests"): +# +# flux install --export > gotk-components.yaml # exact manifests `flux install` would apply +# kustomize overlay = ./components/kustomization.yaml (checked in — the ONLY hand-authored part) +# kubectl apply -k +# +# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION, +# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is +# a temp dir — the repo tree is never written to. +# +# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO +# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so. +echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}" +flux_install_args=(--namespace="${FLUX_NAMESPACE}") +[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}") + +if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then + echo " WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED." + echo " StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN." + flux install "${flux_install_args[@]}" else - echo "==> [1/3] flux install (default controllers only)" - flux install --namespace="${FLUX_NAMESPACE}" + components_dir="${script_dir}/components" + [ -f "${components_dir}/kustomization.yaml" ] || { + echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2 + echo " (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2 + exit 1 + } + render_dir="$(mktemp -d)" + trap 'rm -rf "${render_dir}"' EXIT + echo " rendering flux install --export through ${components_dir}" + flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml" + cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml" + # Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux + # version bump) — better a hard stop than silently installing without the feature gate. + kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml" + grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || { + echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2 + echo " Check ./components/kustomization.yaml against this flux CLI version." >&2 + exit 1 + } + kubectl apply -f "${render_dir}/rendered.yaml" fi # --- 2. verify the out-of-band age key Secret exists ------------------------- diff --git a/bootstrap/flux/components/kustomization.yaml b/bootstrap/flux/components/kustomization.yaml new file mode 100644 index 0000000..0a428ec --- /dev/null +++ b/bootstrap/flux/components/kustomization.yaml @@ -0,0 +1,64 @@ +--- +# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments). +# +# WHY THIS DIRECTORY EXISTS +# ========================= +# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the +# reconciliation graph is authored by hand in clusters/base, and the controllers are installed +# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01: +# +# kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}' +# -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1 +# +# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no +# existing mechanism to customize them, and no amount of editing under clusters/** can reach them. +# +# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" — +# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests +# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result. +# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported +# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the +# repo, so the manifests can never drift from the pinned FLUX_VERSION +# (infra/devcontainer/versions.env) the way a vendored copy would. +# +# Every future controller customization (vertical scaling, tolerations, extra feature gates, +# --concurrent) belongs HERE, as another entry under `patches:`. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + # Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in. + - gotk-components.yaml +patches: + # ------------------------------------------------------------------------------------------- + # StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED. + # + # Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the + # EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium + # HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this + # gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state. + # + # PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with + # `kubectl kustomize --load-restrictor LoadRestrictionsNone | flux envsubst --strict` and + # exited 0, with a negative control reproducing the incident. The permanent guard that keeps it + # true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/. + # + # NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) — + # Flux exempts them by design. That class is what the L0 gate covers; the two are complementary. + # + # This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in + # kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in. + # + # JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic + # list, so a strategic-merge patch would REPLACE every arg `flux install` generates + # (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election) + # and silently drop any the CLI adds in a future version. + - target: + group: apps + version: v1 + kind: Deployment + name: kustomize-controller + namespace: flux-system + patch: |- + - op: add + path: /spec/template/spec/containers/0/args/- + value: --feature-gates=StrictPostBuildSubstitutions=true diff --git a/clusters/base/platform.yaml b/clusters/base/platform.yaml index 1f3dec6..02251e1 100644 --- a/clusters/base/platform.yaml +++ b/clusters/base/platform.yaml @@ -13,6 +13,33 @@ spec: timeout: 5m prune: true # drift correction: delete resources removed from git (R-GITOPS-03) wait: true # block `apps` until platform is healthy + # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium). + # + # `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root + # `flux-system` Kustomization (clusters//sync.yaml), not by this one. Without an edge here, + # `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a + # NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING. + # That is how `k8sServiceHost: ""` reached the Cilium HelmRelease. + # + # `dependsOn` is REVISION-gated, not merely Ready-gated (verified against + # kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version): + # when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile + # revision N until the dependency has APPLIED revision N. Applying revision N is exactly what + # writes a newly-added `cluster-vars` knob — so this one edge closes the race. + # + # `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting + # consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty, + # velero) chains transitively through it. One edge therefore protects the whole graph. + # + # TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest + # ANYWHERE under the root's path (clusters/, which includes clusters/base) fails the root + # Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge + # the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken + # entrypoint should stop the line, not let consumers render against a half-applied variable set. + # No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never + # health-gates its own children (see clusters/dev/sync.yaml). + dependsOn: + - name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars sourceRef: kind: GitRepository name: flux-system diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 5b8b0f9..171eda3 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -5,10 +5,20 @@ # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit. +# +# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD *** +# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution- +# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml` +# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would +# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing +# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml +# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT +# done here: prod is design-only and its root object is the prod bring-up owner's call.) apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base + - sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below) patches: - target: kind: ConfigMap diff --git a/clusters/prod/sync.yaml b/clusters/prod/sync.yaml new file mode 100644 index 0000000..9130eee --- /dev/null +++ b/clusters/prod/sync.yaml @@ -0,0 +1,32 @@ +--- +# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml). +# +# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars +# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object +# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs +# its own sync.yaml -> ./clusters/prod"; this is it. +# +# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained +# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent +# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no +# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does +# not exist, and NOTHING downstream would ever reconcile. +# +# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches +# Ready on apply success without health-gating its children (platform does its own wait/health +# gating). Do not "harden" this to true. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: flux-system + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + sourceRef: + kind: GitRepository + name: flux-system + path: ./clusters/prod diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index fa959a9..a337754 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -311,6 +311,19 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be own comment flags this as owed "when hardware is built." - [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`. +- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing + the `dependsOn` fix. Only the first is now closed: + - [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became + *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries + `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block + forever on a dependency that does not exist and nothing downstream would reconcile. + - [ ] **SOPS decryption** — `clusters/dev/kustomization.yaml` patches every Kustomization with + `decryption: {provider: sops, secretRef: sops-age}`. Prod has no equivalent, so every + SOPS-encrypted Secret would fail to decrypt. Needs the prod age key decision first + (same key as dev, or a prod-specific key — a real security choice, not a copy). + - [ ] **GitRepository source URL** — dev patches in the real Gitea URL. Prod needs its own, + plus a decision on whether prod tracks the same branch as dev or a separate one. + Tracking the same branch means a dev-facing merge reconciles prod. - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).