jakman85andClaude Opus 5
63f98aab26
Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.
New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.
*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.
TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
moving resources between Kustomizations. Chosen over the per-resource
`kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
say name=platform. The new cilium Kustomization relabels all three to name=cilium on
apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".
NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.
Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.
MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
87 lines
5.4 KiB
YAML
87 lines
5.4 KiB
YAML
---
|
|
# Flux Kustomization: PLATFORM services (cluster-wide). Reconciles ./platform.
|
|
# Phase 0 STUB. Realizes R-GITOPS-01/03 (pull-based reconcile + drift correction) and
|
|
# R-ENV-02 (env differences supplied ONLY via substituteFrom cluster-vars).
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform
|
|
namespace: flux-system
|
|
spec:
|
|
interval: 10m
|
|
retryInterval: 2m
|
|
timeout: 5m
|
|
# ======================= TEMPORARY — prune:false TRANSITION STATE =======================
|
|
# RESTORE TO `prune: true` IN THE VERY NEXT MERGE. This is step 1 of the documented Flux
|
|
# procedure for MOVING RESOURCES BETWEEN KUSTOMIZATIONS (fluxcd.io/flux/faq: "set spec.prune to
|
|
# false, sync, move the manifests, then re-enable pruning"), used here to move ./platform/cilium
|
|
# out of this tier and into its own `cilium` Kustomization (clusters/base/cilium.yaml).
|
|
#
|
|
# WHY IT IS NOT OPTIONAL: `platform/cilium/namespace.yaml` is the ONLY declaration of
|
|
# Namespace/kube-system anywhere under ./platform, and the live object carries
|
|
# `kustomize.toolkit.fluxcd.io/name=platform`. The moment ./platform/cilium leaves this
|
|
# Kustomization's rendered path, Namespace/kube-system + the Cilium HelmRepository/HelmRelease
|
|
# become STALE in this Kustomization's inventory. With prune:true that is not "uninstall the
|
|
# CNI" — it is `kubectl delete namespace kube-system`, which takes the whole cluster.
|
|
#
|
|
# A SECOND, INDEPENDENT PROTECTION IS ALSO IN PLACE (belt and braces, deliberately):
|
|
# kustomize-controller's pruner passes `Inclusions: GetOwnerLabels(platform, flux-system)`, so it
|
|
# only deletes objects whose LIVE labels still say they belong to `platform`. The new `cilium`
|
|
# Kustomization relabels all three objects to `name=cilium` when it applies them, and the
|
|
# `dependsOn: [cilium]` edge below is REVISION-gated — `platform` cannot reconcile revision N
|
|
# until `cilium` has APPLIED revision N. So the relabel provably precedes the prune. Neither
|
|
# mechanism is trusted alone for an object of this blast radius.
|
|
prune: false # <-- TEMPORARY; see block above. Normal value: true (R-GITOPS-03)
|
|
wait: true # block `apps` until platform is healthy
|
|
# ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
|
|
#
|
|
# `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
|
|
# `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
|
|
# `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
|
|
# NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
|
|
# That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
|
|
#
|
|
# `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
|
|
# kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
|
|
# when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
|
|
# revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
|
|
# writes a newly-added `cluster-vars` knob — so this one edge closes the race.
|
|
#
|
|
# `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
|
|
# consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
|
|
# velero) chains transitively through it. One edge therefore protects the whole graph.
|
|
#
|
|
# TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
|
|
# ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
|
|
# Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
|
|
# the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
|
|
# entrypoint should stop the line, not let consumers render against a half-applied variable set.
|
|
# No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
|
|
# health-gates its own children (see clusters/dev/sync.yaml).
|
|
dependsOn:
|
|
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
|
# THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
|
|
# (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
|
|
# suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
|
|
# nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
|
|
# is healthy. It is also revision-gated, which is what makes the prune-safety argument above
|
|
# deterministic rather than a race.
|
|
- name: cilium
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: flux-system
|
|
path: ./platform
|
|
# R-ENV-02: the ONLY place environment differences enter platform manifests.
|
|
# dev/prod overlays patch the cluster-vars ConfigMap; the base manifests are never forked.
|
|
postBuild:
|
|
substituteFrom:
|
|
- kind: ConfigMap
|
|
name: cluster-vars
|
|
# TODO(WS-SEC, Phase 1): enable SOPS decryption once the age key exists (ADR-0007).
|
|
# decryption:
|
|
# provider: sops
|
|
# secretRef:
|
|
# name: sops-age
|
|
# TODO(PM): the top-level ./platform/kustomization.yaml must include platform/kyverno/
|
|
# (owned by the policy engineer, WS-TEST). See platform/README.md integration TODO.
|