jakman85andClaude Opus 5
63f98aab26
Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.
New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.
*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.
TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
moving resources between Kustomizations. Chosen over the per-resource
`kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
say name=platform. The new cilium Kustomization relabels all three to name=cilium on
apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".
NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.
Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.
MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
61 lines
3.5 KiB
YAML
61 lines
3.5 KiB
YAML
---
|
|
# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
|
|
# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
|
|
#
|
|
# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
|
|
#
|
|
# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
|
|
# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
|
|
# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
|
|
# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
|
|
#
|
|
# * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
|
|
# * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
|
|
# tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
|
|
# `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
|
|
# the valve the 2026-09-01 substitution incident wanted and did not have.
|
|
# * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
|
|
# `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
|
|
#
|
|
# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
|
|
# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
|
|
# expressed as an explicit edge instead of an accident of co-tenancy.
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: cilium
|
|
namespace: flux-system
|
|
spec:
|
|
# A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
|
|
# kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
|
|
interval: 1h
|
|
retryInterval: 2m
|
|
# 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
|
|
# rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
|
|
timeout: 10m
|
|
prune: true
|
|
# TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
|
|
wait: true
|
|
# Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
|
|
# by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
|
|
# is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
|
|
# revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
|
|
# `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
|
|
dependsOn:
|
|
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: flux-system
|
|
path: ./platform/cilium
|
|
# MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
|
|
# kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
|
|
# this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
|
|
# renders empty). That is the intended fail-closed behaviour — but it means the stanza is
|
|
# load-bearing for the CNI coming up at all. Do not remove it.
|
|
postBuild:
|
|
substituteFrom:
|
|
- kind: ConfigMap
|
|
name: cluster-vars
|
|
# No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
|
|
# oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).
|