jakman85andClaude Opus 5
63f98aab26
Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.
New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.
*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.
TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
moving resources between Kustomizations. Chosen over the per-resource
`kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
say name=platform. The new cilium Kustomization relabels all three to name=cilium on
apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".
NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.
Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.
MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
49 lines
4.2 KiB
YAML
49 lines
4.2 KiB
YAML
---
|
|
# Base Flux entrypoint set. Shared by BOTH environment overlays (clusters/dev, clusters/prod).
|
|
# Overlays reference this base and patch ONLY the cluster-vars ConfigMap (R-ENV-02).
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- flux-system.yaml # GitRepository (reconciliation source)
|
|
# Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
|
|
# of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
|
|
# Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
|
|
# `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
|
|
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
|
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
|
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
|
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
|
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
|
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
|
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
|
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
|
# GENERATED by `catalog/render.py --emit-waves`; add a line here only when a NEW deepest wave
|
|
# is introduced (a service deeper than any existing wave). See DESIGN-waves.md.
|
|
- apps-wave-0.yaml # Flux Kustomization -> ./apps/wave-0 (dependsOn platform-config)
|
|
- apps-wave-1.yaml # Flux Kustomization -> ./apps/wave-1 (dependsOn apps-wave-0)
|
|
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
|
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
|
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
|
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
|
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
|
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
|
- oauth2-proxy.yaml # Flux Kustomization -> ./platform/oauth2-proxy (dependsOn platform-config)
|
|
# headlamp dashboard — POST-platform tier (dependsOn platform-config, wait:false). A leaf UI whose
|
|
# HelmRelease can crashloop (upstream headlamp-0.40.1 `-session-ttl` chart/image desync) must NOT
|
|
# sit in the gating `platform` tier, where its unhealth stalled platform-config + oauth2-proxy +
|
|
# every apps wave/vm on "dependency not ready" (DECISIONS-LOG: "Headlamp gating stall").
|
|
- headlamp.yaml # Flux Kustomization -> ./platform/headlamp (dependsOn platform-config)
|
|
# Homepage apex landing dashboard — POST-platform tier (dependsOn platform-config, wait:false).
|
|
# A leaf UI at home.keithjackson.dev; same non-gating rationale as headlamp/oauth2-proxy (a UI must
|
|
# never gate the platform chain). Carries no SOPS secret, so clusters/dev adds no decryption patch.
|
|
- homepage.yaml # Flux Kustomization -> ./platform/homepage (dependsOn platform-config)
|
|
# Crafty Controller — Minecraft management UI, web UI Keycloak-gated by a per-pod oauth2-proxy
|
|
# reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
|
|
# non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
|
|
- crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
|
|
# Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
|
|
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
|
|
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
|
|
- backup.yaml # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
|
|
- cluster-vars.yaml # overlay knob surface (base defaults)
|