jakman85andClaude Opus 5
63f98aab26
Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.
New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.
*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.
TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
moving resources between Kustomizations. Chosen over the per-resource
`kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
say name=platform. The new cilium Kustomization relabels all three to name=cilium on
apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".
NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.
Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.
MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
56 lines
4.7 KiB
YAML
56 lines
4.7 KiB
YAML
---
|
|
# Top-level platform kustomization — the entrypoint the Flux `platform` Kustomization renders
|
|
# (clusters/base/platform.yaml, path: ./platform).
|
|
#
|
|
# UNIFIED GRAPH (R-ENV-02): dev and prod both render THIS single list; overlays differ only by
|
|
# cluster-vars. Components are enabled here PER PHASE as their phase opens (roadmap/phases.md) —
|
|
# the same pattern by which kyverno was previously deferred. Credentials each component needs are
|
|
# provisioned upfront as SOPS-encrypted Secrets (bootstrap/sops/, .sops.yaml), so enabling a
|
|
# component is never gated on a credential ask — only on its phase and readiness.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
# ---- Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress ----
|
|
- kyverno # admission enforcer FIRST — policies present before other components reconcile
|
|
# cilium (the CNI) is DELIBERATELY NOT here: it now reconciles in its OWN Flux Kustomization
|
|
# (clusters/base/cilium.yaml, dependsOn flux-system, wait:true), and `platform` dependsOn IT.
|
|
# Not the headlamp/oauth2-proxy rationale (a leaf that must not gate) but its mirror image:
|
|
# the CNI is the one component that SHOULD gate everything, and inside this shared tier its
|
|
# health was indistinguishable from eleven unrelated components, the only emergency valve was
|
|
# freezing the whole tier, and the tier's 10m/5m interval/timeout are wrong for a 3-node
|
|
# DaemonSet rollout. Gating is preserved via the explicit `platform.dependsOn: [cilium]` edge.
|
|
# NOTE: platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system in
|
|
# this tree — that is why the move required a prune-disabled transition. See DECISIONS-LOG.
|
|
- longhorn # replicated block storage / CSI
|
|
- external-snapshotter # CSI snapshot CRDs + snapshot-controller (VolumeSnapshot API; CDI smart-clone)
|
|
- cert-manager # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token`
|
|
- externaldns # auto DNS records — needs SOPS `cloudflare-api-token`
|
|
- gateway # Gateway API listeners (ingress; replaces Traefik)
|
|
- coredns # split-horizon in-cluster DNS: *.home.keithjackson.dev -> Gateway VIP (R-DNS-01/03)
|
|
# ---- Phase 3 — Egress / VPN (ADR-0015): node-level host OpenVPN split-tunnel (Ansible role
|
|
# vpn_egress_gateway) steered by a Cilium Egress Gateway policy, with a host nftables kill-switch.
|
|
# No in-cluster gateway pod (Gluetun retired). See ADR-0015 + DECISIONS-LOG "Phase 3 VPN". ----
|
|
- vpn-egress # CiliumEgressGatewayPolicy steering egress-class=vpn pods to the node tun0 (ADR-0015)
|
|
# ---- Phase 4 — Observability (creds provisioned via SOPS: grafana-admin, alertmanager-smtp) ----
|
|
- monitoring # kube-prometheus-stack + Loki + Alloy + Alertmanager
|
|
# ---- Phase 6 — Identity / SSO (ADR-0014): foundational IdP, onboarded early. Keycloak Operator +
|
|
# CloudNativePG DB. Controllers here; the Keycloak/RealmImport/CNPG Cluster CRs + ServiceMonitors
|
|
# + HTTPRoutes are in platform-config (dependsOn platform). ----
|
|
# oauth2-proxy (forward-auth) is DELIBERATELY NOT here: it depends on Keycloak + trusted TLS +
|
|
# split-horizon DNS, so keeping it in this wait:true tier self-gated its own dependencies into a
|
|
# cold-start deadlock (twice). It now reconciles in its own post-platform Flux Kustomization
|
|
# (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). See DECISIONS-LOG.
|
|
- cnpg # CloudNativePG operator (DB backend for Keycloak) — ghcr.io images (not Bitnami)
|
|
- keycloak # Keycloak Operator (quay.io images) + SOPS admin credential
|
|
# headlamp (Kubernetes dashboard) is DELIBERATELY NOT here: a leaf UI whose HelmRelease can
|
|
# crashloop (upstream headlamp-0.40.1 `-session-ttl` chart/image desync, issue #4883) must not
|
|
# sit in this wait:true tier, where its unhealth stalled platform-config + oauth2-proxy + every
|
|
# apps wave/vm on "dependency not ready". It now reconciles in its own post-platform Flux
|
|
# Kustomization (clusters/base/headlamp.yaml, dependsOn platform-config). See DECISIONS-LOG.
|
|
# ---- Later phases: enabled when their phase opens ----
|
|
# - velero # Phase 4 (Backup/DR): BLOCKED on OQ-BKP (off-cluster backup target decision)
|
|
# - image-automation # Phase 4 (Automated updates): Flux image-automation + Renovate
|
|
# ---- Phase 5 — IaaS proof (KubeVirt): operator CONTROLLERS only; the KubeVirt/CDI CRs +
|
|
# ServiceMonitors/alerts/dashboard are applied by platform-config (dependsOn platform, F-MON1). ----
|
|
- kubevirt # KubeVirt + CDI operators (register kubevirt.io / cdi.kubevirt.io CRDs)
|