jakman85andClaude Opus 5
765ad032db
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
58 lines
5.0 KiB
YAML
58 lines
5.0 KiB
YAML
---
|
|
# Base Flux entrypoint set. Shared by BOTH environment overlays (clusters/dev, clusters/prod).
|
|
# Overlays reference this base and patch ONLY the cluster-vars ConfigMap (R-ENV-02).
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- flux-system.yaml # GitRepository (reconciliation source)
|
|
# Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
|
|
# of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
|
|
# Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
|
|
# `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
|
|
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
|
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
|
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
|
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
|
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
|
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
|
# manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
|
|
# datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
|
|
# platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
|
|
# keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
|
|
# CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
|
|
# cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
|
|
# as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
|
|
- platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
|
|
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
|
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
|
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
|
# GENERATED by `catalog/render.py --emit-waves`; add a line here only when a NEW deepest wave
|
|
# is introduced (a service deeper than any existing wave). See DESIGN-waves.md.
|
|
- apps-wave-0.yaml # Flux Kustomization -> ./apps/wave-0 (dependsOn platform-config)
|
|
- apps-wave-1.yaml # Flux Kustomization -> ./apps/wave-1 (dependsOn apps-wave-0)
|
|
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
|
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
|
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
|
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
|
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
|
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
|
- oauth2-proxy.yaml # Flux Kustomization -> ./platform/oauth2-proxy (dependsOn platform-config)
|
|
# headlamp dashboard — POST-platform tier (dependsOn platform-config, wait:false). A leaf UI whose
|
|
# HelmRelease can crashloop (upstream headlamp-0.40.1 `-session-ttl` chart/image desync) must NOT
|
|
# sit in the gating `platform` tier, where its unhealth stalled platform-config + oauth2-proxy +
|
|
# every apps wave/vm on "dependency not ready" (DECISIONS-LOG: "Headlamp gating stall").
|
|
- headlamp.yaml # Flux Kustomization -> ./platform/headlamp (dependsOn platform-config)
|
|
# Homepage apex landing dashboard — POST-platform tier (dependsOn platform-config, wait:false).
|
|
# A leaf UI at home.keithjackson.dev; same non-gating rationale as headlamp/oauth2-proxy (a UI must
|
|
# never gate the platform chain). Carries no SOPS secret, so clusters/dev adds no decryption patch.
|
|
- homepage.yaml # Flux Kustomization -> ./platform/homepage (dependsOn platform-config)
|
|
# Crafty Controller — Minecraft management UI, web UI Keycloak-gated by a per-pod oauth2-proxy
|
|
# reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
|
|
# non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
|
|
- crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
|
|
# Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
|
|
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
|
|
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
|
|
- backup.yaml # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
|
|
- cluster-vars.yaml # overlay knob surface (base defaults)
|