jakman85andClaude Opus 5
765ad032db
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
136 lines
9.5 KiB
YAML
136 lines
9.5 KiB
YAML
---
|
|
# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
|
|
# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
|
|
#
|
|
# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
|
|
#
|
|
# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
|
|
# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
|
|
# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
|
|
# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
|
|
# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
|
|
# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
|
|
# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
|
|
#
|
|
# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
|
|
# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
|
|
# graph order itself, so a cold cluster needs no human in the loop.
|
|
#
|
|
# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
|
|
# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
|
|
# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
|
|
# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
|
|
# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
|
|
# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
|
|
# kind. One edge, no pre-apply.
|
|
#
|
|
# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
|
|
# and everything else waits on: putting CDI's health on that critical path is the mistake the
|
|
# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
|
|
# StorageProfiles that were in the wrong pass.
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform-config-cdi
|
|
namespace: flux-system
|
|
spec:
|
|
interval: 10m
|
|
# 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
|
|
# very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
|
|
# now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
|
|
# about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
|
|
# cache is one retryInterval. Keep it short.
|
|
retryInterval: 2m
|
|
# 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
|
|
# image to pull and no controller to become Available, so a slow apply here means a real error, not
|
|
# a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
|
|
timeout: 2m
|
|
# TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
|
|
# component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
|
|
# It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
|
|
# second path also declares. See the transition note at the bottom of this file.
|
|
prune: true
|
|
# TRUE, but understand exactly what it buys and what it does NOT.
|
|
#
|
|
# CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
|
|
# provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
|
|
# `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
|
|
# it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
|
|
# is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
|
|
# If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
|
|
# `apps-vms` waits on it.
|
|
#
|
|
# WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
|
|
# provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
|
|
# storageprofile-controller computes that status from our spec override plus inference. So Ready
|
|
# here means "the spec override is accepted by the API server", not "CDI has folded it into
|
|
# status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
|
|
# below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
|
|
# a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
|
|
wait: true
|
|
dependsOn:
|
|
# platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
|
|
# causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
|
|
# ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
|
|
# flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
|
|
- name: platform-config
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: flux-system
|
|
path: ./platform-config-cdi
|
|
# NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
|
|
# no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
|
|
# oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
|
|
# clusters/dev or clusters/prod for this tier.
|
|
#
|
|
# substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
|
|
# substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
|
|
# 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
|
|
# whose Kustomization declares substituteFrom. Omitting it would move
|
|
# cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
|
|
# could ship as a literal string, precisely the class of bug that check exists to stop.
|
|
# 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
|
|
# with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
|
|
# empty. Without the stanza there is no substitution at all and the literal ships silently.
|
|
postBuild:
|
|
substituteFrom:
|
|
- kind: ConfigMap
|
|
name: cluster-vars
|
|
# ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
|
|
# Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
|
|
# Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
|
|
# StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
|
|
# the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
|
|
# dual-ownership fight — the objects are pruned before they are re-applied under this tier's
|
|
# `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
|
|
#
|
|
# Why that is acceptable here — and it is NOT the platform/cilium case:
|
|
# * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
|
|
# nothing else in the repo declares a StorageProfile. The stale set is exactly two
|
|
# cluster-scoped CDI CRs.
|
|
# * The objects do not actually go missing: CDI's storageprofile-controller creates a
|
|
# StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
|
|
# inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
|
|
# absent".
|
|
# * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
|
|
# existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
|
|
# is disturbed and no data path is touched.
|
|
# * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
|
|
# instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
|
|
# permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
|
|
# dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
|
|
# created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
|
|
# could.
|
|
#
|
|
# PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
|
|
# happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
|
|
# config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
|
|
# healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
|
|
# --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
|
|
# StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
|
|
# `flux get kustomization platform-config` must be Ready BEFORE merging.
|
|
# If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
|
|
# temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
|
|
# this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
|
|
# `prune: true` in a second commit.
|