fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
2d7d302c7e
commit
f21e89d9d2
10 files changed
+284
-13
No files matched your search
@@ -97,6 +97,27 @@ Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
|
|||||||
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
|
||||||
|
|
||||||
|
The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
|
||||||
|
controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
|
||||||
|
and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
|
||||||
|
reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
|
||||||
|
before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
|
||||||
|
**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
|
||||||
|
`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
|
||||||
|
have existed for months); it cost 12h on prod's cold start.
|
||||||
|
|
||||||
|
So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
|
||||||
|
platform-config`), and a rule:
|
||||||
|
|
||||||
|
> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
|
||||||
|
> **controller at runtime** rather than by a manifest in git.
|
||||||
|
|
||||||
|
`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
|
||||||
|
`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
|
||||||
|
profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
||||||
|
|
||||||
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
||||||
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
||||||
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
||||||
|
|||||||
@@ -4,8 +4,11 @@
|
|||||||
# NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
|
# NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
|
||||||
# import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
|
# import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
|
||||||
# health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
|
# health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
|
||||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
|
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
|
||||||
# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
|
# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
|
||||||
|
# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
|
||||||
|
# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
|
||||||
|
# adoption concern).
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
metadata:
|
metadata:
|
||||||
@@ -19,6 +22,28 @@ spec:
|
|||||||
wait: false
|
wait: false
|
||||||
dependsOn:
|
dependsOn:
|
||||||
- name: platform-config
|
- name: platform-config
|
||||||
|
# LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
|
||||||
|
# storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
|
||||||
|
# decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
|
||||||
|
# (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
|
||||||
|
# live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
|
||||||
|
# with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
|
||||||
|
# to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
|
||||||
|
# conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
|
||||||
|
#
|
||||||
|
# HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
|
||||||
|
# derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
|
||||||
|
# status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
|
||||||
|
# it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
|
||||||
|
#
|
||||||
|
# This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
|
||||||
|
# deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
|
||||||
|
# dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
|
||||||
|
#
|
||||||
|
# The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
|
||||||
|
# edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
|
||||||
|
# cannot silently drop this tier's KubeVirt/StorageClass gate.
|
||||||
|
- name: platform-config-cdi
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
|
|||||||
@@ -14,6 +14,15 @@ resources:
|
|||||||
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
||||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||||
|
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
||||||
|
# manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
|
||||||
|
# datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
|
||||||
|
# platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
|
||||||
|
# keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
|
||||||
|
# CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
|
||||||
|
# cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
|
||||||
|
# as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
|
||||||
|
- platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
|
||||||
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
||||||
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
||||||
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
||||||
|
|||||||
@@ -0,0 +1,135 @@
|
|||||||
|
---
|
||||||
|
# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
|
||||||
|
# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
|
||||||
|
#
|
||||||
|
# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
|
||||||
|
#
|
||||||
|
# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
|
||||||
|
# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
|
||||||
|
# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
|
||||||
|
# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
|
||||||
|
# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
|
||||||
|
# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
|
||||||
|
# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
|
||||||
|
#
|
||||||
|
# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
|
||||||
|
# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
|
||||||
|
# graph order itself, so a cold cluster needs no human in the loop.
|
||||||
|
#
|
||||||
|
# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
|
||||||
|
# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
|
||||||
|
# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
|
||||||
|
# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
|
||||||
|
# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
|
||||||
|
# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
|
||||||
|
# kind. One edge, no pre-apply.
|
||||||
|
#
|
||||||
|
# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
|
||||||
|
# and everything else waits on: putting CDI's health on that critical path is the mistake the
|
||||||
|
# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
|
||||||
|
# StorageProfiles that were in the wrong pass.
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: platform-config-cdi
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
interval: 10m
|
||||||
|
# 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
|
||||||
|
# very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
|
||||||
|
# now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
|
||||||
|
# about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
|
||||||
|
# cache is one retryInterval. Keep it short.
|
||||||
|
retryInterval: 2m
|
||||||
|
# 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
|
||||||
|
# image to pull and no controller to become Available, so a slow apply here means a real error, not
|
||||||
|
# a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
|
||||||
|
timeout: 2m
|
||||||
|
# TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
|
||||||
|
# component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
|
||||||
|
# It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
|
||||||
|
# second path also declares. See the transition note at the bottom of this file.
|
||||||
|
prune: true
|
||||||
|
# TRUE, but understand exactly what it buys and what it does NOT.
|
||||||
|
#
|
||||||
|
# CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
|
||||||
|
# provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
|
||||||
|
# `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
|
||||||
|
# it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
|
||||||
|
# is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
|
||||||
|
# If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
|
||||||
|
# `apps-vms` waits on it.
|
||||||
|
#
|
||||||
|
# WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
|
||||||
|
# provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
|
||||||
|
# storageprofile-controller computes that status from our spec override plus inference. So Ready
|
||||||
|
# here means "the spec override is accepted by the API server", not "CDI has folded it into
|
||||||
|
# status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
|
||||||
|
# below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
|
||||||
|
# a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
|
||||||
|
wait: true
|
||||||
|
dependsOn:
|
||||||
|
# platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
|
||||||
|
# causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
|
||||||
|
# ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
|
||||||
|
# flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
|
||||||
|
- name: platform-config
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
path: ./platform-config-cdi
|
||||||
|
# NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
|
||||||
|
# no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
|
||||||
|
# oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
|
||||||
|
# clusters/dev or clusters/prod for this tier.
|
||||||
|
#
|
||||||
|
# substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
|
||||||
|
# substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
|
||||||
|
# 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
|
||||||
|
# whose Kustomization declares substituteFrom. Omitting it would move
|
||||||
|
# cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
|
||||||
|
# could ship as a literal string, precisely the class of bug that check exists to stop.
|
||||||
|
# 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
|
||||||
|
# with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
|
||||||
|
# empty. Without the stanza there is no substitution at all and the literal ships silently.
|
||||||
|
postBuild:
|
||||||
|
substituteFrom:
|
||||||
|
- kind: ConfigMap
|
||||||
|
name: cluster-vars
|
||||||
|
# ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
|
||||||
|
# Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
|
||||||
|
# Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
|
||||||
|
# StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
|
||||||
|
# the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
|
||||||
|
# dual-ownership fight — the objects are pruned before they are re-applied under this tier's
|
||||||
|
# `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
|
||||||
|
#
|
||||||
|
# Why that is acceptable here — and it is NOT the platform/cilium case:
|
||||||
|
# * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
|
||||||
|
# nothing else in the repo declares a StorageProfile. The stale set is exactly two
|
||||||
|
# cluster-scoped CDI CRs.
|
||||||
|
# * The objects do not actually go missing: CDI's storageprofile-controller creates a
|
||||||
|
# StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
|
||||||
|
# inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
|
||||||
|
# absent".
|
||||||
|
# * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
|
||||||
|
# existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
|
||||||
|
# is disturbed and no data path is touched.
|
||||||
|
# * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
|
||||||
|
# instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
|
||||||
|
# permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
|
||||||
|
# dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
|
||||||
|
# created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
|
||||||
|
# could.
|
||||||
|
#
|
||||||
|
# PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
|
||||||
|
# happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
|
||||||
|
# config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
|
||||||
|
# healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
|
||||||
|
# --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
|
||||||
|
# StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
|
||||||
|
# `flux get kustomization platform-config` must be Ready BEFORE merging.
|
||||||
|
# If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
|
||||||
|
# temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
|
||||||
|
# this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
|
||||||
|
# `prune: true` in a second commit.
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
---
|
||||||
|
# platform-config-cdi — the SECOND-ORDER config wave: custom resources whose CRD is created at
|
||||||
|
# RUNTIME by a controller that a `platform-config` custom resource switches on.
|
||||||
|
#
|
||||||
|
# ── THE BUG THIS DIRECTORY EXISTS TO FIX (observed on PROD, 2026-09-05) ───────────────────────────
|
||||||
|
# The repo already splits CONTROLLERS (`platform`) from CONFIGS (`platform-config`) because a CR
|
||||||
|
# cannot be dry-run before its CRD is registered. That split assumes a CRD is registered by the
|
||||||
|
# controller INSTALL. CDI breaks the assumption: `cdi-operator.yaml` registers exactly ONE CRD
|
||||||
|
# (`cdis.cdi.kubevirt.io`). Every OTHER CDI CRD — `storageprofiles`, `datavolumes`, `datasources`,
|
||||||
|
# `dataimportcrons`, … — is created by cdi-operator only AFTER it has reconciled the `CDI` CR.
|
||||||
|
#
|
||||||
|
# So `cdi-cr.yaml` is a FIRST-order CR (its CRD comes from the install) and belongs in
|
||||||
|
# platform-config, while `cdi-storageprofiles.yaml` is a SECOND-order CR (its CRD comes from
|
||||||
|
# reconciling the first-order CR). Having both in ONE Flux Kustomization is a true DEADLOCK, not a
|
||||||
|
# race: kustomize-controller server-side dry-runs EVERY object before applying ANY of them, so the
|
||||||
|
# unknown `StorageProfile` kind aborts the whole pass → the `CDI` CR is never applied → the
|
||||||
|
# StorageProfile CRD is never created → platform-config can NEVER converge, and every tier that
|
||||||
|
# dependsOn it stays blocked forever. On prod, cdi-operator sat Available for 12h with no CR:
|
||||||
|
#
|
||||||
|
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile"
|
||||||
|
# in version "cdi.kubevirt.io/v1beta1"
|
||||||
|
#
|
||||||
|
# Invisible on dev (whose CDI CRDs have existed for months) and fatal on any COLD cluster.
|
||||||
|
#
|
||||||
|
# ── THE RULE, so this is not rediscovered a third time ────────────────────────────────────────────
|
||||||
|
# A CR belongs HERE (not in platform-config) when the CRD it needs is created by a CONTROLLER AT
|
||||||
|
# RUNTIME rather than by a manifest in git. Grep test: if `grep -rn "kind: CustomResourceDefinition"`
|
||||||
|
# over ./platform (following remote bases) does not produce the CRD, and no chart installs it, then
|
||||||
|
# some operator creates it after being switched on — and this is the tier for it.
|
||||||
|
#
|
||||||
|
# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
|
||||||
|
# manifests stay beside the component they configure). Build/verify locally with:
|
||||||
|
# kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config-cdi
|
||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
# CDI StorageProfiles — snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
|
||||||
|
# `longhorn-vm` virt-default class (clears CDIDefaultStorageClassDegraded). `StorageProfile`'s CRD
|
||||||
|
# is created by cdi-operator only once the `CDI` CR (applied by platform-config) is reconciled,
|
||||||
|
# which is exactly why these cannot live in the same pass as that CR.
|
||||||
|
- ../platform/kubevirt/cdi-storageprofiles.yaml
|
||||||
@@ -41,9 +41,23 @@ resources:
|
|||||||
# ConfigMap kept beside them for locality.
|
# ConfigMap kept beside them for locality.
|
||||||
- ../platform/kubevirt/kubevirt-cr.yaml
|
- ../platform/kubevirt/kubevirt-cr.yaml
|
||||||
- ../platform/kubevirt/cdi-cr.yaml
|
- ../platform/kubevirt/cdi-cr.yaml
|
||||||
# CDI StorageProfiles: snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
|
# *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
|
||||||
# `longhorn-vm` virt-default class (consume cdi.kubevirt.io; clears CDIDefaultStorageClassDegraded).
|
# The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
|
||||||
- ../platform/kubevirt/cdi-storageprofiles.yaml
|
# (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
|
||||||
|
#
|
||||||
|
# `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
|
||||||
|
# `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
|
||||||
|
# only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
|
||||||
|
# in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
|
||||||
|
# `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
|
||||||
|
# created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
|
||||||
|
# A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
|
||||||
|
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
|
||||||
|
# "cdi.kubevirt.io/v1beta1"
|
||||||
|
# Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
|
||||||
|
#
|
||||||
|
# THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
|
||||||
|
# is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
|
||||||
- ../platform/kubevirt/servicemonitor.yaml
|
- ../platform/kubevirt/servicemonitor.yaml
|
||||||
- ../platform/kubevirt/alerts.yaml
|
- ../platform/kubevirt/alerts.yaml
|
||||||
- ../platform/kubevirt/dashboard.yaml
|
- ../platform/kubevirt/dashboard.yaml
|
||||||
|
|||||||
@@ -20,8 +20,12 @@ Neither k3s nor the Longhorn chart ships this. It is a prerequisite for:
|
|||||||
## Wiring
|
## Wiring
|
||||||
|
|
||||||
- **Pass:** `platform` (controllers). Registers the CRDs.
|
- **Pass:** `platform` (controllers). Registers the CRDs.
|
||||||
- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the
|
- The `VolumeSnapshotClass` CR that consumes the CRDs lives in the `platform-config` pass
|
||||||
`platform-config` pass (`dependsOn: platform`), per the controllers/configs split.
|
(`dependsOn: platform`), per the controllers/configs split.
|
||||||
|
- The CDI `StorageProfile`s live one pass further out, in `platform-config-cdi`
|
||||||
|
(`dependsOn: platform-config`) — `storageprofiles.cdi.kubevirt.io` is created by cdi-operator only
|
||||||
|
after it reconciles the `CDI` CR, so they cannot share a pass with it. See
|
||||||
|
`clusters/base/platform-config-cdi.yaml`.
|
||||||
|
|
||||||
## Version
|
## Version
|
||||||
|
|
||||||
|
|||||||
@@ -9,8 +9,10 @@
|
|||||||
# * no VolumeSnapshotClass can be created (item 2), and
|
# * no VolumeSnapshotClass can be created (item 2), and
|
||||||
# * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
|
# * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
|
||||||
# which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
|
# which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
|
||||||
# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI
|
# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass that CONSUMES them
|
||||||
# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform).
|
# is applied by platform-config (dependsOn: platform), and the CDI StorageProfiles that set
|
||||||
|
# `cloneStrategy: snapshot` by platform-config-cdi (dependsOn: platform-config — their own CRD is
|
||||||
|
# created by cdi-operator only after the `CDI` CR reconciles; see clusters/base/platform-config-cdi.yaml).
|
||||||
#
|
#
|
||||||
# CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
|
# CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
|
||||||
# CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment
|
# CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment
|
||||||
|
|||||||
@@ -1,9 +1,19 @@
|
|||||||
---
|
---
|
||||||
# CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs
|
# CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs
|
||||||
# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so —
|
# (ADR-0005).
|
||||||
# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization
|
#
|
||||||
# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in
|
# APPLIED BY `platform-config-cdi`, NOT `platform-config` (moved 2026-09-05). These are SECOND-ORDER
|
||||||
# place from platform-config/kustomization.yaml.
|
# CRs: `storageprofiles.cdi.kubevirt.io` is NOT registered by the CDI operator install (cdi-operator.yaml
|
||||||
|
# ships only `cdis.cdi.kubevirt.io`) — cdi-operator creates it only AFTER it has reconciled the `CDI` CR
|
||||||
|
# that platform-config applies. Flux server-side dry-runs every object in a pass before applying any of
|
||||||
|
# them, so co-locating these with that CR was a TRUE DEADLOCK: unknown kind -> pass aborts -> CR never
|
||||||
|
# applied -> CRD never created -> platform-config never converges -> every tier below it blocked. It was
|
||||||
|
# invisible on dev (CDI CRDs long present) and cost 12h on prod's cold start, 2026-09-05.
|
||||||
|
# They live here beside the KubeVirt/CDI component and are referenced in place from
|
||||||
|
# platform-config-cdi/kustomization.yaml (dependsOn: platform-config, revision-gated).
|
||||||
|
#
|
||||||
|
# `apps-vms` dependsOn platform-config-cdi for these: a DataVolume created before the longhorn-vm
|
||||||
|
# profile below exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
||||||
#
|
#
|
||||||
# CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities).
|
# CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities).
|
||||||
# The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it
|
# The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it
|
||||||
|
|||||||
@@ -8,6 +8,16 @@
|
|||||||
# (dependsOn: platform, wait) so the CRDs exist BEFORE the CRs apply. Applying both in one atomic pass
|
# (dependsOn: platform, wait) so the CRDs exist BEFORE the CRs apply. Applying both in one atomic pass
|
||||||
# deadlocks ("no matches for kind KubeVirt"), exactly the failure F-MON1 documents.
|
# deadlocks ("no matches for kind KubeVirt"), exactly the failure F-MON1 documents.
|
||||||
#
|
#
|
||||||
|
# THERE IS A THIRD PASS FOR CDI, and it is not optional (2026-09-05). The two operator manifests below
|
||||||
|
# register exactly ONE CDI CRD, `cdis.cdi.kubevirt.io`. Every other CDI CRD — `storageprofiles`,
|
||||||
|
# `datavolumes`, `datasources`, `dataimportcrons` — is created by cdi-operator only AFTER it reconciles
|
||||||
|
# the `CDI` CR. So `cdi-storageprofiles.yaml` is a SECOND-ORDER CR and CANNOT share a pass with
|
||||||
|
# `cdi-cr.yaml`: the dry-run of the unknown `StorageProfile` kind aborts the pass, the CR is never
|
||||||
|
# applied, the CRD is never created, and platform-config deadlocks (cost 12h on prod's cold start).
|
||||||
|
# It is therefore applied by a third Kustomization, `platform-config-cdi` (dependsOn platform-config).
|
||||||
|
# Before adding any new CDI CR here, ask which pass owns it: CRD in git -> platform-config;
|
||||||
|
# CRD created by cdi-operator at runtime -> platform-config-cdi.
|
||||||
|
#
|
||||||
# Namespaces: the operator manifests each create their own namespace (`kubevirt`, `cdi`) with the
|
# Namespaces: the operator manifests each create their own namespace (`kubevirt`, `cdi`) with the
|
||||||
# pod-security profile they need (kubevirt is privileged) — so we do NOT redeclare them here (that
|
# pod-security profile they need (kubevirt is privileged) — so we do NOT redeclare them here (that
|
||||||
# would be a duplicate-resource kustomize error). These are PLATFORM namespaces: they intentionally
|
# would be a duplicate-resource kustomize error). These are PLATFORM namespaces: they intentionally
|
||||||
|
|||||||
Reference in new issue
Block a user