jakman85andClaude Opus 5
f21e89d9d2
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
42 lines
3.0 KiB
YAML
42 lines
3.0 KiB
YAML
---
|
|
# platform-config-cdi — the SECOND-ORDER config wave: custom resources whose CRD is created at
|
|
# RUNTIME by a controller that a `platform-config` custom resource switches on.
|
|
#
|
|
# ── THE BUG THIS DIRECTORY EXISTS TO FIX (observed on PROD, 2026-09-05) ───────────────────────────
|
|
# The repo already splits CONTROLLERS (`platform`) from CONFIGS (`platform-config`) because a CR
|
|
# cannot be dry-run before its CRD is registered. That split assumes a CRD is registered by the
|
|
# controller INSTALL. CDI breaks the assumption: `cdi-operator.yaml` registers exactly ONE CRD
|
|
# (`cdis.cdi.kubevirt.io`). Every OTHER CDI CRD — `storageprofiles`, `datavolumes`, `datasources`,
|
|
# `dataimportcrons`, … — is created by cdi-operator only AFTER it has reconciled the `CDI` CR.
|
|
#
|
|
# So `cdi-cr.yaml` is a FIRST-order CR (its CRD comes from the install) and belongs in
|
|
# platform-config, while `cdi-storageprofiles.yaml` is a SECOND-order CR (its CRD comes from
|
|
# reconciling the first-order CR). Having both in ONE Flux Kustomization is a true DEADLOCK, not a
|
|
# race: kustomize-controller server-side dry-runs EVERY object before applying ANY of them, so the
|
|
# unknown `StorageProfile` kind aborts the whole pass → the `CDI` CR is never applied → the
|
|
# StorageProfile CRD is never created → platform-config can NEVER converge, and every tier that
|
|
# dependsOn it stays blocked forever. On prod, cdi-operator sat Available for 12h with no CR:
|
|
#
|
|
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile"
|
|
# in version "cdi.kubevirt.io/v1beta1"
|
|
#
|
|
# Invisible on dev (whose CDI CRDs have existed for months) and fatal on any COLD cluster.
|
|
#
|
|
# ── THE RULE, so this is not rediscovered a third time ────────────────────────────────────────────
|
|
# A CR belongs HERE (not in platform-config) when the CRD it needs is created by a CONTROLLER AT
|
|
# RUNTIME rather than by a manifest in git. Grep test: if `grep -rn "kind: CustomResourceDefinition"`
|
|
# over ./platform (following remote bases) does not produce the CRD, and no chart installs it, then
|
|
# some operator creates it after being switched on — and this is the tier for it.
|
|
#
|
|
# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
|
|
# manifests stay beside the component they configure). Build/verify locally with:
|
|
# kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config-cdi
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
# CDI StorageProfiles — snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
|
|
# `longhorn-vm` virt-default class (clears CDIDefaultStorageClassDegraded). `StorageProfile`'s CRD
|
|
# is created by cdi-operator only once the `CDI` CR (applied by platform-config) is reconciled,
|
|
# which is exactly why these cannot live in the same pass as that CR.
|
|
- ../platform/kubevirt/cdi-storageprofiles.yaml
|