fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
32d67660a6
commit
765ad032db
11 files changed
+330
-17
No files matched your search
+46
-4
@@ -141,10 +141,52 @@ before applying any of them, so one unknown `kind` aborts the entire tier.**
|
||||
Flux. It carries no `${}` tokens, so applying it here cannot diverge from what Flux would render.
|
||||
The CRD appeared in 10s.
|
||||
|
||||
**The durable fix is a repo change, not this script:** move `cdi-storageprofiles.yaml` out of
|
||||
`platform-config` into its own Kustomization with `dependsOn: platform-config`, so the CR lands one
|
||||
reconcile before the resources that need its CRDs. Until that lands, C2 is a prerequisite for C5 on
|
||||
any cold cluster.
|
||||
**REPAIRED IN THE GRAPH (2026-09-05).** `cdi-storageprofiles.yaml` now lives in its own tier,
|
||||
`./platform-config-cdi`, with `dependsOn: platform-config` and `wait: true`. Because `dependsOn` is
|
||||
**revision**-gated and `platform-config` runs `wait: true`, the CDI CR must be applied *and healthy*
|
||||
— which is precisely the event that creates the `StorageProfile` CRD — before the profiles are
|
||||
applied. A cold cluster now orders itself. The `crds` step is retained as a fallback for a cluster
|
||||
already stuck in the broken state, and because it is idempotent.
|
||||
|
||||
**The generalisable rule, now recorded in `clusters/README.md`:** a CR whose CRD is created by a
|
||||
*controller at runtime* is **second-order** and needs its own pass. `cdi-operator.yaml` registers
|
||||
exactly one CRD (`cdis.cdi.kubevirt.io`); `storageprofiles`, `datavolumes`, `datasources` and
|
||||
`dataimportcrons` appear only after cdi-operator reconciles the CDI CR. An audit of
|
||||
`platform-config` found `StorageProfile` is the only second-order kind there.
|
||||
|
||||
`apps-vms` gained `dependsOn: platform-config-cdi` — load-bearing, not cosmetic:
|
||||
`apps/vms/unifi/20-datavolume.yaml` provisions on `storageClassName: longhorn-vm`, and its comment
|
||||
previously claimed `platform-config` guaranteed the profiles, which the move would have silently
|
||||
falsified.
|
||||
|
||||
#### Merge precondition and the one-time transient
|
||||
|
||||
`platform-config` has `prune: true`, so on the first reconcile after merge it **deletes** both
|
||||
StorageProfiles, CDI recreates them within seconds with an empty spec, and `platform-config-cdi`
|
||||
re-applies the overrides ~30–60s later (kustomize-controller's dependency requeue).
|
||||
|
||||
**Measured on dev, so the exposure is known rather than assumed** — comparing our override against
|
||||
what CDI infers unaided for the same Longhorn provisioner:
|
||||
|
||||
| profile | `spec` (ours) | CDI-derived `status` |
|
||||
|---|---|---|
|
||||
| `longhorn-vm` (we override) | RWX / Block, `snapshot` | **ReadWriteMany** / Block, `snapshot` |
|
||||
| `longhorn-static` (no override) | *unset* | **ReadWriteOnce** / Block, `snapshot` |
|
||||
|
||||
So **`cloneStrategy` is not at risk** — CDI infers `snapshot` for Longhorn anyway. **`accessModes`
|
||||
is:** unaided, CDI produces **RWO**. A DataVolume provisioned on `longhorn-vm` *inside the window*
|
||||
would be stamped RWO/Block and be **permanently non-live-migratable**. Flux can no longer cause that
|
||||
(`apps-vms` now gates on the new tier); **do not start a VM disk import, clone or DataVolume by hand
|
||||
during the window.**
|
||||
|
||||
- [ ] **Before merging, confirm `flux get kustomization platform-config` is Ready on the target.** If
|
||||
it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in
|
||||
exactly that state hours earlier. The fallback is the two-step the Cilium move used: merge with
|
||||
`platform-config.prune: false`, confirm the ownership label transferred, restore `prune: true` in
|
||||
a second commit.
|
||||
- [ ] After: both profiles carry `spec.cloneStrategy: snapshot`, `longhorn-vm` carries
|
||||
`claimPropertySets` RWX/Block, and both are labelled
|
||||
`kustomize.toolkit.fluxcd.io/name=platform-config-cdi`.
|
||||
|
||||
### C3 — unwedge stuck Helm releases
|
||||
|
||||
|
||||
Reference in new issue
Block a user