jakman85andClaude Opus 5
1b056dc9d9
`platform-config` stopped dead on prod:
StorageProfile/longhorn dry-run failed: no matches for kind
"StorageProfile" in version "cdi.kubevirt.io/v1beta1"
A TRUE DEADLOCK, not a transient. platform-config applies the CDI CR *and*
platform/kubevirt/cdi-storageprofiles.yaml in the SAME pass. StorageProfile's
CRD does not exist until cdi-operator has reconciled that CR — and Flux
server-side dry-runs EVERY object before applying ANY of them, so the unknown
kind aborts the whole tier, the CR is never applied, the CRD is never created,
and platform-config can never converge. Every tier below it stays blocked.
Same root cause as the Kyverno CRD bundle, different operator: Flux cannot order
a resource against a CRD that one of its own siblings is responsible for
creating. It is INVISIBLE on an established cluster — dev's CDI CRDs have been
there for months — and fatal on a cold one. Observed on prod: cdi-operator
Available for 12h, no CDI CR, no storageprofiles CRD.
`crds` now breaks it by applying ONLY the CDI CR — the minimum — and leaving
every other object to Flux. The CR carries no ${} tokens, so applying it here
cannot diverge from what Flux would render. It refuses if the `cdi` namespace is
absent (platform has not run yet) rather than applying into nothing, waits for
the CRD, and requires it to reach Established. On prod the CRD appeared in 10s
and platform-config was unblocked.
`preflight` now reports the CRDs that later tiers apply resources against
(storageprofiles/cdis/kubevirts/volumesnapshotclasses) and dumps the cdi
namespace when one is missing, so this is caught in seconds instead of after
`resume` burns its 900s wait discovering it. It also names the stale-discovery-
cache variant, which presents identically but clears on a forced reconcile.
THE DURABLE FIX IS A REPO CHANGE, deliberately NOT made here: move
cdi-storageprofiles.yaml out of platform-config into its own Kustomization with
dependsOn: platform-config, so the CR lands one reconcile before the resources
needing its CRDs. That edits the reconciliation graph both clusters share, and
prod currently reads a pinned revision from the bootstrap Gitea while dev is
mid-migration on a separate branch — so it wants doing deliberately, not at
02:00 to unblock a bring-up. Until it lands, C2 is a prerequisite for C5 on any
cold cluster, and PROD-STANDUP.md says so.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
bootstrap/ — Flux + SOPS age-key bootstrap (Phase 1 T-1.4)
sops/: age key generation + Flux SOPS decryption wiring (private key NEVER committed).
flux/: flux bootstrap config pointing at clusters/dev. Spec: ADR-0004, ADR-0007.