jakman85andClaude Opus 5
f21e89d9d2
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
69 lines
4.1 KiB
YAML
69 lines
4.1 KiB
YAML
---
|
|
# Flux Kustomization: VM tier (KubeVirt VirtualMachines). Reconciles ./apps/vms.
|
|
#
|
|
# NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
|
|
# import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
|
|
# health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
|
|
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
|
|
# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
|
|
# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
|
|
# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
|
|
# adoption concern).
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-vms
|
|
namespace: flux-system
|
|
spec:
|
|
interval: 10m
|
|
retryInterval: 2m
|
|
timeout: 5m
|
|
prune: true
|
|
wait: false
|
|
dependsOn:
|
|
- name: platform-config
|
|
# LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
|
|
# storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
|
|
# decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
|
|
# (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
|
|
# live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
|
|
# with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
|
|
# to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
|
|
# conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
|
|
#
|
|
# HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
|
|
# derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
|
|
# status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
|
|
# it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
|
|
#
|
|
# This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
|
|
# deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
|
|
# dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
|
|
#
|
|
# The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
|
|
# edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
|
|
# cannot silently drop this tier's KubeVirt/StorageClass gate.
|
|
- name: platform-config-cdi
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: flux-system
|
|
path: ./apps/vms
|
|
# postBuild.substituteFrom ENABLED 2026-08-31 (R-ENV-02). It was previously omitted because Flux's
|
|
# envsubst runs over the ENTIRE manifest — including VM cloud-init shell scripts — and would mangle
|
|
# every `${...}`; the omission was safe only while no VM manifest referenced a cluster-var. It now
|
|
# does: apps/vms/unifi/50-service-l4.yaml pins the UniFi L4 VIP from ${UNIFI_L4_VIP} instead of the
|
|
# hardcoded dev-LAN literal that would have shipped 192.168.1.54 onto the PROD LAN.
|
|
#
|
|
# The escape contract the old comment prescribed is now in force and MUST be honoured by every VM
|
|
# manifest under this path (verified with `flux envsubst`):
|
|
# `${VAR}` -> substituted (empty when unset) => escape shell braces as `$${VAR}`, which
|
|
# envsubst emits back as a literal `${VAR}`.
|
|
# `$VAR` -> left alone by envsubst => bare shell vars need NO escaping.
|
|
# `$(cmd)` -> left alone by envsubst => command substitution needs NO escaping.
|
|
# `${ARR[0]}` (e.g. ${PIPESTATUS[0]}) -> HARD ERROR "missing closing brace", which fails this
|
|
# whole Kustomization. NEVER use an indexed expansion here, escaped or not.
|
|
postBuild:
|
|
substituteFrom:
|
|
- kind: ConfigMap
|
|
name: cluster-vars
|