jakman85andClaude Opus 5
f21e89d9d2
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
55 lines
3.9 KiB
YAML
55 lines
3.9 KiB
YAML
---
|
|
# external-snapshotter — the cluster-wide CSI snapshot control plane (Phase 2, storage).
|
|
#
|
|
# WHY: Longhorn's CSI driver (driver.longhorn.io) can create CSI VolumeSnapshots, but the
|
|
# `snapshot.storage.k8s.io` API (VolumeSnapshot / VolumeSnapshotContent / VolumeSnapshotClass)
|
|
# and the snapshot-controller that reconciles them are NOT part of k3s or of Longhorn — they are
|
|
# a SEPARATE, cluster-singleton component (kubernetes-csi/external-snapshotter). Without it the
|
|
# CRDs are absent (only `snapshots.longhorn.io` + the kubevirt.io snapshot CRDs exist), so:
|
|
# * no VolumeSnapshotClass can be created (item 2), and
|
|
# * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
|
|
# which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
|
|
# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass that CONSUMES them
|
|
# is applied by platform-config (dependsOn: platform), and the CDI StorageProfiles that set
|
|
# `cloneStrategy: snapshot` by platform-config-cdi (dependsOn: platform-config — their own CRD is
|
|
# created by cdi-operator only after the `CDI` CR reconciles; see clusters/base/platform-config-cdi.yaml).
|
|
#
|
|
# CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
|
|
# CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment
|
|
# (into kube-system, upstream default) reconciles VolumeSnapshots. Custom resources that use these
|
|
# CRDs (the Longhorn VolumeSnapshotClass, item 2) live in platform-config so the CRDs exist first.
|
|
#
|
|
# VERSION (Renovate-maintained): external-snapshotter v8.2.0.
|
|
# * The v8.x line updated its Kubernetes deps to 1.31 at v8.1.0. This cluster is now k3s v1.36.4,
|
|
# so the v8.2.0 pin is NO LONGER verified against the running Kubernetes version — RE-CHECK it
|
|
# for 1.36 (a newer external-snapshotter line may be the version-matched choice). Pin left as-is
|
|
# pending that revalidation.
|
|
# * snapshot.storage.k8s.io CRDs are served at v1 (GA) — driver.longhorn.io speaks v1, so
|
|
# Longhorn CSI snapshots + CDI snapshot-clone work against these CRDs.
|
|
# * The upstream setup manifest pins the controller image to an older v8.0.x tag; the `images`
|
|
# transformer below re-pins it to v8.2.0 so the running controller matches the CRD version.
|
|
# * The `client/config/crd` base also ships the beta VolumeGroupSnapshot CRDs; that feature is
|
|
# gated OFF in the controller (no --enable-volume-group-snapshots arg), so they are inert.
|
|
#
|
|
# VENDORED (was remote github bases). The two upstream kustomize dirs are built to LOCAL files
|
|
# (crds.yaml, snapshot-controller.yaml) at v8.2.0 and committed here. Previously these were REMOTE
|
|
# bases (github.com/...?ref=v8.2.0) that Flux's kustomize-controller `git fetch`ed on EVERY reconcile;
|
|
# those fetches intermittently hit the controller's 27s timeout -> "kustomize build failed" -> the
|
|
# WHOLE `platform` tier flaked (a transient GitHub slowness stalled every downstream app). Local files
|
|
# make the build OFFLINE + deterministic. NOTE: platform/kubevirt still uses the same remote-base
|
|
# pattern and carries the same latent fragility (vendor it too if it flakes).
|
|
# RE-VENDOR on a version bump:
|
|
# kubectl kustomize "https://github.com/kubernetes-csi/external-snapshotter/client/config/crd?ref=vX.Y.Z" > crds.yaml
|
|
# kubectl kustomize "https://github.com/kubernetes-csi/external-snapshotter/deploy/kubernetes/snapshot-controller?ref=vX.Y.Z" > snapshot-controller.yaml
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- crds.yaml
|
|
- snapshot-controller.yaml
|
|
images:
|
|
# renovate: datasource=github-releases depName=kubernetes-csi/external-snapshotter
|
|
# Controller image == CRD version. A version bump = re-vendor crds.yaml + snapshot-controller.yaml
|
|
# at the new tag (see header) AND bump this newTag.
|
|
- name: registry.k8s.io/sig-storage/snapshot-controller
|
|
newTag: v8.2.0
|