jakman85andClaude Opus 5
1b056dc9d9
`platform-config` stopped dead on prod:
StorageProfile/longhorn dry-run failed: no matches for kind
"StorageProfile" in version "cdi.kubevirt.io/v1beta1"
A TRUE DEADLOCK, not a transient. platform-config applies the CDI CR *and*
platform/kubevirt/cdi-storageprofiles.yaml in the SAME pass. StorageProfile's
CRD does not exist until cdi-operator has reconciled that CR — and Flux
server-side dry-runs EVERY object before applying ANY of them, so the unknown
kind aborts the whole tier, the CR is never applied, the CRD is never created,
and platform-config can never converge. Every tier below it stays blocked.
Same root cause as the Kyverno CRD bundle, different operator: Flux cannot order
a resource against a CRD that one of its own siblings is responsible for
creating. It is INVISIBLE on an established cluster — dev's CDI CRDs have been
there for months — and fatal on a cold one. Observed on prod: cdi-operator
Available for 12h, no CDI CR, no storageprofiles CRD.
`crds` now breaks it by applying ONLY the CDI CR — the minimum — and leaving
every other object to Flux. The CR carries no ${} tokens, so applying it here
cannot diverge from what Flux would render. It refuses if the `cdi` namespace is
absent (platform has not run yet) rather than applying into nothing, waits for
the CRD, and requires it to reach Established. On prod the CRD appeared in 10s
and platform-config was unblocked.
`preflight` now reports the CRDs that later tiers apply resources against
(storageprofiles/cdis/kubevirts/volumesnapshotclasses) and dumps the cdi
namespace when one is missing, so this is caught in seconds instead of after
`resume` burns its 900s wait discovering it. It also names the stale-discovery-
cache variant, which presents identically but clears on a forced reconcile.
THE DURABLE FIX IS A REPO CHANGE, deliberately NOT made here: move
cdi-storageprofiles.yaml out of platform-config into its own Kustomization with
dependsOn: platform-config, so the CR lands one reconcile before the resources
needing its CRDs. That edits the reconciliation graph both clusters share, and
prod currently reads a pinned revision from the bootstrap Gitea while dev is
mid-migration on a separate branch — so it wants doing deliberately, not at
02:00 to unblock a bring-up. Until it lands, C2 is a prerequisite for C5 on any
cold cluster, and PROD-STANDUP.md says so.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ — node provisioning and the toolchain
OpenTofu (Proxmox VE) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
| Path | What |
|---|---|
tofu/ |
The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (bpg/proxmox provider) |
cloud-init/ |
First-boot baseline shipped as a Proxmox user-data snippet |
ansible/ |
The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
devcontainer/ |
The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm and the whole CI gate toolchain) |
cluster.sh |
RETIRED — drove the old libvirt VMs; does not work against Proxmox |
PHASE1-STATUS.md |
Current state of the cluster and how to reconnect to it |
PHASE1-SCOPE.md |
Historical Phase-1 scoping doc (libvirt era — superseded) |
Where the cluster actually lives
The dev cluster is not on this workstation. It runs on a Proxmox VE host; the nodes sit on the Proxmox LAN and are reached across a pfSense NAT:
| Nodes | k3s-dev-1/2/3 at 192.168.1.50/.51/.52 — 6 vCPU, 24 GiB, Debian 13, k3s v1.36.4 |
| kube-API | kube-vip VIP 192.168.1.49:6443, reached from the workstation via NAT 192.168.2.50:6443 |
| Node SSH | pfSense NAT 192.168.2.102:2250/2251/2252 → node :22 (Ansible only) |
| Ingress VIP | 192.168.1.53 (Cilium L2, shared Gateway); .53–.60 reserved for LB VIPs |
The earlier libvirt/KVM cluster on 10.73.0.0/24 is retired. Any doc or command referencing
virsh, qemu:///system, or 10.73.0.x for the cluster is historical.