6f61c9622daf94076caa656b3cf74aff0d28c34d
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
765ad032db |
fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
7a3da96d9f |
fix(cdi): single-replica scratch class + 40Gi UniFi disk (fix insufficient-storage import)
The UniFi Block import got past the device-permission fix but then failed ReplicaSchedulingFailure / "insufficient storage": CDI sizes scratch ~= the disk virtual size and, on the default 3-replica `longhorn` class, the ~60Gi scratch x3 (189G) on top of the disk's x3 (180G) overflowed the Longhorn tier (~125G/node). - platform/longhorn/storageclasses.yaml: NEW `longhorn-scratch` StorageClass (numberOfReplicas: 1) — CDI scratch is ephemeral/throwaway, replicating it 3x is waste; if a node dies mid-import CDI just retries. - platform/kubevirt/cdi-cr.yaml: spec.config.scratchSpaceStorageClass -> longhorn-scratch. - apps/vms/unifi/20-datavolume.yaml: boot disk 60Gi -> 40Gi (UOS min is 25Gi; 40Gi is comfortable for a home site and lighter on the replicated tier). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
3586769460 |
feat(storage): CSI snapshots + smart-clone + migratable RWX for KubeVirt live-migration
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration): - platform/external-snapshotter (NEW, controllers pass): the cluster-singleton snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0; this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io and kubevirt.io snapshot CRDs existed. - platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster). - platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone + claimPropertySets RWX/Block. Non-degraded default virt class clears CDIDefaultStorageClassDegraded. - platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass — migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on 1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default). - platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9 (no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains); live-migration stays per-VM opt-in on a longhorn-vm disk. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
b57b3e9d71 |
fix(kubevirt): correct KubeVirtControlPlaneDown expr — wrong metric name + job label
The alert fired continuously for ~21h while the control plane was fully
healthy (both virt-controller and virt-api pods 1/1 Ready, KubeVirt CR
Deployed/Available=True). Two bugs, each making it fire regardless of state:
1. `kubevirt_virt_controller_ready` does not exist — the real metric is
`kubevirt_virt_controller_ready_status` (KubeVirt suffixes `_status`),
so that clause was always ABSENT (never matched).
2. `absent(up{job="kubevirt"} == 1)` — the KubeVirt scrape job label is
`kubevirt-prometheus-metrics` (the Service name), not `kubevirt`, so
`up{job="kubevirt"}` matched nothing and absent() was permanently 1.
Fix: use `sum(kubevirt_virt_{controller,api}_ready_status) == 0` (fires only
when every replica is unready) and the correct job label. virt-api added to
match the alert's stated scope (virt-controller/virt-api).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
6a7e9d15e7 |
fix(kubevirt): drop broken http CDI ServiceMonitor causing false CDIOperatorDown
The cdi-operator manages its own `service-monitor-cdi` (scheme https + cdi-metrics-reader-token) which backs CDI's shipped recording rules (kubevirt_cdi_operator_up / cluster:kubevirt_cdi_operator_up:sum) and the CDIOperatorDown alert. Our extra `cdi` ServiceMonitor scraped the same cdi-prometheus-metrics service (HTTPS :8443) over http, so every scrape failed with 400 Bad Request, adding down (up=0) targets for the CDI pods. That poisoned the up-aggregation the recording rule sums, driving it to 0 and firing a false CDIOperatorDown even though the operator was Running, Ready, and Available=True. Remove the duplicate; keep the kubevirt SM (virt-operator ships none of its own). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
3a59ef1918 |
fix(kubevirt): boot VMs via CDI DataVolume + EFI + real KVM (Phase 5)
VMs did not boot on the dev cluster. Three distinct live-verified causes, all fixed GitOps-side: 1. containerDisk root failed at OCI init: exec "/container-disk-binary/usr/bin/container-disk": no such file. KubeVirt v1.9 supplies that binary to the volumerootdisk init container via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA `ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver silently prunes the image volume + its mount -> binary missing -> crash. (Proved: a server-side dry-run of a pod with an `image` volume drops it.) Fix: convert the VM roots in all three tests to CDI DataVolumes imported onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume path). This is also the spec-required disk model (docs/06, R-STATE-01). vm-lifecycle now PASSES green live (VMI Running on start + restart). 2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out -> local-fs.target fails -> cloud-init runcmd/poweroff never runs. Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI. 3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes until one went NotReady and VMs were evicted). Cause: useEmulation:true in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/ kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a virt-launcher compute container = "No such file"; qemu accel = tcg.) Fix: useEmulation:false so the kvm device is requested and VMs run on real (nested) KVM. Requires Flux reconcile to take effect live (PM action). The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller (294 labels, hypervisor=true), device plugins, and VMI scheduling all work. Also raised test exec timeouts to 30m (CDI import + boot) and, in vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy so the CDI importer's egress to quay.io is not severed by the policy. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
dbffb8fab8 |
Phase 5: KubeVirt + CDI IaaS proof (R-PLAT-03/04, R-STATE-01, R-NET-01)
- platform/kubevirt: pinned KubeVirt v1.9.0 + CDI v1.66.0 operators (remote bases); KubeVirt/CDI CRs tuned for Longhorn; useEmulation=true (prefer real KVM, emulate fallback). - Controllers/configs split (F-MON1): CRs + ServiceMonitors + alerts via platform-config. - Observability: ServiceMonitors, KubeVirt platform-health alerts (R-OBS-04), VM dashboard. - catalog/vm-demo.yaml (kind: vm) + rendered VirtualMachine/DataVolume fixture. - Conformance: vm-lifecycle (R-PLAT-03/04), vm-data-survival (R-STATE-01, L3), vm-default-deny (R-NET-01) — self-skipping until KubeVirt+longhorn exist. - ci/SPEC_PIN: adopt accepted spec b633b7f (ADR-0015 egress work already merged). - conformance/run.sh: document R-ENV-02/R-GITOPS-01 coverage gaps (WS-GITOPS/INFRA-owned). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
af4f6c07ae |
T-0.1 Flux GitOps layout + dev/prod overlays + platform component skeletons (R-GITOPS-01/03, R-ENV-02, R-PLAT-01)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |