home-network-k3s-impl

531 Commits 139 Branches 0 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	765ad032db	fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply `platform-config` applied the CDI CR and cdi-storageprofiles.yaml in ONE pass. StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and Flux server-side dry-runs every object before applying any — so the unknown kind aborted the tier, the CR was never applied, the CRD was never created, and the tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI CR, no storageprofiles CRD, every downstream tier blocked. The profiles now live in their own tier ./platform-config-cdi with dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated and platform-config runs wait: true, the CDI CR must be applied AND healthy — exactly the event that creates the CRD — before the profiles are applied. Cold clusters order themselves. `standup.sh crds` is retained as a fallback for clusters already stuck, and because it is idempotent. THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is created by a CONTROLLER AT RUNTIME is second-order and needs its own pass. cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles, datavolumes, datasources and dataimportcrons appear only after it reconciles the CDI CR. An audit found StorageProfile is the only second-order kind in platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster, VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium L2 CRs all have their CRD in git or in a chart. Moving the CDI CR up into `platform` was rejected: that would put CDI's health on the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and headlamp splits exist to undo. apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic: apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and its comment claimed platform-config guaranteed the profiles — which this move would have silently falsified. THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has prune: true, so it deletes both profiles once, CDI recreates them with empty spec, and the new tier re-applies the overrides ~30-60s later. Compared our override against CDI's unaided inference for the same Longhorn provisioner on dev: longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block longhorn-static (no override) spec unset -> status ReadWriteOnce/Block Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can no longer cause it (apps-vms gates on the new tier); a human must not do it by hand. Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated — CDI creating and inferring them with no Flux involvement. MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the target first. If it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in that state hours earlier. Fallback is the two-step the Cilium move used (merge with prune: false, verify the ownership label moved, restore prune: true). substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN}, deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits paths whose Kustomization declares it, so omitting it would move the file OUT of the audited set — a blind spot for a future literal-token ship. Gate confirms it stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS decryption stanza: the moved file is two plain CRs, so no overlay change was needed. VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the two profiles; the graph is byte-identical across dev and prod with apps-vms carrying both edges and no cycle. Full CI gate PASS. KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include platform-config-cdi, so `resume` will not wait on it. Another agent holds that file; the PM will add it. Refs R-GITOPS-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-05 11:53:45 -05:00
jakman85andClaude Opus 4.8	cbc2301b30	docs: reconcile docs/comments with post-upgrade cluster state Text-only cleanup after the upgrade campaign. Updates stale version strings, prose, and comments to the verified current facts; no Kubernetes behavior or version pins changed except the kubectl toolchain bump (below). Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie, Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB, dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18, Grafana 13.2.0. - Version strings across infra/ and platform/ READMEs + comments - kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew) - Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh) - COMPLETED/historical banners on the three executed plan docs - Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges - Appended dated DECISIONS-LOG entry (2026-08-30) superseding the 2026-08-26 LIVE STATE line Left as revalidation items (not blind-edited): external-snapshotter v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1, conformance ImageVolume feature-gate assertions under 1.36. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-30 15:13:12 -05:00
jakman85andClaude Opus 4.8	3586769460	feat(storage): CSI snapshots + smart-clone + migratable RWX for KubeVirt live-migration Adds the VM/CDI storage tier approved by the PM (RWX for live-migration): - platform/external-snapshotter (NEW, controllers pass): the cluster-singleton snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0; this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io and kubevirt.io snapshot CRDs existed. - platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster). - platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone + claimPropertySets RWX/Block. Non-degraded default virt class clears CDIDefaultStorageClassDegraded. - platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass — migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on 1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default). - platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9 (no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains); live-migration stays per-VM opt-in on a longhorn-vm disk. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-26 12:06:15 -05:00

Author

SHA1

Message

Date

jakman85andClaude Opus 5

765ad032db

fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply

`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.

The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.

THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.

Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.

apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.

THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
  longhorn-vm (we override)     spec RWX/Block  -> status ReadWriteMany/Block
  longhorn-static (no override) spec unset      -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.

MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).

substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.

VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.

KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-05 11:53:45 -05:00

jakman85andClaude Opus 4.8

cbc2301b30

docs: reconcile docs/comments with post-upgrade cluster state

Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).

Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.

- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
  versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
  with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
  2026-08-26 LIVE STATE line

Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-08-30 15:13:12 -05:00

jakman85andClaude Opus 4.8

3586769460

feat(storage): CSI snapshots + smart-clone + migratable RWX for KubeVirt live-migration

Adds the VM/CDI storage tier approved by the PM (RWX for live-migration):

- platform/external-snapshotter (NEW, controllers pass): the cluster-singleton
  snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in
  kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0;
  this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io
  and kubevirt.io snapshot CRDs existed.
- platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the
  driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster).
- platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile
  longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone
  + claimPropertySets RWX/Block. Non-degraded default virt class clears
  CDIDefaultStorageClassDegraded.
- platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass —
  migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on
  1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default).
- platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9
  (no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains);
  live-migration stays per-VM opt-in on a longhorn-vm disk.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-08-26 12:06:15 -05:00