feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite

OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

catalog/render.py

@@ -47,8 +47,7 @@ LBL = "homelab.keithjackson.dev"
 # The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
 # Every rendered manifest lands under a Flux path whose Kustomization declares
 # postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
-# reconcile time: dev renders dev.home.keithjackson.dev, prod renders home.keithjackson.dev (the
+# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
-# APEX — operator decision 2026-09-04; prod IS the apex, so there is no `prod.` label and no CNAMEs).
 # It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
 # the escape contract below reserves `$${...}` for expansions meant for a container's shell.
 # Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
@@ -607,7 +606,7 @@ def r_internet_egress(s: dict) -> dict:
 def r_httproutes(s: dict) -> list:
     """One HTTPRoute per ingressRoute, attaching host -> service port on the shared Gateway.
 
-    TLS is the Gateway's wildcard *.${BASE_DOMAIN} cert (no per-service cert step);
+    TLS is the Gateway's wildcard *.home.keithjackson.dev cert (no per-service cert step);
     ExternalDNS manages the record from the HTTPRoute (R-TLS/R-DNS, renderer-contract).
     """
     out = []

clusters/README.md

@@ -97,6 +97,27 @@ Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
 Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)
 ```
 
+### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
+
+The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
+controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
+and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
+reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
+before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
+**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
+`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
+have existed for months); it cost 12h on prod's cold start.
+
+So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
+platform-config`), and a rule:
+
+> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
+> **controller at runtime** rather than by a manifest in git.
+
+`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
+`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
+profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
+
 Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
 exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
 (`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own

clusters/base/apps-vms.yaml

@@ -4,8 +4,11 @@
 # NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
 # import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
 # health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
-# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
+# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
-# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
+# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
+# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
+# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
+# adoption concern).
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
@@ -19,6 +22,28 @@ spec:
   wait: false
   dependsOn:
     - name: platform-config
+    # LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
+    # storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
+    # decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
+    # (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
+    # live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
+    # with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
+    # to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
+    # conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
+    #
+    # HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
+    # derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
+    # status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
+    # it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
+    #
+    # This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
+    # deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
+    # dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
+    #
+    # The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
+    # edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
+    # cannot silently drop this tier's KubeVirt/StorageClass gate.
+    - name: platform-config-cdi
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/base/cluster-vars.yaml

@@ -26,49 +26,36 @@ data:
   # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
   # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
   # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
-  # suffix, so the two clusters can no longer describe the same name.
+  # subdomain, so the two clusters can no longer describe the same name.
   #
-  #   dev   -> dev.home.keithjackson.dev        (a subdomain; = this base default)
+  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
-  #   prod  -> home.keithjackson.dev            (THE APEX — prod IS the apex, no `prod.` label)
+  #   prod  -> prod.home.keithjackson.dev
   #
-  # OPERATOR DECISION 2026-09-04 (supersedes the interim split committed earlier the same day, which
+  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
-  # had prod on `prod.home.` with apex CNAMEs pointing at it). PROD IS THE APEX. That deletes the
+  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
-  # whole CNAME layer — `<svc>.home.keithjackson.dev` is prod's real name, not an alias — and it is
+  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
-  # DEV that carries a label. Convention is unchanged from GATEWAY_VIP / LONGHORN_DATA_PATH: the base
+  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
-  # default is the DEV value (dev restates it verbatim) and prod overrides it. What changed is which
+  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
-  # string dev holds.
+  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
-  #
+  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
-  # DEV'S VALUE IS A MIGRATION, NOT A VARIABLE CHANGE. The live dev cluster is serving the APEX right
+  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
-  # now: its wildcard cert, every public DNS record, its Keycloak issuer, ~12 OIDC clients' redirect
+  BASE_DOMAIN: "home.keithjackson.dev"
-  # URIs, kubectl SSO, and gitea.home.keithjackson.dev — which IS this repo's GitOps source AND the
-  # push target. Flipping this value on dev without first standing up the new host set STRANDS DEV:
-  # Flux's GitRepository url moves to gitea.dev.home.keithjackson.dev, which has no DNS record and no
-  # cert, so dev stops reconciling and cannot be fixed by pushing. ORDERED CUTOVER, operator present:
-  # infra/PROD-PROVISIONING.md §10.
-  #
-  # DEV MUST VACATE THE APEX BEFORE PROD PUBLISHES THERE. Both clusters describing
-  # `*.home.keithjackson.dev` at once is the 2026-09-04 incident precondition, re-created.
-  BASE_DOMAIN: "dev.home.keithjackson.dev"
 
   # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
   # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
   # so it cannot be created, updated or deleted regardless of TXT ownership.
   #
-  # BOTH ENVIRONMENTS USE THE CLOUDFLARE **ZONE**, `keithjackson.dev`. This value is NOT a per-
+  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
-  # environment isolation knob and must not be narrowed to either environment's BASE_DOMAIN.
+  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
+  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
+  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
+  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
   #
-  # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) — the Cloudflare provider uses --domain-filter for ZONE
+  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
-  # SELECTION as well as name filtering, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
-  # matches NO zone: ExternalDNS generates the record and then skips it with "no hosted zone matching
+  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
-  # record DNS Name", publishing NOTHING. This has been tried and reverted more than once (an agent
+  # — it can never delete another environment's records — and the fallback is to set prod's value
-  # narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in review). DO NOT "harden"
+  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
-  # it — the change is not defence in depth, it is a total outage of DNS publication.
-  #
-  # WHERE ISOLATION ACTUALLY COMES FROM, and it is sufficient:
-  #   (1) the two environments request DISJOINT hostname sets (dev.home.… vs the apex), and
-  #   (2) EXTERNALDNS_TXT_OWNER_ID differs, so neither registry ever claims the other's records.
-  # That pairing is what the 2026-09-04 incident needed; a domain filter cannot provide it, because
-  # both environments live in the one zone either way.
   EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
 
   # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
@@ -77,35 +64,14 @@ data:
   # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
   # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
   #
-  #   dev -> `dev`      prod -> `prod`      base default -> a non-colliding sentinel (below)
+  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
+  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
+  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
   #
-  # DEV IS `dev`, NOT `default` (changed 2026-09-04 with the move to dev.home.keithjackson.dev). Until
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
-  # this migration dev HAD to render `default`, because its LIVE apex records were already registered
+  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
-  # in Cloudflare's TXT registry under that owner and renaming would have orphaned every one of them.
+  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
-  # THAT CONSTRAINT NO LONGER BINDS: dev is publishing an ENTIRELY NEW record set under
+  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
-  # *.dev.home.keithjackson.dev, with no pre-existing registry entries to recognise, so the owner id
-  # is now free to be chosen on its merits. Two reasons to take `dev`:
-  #   - `default` is THE VALUE YOU GET BY FORGETTING (it is the external-dns chart default, and both
-  #     clusters silently shared it, which IS the incident). An environment whose identity is the
-  #     forget-value can never be told apart from a misconfigured newcomer. No environment should
-  #     hold it.
-  #   - It makes vacating the apex an EXPLICIT, ORDERED, VERIFIABLE step. Dev's old apex records stay
-  #     owned by `default`, which no environment now claims, so nothing deletes them implicitly on
-  #     some future reconcile — they are retired deliberately, and prod's publish is gated on that
-  #     retirement being confirmed in Cloudflare (infra/PROD-PROVISIONING.md §10) rather than on a
-  #     race between two reconcile loops.
-  #
-  # PROD STAYS `prod` AND MUST NOT BECOME `default`, even though prod now serves the apex names that
-  # dev's `default`-owned records currently occupy. Giving prod `default` would hand prod ownership of
-  # the LIVE dev cluster's records while dev is still serving them, and under `policy: sync` prod
-  # would delete the ones no prod source claims. That is the 2026-09-04 incident, re-armed, with the
-  # roles swapped. Prod inherits the apex NAMES; it must never inherit the apex OWNER ID.
-  #
-  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default` and not any real
-  # environment's id. A new environment that forgets this knob must not silently inherit another's
-  # identity — that is precisely the incident. With this sentinel such a cluster owns nothing, so it
-  # can DELETE nothing; it would publish under a visibly wrong owner id (grep the TXT records)
-  # instead of destroying a live environment's DNS.
   EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
 
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
@@ -113,7 +79,7 @@ data:
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
                                             # ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
                                             # takes the pool's first free IP, and coredns-custom answers
-                                            # *.${BASE_DOMAIN} with THIS so pod -> public-hostname traffic
+                                            # *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
                                             # hairpins through the Gateway (correct SNI -> wildcard cert).
                                             # Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
                                             # long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
@@ -135,10 +101,37 @@ data:
   # ship the literal string into the cluster.
 
   # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
-  # [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
+  #
-  # main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
+  # *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
-  # reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
+  # 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
-  K3S_API_HOST: "192.168.1.50"
+  # control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
+  # live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
+  # own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
+  # Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
+  # outage you would notice. A virtual address that follows the control plane is the fix, and it is
+  # what kube-vip was stood up to provide.
+  #
+  #   dev  -> 192.168.1.49   (= kube_vip_address, group_vars/all.yml; base default, below)
+  #   prod -> 192.168.2.9    (= kube_vip_address, group_vars/k3s_prod/vars.yml)
+  #
+  # THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
+  # unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
+  # move. Read it as "the API host Cilium dials", which is what it always meant.
+  #
+  # [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
+  # (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
+  # already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
+  # the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
+  # already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
+  # block at the top of group_vars/all.yml.
+  #
+  # Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
+  # in both envs, so it is NOT a knob.
+  #
+  # TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
+  # both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
+  # https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
+  K3S_API_HOST: "192.168.1.49"
 
   # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
   # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the

clusters/base/kustomization.yaml

@@ -14,6 +14,15 @@ resources:
   - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
+  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
+  # manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
+  # datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
+  # platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
+  # keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
+  # CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
+  # cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
+  # as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
+  - platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
   # Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
   # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.

clusters/base/platform-config-cdi.yaml

@@ -0,0 +1,135 @@
+---
+# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
+# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
+#
+# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
+#
+# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
+# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
+# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
+# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
+# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
+# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
+# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
+#
+# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
+# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
+# graph order itself, so a cold cluster needs no human in the loop.
+#
+# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
+# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
+# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
+# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
+# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
+# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
+# kind. One edge, no pre-apply.
+#
+# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
+# and everything else waits on: putting CDI's health on that critical path is the mistake the
+# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
+# StorageProfiles that were in the wrong pass.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: platform-config-cdi
+  namespace: flux-system
+spec:
+  interval: 10m
+  # 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
+  # very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
+  # now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
+  # about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
+  # cache is one retryInterval. Keep it short.
+  retryInterval: 2m
+  # 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
+  # image to pull and no controller to become Available, so a slow apply here means a real error, not
+  # a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
+  timeout: 2m
+  # TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
+  # component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
+  # It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
+  # second path also declares. See the transition note at the bottom of this file.
+  prune: true
+  # TRUE, but understand exactly what it buys and what it does NOT.
+  #
+  # CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
+  # provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
+  # `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
+  # it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
+  # is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
+  # If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
+  # `apps-vms` waits on it.
+  #
+  # WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
+  # provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
+  # storageprofile-controller computes that status from our spec override plus inference. So Ready
+  # here means "the spec override is accepted by the API server", not "CDI has folded it into
+  # status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
+  # below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
+  # a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
+  wait: true
+  dependsOn:
+    # platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
+    # causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
+    # ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
+    # flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform-config-cdi
+  # NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
+  # no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
+  # oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
+  # clusters/dev or clusters/prod for this tier.
+  #
+  # substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
+  # substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
+  #   1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
+  #      whose Kustomization declares substituteFrom. Omitting it would move
+  #      cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
+  #      could ship as a literal string, precisely the class of bug that check exists to stop.
+  #   2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
+  #      with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
+  #      empty. Without the stanza there is no substitution at all and the literal ships silently.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
+  # Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
+  # Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
+  # StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
+  # the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
+  # dual-ownership fight — the objects are pruned before they are re-applied under this tier's
+  # `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
+  #
+  # Why that is acceptable here — and it is NOT the platform/cilium case:
+  #   * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
+  #     nothing else in the repo declares a StorageProfile. The stale set is exactly two
+  #     cluster-scoped CDI CRs.
+  #   * The objects do not actually go missing: CDI's storageprofile-controller creates a
+  #     StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
+  #     inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
+  #     absent".
+  #   * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
+  #     existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
+  #     is disturbed and no data path is touched.
+  #   * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
+  #     instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
+  #     permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
+  #     dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
+  #     created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
+  #     could.
+  #
+  # PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
+  # happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
+  # config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
+  # healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
+  # --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
+  # StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
+  #   `flux get kustomization platform-config` must be Ready BEFORE merging.
+  # If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
+  # temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
+  # this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
+  # `prune: true` in a second commit.

clusters/dev/kustomization.yaml

@@ -5,20 +5,6 @@
 # reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
 # It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
 # on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
-#
-# ##############################################################################################
-# ###  *** DO NOT MERGE THIS FILE TO `main` UNATTENDED (as of 2026-09-04) ***                 ###
-# ###                                                                                        ###
-# ###  This overlay now carries dev's DNS MIGRATION target: BASE_DOMAIN moves from the apex   ###
-# ###  `home.keithjackson.dev` to `dev.home.keithjackson.dev`, and the GitRepository url       ###
-# ###  moves with it. sync.yaml reconciles ./clusters/dev from `main` EVERY 10 MINUTES, so a   ###
-# ###  merge to main IS the cutover — it executes on the live dev cluster with nobody present. ###
-# ###  Dev's Gitea is both its GitOps source and the push target: if the new host is not up    ###
-# ###  with a Ready cert first, dev stops reconciling and pushing cannot repair it.            ###
-# ###                                                                                        ###
-# ###  Run the ordered runbook: infra/PROD-PROVISIONING.md §10 (steps D1-D8), operator        ###
-# ###  present. Merging is step D3, not step one.                                             ###
-# ##############################################################################################
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
@@ -145,24 +131,7 @@ patches:
         # via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
         # secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
         # revert this url (docs/05).
-        #
+        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
-        # *** THE STRAND. THIS URL IS THE MOST DANGEROUS LINE IN THE REPOSITORY. ***
-        # It is BOTH dev's GitOps source AND the operator's push target, and it is NOT tokenized —
-        # it cannot be: the `flux-system` root Kustomization that renders ./clusters/dev declares NO
-        # postBuild.substituteFrom, so `gitea.${BASE_DOMAIN}` here would ship the LITERAL token and
-        # break the source outright (the L0 flux-postbuild-substitution check asserts exactly this).
-        # So it is a hand-maintained literal that must be moved in lockstep with BASE_DOMAIN below.
-        #
-        # It has been moved to dev's NEW suffix as part of the 2026-09-04 migration authoring. That
-        # makes this file the TARGET state, NOT a safe unattended merge: dev's root Kustomization
-        # (sync.yaml) reconciles ./clusters/dev from `main` every 10m, so MERGING THIS TO main WHILE
-        # DEV IS LIVE MIGRATES DEV UNATTENDED. If gitea.dev.home.keithjackson.dev does not already
-        # resolve and serve a valid cert at that instant, dev stops reconciling and CANNOT BE FIXED
-        # BY PUSHING — the broken thing is the path Flux uses to read pushes.
-        # ORDERING THAT AVOIDS IT (infra/PROD-PROVISIONING.md §10, steps D1-D3): suspend dev's root
-        # Kustomization or park this url on the bootstrap Gitea (http://192.168.2.132:3000/...) for
-        # the migration window, and only restore the in-cluster url after the new host is proven.
-        url: https://gitea.dev.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   # --- dev-scoped cluster-vars (R-ENV-02) ---
@@ -179,60 +148,33 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
-      # ==========================================================================================
+      # --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
-      # DEV DNS IDENTITY (R-DNS-01/02/03) — DEV MOVES OFF THE APEX. **NOT YET APPLIED.**
+      # DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
-      # ==========================================================================================
+      # like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
-      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day:
+      # Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
-      #
+      # Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
-      #   dev   -> dev.home.keithjackson.dev      (this knob; = the base default, restated)
+      # (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
-      #   prod  -> home.keithjackson.dev          (THE APEX — prod IS the apex, no `prod.` label,
+      # prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
-      #                                            and therefore no apex CNAME layer at all)
+      # not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
-      #
+      # not a variable change: infra/PROD-PROVISIONING.md §10.
-      # *** THIS VALUE IS AHEAD OF THE LIVE CLUSTER. *** The dev cluster is serving the APEX right
-      # now. Merging this file does not perform the migration — it DEFINES the target. The cutover is
-      # an ORDERED, OPERATOR-PRESENT runbook: infra/PROD-PROVISIONING.md §10.
-      #
-      # WHY IT CANNOT BE A PLAIN VARIABLE FLIP — THE STRAND. `spec.url` of the GitRepository ~20
-      # lines above is https://gitea.home.keithjackson.dev/... . Dev's Gitea is BOTH dev's GitOps
-      # source AND the operator's push target. The moment BASE_DOMAIN moves, dev's Gitea HTTPRoute,
-      # DNS record and wildcard cert all move to gitea.dev.home.keithjackson.dev. If that host does
-      # not ALREADY resolve with a valid, Ready cert, dev stops reconciling — and it CANNOT BE FIXED
-      # BY PUSHING, because the thing that is broken is the path Flux uses to read pushes. Recovery
-      # is then hands-on (point the source at the bootstrap Gitea on 192.168.2.132:3000, or repair
-      # in-cluster by hand). §10 states the ordering that avoids it: publish and verify the new host
-      # FIRST, or park the source on the bootstrap Gitea for the window.
-      #
-      # ALSO MOVED BY THIS ONE LINE, all on a live cluster: the `*.` wildcard Certificate (a fresh
-      # ACME order), Keycloak's issuer URL and ~12 OIDC clients' redirect URIs, kubectl SSO (a k3s
-      # server flag => a rolling restart + every user re-issuing their kubeconfig), coredns-custom's
-      # split-horizon zone, Homepage's links, and 6 L2 chainsaw tests that hardcode the apex.
       - op: replace
         path: /data/BASE_DOMAIN
-        value: "dev.home.keithjackson.dev"
+        value: "home.keithjackson.dev"
-      # ExternalDNS scope. MUST stay the Cloudflare ZONE — NOT dev.home.keithjackson.dev. FINDING
+      # ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
-      # F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this for ZONE SELECTION, so a
+      # and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
-      # strict-subdomain filter matches no zone and publishes NOTHING ("no hosted zone matching
+      # the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
-      # record DNS Name"). Dev moving to a subdomain does NOT make narrowing safe — it makes it
-      # newly tempting, which is why this comment is here. Same value the retired DNS_ZONE knob held.
       - op: replace
         path: /data/EXTERNALDNS_DOMAIN_FILTER
         value: "keithjackson.dev"
-      # OWNER ID: `dev` (was `default` until 2026-09-04). The old value was FORCED — dev's live apex
+      # DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
-      # records were already in Cloudflare's TXT registry under owner `default`, so renaming would
+      # registry under owner `default` (the external-dns chart default, which is what both clusters
-      # have orphaned every one of them. THE MIGRATION DISSOLVES THAT CONSTRAINT: on the new suffix
+      # silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
-      # dev publishes a brand-new record set with no prior registry entries to recognise, so the id
+      # BY this string: any other value and it stops recognising every record it manages, then
-      # is chosen on its merits, and `default` has none — it is the value external-dns uses when you
+      # re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
-      # FORGET the flag, which is exactly how both clusters came to share it. Consequences, both
+      # whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
-      # deliberate:
-      #   - dev's OLD apex records remain owned by `default`, which no environment now claims, so
-      #     nothing deletes them implicitly. Retiring them is an explicit §10 step, and prod's first
-      #     publish is gated on that retirement being CONFIRMED — not on two reconcile loops racing.
-      #   - prod stays `prod` and must NEVER be given `default`, or prod would own the live dev
-      #     cluster's apex records and sync-delete them. Prod inherits the apex NAMES, not its OWNER.
-      # dev != prod is the load-bearing invariant; the base default stays a non-colliding sentinel.
       - op: replace
         path: /data/EXTERNALDNS_TXT_OWNER_ID
-        value: "dev"
+        value: "default"
       # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
       # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
       # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -241,15 +183,29 @@ patches:
       #
       # Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
       # (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
-      # *.dev.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
+      # *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
       - op: replace
         path: /data/GATEWAY_VIP
         value: "192.168.1.53"
-      # k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
+      # k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
-      # (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
+      # the address kube-vip floats, NOT the primary node .50.
+      # [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
+      # (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
+      #
+      # *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
+      # Every other value in this block is what dev is running right now. This one is not:
+      # the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
+      # DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
+      # AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
+      # in parity for exactly this reason). Do not push this to dev-live casually, and not
+      # while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
+      # but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
+      # already (k3s_tls_sans) and dev's own kubeconfig validates against it via
+      # tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
+      # same LAN as the nodes and ARP-announced by kube-vip on every server.
       - op: replace
         path: /data/K3S_API_HOST
-        value: "192.168.1.50"
+        value: "192.168.1.49"
       # Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
       # (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
       - op: replace

clusters/prod/kustomization.yaml

@@ -129,23 +129,14 @@ patches:
   #
   # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
   # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
-  # REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.),
+  # Two independent reasons, both consequences of the split-horizon design:
-  # which changes WHY the blocker holds but not THAT it holds:
+  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
-  #   1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is
+  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
-  #      pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source
+  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
-  #      inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster)
+  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
-  #      BEFORE dev is retired, or prod loses its source of truth.
+  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
-  #      NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now
+  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
-  #      gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at
+  #      gitea.home.keithjackson.dev to itself and never reach a repository.
-  #      the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the
-  #      cutover below a one-line change once prod actually runs Gitea.
-  #   2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is
-  #      easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole
-  #      ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant
-  #      prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea
-  #      listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev
-  #      to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What
-  #      remains is purely reason 1 — the Gitea is not there YET.
   #
   # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
   #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
@@ -211,41 +202,28 @@ patches:
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
       # ==========================================================================================
-      # PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX.
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
       # ==========================================================================================
-      # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day
+      # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
-      # (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here:
+      # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
-      #
+      # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
-      #   dev   -> dev.home.keithjackson.dev     (dev carries the label; clusters/dev)
+      # of which already happened:
-      #   prod  -> home.keithjackson.dev         (THE APEX — this knob)
-      #
-      # There is NO `prod.` label and NO CNAME layer. `<svc>.home.keithjackson.dev` is prod's REAL
-      # name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its
-      # Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`.
-      #
-      # BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was
-      # HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the
-      # EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two
-      # consequences, one of which already happened:
       #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
       #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
       #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
       #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
       #      have raced each other's `_acme-challenge` TXT records on that one zone.
+      # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
+      # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
       #
-      # *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. ***
+      #   dev   -> home.keithjackson.dev         (UNCHANGED — dev is live; phase 2 moves it)
-      # Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are
+      #   prod  -> prod.home.keithjackson.dev    (this knob)
-      # therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream
+      #
-      # tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
+      # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
-      #   (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
+      # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
-      #   (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
+      # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
-      #       (verified against Cloudflare directly — a pfSense override is NOT evidence about public
+      # `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
-      #       DNS; that mistake was made on 2026-09-04), AND
+      # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
-      #   (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
-      #       that same name set while dev's still exists counts against LE's duplicate-certificate
-      #       limit (5 per week per identical name set).
-      # Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates
-      # the 2026-09-04 incident with the roles reversed.
       #
       # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
       # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
@@ -253,33 +231,30 @@ patches:
       # trusts a different Keycloak than the one prod actually runs.
       - op: replace
         path: /data/BASE_DOMAIN
-        value: "home.keithjackson.dev"
+        value: "prod.home.keithjackson.dev"
+      # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
+      # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
+      # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
+      # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
+      # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
+      # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
+      # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
+      # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
       - op: replace
         path: /data/EXTERNALDNS_DOMAIN_FILTER
-        # MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks
+        # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
-        # like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING
+        # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
-        # F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the
+        # record then skipped it with "no hosted zone matching record DNS Name", so prod would
-        # zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching
+        # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
-        # record DNS Name", so prod publishes NOTHING. An agent narrowed this to
+        # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
-        # prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it.
+        # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
-        # Isolation from dev does NOT come from this filter — it comes from
-        # (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and
-        # (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records.
         # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
         value: "keithjackson.dev"
       # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
       # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
-      # its own and, under policy: sync, deleted the records behind them.
+      # its own and, under policy: sync, deleted the records behind them. This value MUST differ from
-      #
+      # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
-      # PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex
+      # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
-      # NAMES, and dev's existing apex records are registered under owner `default` — so `default` is
-      # the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no
-      # prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must
-      # never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are
-      # invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it
-      # simply fails to publish until those records are retired, which is the correct behaviour.
-      # dev is `dev` (clusters/dev, changed with dev's migration); the base default is a
-      # non-colliding sentinel. All three differ.
       - op: replace
         path: /data/EXTERNALDNS_TXT_OWNER_ID
         value: "prod"
@@ -365,14 +340,32 @@ patches:
       - op: replace
         path: /data/CILIUM_LB_POOL_STOP
         value: "192.168.2.30"
-      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
       # k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
-      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
+      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
-      # Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
+      #
-      # platform/cilium/helmrelease.yaml. Not yet decided.
+      # *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
+      # read "Not yet decided" while shipping .10 — the founder's own address — and that is what
+      # the LIVE prod cluster is running: all three cilium-agents carry
+      # KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
+      # 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
+      # the datapath keeps running off Cilium's cache while no service or policy update lands, with
+      # nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
+      # control plane is, which is the whole reason kube-vip exists here.)
+      #
+      # It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
+      # this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
+      # equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
+      # that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
+      # `--server` join needs a host that is up before any VIP is floated. Two addresses, two
+      # variables, deliberately.
+      #
+      # .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
+      # LB-IPAM can never be handed the same IP — asserted by
+      # conformance/tests/l0-static/ha-control-plane-as-code.
       - op: replace
         path: /data/K3S_API_HOST
-        value: "192.168.2.10"
+        value: "192.168.2.9"
+      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
       # Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
       # prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
       # they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not

infra/PROD-PROVISIONING.md

@@ -1172,7 +1172,7 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
 | `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
 | `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
 | `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
-| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real. Prod's value was inverted the same day to the APEX `home.keithjackson.dev`, with dev moving to `dev.home.keithjackson.dev` — see §10a, which is authored but NOT yet applied. |
+| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real, and prod serves `prod.home.keithjackson.dev`. |
 
 - [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
       overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
@@ -1283,20 +1283,12 @@ single-server-etcd window, and avoids a later member-add dance.
       specifically.
 - [ ] Monitoring and alerts live on prod.
 - [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
-      Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10a
+      Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10 phase 2.
-      "The L2 tests hardcode the apex".
+- [ ] **Prod serves its own DNS/TLS identity** and dev's records are provably untouched (§10):
-- [ ] **Prod serves the apex DNS/TLS identity** and dev's records are provably untouched (§10/§10a):
+      distinct `EXTERNALDNS_TXT_OWNER_ID`, prod wildcard cert `*.prod.home.keithjackson.dev`, and
-      distinct `EXTERNALDNS_TXT_OWNER_ID` (`prod` vs `dev`), prod wildcard cert
+      prod's own Keycloak realm + per-service client secrets.
-      `*.home.keithjackson.dev`, dev migrated to `*.dev.home.keithjackson.dev` with its old apex
-      records retired (§10a step D8), and prod's own Keycloak realm + per-service client secrets.
 
-## 10. DNS/TLS environment split — **the 2026-09-04 incident, and the target architecture**
+## 10. DNS/TLS environment split — **the 2026-09-04 incident, and phase 1 of the fix**
-
-> **Current state (2026-09-04):** the target below is **AUTHORED, NOT APPLIED.** Prod's
-> `platform` tier and its 10 downstream tiers are SUSPENDED. Dev is **live on the apex**. The
-> ordered cutover is **§10a**, and merging the authored change to `main` is step **D3** of it,
-> not step one — dev self-reconciles `main`, so a merge outside the runbook window executes
-> the migration unattended.
 
 ### What happened
 
@@ -1328,82 +1320,43 @@ values; for DNS and TLS that promise was false, so `clusters/prod` was *authored
 public identity. Two clusters that describe the same DNS name will fight over it no matter how the
 DNS controller is configured.
 
-### The architecture decision (operator) — **INVERTED 2026-09-04, this is the current target**
+### The architecture decision (operator)
 
 ```
-dev   ->  *.dev.home.keithjackson.dev      gitea.dev.home.keithjackson.dev
+dev   ->  *.dev.home.keithjackson.dev        gitea.dev.home.keithjackson.dev
-prod  ->  *.home.keithjackson.dev          gitea.home.keithjackson.dev     (THE APEX)
+prod  ->  *.prod.home.keithjackson.dev       gitea.prod.home.keithjackson.dev
+apex  ->  CNAME <svc>.home.keithjackson.dev  ->  <svc>.prod.home.keithjackson.dev
 ```
 
-**Prod *is* the apex. There is no `prod.` label and no CNAME layer at all** —
+### PHASE 1 — **prod only. Dev did not move.** (this change)
-`<svc>.home.keithjackson.dev` is prod's real name, not an alias for one. This **supersedes** the
-interim split committed earlier the same day (`c8c1d29`), which put prod on
-`*.prod.home.keithjackson.dev` behind apex CNAMEs pointing at it. That layer is deleted: fewer
-moving parts, no CNAME chain to keep in step, and prod ends up on the names users already type.
 
-The cost of the inversion is that **dev must move**, and dev is the live cluster. The interim split
+`BASE_DOMAIN` is now real: every hostname in every Flux-reconciled manifest derives from it.
-was chosen precisely because it moved only the unbuilt environment; this one does not have that
-property, so the whole of §10's remaining risk now sits on dev's side, and it is sequenced below.
 
-### The values, per environment
+| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` |
+|---|---|---|---|
+| base default | `home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` |
+| **dev** | `home.keithjackson.dev` (**unchanged**) | `default` (**must not change**) | `keithjackson.dev` |
+| **prod** | `prod.home.keithjackson.dev` | `prod` | `prod.home.keithjackson.dev` |
 
-| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` | Ansible `k3s_base_domain` |
+**Dev's effective hostname is deliberately still the apex `home.keithjackson.dev`.** Moving prod
-|---|---|---|---|---|
+alone ends the collision and unblocks prod, and it is the whole of the risk that had to be taken
-| base default | `dev.home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` | `dev.home.keithjackson.dev` (role default) |
+while the operator is away. Dev's rendered output is **byte-identical** except for one added line,
-| **dev** | `dev.home.keithjackson.dev` | `dev` | `keithjackson.dev` | (inherits the role default) |
+`txtOwnerId: default`, which is the value dev's external-dns was already using implicitly — verified
-| **prod** | `home.keithjackson.dev` | `prod` | `keithjackson.dev` | `home.keithjackson.dev` (`group_vars/k3s_prod`) |
+by rendering all 12 `substituteFrom` paths through `kubectl kustomize | flux envsubst --strict` for
+both overlays and diffing.
 
-`[parity]`: `k3s_base_domain` must equal the same cluster's `BASE_DOMAIN`. The kube-apiserver's
+**Two independent guards now, either sufficient alone:**
-`oidc-issuer-url` is a k3s **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux
-post-build substitution can never reach it — hence one fact in two places. If they disagree the
-apiserver validates tokens against a different Keycloak than the cluster runs: kubectl SSO fails
-while the cert-based admin kubeconfig keeps working, so it presents as "SSO is broken", not as skew.
 
-**`EXTERNALDNS_DOMAIN_FILTER` is the Cloudflare ZONE for BOTH, and must stay that way.** FINDING
+1. **`txtOwnerId`** — per-environment identity. A cluster only ever touches records it owns. This is
-F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for **zone selection** as
+   the direct fix for the incident. Dev **must** render exactly `default`: its live records are
-well as record matching, so a filter that is a strict subdomain of the hosted zone matches **no
+   already registered under that owner, and any other value would make dev's own external-dns stop
-zone** — external-dns generates the record and then skips it with *"no hosted zone matching record
+   recognising them and re-create duplicates beside the orphans.
-DNS Name"*, publishing **nothing**. This has been "hardened" and reverted more than once. It is not
+2. **`domainFilters`** — per-environment name scope. Prod cannot see a name outside its own subtree.
-defence in depth; it is a total outage of DNS publication. Isolation comes from the two guards below,
+   Defence in depth, added because the first guard is a single string in a single file.
-which a domain filter cannot provide anyway: both environments live in the one zone regardless.
 
-**The two guards that actually work:**
+The base default for the owner id is a **non-colliding sentinel, deliberately not `default`**: a
-
+future environment that forgets the knob must not silently inherit dev's identity — that is exactly
-1. **`txtOwnerId`, per-environment identity** — a cluster only ever reconciles records its own owner
+what happened here. With the sentinel such a cluster owns nothing, so it can delete nothing.
-   id claims in the TXT registry. This is the direct fix for the incident.
-2. **Disjoint hostname sets** — after the migration the two clusters never describe the same record,
-   so there is nothing to fight over. *This guard does not exist during the migration window*, which
-   is the entire reason the ordering below is mandatory.
-
-#### Why dev's owner id changed from `default` to `dev`
-
-Until this migration dev **had** to render `default`: its live apex records were already registered
-in Cloudflare's TXT registry under that owner, and renaming would have orphaned every one of them.
-**The migration dissolves that constraint** — on `*.dev.home.keithjackson.dev` dev publishes an
-entirely new record set with no prior registry entries to recognise, so the id is free and is chosen
-on its merits. `default` has none: it is the value external-dns uses when you **forget** the flag,
-which is exactly how both clusters came to share it, and an environment whose identity is the
-forget-value can never be told apart from a misconfigured newcomer. Consequences, both deliberate:
-
-- Dev's **old apex records stay owned by `default`**, which no environment now claims, so nothing
-  deletes them implicitly on some future reconcile. Retiring them is an explicit step (**D7**) and
-  prod's first publish is gated on that retirement being *confirmed* — not on two reconcile loops
-  racing each other.
-- **Prod stays `prod` and must never be given `default`.** Prod is taking over the apex *names*, and
-  dev's existing apex records are registered under `default` — so `default` is the one value that
-  would let prod adopt them, and under `policy: sync` delete the ones no prod source claims, on a
-  cluster dev is still serving. That is the 2026-09-04 incident re-armed with the roles swapped.
-  Prod inherits the apex **names**; it must never inherit the apex **owner id**.
-
-All three values differ, and the base default remains a sentinel so an environment that forgets the
-knob owns nothing and can therefore delete nothing.
-
-### PHASE 1 — prod tokenized and moved off dev's names (`c8c1d29`, done; suffix since inverted)
-
-`BASE_DOMAIN` was a **dead knob** — declared in `clusters/base/cluster-vars.yaml` and referenced by
-no reconciled manifest, because every hostname was a hardcoded literal. `c8c1d29` made it real (141
-references across the reconciled set) and gave prod its own suffix. That work stands; only the
-*value* prod holds has changed, from `prod.home.keithjackson.dev` to the apex.
 
 ### What could NOT be tokenized, and why
 
@@ -1412,195 +1365,70 @@ references across the reconciled set) and gave prod its own suffix. That work st
   Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and
   **cannot be verified by the offline render harness** (`kustomize` does not decrypt), so a mistake
   would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately.
-  **Both environments now need action:** dev's copy must move to `keycloak.dev.home.keithjackson.dev`
+  **Prod action required:** prod's Headlamp will otherwise authenticate against **dev's** Keycloak.
-  as part of step **D5**, and prod needs its own copy with `keycloak.home.keithjackson.dev` and a
+  Prod needs its own copy of this Secret with `keycloak.prod.home.keithjackson.dev` and a client
-  client secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client
+  secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client secret
-  secret is realm-scoped.
+  is per-realm.
 - **`oidc-issuer-url` in the kube-apiserver flags** (`roles/k3s_server/tasks/main.yml`) — a k3s
-  server flag, not a manifest. Handled on the **Ansible** seam instead: role default
+  **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux post-build substitution
-  `k3s_base_domain` (dev's value) + derived `k3s_oidc_issuer_url`, overridden for prod in
+  can never reach it. Correctly tokenized on the **Ansible** side instead: new role default
-  `inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`,
+  `k3s_base_domain` (dev value) + derived `k3s_oidc_issuer_url`, overridden for prod in
-  so a fourth prod server inherits it from group membership. Verified per host by a read-only probe
+  `inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`.
-  (`connection: local`, zero SSH, `changed=0`): all three dev servers resolve
+  `[parity]`: `k3s_base_domain` **must** equal the same cluster's `BASE_DOMAIN`, or the apiserver
-  `https://keycloak.dev.home.keithjackson.dev/realms/homelab`, all three prod servers resolve
+  validates tokens against a different Keycloak than the cluster runs and kubectl SSO fails while the
-  `https://keycloak.home.keithjackson.dev/realms/homelab`.
+  cert-based admin kubeconfig keeps working (so it looks like "SSO is broken", not like config skew).
-- **`spec.url` of dev's `GitRepository`** (`clusters/dev/kustomization.yaml`) — see the strand below.
-  It *cannot* be tokenized: the `flux-system` root Kustomization that renders `./clusters/dev`
-  declares no `postBuild.substituteFrom`, so `gitea.${BASE_DOMAIN}` there would ship the **literal
-  token** and break the source outright (the L0 `flux-postbuild-substitution` check asserts exactly
-  this). It is a hand-maintained literal that must move in lockstep with `BASE_DOMAIN`.
 
----
+### A pre-existing prod problem this incidentally FIXED
 
-## 10a. THE MIGRATION RUNBOOK — ordering is load-bearing
+§4 recorded that prod nodes use **public** resolvers, so the kube-apiserver would resolve its OIDC
+issuer publicly and get **dev's** Gateway VIP (192.168.1.53) — the wrong cluster on the wrong LAN,
+making prod kubectl SSO impossible, and requiring an `/etc/hosts` pin as a workaround. With prod on
+its own subdomain the public record for `keycloak.prod.home.keithjackson.dev` is published by
+**prod's own** external-dns pointing at **prod's** Gateway VIP (192.168.2.20), so the public answer
+and the split-horizon answer agree. **No `/etc/hosts` pin is needed.** This holds only once prod's
+external-dns has actually published the record; until then use the break-glass cert kubeconfig.
 
-> **Prod's `platform` tier and its 10 downstream tiers are SUSPENDED and must stay suspended until
+### Verify on prod's first reconcile
-> step P1.** Nothing below is safe to run out of order, and nothing below should be run while the
-> operator is away.
 
-### The two ways this goes wrong
+- [ ] **`domainFilters` zone selection.** The Cloudflare provider uses this value for *zone* lookup
+      as well as record matching, and in 2026-08 a strict-subdomain filter dropped the parent zone
+      ("no hosted zone matching record DNS Name"). If prod's external-dns logs no matching zone it
+      will create **nothing** — fail-closed, never destructive. Fallback: set prod's
+      `EXTERNALDNS_DOMAIN_FILTER` back to `keithjackson.dev` and rely on `txtOwnerId`, which is
+      sufficient on its own.
+- [ ] **Prod's wildcard cert is `*.prod.home.keithjackson.dev`** and solves DNS-01 without touching
+      dev's `_acme-challenge` records.
+- [ ] **Dev's records are untouched** after prod's tiers are resumed. Confirm dev's TXT registry
+      entries still read owner `default` and that prod's read owner `prod`.
+- [ ] **Prod's own Keycloak realm + per-service client secrets** exist (every `*.sops.yaml` client
+      secret is realm-scoped; prod cannot reuse dev's).
 
-**(1) BOTH CLUSTERS ON THE APEX — the 2026-09-04 incident, recreated.** If prod publishes
+### PHASE 2 — dev migration + apex CNAMEs (**deferred, not done**)
-`*.home.keithjackson.dev` before dev has vacated it, two clusters describe the same records in the
-same zone again. Distinct `txtOwnerId`s stop prod from *deleting* dev's records — that guard holds —
-but they do not stop the second failure mode: both clusters requesting a `letsencrypt-prod` wildcard
-for the **same name set** and racing each other's `_acme-challenge` TXT records, so neither reliably
-solves DNS-01. **Rule: dev vacates the apex first, and the vacate is verified against Cloudflare
-before prod is unsuspended.**
 
-**(2) THE STRAND — dev's GitOps source is the thing that moves.** `spec.url` of dev's
+Deliberately not attempted here: dev is **live** (18 Longhorn volumes, working SSO) and it hosts the
-`GitRepository` is `https://gitea.home.keithjackson.dev/...`, which is **both** dev's GitOps source
+Gitea that is currently this repo's GitOps source. Moving it is a migration, not a variable change.
-**and** the PM's push target. Flipping dev's `BASE_DOMAIN` moves Gitea's HTTPRoute, its public DNS
+Required, roughly in order:
-record and the wildcard cert covering it to `gitea.dev.home.keithjackson.dev` all at once. If that
-host does not already resolve and serve a valid, Ready cert at that instant, **dev stops reconciling
-and cannot be fixed by pushing** — the broken thing *is* the path Flux uses to read pushes. Recovery
-is then hands-on and needs LAN access to the cluster.
 
-**(2b) AND THE MERGE ITSELF IS THE TRIGGER.** `clusters/dev/sync.yaml` is a self-managing root
+1. **`BASE_DOMAIN` → `dev.home.keithjackson.dev`** in `clusters/dev`, plus `k3s_base_domain` for the
-Kustomization reconciling `./clusters/dev` from **`main`** every 10 minutes, and prod tracks `main`
+   dev inventory group.
-too. **A merge to `main` therefore *executes* this migration on the live dev cluster, unattended.**
+2. **A new `*.dev.home.keithjackson.dev` wildcard cert** issued and Ready *before* cutover.
-The authored change is a *target*, not a deployment. **Do not merge these files to `main` outside the
+3. **Keycloak issuer change.** The realm's issuer URL is baked into every token; every OIDC client
-runbook window.** Park them on a branch until the operator is at the keyboard and step **D1** has
+   (`grafana`, `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`,
-been done.
+   `qbittorrent`, `radarr`, `sonarr`, and the `kubernetes` public client) needs its redirect URIs
-
+   updated in the same change, and the k3s apiserver `oidc-issuer-url` with it — which means a
-### D — dev vacates the apex (operator present, in this order)
+   **rolling k3s restart** on the dev servers.
-
+4. **kubectl SSO re-issue** for every user (issuer changed).
-- [ ] **D0. Snapshot the escape routes before touching anything.** Confirm a working break-glass
+5. **`gitea.dev.home.keithjackson.dev`** — this is the GitOps source URL in
-      **cert-based** admin kubeconfig for dev (not the OIDC one — OIDC breaks in D6). Confirm the
+   `clusters/dev/kustomization.yaml`. Flux must be able to resolve and trust the new host *before*
-      retained bootstrap Gitea can be started (`bootstrap/gitea/bootstrap-gitea.sh`; dev's historical
+   the URL is switched, or dev stops reconciling and the change cannot be pushed to fix itself.
-      recovery host was `192.168.2.103:3000`, prod's current bootstrap is `192.168.2.132:3000`).
+   Sequence this last, and keep the bootstrap Gitea available as the recovery path (docs/05).
-      Record dev's current Cloudflare record set (names + TXT registry owners) so D7 has a checklist
+6. **`EXTERNALDNS_TXT_OWNER_ID` for dev.** Once dev's names move, its existing `default`-owned
-      and so an accidental deletion is detectable.
+   records are for names dev no longer serves; retire them deliberately rather than letting a
-- [ ] **D1. Break the automatic trigger.** Either `flux suspend kustomization flux-system -n
+   renamed owner orphan them.
-      flux-system` on dev (stops the root from self-applying the merge), **or** re-point dev's
+7. **Apex CNAMEs** `<svc>.home.keithjackson.dev` → `<svc>.prod.home.keithjackson.dev`. This is what
-      `GitRepository` url at the bootstrap Gitea for the window. Do this **before** the merge, not
+   actually **moves user traffic to prod**, and it must not land before prod serves those services.
-      after. This is what converts "merging strands dev" into "merging is inert until you say go".
+8. **The L2 behavioural conformance tests hardcode `home.keithjackson.dev`** (`dns-exposure`,
-- [ ] **D2. Publish and prove the new host set *while dev still serves the apex*.** The new names
+   `dns-internal-resolution`, `gateway-ingress-reachable`, `oidc-sso-grafana`, `sso-forward-auth`,
-      must exist before anything depends on them:
+   `trusted-tls`). They are test inputs, not reconciled manifests, so they were left alone in phase 1
-      - a `*.dev.home.keithjackson.dev` + `dev.home.keithjackson.dev` **wildcard Certificate issued
+   — but they will fail against prod and must be parameterized before §9's "suite passes green
-        and `Ready`** (LE prod, DNS-01) — see the ACME note below before ordering it;
+   against the prod cluster" can be claimed.
-      - public DNS for `gitea.dev.home.keithjackson.dev` resolving to dev's Gateway VIP
-        (192.168.1.53), **checked against Cloudflare / a public resolver, never through the pfSense
-        override** — that mistake was made on 2026-09-04 and produced a false healthy reading;
-      - `curl -sS https://gitea.dev.home.keithjackson.dev/api/v1/version` succeeding with a
-        publicly-trusted cert from a host on the dev LAN.
-      Until D2 is green, D3 is not safe.
-- [ ] **D3. Merge the authored change to `main`** (`clusters/base/cluster-vars.yaml`,
-      `clusters/dev/`, `clusters/prod/`, the two Ansible files, `platform/externaldns`,
-      `catalog/render.py`). With D1 in place nothing moves yet.
-- [ ] **D4. Release dev's reconciliation deliberately.** Resume the root Kustomization (or restore
-      the in-cluster url, now pointing at `gitea.dev.home.keithjackson.dev`) and watch
-      `flux get kustomizations -A` converge. Dev's Gateway listeners, HTTPRoutes, coredns-custom zone
-      and Homepage links all move here. Dev's external-dns publishes the new record set under owner
-      `dev`.
-- [ ] **D5. Keycloak: issuer + every client.** Dev's realm issuer becomes
-      `https://keycloak.dev.home.keithjackson.dev/realms/homelab`. The issuer is baked into every
-      token, and **~12 OIDC clients** need their redirect URIs moved in the same change: `grafana`,
-      `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`, `qbittorrent`,
-      `radarr`, `sonarr`, plus the `kubernetes` public client. Most are reconciled from
-      `platform/keycloak/keycloak-realm-import.yaml` and move with D4; **headlamp's is the exception**
-      — its issuer/callback live in a SOPS-encrypted Secret and must be re-encrypted by hand.
-- [ ] **D6. Rolling k3s restart + kubectl SSO re-issue.** `k3s_base_domain` now renders a different
-      `oidc-issuer-url` into `/etc/rancher/k3s/config.yaml`, so the `k3s_server` role is **no longer
-      a no-op on dev**: it rewrites the file and fires the Restart handler on all three servers. Run
-      it **one server at a time**, confirming `kubectl get nodes` between each. Every user's OIDC
-      kubeconfig is invalidated and must be re-issued; the cert-based admin kubeconfig keeps working
-      throughout, which is what makes this step recoverable. **Do not reboot nodes** — restart the
-      `k3s` service only.
-- [ ] **D7. Retire dev's old apex records.** They are owned by `default`, which no environment now
-      claims, so nothing removes them automatically. Delete the `<svc>.home.keithjackson.dev` A
-      records, the apex record, **and their TXT registry entries** (`external-dns-*` TXT), plus any
-      stale `_acme-challenge.home.keithjackson.dev`. *Alternative, if you prefer external-dns to do
-      its own cleanup:* set dev's `EXTERNALDNS_TXT_OWNER_ID` back to `default` for exactly one
-      reconcile after D4 — `policy: sync` will then garbage-collect the apex set it no longer has
-      sources for — and only then switch it to `dev`. That is two commits and needs watching; the
-      direct deletion is preferred because it is verifiable in one place.
-- [ ] **D8. Verify the apex is EMPTY, against Cloudflare.** `dig +short <svc>.home.keithjackson.dev
-      @1.1.1.1` returns nothing for every service name, and no TXT registry entry for those names
-      remains. **This is the gate on the whole P sequence.** A pfSense/PiHole override is not
-      evidence about public DNS.
-
-### P — prod takes the apex
-
-- [ ] **P1. Only now unsuspend prod**, in dependency order: `flux resume kustomization platform`,
-      then the 10 downstream tiers. Confirm D8 first, in Cloudflare, not from a local resolver.
-- [ ] **P2. Prod's external-dns publishes under owner `prod`.** Check the logs for a matching hosted
-      zone (F-DNS1 fail-closed mode: no zone selected ⇒ nothing created — which is safe, but means
-      the filter was narrowed against instruction). Confirm the new TXT registry entries read owner
-      `prod` and that no record owned by `dev` was touched.
-- [ ] **P3. Prod's wildcard cert `*.home.keithjackson.dev` + `home.keithjackson.dev` is `Ready`** and
-      solved DNS-01 without colliding with any `_acme-challenge` dev still holds.
-- [ ] **P4. Prod's kubectl SSO works without an `/etc/hosts` pin.** Prod nodes resolve
-      `keycloak.home.keithjackson.dev` through **public** upstreams (§4), so this only becomes true
-      once P2 has published that name at prod's Gateway VIP (192.168.2.20). Before then a prod node
-      resolving its own issuer reaches **dev**. Use the break-glass cert kubeconfig until P2 is done.
-- [ ] **P5. Cut prod's GitOps source over** from the workstation bootstrap Gitea
-      (`http://192.168.2.132:3000/...`) to `https://gitea.home.keithjackson.dev/...` once prod runs
-      its own Gitea — the block in `clusters/prod/kustomization.yaml` owns that procedure. Note the
-      inversion **simplified** this: prod's apex Gitea name and dev's `gitea.dev.home...` are now
-      different hosts, so prod pointing at the apex means "prod's own Gitea", never dev's.
-
-### ACME: the duplicate-certificate limit
-
-Let's Encrypt rate-limits **duplicate certificates — 5 per week for an identical set of names.**
-Prod's wildcard is `*.home.keithjackson.dev` + `home.keithjackson.dev`, which is **exactly the name
-set dev holds today**. Every prod order therefore counts against the same bucket as dev's historical
-renewals of that pair, and a retry loop during P3 can exhaust it and leave prod with **no trusted
-cert for up to a week**. Mitigations:
-
-- Order prod's cert **after** D8, so dev is no longer renewing the same pair.
-- Let dev's old `*.home.keithjackson.dev` `Certificate`/Secret be **deleted** in D4/D7 rather than
-  left to renew in the background.
-- Rehearse against **letsencrypt-staging** first (`ACME_CLUSTER_ISSUER`) — staging has its own,
-  far looser limits — and only switch prod to `letsencrypt-prod` when the DNS-01 solve is proven.
-- Do **not** delete-and-recreate prod's `Certificate` to "retry"; each recreation is a fresh order.
-
-### The L2 tests hardcode the apex
-
-Six behavioural (L2) chainsaw tests embed `home.keithjackson.dev` as a literal:
-
-| test | file |
-|---|---|
-| `dns-exposure` | `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml` |
-| `dns-internal-resolution` | `conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml` |
-| `gateway-ingress-reachable` | `conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml` (+ `fixtures/app-and-probe.yaml`) |
-| `oidc-sso-grafana` | `conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml` |
-| `sso-forward-auth` | `conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml` |
-| `trusted-tls` | `conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml` |
-
-They are **test inputs, not reconciled manifests**, so they are deliberately left alone by the
-authoring change: they are the only thing still asserting the apex, and while dev is live on the apex
-they are *correct*. Under the new direction the literal happens to become right for **prod** and
-wrong for **dev**, so they must be **parameterized per environment** (not merely edited) before
-either §9's *"suite passes green against the prod cluster"* or a green L2 run against migrated dev
-can be claimed. Until then, expect the dev L2 suite to fail between D4 and that parameterization —
-a known, scheduled failure, not a regression.
-
-The L1 *rendering* tests need no change: their goldens carry `${BASE_DOMAIN}` as a token, and the
-literal hostnames in them come from each test's own fixture `ingressRoutes[].host`, which is
-environment-independent by construction.
-
-### Offline proof carried out for the authoring change (no cluster touched)
-
-All 12 `substituteFrom` paths were rendered for **both** overlays through the real toolchain
-(`kubectl kustomize --load-restrictor LoadRestrictionsNone <spec.path> | flux envsubst --strict`,
-with each overlay's effective `cluster-vars` taken from `kubectl kustomize clusters/<env>`), before
-and after the change.
-
-- **Dev's diff is *only* the hostname move.** Proven by inverse substitution rather than by reading
-  the diff: applying `s/dev.home.keithjackson.dev/home.keithjackson.dev/` to the new render restores
-  the baseline **byte-for-byte** in all 12 paths, plus 2 declared `txtOwnerId: default → dev` lines.
-  145 changed line pairs, all classified: 143 hostname-move, 2 declared owner-id, **0 unexplained**,
-  and **0 bare-apex remnants**. The 2 paths with no change (`cilium`, `velero`) provably contain no
-  hostname at all (`grep -c keithjackson.dev` = 0).
-- **Prod lands on the bare apex:** **0** occurrences of `prod.home.keithjackson.dev` and **0** of
-  `dev.home.keithjackson.dev` anywhere in prod's rendered output; 143 apex names.
-- **Certs:** dev `*.dev.home.keithjackson.dev` + `dev.home.keithjackson.dev`; prod
-  `*.home.keithjackson.dev` + `home.keithjackson.dev`.
-- **Positive controls** (the harness is provably not blind): perturbing an unrelated knob
-  (`GATEWAY_VIP`) surfaces 1 unexplained line; perturbing `BASE_DOMAIN` to a wrong value fails 10 of
-  12 paths; injecting a single un-moved hostname is caught by the independent bare-apex scan (which
-  the byte-identity check alone cannot see — which is why both exist).
-- Gates: `ci/l0.sh`, `ci/l1.sh`, `ci/render-golden.sh`, `ci/lint.sh`,
-  `flux-postbuild-substitution/check.sh` (254 assertions), `catalog/render.py --selftest`,
-  `ci/spec-immutability.sh` — all **PASS**.

infra/ansible/README.md

@@ -79,11 +79,30 @@ Ansible installs Cilium; the Flux `platform/cilium` HelmRelease adopts config
 afterward. The parity-critical values **must match** between this role's
 `templates/cilium-values.yaml.j2` and that HelmRelease:
 - `kubeProxyReplacement: true`
-- `k8sServiceHost: {{ k3s_api_host }}` (= primary node IP, `10.73.0.11` by default)
+- `k8sServiceHost: {{ k3s_apiserver_endpoint }}` — **the kube-vip CONTROL-PLANE VIP**
-- `k8sServicePort: 6443`
+  (`= kube_vip_address`: dev `192.168.1.49`, prod `192.168.2.9`), **not** `k3s_api_host`
+- `k8sServicePort: {{ k3s_api_port }}` (6443 in both environments)
 
-The HelmRelease currently has `k8sServiceHost: "REPLACE-ME"` — WS-NET must set it
+### Two API addresses, split 2026-09-06 — do not re-merge them
-to the same value this role uses (surfaced by `tofu output primary_node_ip`).
+`k3s_api_host` and `k3s_apiserver_endpoint` were one variable, and the conflation put every
+prod `cilium-agent` on `192.168.2.10` (`k3s-prod-1`'s own address) on a cluster whose point is
+HA — verified live. Lose that node and the datapath keeps serving from Cilium's cache while no
+service or policy update ever lands, which is a silent failure, not an outage.
+
+| variable | value | consumed by | why it must be that address |
+|---|---|---|---|
+| `k3s_api_host` | the founder's own node IP (dev `192.168.1.50`, prod `192.168.2.10`) | `roles/k3s_server`: the secondary `--server https://…:6443` join URL, and one `k3s_tls_sans` entry | on a **cold** build no VIP exists yet (kube-vip is a k3s auto-deploy manifest, so it needs a cluster first). The founder just ran `--cluster-init` and is up |
+| `k3s_apiserver_endpoint` | `{{ kube_vip_address }}` (dev `192.168.1.49`, prod `192.168.2.9`) | `roles/cilium` → `k8sServiceHost`; any steady-state client | it must survive losing any single server |
+
+Consequences already handled: `playbooks/site.yml` imports `kube-vip.yml` **between** `k3s.yml`
+and `cilium.yml` (the VIP has to be floating before the CNI dials it), and `roles/kube_vip`'s
+last task blocks until the VIP answers on 6443. `k3s_tls_sans` already carried both addresses,
+so no cert reissue and no k3s restart are involved.
+
+The Flux side of the same contract is `${K3S_API_HOST}` in `platform/cilium/helmrelease.yaml`,
+set per overlay in `clusters/<env>/kustomization.yaml`; `[parity]` means it must equal
+`k3s_apiserver_endpoint` for the SAME environment. Asserted by
+`conformance/tests/l0-static/ha-control-plane-as-code`.
 
 ## Interface assumed of `bootstrap/` (authored by other workers, NOT here)
 `roles/flux_bootstrap` invokes two entrypoints via overridable command vars:
@@ -148,7 +167,9 @@ ansible-playbook playbooks/site.yml \
 What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
 `k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
 target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
-node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
+node-IP list), `k3s_api_host=192.168.2.10` (the founder, bootstrap-join only),
+`kube_vip_address=192.168.2.9` — and therefore `k3s_apiserver_endpoint=192.168.2.9`, which is what
+Cilium dials — and
 `cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
 `.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
 

infra/ansible/inventory/group_vars/all.yml

@@ -27,19 +27,85 @@ k3s_servers_group: k3s_servers
 # a kubeconfig is a credential.)
 cluster_kubeconfig_name: dev.kubeconfig
 
-# The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP.
+# --- TWO API ADDRESSES, AND THEY ARE NOT INTERCHANGEABLE (split 2026-09-06) ----------
+# This used to be ONE variable, k3s_api_host, consumed by BOTH the cold-bootstrap join and
+# Cilium's k8sServiceHost. That conflation is what pointed every cilium-agent on the LIVE
+# prod cluster at 192.168.2.10 — k3s-prod-1's own address — on a cluster whose entire
+# purpose is HA: verified 2026-09-06, all three agents carried
+# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held 192.168.2.9. Lose that one node
+# and no service/policy update reaches the CNI until it returns (the existing datapath
+# survives on Cilium's cache, so it fails SILENTLY). The two consumers have genuinely
+# different requirements, so they now have two variables.
+#
+#   k3s_api_host            THE FOUNDER'S OWN ADDRESS. Bootstrap-time only.
+#   k3s_apiserver_endpoint  THE HA ENDPOINT (the kube-vip VIP). Steady-state.
+#
+# DO NOT "simplify" these back into one. See each block below for why each half must be
+# the address it is.
+
+# THE FOUNDER'S OWN NODE IP — a BOOTSTRAP-TIME address, deliberately NOT the VIP.
 # NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node
-# IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san /
+# IP — so we read the real 192.168.1.x from k3s_node_ip instead.
-# secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux
+#
-# HelmRelease stay in PARITY (docs/02 contract).
+# EXACTLY TWO CONSUMERS, and both of them need the founder specifically:
+#   1. roles/k3s_server — the secondary `--server https://<host>:<port>` JOIN URL. On a COLD
+#      build the VIP DOES NOT EXIST YET at join time: kube-vip is a k3s auto-deploy manifest
+#      that only lands once a cluster exists, so a join aimed at the VIP is a join aimed at
+#      an unanswered address and the cold build is unjoinable. The founder is up (it just ran
+#      --cluster-init), which is the whole reason this variable is the founder.
+#   2. roles/k3s_server — one entry of k3s_tls_sans, so the founder's own address is a SAN
+#      alongside the VIP and every other server's IP.
+#
+# ANYTHING THAT RUNS AGAINST A CLUSTER THAT ALREADY EXISTS MUST NOT USE THIS. It is a
+# single point of failure by construction. Use k3s_apiserver_endpoint below.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}"
+
+# Shared by both halves: k3s serves the API on 6443 in every environment, so the port is
+# not a per-environment knob and is not duplicated per address.
 k3s_api_port: 6443
 
+# THE HA kube-apiserver ENDPOINT — a VIRTUAL address that reaches the cluster wherever the
+# control plane currently is, not a machine. OPERATOR DECISION 2026-09-06: in-cluster
+# components dial the VIP, as kube-vip was always intended to provide.
+#
+# DERIVED from kube_vip_address (below), never restated: an environment overrides the VIP in
+# ONE place (group_vars/k3s_prod/vars.yml sets kube_vip_address: 192.168.2.9) and this
+# follows automatically, so prod needs no override of its own and a fourth environment
+# cannot set one of the two and forget the other.
+#
+# NAMED FOR THE ROLE, NOT THE MECHANISM. Consumers want "the address that reaches the
+# apiserver, HA"; that it is currently floated by kube-vip is an implementation detail. If
+# the mechanism is ever something else (an external LB, a DNS name), this one line moves and
+# no consumer changes.
+#
+# CONSUMERS:
+#   - roles/cilium -> cilium-values.yaml.j2 -> k8sServiceHost.  [parity] with the Flux side:
+#     clusters/base/cluster-vars.yaml K3S_API_HOST (and each overlay's patch) MUST equal
+#     THIS value for that environment, or Flux and the bootstrap fight over the CNI's
+#     control-plane path. See platform/cilium/helmrelease.yaml's PARITY CONTRACT header.
+#
+# TLS IS ALREADY COVERED — verified, not assumed: k3s_tls_sans (roles/k3s_server/defaults)
+# has always included kube_vip_address, and on 2026-09-06 the prod apiserver was reached
+# through https://192.168.2.9:6443 with FULL CA verification and tls-server-name=192.168.2.9
+# (infra/ansible/.cache/prod.kubeconfig, no insecure-skip-tls-verify). So the serving cert
+# needs no change and no k3s restart for this switch.
+#
+# ORDERING CONSEQUENCE, and it is why playbooks/site.yml changed too: Cilium now needs the
+# VIP to be UP before it installs. site.yml therefore imports kube-vip.yml BETWEEN k3s.yml
+# and cilium.yml, and roles/kube_vip waits for the VIP to answer before returning. kube-vip
+# can precede the CNI because its DaemonSet is hostNetwork with
+# KUBERNETES_SERVICE_HOST=127.0.0.1 — it needs neither a CNI nor kube-proxy.
+k3s_apiserver_endpoint: "{{ kube_vip_address }}"
+
 # Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover
 # endpoint for the kube-apiserver, fronted by an operator pfSense port-forward at
-# 192.168.1.49:6443. Shared here so BOTH kube_vip AND the k3s_server tls-san list
+# 192.168.1.49:6443. Shared here so THREE things agree on the one address:
-# agree on the address (the serving cert must carry this IP or TLS via the VIP
+#   - roles/kube_vip          — what to ARP and lease-elect for;
-# fails to validate).
+#   - roles/k3s_server        — the tls-san entry (the serving cert must carry this IP or
+#                               TLS via the VIP fails to validate);
+#   - k3s_apiserver_endpoint  — above; what in-cluster components actually dial.
+# THIS is the one line an environment overrides (prod: 192.168.2.9, in
+# group_vars/k3s_prod/vars.yml); all three follow.
 kube_vip_address: "192.168.1.49"
 
 # Controller (dev container) reaches the kube-API DIRECTLY via the pfSense NAT

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -51,12 +51,18 @@ k3s_servers_group: k3s_prod
 # than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential.
 cluster_kubeconfig_name: prod.kubeconfig
 
-# --- k3s API host — safe to set now, the primary's real IP is already known -------
+# --- k3s API host — THE FOUNDER'S ADDRESS, bootstrap-time only --------------------
 # Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
 # dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
 # has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
 # address (see the KEY DIFFERENCE note there) — so point straight at it instead of
 # needing a k3s_node_ip split.
+#
+# WHAT THIS IS *NOT*, since 2026-09-06: it is not the endpoint anything steady-state
+# dials. That is k3s_apiserver_endpoint (group_vars/all.yml), which derives from
+# kube_vip_address below and therefore needs NO override here. k3s_api_host is consumed
+# only by the secondary `--server` cold-join URL and by one tls-san entry. Read the
+# TWO API ADDRESSES block at the top of group_vars/all.yml before touching either.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
 # by template reference, so it needs no separate override here — it picks up the real
@@ -75,8 +81,15 @@ k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # round (control-plane .49, ingress pool .53-.60).
 #
 # This is the address kube-vip floats for the API and that k3s tls-san's onto its
-# serving cert; secondaries use it as their --server join target. Dev's equivalent is
+# serving cert. Dev's equivalent is kube_vip_address: 192.168.1.49 in group_vars/all.yml.
-# kube_vip_address: 192.168.1.49 in group_vars/all.yml.
+#
+# CORRECTION 2026-09-06: an earlier version of this comment claimed "secondaries use it as
+# their --server join target". They do not, and they must not — on a COLD build the VIP is
+# not up when the secondaries join (kube-vip is a k3s auto-deploy manifest and needs a
+# cluster first), so the join uses k3s_api_host, the founder. What DOES use this address is
+# k3s_apiserver_endpoint (group_vars/all.yml, derived from this line), i.e. Cilium's
+# k8sServiceHost and any other steady-state client. Overriding kube_vip_address here moves
+# BOTH the VIP and prod's Cilium endpoint together, which is the point of deriving it.
 #
 # HISTORY: an earlier pass here wrongly set this to 192.168.2.20 — that is the INGRESS
 # VIP, not the control-plane VIP. Corrected. Keep the two distinct.
@@ -293,47 +306,35 @@ node_baseline_dns_servers:
   # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
   # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
   #
-  # ORDERING HAZARD, NOT YET CLEARED. Prod's issuer is keycloak.home.keithjackson.dev — an APEX
+  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # name, because prod IS the apex under the 2026-09-04 inversion (k3s_base_domain below).
+  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # That is the name the LIVE DEV CLUSTER answers today, and its public A record currently
+  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # points at DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. A prod
+  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
-  # node resolving its own issuer through these public upstreams therefore reaches dev until:
+  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
-  #   (1) dev has migrated to *.dev.home.keithjackson.dev and RETIRED its apex records, and
+  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
-  #   (2) PROD's own external-dns has published keycloak.home.keithjackson.dev at PROD's
+  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
-  #       Gateway VIP (192.168.2.20).
+  #
-  # Only then do the public answer and the in-cluster split-horizon answer AGREE, and no
+  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
-  # /etc/hosts pin is needed. Until then kubectl OIDC SSO on prod cannot work; the cert-based
+  # external-dns has actually published the record. Until then the issuer does not resolve
-  # admin kubeconfig is the break-glass path and always works. Ordering, not a design gap —
+  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
-  # the ordered steps are infra/PROD-PROVISIONING.md §10.
+  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
 
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
-# PROD **IS** THE APEX. [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml —
+# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
-# the same one-fact-in-two-places contract as k3s_data_dir and k3s_api_host. The Flux side
+# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
-# cannot supply this: the kube-apiserver's oidc-issuer-url is a k3s server FLAG written to
+# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
-# /etc/rancher/k3s/config.yaml, not a manifest, so postBuild.substituteFrom never sees it.
+# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
-# Full rationale in roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url derives.
+# postBuild.substituteFrom never sees it. Full rationale in
+# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
 #
-# WHY THE SUFFIXES SPLIT (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve
+# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
-# the SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
+# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
-# deleted every one of dev's public records.
+# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
-#
+# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
-# WHICH WAY ROUND — INVERTED 2026-09-04 by operator decision, superseding the interim split
+# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
-# committed earlier the same day (prod on `prod.home.` behind apex CNAMEs). The simpler target:
+# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
-#   prod -> home.keithjackson.dev       (THE APEX; this variable — no `prod.` label, no CNAMEs)
+k3s_base_domain: "prod.home.keithjackson.dev"
-#   dev  -> dev.home.keithjackson.dev   (roles/k3s_server/defaults/main.yml)
-# So prod's Keycloak issuer is https://keycloak.home.keithjackson.dev/realms/homelab, and the
-# `kubernetes` OIDC client in PROD's OWN Keycloak realm is what prod's kubectl SSO authenticates
-# against.
-#
-# ORDERING HAZARD SPECIFIC TO THIS VARIABLE. keycloak.home.keithjackson.dev is the name the LIVE
-# dev cluster answers today, and prod's nodes resolve the issuer through PUBLIC upstreams (see
-# node_baseline_dns_search above). So until dev has vacated the apex and PROD's external-dns has
-# published that name at PROD's Gateway VIP (192.168.2.20), a prod node resolving its own issuer
-# gets DEV's VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. kubectl OIDC SSO on prod
-# therefore does not work until §10's dev-vacate steps are complete; the cert-based admin
-# kubeconfig is the break-glass path and always works. Ordering, not a design gap.
-k3s_base_domain: "home.keithjackson.dev"
 
 # --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
 # roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE

infra/ansible/playbooks/kube-vip.yml

@@ -3,6 +3,13 @@
 # Control-plane ONLY (svc_enable=false) — Cilium owns Service LoadBalancing.
 # Writes the manifest into each server's k3s auto-deploy dir; k3s applies it.
 #
+# PART OF THE ONE-COMMAND BRING-UP SINCE 2026-09-06, not a separate operator step:
+# playbooks/site.yml imports this play BETWEEN k3s.yml and cilium.yml, because Cilium's
+# k8sServiceHost is now this VIP (k3s_apiserver_endpoint, group_vars/all.yml) and the CNI
+# cannot be installed against an endpoint nothing announces yet. The role's final task waits
+# for the VIP to answer on 6443, so a failure to float it stops the run here with a named
+# cause instead of surfacing later as unreachable cilium-agents.
+#
 # TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
 #   ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
 # (the VIP itself comes from kube_vip_address, already scoped per environment in

infra/ansible/playbooks/site.yml

@@ -1,7 +1,10 @@
 ---
 # Full dev/test cluster bring-up, in order (docs/02 sequencing; run AFTER
 # `tofu apply` has created and cloud-init has baselined the VMs):
-#   k3s HA install -> Cilium CNI -> verify readiness -> Flux + SOPS hand-off.
+#   k3s HA install -> kube-vip control-plane VIP -> Cilium CNI -> verify readiness
+#   -> Flux + SOPS hand-off.
+# (kube-vip joined the chain 2026-09-06 — see the note at its import below; it is a
+# prerequisite of the CNI now, not a separate operator step.)
 #
 # TARGET (R-ENV-02): every imported play takes its host pattern from a var that
 # defaults to the dev group, so `ansible-playbook playbooks/site.yml` with no extra
@@ -20,6 +23,29 @@
 - name: Import - install k3s HA control plane
   ansible.builtin.import_playbook: k3s.yml
 
+# ADDED 2026-09-06, AND ITS POSITION IS LOAD-BEARING: kube-vip MUST come up between k3s and
+# Cilium. Cilium's k8sServiceHost is now the control-plane VIP (k3s_apiserver_endpoint =
+# kube_vip_address) rather than the founder's own node IP, so installing Cilium before the
+# VIP is announced would point every agent at an address nothing answers on — the exact
+# shape of the 2026-09-01 incident, where agents that could not reach the API server
+# deadlocked on a cold cluster.
+#
+# IT CANNOT COME EARLIER. kube-vip ships as a k3s auto-deploy manifest
+# (<data-dir>/server/manifests), so a cluster must exist before it can be applied — hence
+# after k3s.yml, not before.
+#
+# IT DOES NOT NEED THE CNI, which is what makes this ordering possible at all: the kube-vip
+# DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 (roles/kube_vip's template
+# explains why), so it reaches the node-local apiserver with no CNI and no kube-proxy, and
+# DaemonSet pods carry the default not-ready/network-unavailable tolerations that let them
+# schedule onto a still-NotReady node. roles/kube_vip blocks until the VIP actually answers
+# on 6443, so cilium.yml below cannot start against a VIP that is not up.
+#
+# Re-running this against the LIVE dev cluster is a no-op for this play: dev already runs
+# kube-vip with the same rendered manifest, and the wait finds the VIP already answering.
+- name: Import - float the control-plane HA VIP (kube-vip) BEFORE the CNI dials it
+  ansible.builtin.import_playbook: kube-vip.yml
+
 - name: Import - install Cilium CNI
   ansible.builtin.import_playbook: cilium.yml
 

infra/ansible/roles/cilium/defaults/main.yml

@@ -2,7 +2,17 @@
 # cilium role defaults (ADR-0002, docs/02 Cilium-ownership contract).
 # This role performs the INITIAL Cilium install; the Flux platform/cilium
 # HelmRelease adopts config afterward. The parity-critical values
-# (kubeProxyReplacement, k8sServiceHost/Port) are templated from k3s_api_host.
+# (kubeProxyReplacement, k8sServiceHost/Port) are templated from
+# k3s_apiserver_endpoint / k3s_api_port (group_vars/all.yml).
+#
+# k8sServiceHost IS THE HA VIP, NOT THE FOUNDER (changed 2026-09-06). It reads
+# k3s_apiserver_endpoint — which derives from kube_vip_address, so it is dev 192.168.1.49 /
+# prod 192.168.2.9 — and deliberately NOT k3s_api_host, the founder's own node IP. Those two
+# were one variable until 2026-09-06, and the conflation is what left all three prod
+# cilium-agents dialing k3s-prod-1 on an HA cluster. Do not re-merge them.
+#
+# THIS ROLE THEREFORE DEPENDS ON THE VIP EXISTING. playbooks/site.yml runs kube-vip.yml
+# before cilium.yml and roles/kube_vip blocks until the VIP answers on 6443.
 
 # Exact pin — MUST equal the platform/cilium HelmRelease chart.spec.version (1.20.1, an
 # exact pin, not a range). Renovate bumps both together.

infra/ansible/roles/cilium/tasks/main.yml

@@ -89,7 +89,7 @@
   run_once: true
   changed_when: false
 
-- name: Render the Cilium values file (parity-critical settings from k3s_api_host)
+- name: Render the Cilium values file (parity-critical settings from k3s_apiserver_endpoint)
   ansible.builtin.template:
     src: cilium-values.yaml.j2
     dest: "{{ cilium_values_path }}"

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -4,9 +4,24 @@
 ---
 # [parity] k3s kube-proxy is disabled → Cilium replaces it (ADR-0001/0002).
 kubeProxyReplacement: true
-# [parity] = the k3s API endpoint (primary node IP). The Flux HelmRelease's
+# [parity] = k3s_apiserver_endpoint, THE HA CONTROL-PLANE VIP (kube_vip_address) — dev
-# k8sServiceHost "REPLACE-ME" must be set to this same value.
+# 192.168.1.49, prod 192.168.2.9. NOT k3s_api_host, which is the FOUNDER'S OWN node IP:
-k8sServiceHost: "{{ k3s_api_host }}"
+# pointing the CNI's kube-proxy replacement at one machine on a 3-server HA cluster means
+# losing that machine stops every service/policy update reaching the datapath, silently
+# (the existing datapath keeps working from Cilium's cache, so nothing looks broken). That
+# was the live state of prod until 2026-09-06 — all three cilium-agents carried
+# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held .9. See group_vars/all.yml.
+#
+# [parity] The Flux side is platform/cilium/helmrelease.yaml's k8sServiceHost:
+# "${K3S_API_HOST}", whose per-environment value comes from clusters/<env>/kustomization.yaml.
+# THAT VALUE MUST EQUAL k3s_apiserver_endpoint for the SAME environment. If it does not,
+# Flux and this bootstrap render different control-plane paths and fight over the CNI.
+#
+# THE VIP MUST BE UP BEFORE THIS ROLE RUNS. playbooks/site.yml imports kube-vip.yml between
+# k3s.yml and cilium.yml for exactly this reason, and roles/kube_vip blocks until the VIP
+# answers. Rendering this against a VIP that nothing announces yet reproduces the
+# 2026-09-01 shape: agents that cannot reach the API server on a cold cluster.
+k8sServiceHost: "{{ k3s_apiserver_endpoint }}"
 k8sServicePort: {{ k3s_api_port }}
 
 # Ingress via Gateway API — no Traefik (docs/09, ADR-0002).

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -59,6 +59,17 @@ k3s_server_mode: "secondary"
 # serving cert on restart when this set changes. Deduplicated (the primary IP is
 # also k3s_api_host).
 #
+# THIS LIST COVERS BOTH API ADDRESSES, WHICH IS WHY THE 2026-09-06 SPLIT NEEDED NO CERT
+# CHANGE AND NO k3s RESTART. group_vars/all.yml now distinguishes k3s_api_host (the
+# founder, for the cold join) from k3s_apiserver_endpoint (= kube_vip_address, what Cilium
+# and every steady-state client dial). Both are already in this set, so repointing a
+# consumer from one to the other cannot produce a TLS failure. VERIFIED LIVE 2026-09-06,
+# not assumed: prod's apiserver answered through https://192.168.2.9:6443 with full CA
+# verification and tls-server-name=192.168.2.9. Do not remove either entry — dropping
+# kube_vip_address breaks every client of the VIP; dropping k3s_api_host breaks nothing
+# today only because it is also in the k3s_servers_group node-IP list below, and that
+# coincidence is not a contract.
+#
 # k3s_servers_group — NOT a literal group name — is what makes this role environment-
 # agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
 # group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
@@ -84,26 +95,15 @@ k3s_tls_sans: >-
 # while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
 # not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
 #
-# THE DEFAULT IS THE DEV VALUE — the same convention as k3s_data_dir above and cluster-vars'
+# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
-# GATEWAY_VIP / LONGHORN_DATA_PATH: the role default is dev's value and prod overrides it in
+# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
-# inventory/group_vars/k3s_prod/vars.yml (the same inventory seam k3s_data_dir / k3s_api_host /
+# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
-# kube_vip_address use, so a FOURTH prod server inherits it from group membership rather than
+# what it already is and the Restart handler does NOT fire on a re-run.
-# from anyone remembering).
 #
-# *** WHICH DEV VALUE — UPDATED 2026-09-04, AND IT IS AHEAD OF THE LIVE DEV CLUSTER. ***
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
-# The operator inverted the suffix split: PROD is now the apex `home.keithjackson.dev` and DEV
+# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
-# carries the label `dev.home.keithjackson.dev`. This default therefore moved from the apex to
+# from group membership rather than from anyone remembering.
-# dev's subdomain, mirroring BASE_DOMAIN in clusters/dev/kustomization.yaml.
+k3s_base_domain: "home.keithjackson.dev"
-#
-# CONSEQUENCE — this default no longer matches what the live dev nodes are running, so unlike
-# every previous edit to this file A DEV RE-RUN IS NOT A NO-OP. Rendering config.yaml with the
-# new issuer CHANGES the file, FIRES the Restart handler, and rolls k3s on all three dev
-# servers, which also invalidates every user's OIDC kubeconfig (SSO must be re-issued). That is
-# a deliberate step of the dev DNS migration, NOT a side effect to discover during unrelated
-# work: DO NOT run the k3s_server role against dev until step D6 of
-# infra/PROD-PROVISIONING.md §10, and never while the operator is away. The break-glass
-# cert-based admin kubeconfig keeps working throughout, which is what makes the step safe.
-k3s_base_domain: "dev.home.keithjackson.dev"
 
 # Derived, so the realm path is stated once. Override k3s_base_domain, not this.
 k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -155,6 +155,18 @@
          | default(k3s_primary_token_raw.content | b64decode | trim, true) }}
   when: k3s_server_mode == 'secondary'
 
+# THE JOIN TARGET IS k3s_api_host — THE FOUNDER — AND MUST NOT BECOME THE VIP.
+# group_vars/all.yml splits the two API addresses (k3s_api_host = the founder's own node IP,
+# k3s_apiserver_endpoint = the kube-vip HA VIP) and this is the canonical founder-only
+# consumer. On a COLD build the VIP does not exist at this point in the sequence: kube-vip
+# is a k3s auto-deploy manifest, so it cannot be applied until a cluster exists, and
+# playbooks/site.yml only imports kube-vip.yml AFTER k3s.yml. A `--server` pointed at the
+# VIP would therefore dial an address nothing announces and the cold build would be
+# unjoinable — with no diagnostic beyond a connection timeout.
+# On a WARM single-node rejoin (`--limit k3s-dev-N`) the founder is up and answering too, so
+# the founder works in both cases and the VIP works in only one. Cilium's k8sServiceHost is
+# the opposite trade (steady state, must survive losing the founder) and uses the endpoint
+# variable instead.
 - name: Install k3s secondary (join as server)
   ansible.builtin.command:
     cmd: >-

infra/ansible/roles/kube_vip/defaults/main.yml

@@ -24,6 +24,16 @@ kube_vip_port: 6443
 # only if detection is wrong.
 kube_vip_interface: ""
 
+# How long to wait for the VIP to actually ANSWER on kube_vip_port after the manifest is
+# written (seconds). This is a real gate, not cosmetics: since 2026-09-06 Cilium's
+# k8sServiceHost is this VIP (k3s_apiserver_endpoint), and playbooks/site.yml runs this role
+# immediately before cilium.yml — so if the VIP never comes up, the run must stop HERE with
+# a clear message rather than proceed to install a CNI pointed at an address nothing
+# answers. On a cold cluster the budget covers k3s's deploy controller applying the
+# manifest, the ghcr.io image pull and leader election; on an already-running cluster the
+# check returns immediately.
+kube_vip_wait_timeout: 300
+
 # Leader-election tuning (seconds) — a snappy failover without thrashing.
 kube_vip_lease_duration: 15
 kube_vip_renew_deadline: 10

infra/ansible/roles/kube_vip/tasks/main.yml

@@ -21,6 +21,14 @@
     executable: /bin/bash
   register: kube_vip_iface_detect
   changed_when: false
+  # RUN THIS EVEN UNDER --check. It is `ip -o -4 addr show`: it reads and changes nothing.
+  # Without check_mode: false the shell module is SKIPPED in check mode, kube_vip_iface_detect
+  # has no stdout, and the assert two tasks down aborts the play with "Could not detect the
+  # interface ... set kube_vip_interface explicitly" — a misleading failure that says nothing
+  # about the node. That mattered less while this play was an out-of-band operator step; it
+  # matters now that playbooks/site.yml imports it, because `--check` on the bring-up before a
+  # cold prod build is exactly the rehearsal an operator should be able to run.
+  check_mode: false
 
 - name: Resolve the kube-vip VIP interface (override wins, else detected)
   ansible.builtin.set_fact:
@@ -52,3 +60,34 @@
     owner: root
     group: root
     mode: "0600"
+
+# PROVE THE VIP IS ACTUALLY UP BEFORE THIS PLAY RETURNS (added 2026-09-06).
+#
+# Templating the manifest is not the same as having a floating endpoint: k3s's deploy
+# controller still has to apply it, the image still has to pull, and leader election still
+# has to pick an owner. Nothing used to check, which was tolerable while the VIP was only an
+# operator convenience. It is not tolerable now — Cilium's k8sServiceHost IS this VIP
+# (k3s_apiserver_endpoint) and playbooks/site.yml runs cilium.yml straight after this play,
+# so an unannounced VIP would mean installing a CNI that cannot reach the API server. That
+# failure is the 2026-09-01 shape and it is very expensive to diagnose from the far side;
+# failing here instead names the actual cause.
+#
+# run_once: the VIP is ONE address for the whole environment, so one probe answers for the
+# environment. Probing from a server host (not the controller) is deliberate: the servers
+# are on the VIP's own LAN in both environments, whereas the dev controller reaches dev's
+# API only through a pfSense NAT — a controller-side probe would have been a property of
+# where the play was launched from. READ-ONLY: it opens a TCP connection and changes nothing.
+- name: Wait for the control-plane VIP to answer (the CNI's endpoint depends on it)
+  ansible.builtin.wait_for:
+    host: "{{ kube_vip_address }}"
+    port: "{{ kube_vip_port }}"
+    timeout: "{{ kube_vip_wait_timeout }}"
+  run_once: true
+  changed_when: false
+  vars:
+    ansible_become: false
+  # DELIBERATELY NOT check_mode: false, unlike the interface probe above — so a `--check`
+  # rehearsal SKIPS this and does not sit on a socket for kube_vip_wait_timeout. wait_for
+  # declares no check-mode support, so the skip is the module's own behaviour and this comment
+  # is here only so the skip does not read as an oversight. CONSEQUENCE, stated plainly: a
+  # `--check` run does NOT verify the VIP is up. Only a real run gates the CNI on it.

platform-config-cdi/kustomization.yaml

@@ -0,0 +1,41 @@
+---
+# platform-config-cdi — the SECOND-ORDER config wave: custom resources whose CRD is created at
+# RUNTIME by a controller that a `platform-config` custom resource switches on.
+#
+# ── THE BUG THIS DIRECTORY EXISTS TO FIX (observed on PROD, 2026-09-05) ───────────────────────────
+# The repo already splits CONTROLLERS (`platform`) from CONFIGS (`platform-config`) because a CR
+# cannot be dry-run before its CRD is registered. That split assumes a CRD is registered by the
+# controller INSTALL. CDI breaks the assumption: `cdi-operator.yaml` registers exactly ONE CRD
+# (`cdis.cdi.kubevirt.io`). Every OTHER CDI CRD — `storageprofiles`, `datavolumes`, `datasources`,
+# `dataimportcrons`, … — is created by cdi-operator only AFTER it has reconciled the `CDI` CR.
+#
+# So `cdi-cr.yaml` is a FIRST-order CR (its CRD comes from the install) and belongs in
+# platform-config, while `cdi-storageprofiles.yaml` is a SECOND-order CR (its CRD comes from
+# reconciling the first-order CR). Having both in ONE Flux Kustomization is a true DEADLOCK, not a
+# race: kustomize-controller server-side dry-runs EVERY object before applying ANY of them, so the
+# unknown `StorageProfile` kind aborts the whole pass → the `CDI` CR is never applied → the
+# StorageProfile CRD is never created → platform-config can NEVER converge, and every tier that
+# dependsOn it stays blocked forever. On prod, cdi-operator sat Available for 12h with no CR:
+#
+#   StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile"
+#   in version "cdi.kubevirt.io/v1beta1"
+#
+# Invisible on dev (whose CDI CRDs have existed for months) and fatal on any COLD cluster.
+#
+# ── THE RULE, so this is not rediscovered a third time ────────────────────────────────────────────
+# A CR belongs HERE (not in platform-config) when the CRD it needs is created by a CONTROLLER AT
+# RUNTIME rather than by a manifest in git. Grep test: if `grep -rn "kind: CustomResourceDefinition"`
+# over ./platform (following remote bases) does not produce the CRD, and no chart installs it, then
+# some operator creates it after being switched on — and this is the tier for it.
+#
+# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
+# manifests stay beside the component they configure). Build/verify locally with:
+#   kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config-cdi
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  # CDI StorageProfiles — snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
+  # `longhorn-vm` virt-default class (clears CDIDefaultStorageClassDegraded). `StorageProfile`'s CRD
+  # is created by cdi-operator only once the `CDI` CR (applied by platform-config) is reconciled,
+  # which is exactly why these cannot live in the same pass as that CR.
+  - ../platform/kubevirt/cdi-storageprofiles.yaml

platform-config/kustomization.yaml

@@ -25,6 +25,14 @@ resources:
   - ../platform/kyverno/policies/restrict-host-namespaces.yaml
   - ../platform/kyverno/policies/require-egress-class-label.yaml
   - ../platform/kyverno/policies/require-auth-mode-label.yaml
+  # COLD-START FIX (2026-09-06, prod): a MUTATE policy that must travel WITH the two label
+  # policies above. CDI generates its own short-lived worker pods (importer-*, uploader-*,
+  # cloner-*) inside app namespaces; nothing in git can label them, so both policies DENIED them
+  # and `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled with no disk. This policy adds
+  # the two labels to pods created by ServiceAccount cdi/cdi-sa, so the requirement is SATISFIED
+  # rather than bypassed. Applying it here (not later) matters: if it lands after the validating
+  # policies, the next import is blocked in the gap.
+  - ../platform/kyverno/policies/label-cdi-worker-pods.yaml
   - ../platform/kyverno/admission/secret-provenance.yaml
   # cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
   - cilium/l2-ip-pool.yaml
@@ -41,9 +49,23 @@ resources:
   # ConfigMap kept beside them for locality.
   - ../platform/kubevirt/kubevirt-cr.yaml
   - ../platform/kubevirt/cdi-cr.yaml
-  # CDI StorageProfiles: snapshot smart-clone for `longhorn`, and RWX/Block + snapshot clone for the
+  # *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
-  # `longhorn-vm` virt-default class (consume cdi.kubevirt.io; clears CDIDefaultStorageClassDegraded).
+  # The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
-  - ../platform/kubevirt/cdi-storageprofiles.yaml
+  # (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
+  #
+  # `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
+  # `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
+  # only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
+  # in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
+  # `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
+  # created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
+  # A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
+  #   StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
+  #   "cdi.kubevirt.io/v1beta1"
+  # Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
+  #
+  # THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
+  # is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
   - ../platform/kubevirt/servicemonitor.yaml
   - ../platform/kubevirt/alerts.yaml
   - ../platform/kubevirt/dashboard.yaml

platform/cilium/helmrelease.yaml

@@ -10,7 +10,8 @@
 #
 # SOURCE OF TRUTH (do not diverge without updating both):
 #   infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
-#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version, k3s_api_host/port)
+#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version)
+#   infra/ansible/inventory/group_vars/all.yml     (k3s_apiserver_endpoint, k3s_api_port)
 #
 # Every value tagged [parity] below is asserted identical to the Ansible render. If you change one
 # here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.)
@@ -45,11 +46,35 @@ spec:
     # [parity] MUST equal Ansible: kubeProxyReplacement: true
     kubeProxyReplacement: true
     # [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint
-    # (k3s_api_host/k3s_api_port). This is the primary control-plane node IP.
+    # (**k3s_apiserver_endpoint** / k3s_api_port, infra/ansible/inventory/group_vars/all.yml).
     # Setting these wrong breaks Cilium's connection to the API server on adopt.
-    # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.50 | prod set in clusters/prod.
+    #
-    # The `platform` Flux Kustomization substitutes it; the Ansible side is already parameterized
+    # THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS — operator decision 2026-09-06.
-    # from inventory (k3s_api_host), so the parity contract holds per-environment, not per-literal.
+    # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.49 | prod 192.168.2.9, each equal to
+    # that environment's kube_vip_address. The knob's NAME still says "host"; read it as "the API
+    # host Cilium dials", and see clusters/base/cluster-vars.yaml for why it was not renamed.
+    #
+    # WHY, in one line: this value used to be the PRIMARY NODE's IP, so on a 3-server HA cluster the
+    # CNI's control-plane path depended on one machine. Verified on live prod 2026-09-06 — every
+    # cilium-agent carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1) while kube-vip held .9.
+    # Losing that node does not break the datapath (Cilium keeps serving from cache); it silently
+    # stops every service and policy update from ever landing, which is worse to detect.
+    #
+    # [parity] IS NOW AGAINST k3s_apiserver_endpoint AND NOT k3s_api_host. Those were one Ansible
+    # variable until 2026-09-06 and the conflation is the defect. k3s_api_host still exists and
+    # still holds the FOUNDER's own address, because roles/k3s_server's secondary
+    # `--server https://<host>:6443` join must target a host that is already up — on a COLD build
+    # no VIP exists yet. So: founder for the join, VIP for the CNI. If you change this line, change
+    # cilium-values.yaml.j2 in the same PR, and point it at the SAME variable.
+    #
+    # The `platform` Flux Kustomization substitutes it; the Ansible side is parameterized from
+    # inventory (k3s_apiserver_endpoint derives from kube_vip_address), so the parity contract holds
+    # per-environment, not per-literal.
+    #
+    # NEEDS NO CERT WORK: k3s_tls_sans has always carried kube_vip_address alongside every node IP,
+    # so both addresses are already SANs on the serving cert (verified live 2026-09-06 — prod
+    # answers through https://192.168.2.9:6443 under full CA verification, no insecure skip). This
+    # switch reissues nothing and restarts no apiserver.
     k8sServiceHost: "${K3S_API_HOST}"
     k8sServicePort: 6443
     # --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 ---

platform/cilium/values.yaml

@@ -6,16 +6,20 @@
 #
 # SOURCE OF TRUTH (all [parity] fields MUST equal these):
 #   infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
-#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version: 1.20.1, k3s_api_host/port)
+#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version: 1.20.1)
+#   infra/ansible/inventory/group_vars/all.yml     (k3s_apiserver_endpoint = the kube-vip VIP,
+#                                                   k3s_api_port) — NOT k3s_api_host, which is the
+#                                                   founder's own address and bootstrap-only
 #
 # chart version (helmrelease.yaml chart.spec.version): 1.20.1   # [parity] = cilium_version
 
 # [parity] k3s kube-proxy disabled → Cilium replaces it (ADR-0001/0002)
 kubeProxyReplacement: true
-# [parity] = the k3s API endpoint (primary control-plane node IP) + port.
+# [parity] = the k3s API endpoint + port. THE CONTROL-PLANE VIP (kube-vip), not the primary node —
+# changed 2026-09-06; the Ansible half is k3s_apiserver_endpoint, NOT k3s_api_host.
 # Mirrors the HelmRelease's ${K3S_API_HOST} cluster-vars token (R-ENV-02). This file is NOT a
 # kustomize resource, so nothing substitutes it — the token is shown with its dev value beside it.
-k8sServiceHost: "${K3S_API_HOST}"   # dev: 192.168.1.50
+k8sServiceHost: "${K3S_API_HOST}"   # the CONTROL-PLANE VIP — dev: 192.168.1.49 | prod: 192.168.2.9
 k8sServicePort: 6443
 # [parity] ingress via Gateway API, no Traefik (docs/09)
 gatewayAPI:

platform/external-snapshotter/README.md

@@ -20,8 +20,12 @@ Neither k3s nor the Longhorn chart ships this. It is a prerequisite for:
 ## Wiring
 
 - **Pass:** `platform` (controllers). Registers the CRDs.
-- The `VolumeSnapshotClass` / `StorageProfile` CRs that consume the CRDs live in the
+- The `VolumeSnapshotClass` CR that consumes the CRDs lives in the `platform-config` pass
-  `platform-config` pass (`dependsOn: platform`), per the controllers/configs split.
+  (`dependsOn: platform`), per the controllers/configs split.
+- The CDI `StorageProfile`s live one pass further out, in `platform-config-cdi`
+  (`dependsOn: platform-config`) — `storageprofiles.cdi.kubevirt.io` is created by cdi-operator only
+  after it reconciles the `CDI` CR, so they cannot share a pass with it. See
+  `clusters/base/platform-config-cdi.yaml`.
 
 ## Version
 

platform/external-snapshotter/kustomization.yaml

@@ -9,8 +9,10 @@
 #   * no VolumeSnapshotClass can be created (item 2), and
 #   * CDI cannot do SNAPSHOT smart-clone (item 3) — it falls back to slow host-assisted `copy`,
 #     which leaves the default virt StorageClass "degraded" (CDIDefaultStorageClassDegraded).
-# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass + the CDI
+# Installing it here (controllers pass) registers the CRDs; the VolumeSnapshotClass that CONSUMES them
-# StorageProfiles that CONSUME them are applied by platform-config (dependsOn: platform).
+# is applied by platform-config (dependsOn: platform), and the CDI StorageProfiles that set
+# `cloneStrategy: snapshot` by platform-config-cdi (dependsOn: platform-config — their own CRD is
+# created by cdi-operator only after the `CDI` CR reconciles; see clusters/base/platform-config-cdi.yaml).
 #
 # CONTROLLERS/CONFIGS SPLIT (F-MON1): this is the CONTROLLERS half. The snapshot.storage.k8s.io
 # CRDs are registered by the `client/config/crd` base below; the snapshot-controller Deployment

platform/externaldns/helmrelease.yaml

@@ -39,33 +39,24 @@ spec:
     # garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
     # Full write-up: infra/PROD-PROVISIONING.md §10.
     #
-    # THE TWO GUARDS THAT ACTUALLY PREVENT A RECURRENCE — note that neither is domainFilters:
+    # TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence:
-    #   1. txtOwnerId  — per-environment identity; a cluster only ever touches records it owns.
+    #   1. txtOwnerId       — per-environment identity; a cluster only ever touches records it owns.
-    #   2. DISJOINT HOSTNAME SETS — the two environments serve different suffixes, so they never
+    #   2. domainFilters    — per-environment name scope; prod cannot even see names outside its
-    #      describe the same record in the first place. dev = *.dev.home.keithjackson.dev,
+    #                         own subtree. Defence in depth, added after the incident.
-    #      prod = *.home.keithjackson.dev (prod IS the apex; operator decision 2026-09-04).
     # ---------------------------------------------------------------------------------------------
-    # Which names this instance may manage at all. BOTH ENVIRONMENTS RENDER THE CLOUDFLARE **ZONE**
+    # Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit
-    # `keithjackson.dev` HERE, and this is NOT an oversight to be tightened.
+    # 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is
-    #
+    # a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record
-    # FINDING F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for ZONE
+    # DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the
-    # SELECTION as well as record matching, so a filter that is a STRICT SUBDOMAIN of the hosted zone
+    # apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no
-    # matches NO zone — external-dns generates the record and then skips it with "no hosted zone
+    # matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive)
-    # matching record DNS Name", publishing NOTHING. This has been "improved" and reverted more than
+    # and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving
-    # once (an agent narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in
+    # txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml.
-    # review). It is not defence in depth; it is a total outage of DNS publication. Isolation comes
-    # from the two guards above, which a domain filter cannot provide anyway — both environments live
-    # in the ONE zone regardless of what this says. See clusters/base/cluster-vars.yaml.
     domainFilters:
       - "${EXTERNALDNS_DOMAIN_FILTER}"
     # The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
-    # caused the incident. dev renders `dev`, prod renders `prod`, and the BASE default is a
+    # caused the incident. Dev renders `default` (its live records are already registered under that
-    # non-colliding sentinel so a new environment that forgets the knob owns nothing and can delete
+    # owner; changing it would orphan them and duplicate every record). Prod renders `prod`.
-    # nothing. NOTHING renders `default` any more (the external-dns chart default, and the value both
-    # clusters silently shared): dev held it only because its live apex records were registered under
-    # it, and dev's move to *.dev.home.keithjackson.dev retired that constraint along with those
-    # records. Prod must NOT be given `default` even though it inherits the apex NAMES — that would
-    # hand prod ownership of dev's old apex records and let `policy: sync` below delete them.
     txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
     policy: sync            # create + update + delete records as services come and go — but ONLY
                             # records this owner id owns AND this domainFilter admits (see above).

platform/kubevirt/cdi-storageprofiles.yaml

@@ -1,9 +1,19 @@
 ---
 # CDI StorageProfiles — the storage "recipes" CDI uses when it provisions/clones VM disk PVCs
-# (ADR-0005). CONSUME the cdi.kubevirt.io CRD registered by the CDI operator (platform pass), so —
+# (ADR-0005).
-# like cdi-cr.yaml — these CRs are applied by the `platform-config` Flux Kustomization
+#
-# (dependsOn: platform). They live here beside the KubeVirt/CDI component and are referenced in
+# APPLIED BY `platform-config-cdi`, NOT `platform-config` (moved 2026-09-05). These are SECOND-ORDER
-# place from platform-config/kustomization.yaml.
+# CRs: `storageprofiles.cdi.kubevirt.io` is NOT registered by the CDI operator install (cdi-operator.yaml
+# ships only `cdis.cdi.kubevirt.io`) — cdi-operator creates it only AFTER it has reconciled the `CDI` CR
+# that platform-config applies. Flux server-side dry-runs every object in a pass before applying any of
+# them, so co-locating these with that CR was a TRUE DEADLOCK: unknown kind -> pass aborts -> CR never
+# applied -> CRD never created -> platform-config never converges -> every tier below it blocked. It was
+# invisible on dev (CDI CRDs long present) and cost 12h on prod's cold start, 2026-09-05.
+# They live here beside the KubeVirt/CDI component and are referenced in place from
+# platform-config-cdi/kustomization.yaml (dependsOn: platform-config, revision-gated).
+#
+# `apps-vms` dependsOn platform-config-cdi for these: a DataVolume created before the longhorn-vm
+# profile below exists is stamped RWO/Filesystem and is permanently not live-migratable.
 #
 # CDI AUTO-CREATES a StorageProfile per StorageClass and fills its STATUS (inferred capabilities).
 # The operator owns `.status`; the USER owns `.spec`. Setting `.spec` here does NOT fight CDI — it

platform/kubevirt/kustomization.yaml

@@ -8,6 +8,16 @@
 # (dependsOn: platform, wait) so the CRDs exist BEFORE the CRs apply. Applying both in one atomic pass
 # deadlocks ("no matches for kind KubeVirt"), exactly the failure F-MON1 documents.
 #
+# THERE IS A THIRD PASS FOR CDI, and it is not optional (2026-09-05). The two operator manifests below
+# register exactly ONE CDI CRD, `cdis.cdi.kubevirt.io`. Every other CDI CRD — `storageprofiles`,
+# `datavolumes`, `datasources`, `dataimportcrons` — is created by cdi-operator only AFTER it reconciles
+# the `CDI` CR. So `cdi-storageprofiles.yaml` is a SECOND-ORDER CR and CANNOT share a pass with
+# `cdi-cr.yaml`: the dry-run of the unknown `StorageProfile` kind aborts the pass, the CR is never
+# applied, the CRD is never created, and platform-config deadlocks (cost 12h on prod's cold start).
+# It is therefore applied by a third Kustomization, `platform-config-cdi` (dependsOn platform-config).
+# Before adding any new CDI CR here, ask which pass owns it: CRD in git -> platform-config;
+# CRD created by cdi-operator at runtime -> platform-config-cdi.
+#
 # Namespaces: the operator manifests each create their own namespace (`kubevirt`, `cdi`) with the
 # pod-security profile they need (kubevirt is privileged) — so we do NOT redeclare them here (that
 # would be a duplicate-resource kustomize error). These are PLATFORM namespaces: they intentionally

platform/kyverno/README.md

@@ -22,14 +22,19 @@ platform/kyverno/
 │   ├── disallow-plaintext-secrets.yaml      # R-SEC-01 (CI-only: sops block required)
 │   ├── restrict-host-namespaces.yaml        # R-SEC-02 (+R-NET-01 bypass guard)
 │   ├── require-egress-class-label.yaml      # R-NET-05
-│   └── require-auth-mode-label.yaml         # R-IDP-02
+│   ├── require-auth-mode-label.yaml         # R-IDP-02
+│   └── label-cdi-worker-pods.yaml           # R-NET-05+R-IDP-02 (mutate: CDI's own worker pods)
 ├── servicedefinition/            # ServiceDefinition shape validation (kyverno-json)
 │   ├── validate-servicedefinition.yaml      # R-GITOPS-02
 │   ├── payloads/ (valid-* / invalid-*)      # pass & fail fixtures
 │   └── README.md
 ├── tests/                        # `kyverno test` suite — runnable L0 tests NOW, no cluster
 │   ├── kyverno-test.yaml                     # policies × fixtures × expected results
-│   └── resources/ (good-* / bad-* fixtures + generated golden)
+│   ├── resources/ (good-* / bad-* fixtures + generated golden)
+│   └── controller-worker-pods/               # the CDI-worker mutation, proven BOTH ways
+│       ├── kyverno-test.yaml                 #   2 Tests: cdi-sa requester vs ordinary requester
+│       ├── userinfo-cdi-sa.yaml / userinfo-ordinary.yaml
+│       └── resources/ + patched/             #   real + FORGED importer pods; expected patch
 ├── admission/                    # in-cluster enforcing bundle (scaffolding, T-2.7)
 │   ├── namespace.yaml · helmrelease.yaml · kustomization.yaml
 │   ├── secret-provenance.yaml                # R-SEC-01 admission variant (provenance)
@@ -47,6 +52,7 @@ platform/kyverno/
 | `restrict-host-namespaces` | `disallow-host-namespaces`, `disallow-privileged-containers` | **R-SEC-02** (+R-NET-01) | docs/07 least reach | App pods can't use hostNetwork/PID/IPC or privileged (would bypass default-deny) |
 | `require-egress-class-label` | `require-egress-class` | **R-NET-05** | docs/03, docs/13 | App pods must declare `egress-class` ∈ {internet,vpn} |
 | `require-auth-mode-label` | `require-auth-mode` | **R-IDP-02** | docs/13, docs/15 | App pods must declare `auth-mode` ∈ {oidc,forward-auth,native,none} |
+| `label-cdi-worker-pods` (mutate) | `add-worker-pod-labels` | **R-NET-05**, **R-IDP-02** | docs/03, docs/13 | CDI's **controller-generated** worker pods (`importer-*`/`uploader-*`/`cloner-*`, created by SA `cdi/cdi-sa`) get `egress-class: internet` + `auth-mode: none` **added**, so they satisfy the two policies above instead of being exempted from them. Matched on the **generating ServiceAccount**, never a pod-name prefix |
 | `validate-servicedefinition` (kyverno-json) | shape / dependencies / storage | **R-GITOPS-02** | docs/13 (+03,15) | ServiceDefinition is schema/invariant valid before render; public⇒not `none` auth |
 | `admission/secret-provenance` | `secrets-must-be-flux-managed` | **R-SEC-01** | docs/07 | **Admission-only** variant: app-ns Secrets must come from Flux/SOPS (see below) |
 
@@ -90,6 +96,19 @@ self-contained — no cluster, no external state (reproducibility contract, R-TE
 
 - **Label keys are a proposed contract** (`labels.md`) shared with WS-NET / WS-SEC /
   WS-GITOPS. Values are spec-fixed; keys need owner sign-off (coordination-protocol §2).
+- **Controller-generated pods are a cold-start hazard.** A policy that matches *any* Pod in an
+  app namespace also matches pods that no manifest in git authors. Prod hit this on 2026-09-06:
+  `require-auth-mode-label` + `require-egress-class-label` denied CDI's own `importer-prime-*`
+  pod, so `DataVolume unifi/unifi-rootdisk` stuck in `ImportScheduled` and the UniFi VM had no
+  disk. Fix: `policies/label-cdi-worker-pods.yaml` **mutates** those pods to carry truthful
+  labels (matched on the generating ServiceAccount `cdi/cdi-sa`), rather than excluding them.
+  `virt-launcher-*` pods need nothing — they inherit the VM pod-template labels. KubeVirt
+  hotplug (`hp-volume-*`) and export (`virt-export-*`) pods do **not** inherit them and are
+  unused here; extend the same policy if either is adopted.
+- **`PolicyException` is NOT usable in this install**: the Kyverno admission/background/reports
+  controllers all run `--enablePolicyException=false` (chart default, verified on prod
+  2026-09-06). The CRD exists (from the pre-applied CRD bundle) so a `PolicyException` would
+  APPLY cleanly and be silently ignored — a policy hole that looks like a fix and isn't.
 - **Secrets CI vs admission divergence** is deliberate and documented in
   `admission/README.md` and `policies/disallow-plaintext-secrets.yaml`.
 - The ServiceDefinition JSON Schema (WS-GITOPS T-0.3) is the complementary structural

platform/kyverno/admission/kustomization.yaml

@@ -13,6 +13,10 @@ resources:
   - ../policies/restrict-host-namespaces.yaml
   - ../policies/require-egress-class-label.yaml
   - ../policies/require-auth-mode-label.yaml
+  # mutate: gives CDI's OWN generated worker pods (importer-*/uploader-*/cloner-*) the two
+  # labels the policies above require -- they are controller-generated, so nothing in git can
+  # label them. Ships with them, never separately.
+  - ../policies/label-cdi-worker-pods.yaml
   # --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
   #     (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
   - secret-provenance.yaml

platform/kyverno/labels.md

@@ -15,6 +15,17 @@ Shared contract with WS-NET (egress rendering), WS-SEC (auth), and WS-GITOPS (re
 | `homelab.keithjackson.dev/baseline` | NetworkPolicy | `default-deny` | (convention) identifies the baseline NP | `require-default-deny-baseline` (generate), `validate-default-deny-shape` |
 | `homelab.keithjackson.dev/secret-managed-by` | Secret | `controller` | (convention) exempts controller-generated Secrets | `disallow-plaintext-secrets` (exclude), admission `secret-provenance` |
 
+## Controller-generated pods (who sets the labels when git cannot)
+
+`egress-class` / `auth-mode` are normally projected onto pod templates by the renderer (below).
+Some pods are generated by a **controller at runtime** and therefore have no manifest in git to
+label — CDI's `importer-*` / `uploader-*` / `cloner-*` worker pods are the live example: both
+label policies denied CDI's importer on prod (2026-09-06) and the UniFi VM got no disk. For those,
+`policies/label-cdi-worker-pods.yaml` **adds** the labels at admission (`+(…)` add-if-absent),
+matched on the **generating ServiceAccount** (`cdi/cdi-sa`) — never on a pod-name prefix or on
+CDI's own labels, both of which any pod author in the namespace can copy. `virt-launcher-*` needs
+nothing: KubeVirt propagates the VM pod-template labels onto it (verified on dev).
+
 ## Rendering expectation (for WS-GITOPS)
 
 The ServiceDefinition (`docs/13`) declares intent — `egress:` and `auth.mode:` — at the

platform/kyverno/policies/label-cdi-worker-pods.yaml

@@ -0,0 +1,136 @@
+# Serves: R-NET-05 (egress is a declared property) + R-IDP-02 (auth mode is declared) —
+# this policy does NOT weaken either requirement. It makes the one class of pod that
+# CANNOT carry the labels from git SATISFY them, instead of being exempted from them.
+#
+# THE DEFECT IT FIXES (observed on LIVE prod, 2026-09-05/06):
+#   `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled for hours holding a detached
+#   40Gi `prime-…` scratch PVC. `kubectl -n cdi logs deploy/cdi-deployment` showed:
+#     admission webhook "validate.kyverno.svc-fail" denied the request: resource
+#     Pod/unifi/importer-prime-<uid> was blocked due to the following policies
+#       require-auth-mode-label / require-egress-class-label
+#   CDI generates its own short-lived worker pods (importer-*, uploader-*, cloner-*/
+#   clone-source-*) from inside the cdi-deployment controller. Those pods are not in git,
+#   so nothing can put the platform labels on them, and both label policies match ANY Pod
+#   in a namespace labelled homelab.keithjackson.dev/app-namespace=true. Result: the VM
+#   never gets a disk. This is a COLD-START defect, not a prod quirk — dev's identical
+#   import Succeeded before the policies were enforcing, so dev has never exercised it and
+#   would hit it on any rebuild.
+#
+# WHY MUTATE (and not an exception/exclusion):
+#   The honest reading of R-NET-05/R-IDP-02 is "every pod's egress path and auth posture is
+#   DECLARED". A CDI importer pod has truthful values for both:
+#     egress-class: internet  — it pulls the image from the public internet over https
+#                               (cloud-images.ubuntu.com / quay.io) and must take the
+#                               default WAN path, which is exactly what Cilium's egress
+#                               gateway does for `internet` (ADR-0013/ADR-0015). It is NOT
+#                               a `vpn` workload.
+#     auth-mode: none         — the pod serves no UI and terminates when the import ends;
+#                               there is nothing to put an IdP in front of. docs/13 permits
+#                               `none` for internal, non-public workloads (the
+#                               ServiceDefinition policy forbids only public + none).
+#   So the requirement is SATISFIED rather than bypassed: the invariant "every pod in an app
+#   namespace carries both labels" stays literally true cluster-wide, both label policies
+#   stay untouched (no `exclude`, no PolicyException, background scanning still on), and the
+#   labels are functional — the pod is now selectable by the Cilium egress fabric instead of
+#   being an unlabelled pod that silently falls through to the default path.
+#
+# THE MATCH IS THE GENERATING ServiceAccount, NOT A NAME PREFIX:
+#   `subjects: ServiceAccount cdi/cdi-sa` is the identity in the AdmissionReview's userInfo —
+#   the controller that issued the CREATE. Verified on prod from the Kyverno admission log of
+#   the blocked request:
+#     user={"username":"system:serviceaccount:cdi:cdi-sa", ...} roles=["cdi:cdi-deployment"]
+#     name=importer-prime-be3f9d1c-… namespace=unifi operation=CREATE
+#   A pod NAME (`importer-*`) or CDI's own labels (`cdi.kubevirt.io`) are attacker-CHOOSABLE:
+#   anyone who can create a pod in the namespace can name it `importer-prime-x` and copy the
+#   labels, and would then inherit the exemption. The requesting ServiceAccount cannot be
+#   chosen by the pod author — only by someone who already holds cdi-sa's token, which already
+#   confers "create any pod in any namespace" cluster-wide. So this adds no reachable
+#   privilege. It is also VERSION-ROBUST: it covers every CDI worker kind (importer, uploader,
+#   cloner/clone-source) without enumerating names that change between CDI releases —
+#   confirmed on prod that cdi-sa is the ONLY cdi-namespace ServiceAccount that can create
+#   pods (`kubectl auth can-i create pods -n unifi --as …`: cdi-sa yes; cdi-apiserver no;
+#   cdi-uploadproxy no).
+#
+# DELIBERATELY NOT COVERED — KubeVirt's own generated pods:
+#   virt-launcher-* pods INHERIT the VM's pod-template labels, so a VM authored in this repo
+#   already passes (verified on dev: virt-launcher-unifi-22dg7 carries egress-class=internet,
+#   auth-mode=native from apps/vms/unifi/30-virtualmachine.yaml). Exempting
+#   kubevirt-controller would REMOVE that useful enforcement on VM authors, so it is not done.
+#   KubeVirt's hotplug (`hp-volume-*`) and export (`virt-export-*`) pods do NOT inherit those
+#   labels, but nothing in this repo uses volume hotplug or VirtualMachineExport; if either is
+#   adopted, extend this policy then — with the same ServiceAccount-scoped shape.
+#
+# OPERATOR NOTE — this is the cluster's FIRST resource-MUTATING policy. Verified on prod
+#   2026-09-06 that `kyverno-resource-mutating-webhook-cfg` currently holds 0 webhooks; with the
+#   chart's `autoUpdateWebhooks=true` Kyverno registers `mutate.kyverno.svc-fail` for Pods within
+#   seconds of this policy being applied. Prod runs `forceFailurePolicyIgnore=false`, so that
+#   webhook is failurePolicy: Fail — the same posture as the validating webhook that is ALREADY in
+#   the path for these pods, so no new fail-open surface and no new blocking surface. CDI retries
+#   the importer pod on a loop, so the stuck import proceeds once the policy lands (the detached
+#   `prime-…` scratch PVC left behind by the blocked attempts is a separate operator cleanup).
+#
+# RESIDUAL TRADE-OFF (state it plainly, it is reversible by deleting this file):
+#   Anything created by `cdi:cdi-sa` in an app namespace now gets `egress-class: internet` +
+#   `auth-mode: none` supplied for it instead of being rejected for not declaring them. Whoever
+#   holds cdi-sa's token can therefore run a pod in an app namespace without declaring its egress
+#   class. That identity can already create arbitrary pods in every namespace (ClusterRole `cdi`:
+#   pods create/delete cluster-wide), so it is not a new capability — but the declaration is now
+#   assumed rather than demanded for that one identity. Nothing else changes: the pod is still
+#   subject to restrict-host-namespaces, the namespace default-deny baseline, and every other
+#   policy, and `internet` is the LEAST-reach of the two egress classes (it does not grant the VPN
+#   path).
+#
+# PROOF (offline, no cluster — platform/kyverno/tests/controller-worker-pods/):
+#   (a) a CDI importer pod created by cdi:cdi-sa is mutated and then PASSES both policies;
+#   (b) the SAME namespace's ordinary pod — including one that FORGES the importer name and
+#       CDI's labels — is NOT mutated and is still DENIED by both policies.
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: label-cdi-worker-pods
+  annotations:
+    policies.kyverno.io/title: Label CDI-generated worker pods
+    policies.kyverno.io/category: Network Isolation
+    policies.kyverno.io/severity: medium
+    homelab.keithjackson.dev/requirements: R-NET-05,R-IDP-02
+    policies.kyverno.io/description: >-
+      Adds homelab.keithjackson.dev/egress-class=internet and auth-mode=none to the
+      short-lived worker Pods that the CDI controller (ServiceAccount cdi/cdi-sa)
+      generates in application namespaces, so they satisfy require-egress-class-label
+      and require-auth-mode-label instead of being exempted from them.
+spec:
+  # No validationFailureAction: this is a MUTATE policy, there is nothing to fail.
+  # background MUST be false: `match.subjects` is userInfo, which exists only on an
+  # AdmissionReview. Kyverno rejects a userInfo match with background scanning enabled
+  # ("invalid variable used at path ... policy contains 'userInfo'"), and a background scan
+  # of an already-created pod has no requester to match anyway.
+  background: false
+  rules:
+    - name: add-worker-pod-labels
+      match:
+        # One filter entry: the resource conditions AND the subject must both match.
+        all:
+          - resources:
+              kinds:
+                - Pod
+              operations:
+                - CREATE
+              namespaceSelector:
+                matchLabels:
+                  homelab.keithjackson.dev/app-namespace: "true"
+            subjects:
+              # The CDI controller (cdi-deployment) ServiceAccount — the identity that
+              # CREATEs importer-*/uploader-*/cloner-* pods. Not the pods' own SA.
+              - kind: ServiceAccount
+                namespace: cdi
+                name: cdi-sa
+      mutate:
+        patchStrategicMerge:
+          metadata:
+            labels:
+              # `+(key)` is Kyverno's ADD-IF-ABSENT anchor: an explicit value already on the
+              # pod is never overwritten (so a future CDI worker that legitimately needs
+              # egress-class: vpn can be given it, and virt-launcher-style inherited labels
+              # would survive).
+              +(homelab.keithjackson.dev/egress-class): internet
+              +(homelab.keithjackson.dev/auth-mode): none

platform/kyverno/tests/controller-worker-pods/kyverno-test.yaml

@@ -0,0 +1,91 @@
+# L0 policy-as-code suite for the CDI worker-pod label mutation — runs in CI WITHOUT a cluster
+# (discovered recursively by the gate's `kyverno test platform/kyverno`; ci/l0.sh).
+#
+# WHY A SEPARATE SUITE from ../kyverno-test.yaml: a `kyverno test` manifest carries ONE
+# `userinfo`, and the whole point here is that the SAME pods get different outcomes for
+# different requesters. Two Test documents, two identities, one file:
+#   1. cdi-controller-worker-pods-admitted  (requester = cdi/cdi-sa)     -> mutated, ADMITTED
+#   2. cdi-worker-exemption-is-narrow       (requester = ordinary user)  -> not mutated, DENIED
+# Keeping the mutate policy OUT of ../kyverno-test.yaml is deliberate: that suite declares no
+# userinfo, and with no userinfo the offline engine does not discriminate on `subjects`, which
+# would silently turn its negative (result: fail) rows into passes.
+apiVersion: cli.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: cdi-controller-worker-pods-admitted
+# Namespace labels for offline namespaceSelector resolution (shared with the main suite).
+variables: ../values.yaml
+userinfo: userinfo-cdi-sa.yaml
+policies:
+  - ../../policies/label-cdi-worker-pods.yaml
+  - ../../policies/require-egress-class-label.yaml
+  - ../../policies/require-auth-mode-label.yaml
+resources:
+  - ../resources/namespaces.yaml
+  - resources/cdi-worker-pods.yaml
+results:
+  # ---- the mutation fires for the CDI controller and adds exactly the two labels ----
+  - policy: label-cdi-worker-pods
+    rule: add-worker-pod-labels
+    kind: Pod
+    resources:
+      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
+    patchedResources: patched/importer-mutated.yaml
+    result: pass
+  # ---- R-NET-05 + R-IDP-02: the mutated CDI worker pod now SATISFIES both policies, so the
+  #      import CDI could not start is admitted. (The engine validates the mutated resource.)
+  - policy: require-egress-class-label
+    rule: require-egress-class
+    kind: Pod
+    resources:
+      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
+    result: pass
+  - policy: require-auth-mode-label
+    rule: require-auth-mode
+    kind: Pod
+    resources:
+      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
+    result: pass
+---
+apiVersion: cli.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: cdi-worker-exemption-is-narrow
+variables: ../values.yaml
+userinfo: userinfo-ordinary.yaml
+policies:
+  - ../../policies/label-cdi-worker-pods.yaml
+  - ../../policies/require-egress-class-label.yaml
+  - ../../policies/require-auth-mode-label.yaml
+resources:
+  - ../resources/namespaces.yaml
+  - resources/cdi-worker-pods.yaml
+results:
+  # ---- the mutation does NOT fire for an ordinary requester, not even for a pod that forges
+  #      the importer's name and CDI's labels ----
+  - policy: label-cdi-worker-pods
+    rule: add-worker-pod-labels
+    kind: Pod
+    resources:
+      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
+      - importer-prime-forged-by-an-app
+      - plain-app-pod-no-labels
+    result: skip
+  # ---- so every unlabelled pod in the SAME app namespace is still DENIED (R-NET-05) ----
+  - policy: require-egress-class-label
+    rule: require-egress-class
+    kind: Pod
+    resources:
+      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
+      - importer-prime-forged-by-an-app
+      - plain-app-pod-no-labels
+    result: fail
+  # ---- ... and by R-IDP-02 ----
+  - policy: require-auth-mode-label
+    rule: require-auth-mode
+    kind: Pod
+    resources:
+      - importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
+      - importer-prime-forged-by-an-app
+      - plain-app-pod-no-labels
+    result: fail

platform/kyverno/tests/controller-worker-pods/patched/importer-mutated.yaml

@@ -0,0 +1,16 @@
+# Expected result of the mutation: CDI's own labels untouched, the two platform labels ADDED.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
+  namespace: app-ns
+  labels:
+    app: containerized-data-importer
+    cdi.kubevirt.io: importer
+    homelab.keithjackson.dev/egress-class: internet
+    homelab.keithjackson.dev/auth-mode: none
+spec:
+  restartPolicy: OnFailure
+  containers:
+    - name: importer
+      image: quay.io/kubevirt/cdi-importer:v1.62.0

platform/kyverno/tests/controller-worker-pods/resources/cdi-worker-pods.yaml

@@ -0,0 +1,53 @@
+# Fixtures for the CDI-worker-pod label mutation (policies/label-cdi-worker-pods.yaml).
+#
+# All three pods live in `app-ns` (../resources/namespaces.yaml — carries
+# homelab.keithjackson.dev/app-namespace: "true"), so all three are IN SCOPE of
+# require-egress-class-label + require-auth-mode-label. What separates them is WHO creates
+# them, which is the whole point of the mechanism: the two "forged" pods are byte-for-byte
+# as CDI-looking as the real one.
+#
+# ---- (a) the REAL CDI worker pod: shape copied from the pod CDI actually tried to create on
+#          prod (name importer-prime-<pvc-uid>, CDI's own labels, no platform labels). It is
+#          created by ServiceAccount cdi/cdi-sa — see userinfo-cdi-sa.yaml.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: importer-prime-be3f9d1c-c3ce-455a-a450-3d9e95c7baa2
+  namespace: app-ns
+  labels:
+    app: containerized-data-importer
+    cdi.kubevirt.io: importer
+    # NOTE: no homelab.keithjackson.dev/* labels — CDI generates this pod itself, nothing in
+    # git can label it. This is exactly why the import was blocked.
+spec:
+  restartPolicy: OnFailure
+  containers:
+    - name: importer
+      image: quay.io/kubevirt/cdi-importer:v1.62.0
+---
+# ---- (b) FORGED: an ordinary pod that copies the importer's NAME PREFIX and CDI's labels.
+#          Proves that a name-prefix or label-based exemption would have been walkable, and
+#          that this ServiceAccount-scoped one is not.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: importer-prime-forged-by-an-app
+  namespace: app-ns
+  labels:
+    app: containerized-data-importer
+    cdi.kubevirt.io: importer
+spec:
+  containers:
+    - name: app
+      image: registry.example/app:1.0.0
+---
+# ---- (c) a plain unlabelled application pod in the SAME namespace. Must still be DENIED.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: plain-app-pod-no-labels
+  namespace: app-ns
+spec:
+  containers:
+    - name: app
+      image: registry.example/app:1.0.0

platform/kyverno/tests/controller-worker-pods/userinfo-cdi-sa.yaml

@@ -0,0 +1,21 @@
+# The requester identity Kyverno actually saw on prod when CDI created the importer pod
+# (from the kyverno-admission-controller log of the blocked request, 2026-09-06):
+#   user={"username":"system:serviceaccount:cdi:cdi-sa","groups":["system:serviceaccounts",
+#         "system:serviceaccounts:cdi","system:authenticated"], ...}
+#   roles=["cdi:cdi-deployment"]  clusterroles=["cdi", …]
+# `kyverno test` feeds this to the engine as the AdmissionReview userInfo so
+# `match.subjects` can be evaluated offline.
+apiVersion: cli.kyverno.io/v1alpha1
+kind: UserInfo
+metadata:
+  name: cdi-controller
+clusterRoles:
+  - cdi
+roles:
+  - cdi:cdi-deployment
+userInfo:
+  username: system:serviceaccount:cdi:cdi-sa
+  groups:
+    - system:serviceaccounts
+    - system:serviceaccounts:cdi
+    - system:authenticated

platform/kyverno/tests/controller-worker-pods/userinfo-ordinary.yaml

@@ -0,0 +1,16 @@
+# An ORDINARY requester in the application namespace — the attacker/mistake case. Anything
+# that is not cdi/cdi-sa: an app's own ServiceAccount, a human's kubectl, Flux.
+# The mutation must NOT fire for this identity, so both label policies must still DENY —
+# even for a pod that forged the importer's name and CDI's labels.
+apiVersion: cli.kyverno.io/v1alpha1
+kind: UserInfo
+metadata:
+  name: ordinary-namespace-user
+clusterRoles:
+  - edit
+userInfo:
+  username: system:serviceaccount:app-ns:default
+  groups:
+    - system:serviceaccounts
+    - system:serviceaccounts:app-ns
+    - system:authenticated

platform/vpn-egress/egress-steering.yaml

@@ -1,7 +1,10 @@
 ---
 # ============================================================================
 # Cilium Egress Gateway steering for `egress-class=vpn` pods — ADR-0015.
-# Serves: R-NET-05, R-NET-06, R-NET-07.
+# Serves: R-NET-04, R-NET-05, R-NET-06, R-NET-07.
+# R-NET-04 added 2026-09-06: this file IS the R-NET-04 datapath — the rule that selects between the
+# two egress paths. Omitting it made `run.sh --status` report R-NET-04 as "TEST-ONLY / impl none"
+# even though the implementation is right here.
 #
 # ##########################################################################
 # ##  LIVE (2026-08-24): wired into kustomization.yaml, reconciled by Flux ##
@@ -35,7 +38,7 @@ kind: CiliumEgressGatewayPolicy
 metadata:
   name: vpn-egress-steering
   annotations:
-    homelab.keithjackson.dev/requirements: R-NET-05,R-NET-06,R-NET-07
+    homelab.keithjackson.dev/requirements: R-NET-04,R-NET-05,R-NET-06,R-NET-07
     homelab.keithjackson.dev/adr: ADR-0015
 spec:
   # --- Client selection (INVARIANTS 1 & 2) --------------------------------