feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite

OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/ansible/roles/cilium/defaults/main.yml

@@ -2,7 +2,17 @@
 # cilium role defaults (ADR-0002, docs/02 Cilium-ownership contract).
 # This role performs the INITIAL Cilium install; the Flux platform/cilium
 # HelmRelease adopts config afterward. The parity-critical values
-# (kubeProxyReplacement, k8sServiceHost/Port) are templated from k3s_api_host.
+# (kubeProxyReplacement, k8sServiceHost/Port) are templated from
+# k3s_apiserver_endpoint / k3s_api_port (group_vars/all.yml).
+#
+# k8sServiceHost IS THE HA VIP, NOT THE FOUNDER (changed 2026-09-06). It reads
+# k3s_apiserver_endpoint — which derives from kube_vip_address, so it is dev 192.168.1.49 /
+# prod 192.168.2.9 — and deliberately NOT k3s_api_host, the founder's own node IP. Those two
+# were one variable until 2026-09-06, and the conflation is what left all three prod
+# cilium-agents dialing k3s-prod-1 on an HA cluster. Do not re-merge them.
+#
+# THIS ROLE THEREFORE DEPENDS ON THE VIP EXISTING. playbooks/site.yml runs kube-vip.yml
+# before cilium.yml and roles/kube_vip blocks until the VIP answers on 6443.
 
 # Exact pin — MUST equal the platform/cilium HelmRelease chart.spec.version (1.20.1, an
 # exact pin, not a range). Renovate bumps both together.

infra/ansible/roles/cilium/tasks/main.yml

@@ -89,7 +89,7 @@
   run_once: true
   changed_when: false
 
-- name: Render the Cilium values file (parity-critical settings from k3s_api_host)
+- name: Render the Cilium values file (parity-critical settings from k3s_apiserver_endpoint)
   ansible.builtin.template:
     src: cilium-values.yaml.j2
     dest: "{{ cilium_values_path }}"

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -4,9 +4,24 @@
 ---
 # [parity] k3s kube-proxy is disabled → Cilium replaces it (ADR-0001/0002).
 kubeProxyReplacement: true
-# [parity] = the k3s API endpoint (primary node IP). The Flux HelmRelease's
-# k8sServiceHost "REPLACE-ME" must be set to this same value.
-k8sServiceHost: "{{ k3s_api_host }}"
+# [parity] = k3s_apiserver_endpoint, THE HA CONTROL-PLANE VIP (kube_vip_address) — dev
+# 192.168.1.49, prod 192.168.2.9. NOT k3s_api_host, which is the FOUNDER'S OWN node IP:
+# pointing the CNI's kube-proxy replacement at one machine on a 3-server HA cluster means
+# losing that machine stops every service/policy update reaching the datapath, silently
+# (the existing datapath keeps working from Cilium's cache, so nothing looks broken). That
+# was the live state of prod until 2026-09-06 — all three cilium-agents carried
+# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held .9. See group_vars/all.yml.
+#
+# [parity] The Flux side is platform/cilium/helmrelease.yaml's k8sServiceHost:
+# "${K3S_API_HOST}", whose per-environment value comes from clusters/<env>/kustomization.yaml.
+# THAT VALUE MUST EQUAL k3s_apiserver_endpoint for the SAME environment. If it does not,
+# Flux and this bootstrap render different control-plane paths and fight over the CNI.
+#
+# THE VIP MUST BE UP BEFORE THIS ROLE RUNS. playbooks/site.yml imports kube-vip.yml between
+# k3s.yml and cilium.yml for exactly this reason, and roles/kube_vip blocks until the VIP
+# answers. Rendering this against a VIP that nothing announces yet reproduces the
+# 2026-09-01 shape: agents that cannot reach the API server on a cold cluster.
+k8sServiceHost: "{{ k3s_apiserver_endpoint }}"
 k8sServicePort: {{ k3s_api_port }}
 
 # Ingress via Gateway API — no Traefik (docs/09, ADR-0002).

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -59,6 +59,17 @@ k3s_server_mode: "secondary"
 # serving cert on restart when this set changes. Deduplicated (the primary IP is
 # also k3s_api_host).
 #
+# THIS LIST COVERS BOTH API ADDRESSES, WHICH IS WHY THE 2026-09-06 SPLIT NEEDED NO CERT
+# CHANGE AND NO k3s RESTART. group_vars/all.yml now distinguishes k3s_api_host (the
+# founder, for the cold join) from k3s_apiserver_endpoint (= kube_vip_address, what Cilium
+# and every steady-state client dial). Both are already in this set, so repointing a
+# consumer from one to the other cannot produce a TLS failure. VERIFIED LIVE 2026-09-06,
+# not assumed: prod's apiserver answered through https://192.168.2.9:6443 with full CA
+# verification and tls-server-name=192.168.2.9. Do not remove either entry — dropping
+# kube_vip_address breaks every client of the VIP; dropping k3s_api_host breaks nothing
+# today only because it is also in the k3s_servers_group node-IP list below, and that
+# coincidence is not a contract.
+#
 # k3s_servers_group — NOT a literal group name — is what makes this role environment-
 # agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
 # group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -155,6 +155,18 @@
          | default(k3s_primary_token_raw.content | b64decode | trim, true) }}
   when: k3s_server_mode == 'secondary'
 
+# THE JOIN TARGET IS k3s_api_host — THE FOUNDER — AND MUST NOT BECOME THE VIP.
+# group_vars/all.yml splits the two API addresses (k3s_api_host = the founder's own node IP,
+# k3s_apiserver_endpoint = the kube-vip HA VIP) and this is the canonical founder-only
+# consumer. On a COLD build the VIP does not exist at this point in the sequence: kube-vip
+# is a k3s auto-deploy manifest, so it cannot be applied until a cluster exists, and
+# playbooks/site.yml only imports kube-vip.yml AFTER k3s.yml. A `--server` pointed at the
+# VIP would therefore dial an address nothing announces and the cold build would be
+# unjoinable — with no diagnostic beyond a connection timeout.
+# On a WARM single-node rejoin (`--limit k3s-dev-N`) the founder is up and answering too, so
+# the founder works in both cases and the VIP works in only one. Cilium's k8sServiceHost is
+# the opposite trade (steady state, must survive losing the founder) and uses the endpoint
+# variable instead.
 - name: Install k3s secondary (join as server)
   ansible.builtin.command:
     cmd: >-

infra/ansible/roles/kube_vip/defaults/main.yml

@@ -24,6 +24,16 @@ kube_vip_port: 6443
 # only if detection is wrong.
 kube_vip_interface: ""
 
+# How long to wait for the VIP to actually ANSWER on kube_vip_port after the manifest is
+# written (seconds). This is a real gate, not cosmetics: since 2026-09-06 Cilium's
+# k8sServiceHost is this VIP (k3s_apiserver_endpoint), and playbooks/site.yml runs this role
+# immediately before cilium.yml — so if the VIP never comes up, the run must stop HERE with
+# a clear message rather than proceed to install a CNI pointed at an address nothing
+# answers. On a cold cluster the budget covers k3s's deploy controller applying the
+# manifest, the ghcr.io image pull and leader election; on an already-running cluster the
+# check returns immediately.
+kube_vip_wait_timeout: 300
+
 # Leader-election tuning (seconds) — a snappy failover without thrashing.
 kube_vip_lease_duration: 15
 kube_vip_renew_deadline: 10

infra/ansible/roles/kube_vip/tasks/main.yml

@@ -21,6 +21,14 @@
     executable: /bin/bash
   register: kube_vip_iface_detect
   changed_when: false
+  # RUN THIS EVEN UNDER --check. It is `ip -o -4 addr show`: it reads and changes nothing.
+  # Without check_mode: false the shell module is SKIPPED in check mode, kube_vip_iface_detect
+  # has no stdout, and the assert two tasks down aborts the play with "Could not detect the
+  # interface ... set kube_vip_interface explicitly" — a misleading failure that says nothing
+  # about the node. That mattered less while this play was an out-of-band operator step; it
+  # matters now that playbooks/site.yml imports it, because `--check` on the bring-up before a
+  # cold prod build is exactly the rehearsal an operator should be able to run.
+  check_mode: false
 
 - name: Resolve the kube-vip VIP interface (override wins, else detected)
   ansible.builtin.set_fact:
@@ -52,3 +60,34 @@
     owner: root
     group: root
     mode: "0600"
+
+# PROVE THE VIP IS ACTUALLY UP BEFORE THIS PLAY RETURNS (added 2026-09-06).
+#
+# Templating the manifest is not the same as having a floating endpoint: k3s's deploy
+# controller still has to apply it, the image still has to pull, and leader election still
+# has to pick an owner. Nothing used to check, which was tolerable while the VIP was only an
+# operator convenience. It is not tolerable now — Cilium's k8sServiceHost IS this VIP
+# (k3s_apiserver_endpoint) and playbooks/site.yml runs cilium.yml straight after this play,
+# so an unannounced VIP would mean installing a CNI that cannot reach the API server. That
+# failure is the 2026-09-01 shape and it is very expensive to diagnose from the far side;
+# failing here instead names the actual cause.
+#
+# run_once: the VIP is ONE address for the whole environment, so one probe answers for the
+# environment. Probing from a server host (not the controller) is deliberate: the servers
+# are on the VIP's own LAN in both environments, whereas the dev controller reaches dev's
+# API only through a pfSense NAT — a controller-side probe would have been a property of
+# where the play was launched from. READ-ONLY: it opens a TCP connection and changes nothing.
+- name: Wait for the control-plane VIP to answer (the CNI's endpoint depends on it)
+  ansible.builtin.wait_for:
+    host: "{{ kube_vip_address }}"
+    port: "{{ kube_vip_port }}"
+    timeout: "{{ kube_vip_wait_timeout }}"
+  run_once: true
+  changed_when: false
+  vars:
+    ansible_become: false
+  # DELIBERATELY NOT check_mode: false, unlike the interface probe above — so a `--check`
+  # rehearsal SKIPS this and does not sit on a socket for kube_vip_wait_timeout. wait_for
+  # declares no check-mode support, so the skip is the module's own behaviour and this comment
+  # is here only so the skip does not read as an oversight. CONSEQUENCE, stated plainly: a
+  # `--check` run does NOT verify the VIP is up. Only a real run gates the CNI on it.