feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite

OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach the cluster and provide fault tolerant HA. Without a change it will not work in HA." THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live before changing anything: all three prod cilium-agent pods carried KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated that stale comment as an open question when the evidence already settled it. SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs: k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join target and a tls-san. infra/tofu/outputs.tf GENERATES this name, so renaming would ripple into tofu and the runbooks. k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium consumes. Derived, not restated, so an environment overrides ONE line and the VIP, the SAN and the CNI endpoint move together. Prod needed no new override. THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran node-baseline -> k3s -> cilium, so Cilium would install against an address nothing announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value: site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier (kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so a cold run stops THERE with a named cause instead of surfacing as unreachable agents. Also fixed a pre-existing bug that only mattered once this play joined site.yml: the role's read-only interface probe ran without `check_mode: false`, so under --check it was skipped and the assert fired with a misleading "set kube_vip_interface explicitly". VERIFIED BY ME, not just reported: renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP) cilium template consumes k3s_apiserver_endpoint, not k3s_api_host cold join still `--server https://{{ k3s_api_host }}:6443` — the founder order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux gate full CI gate PASS Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders ['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved — .9 already answers under full CA verification; and --check --diff of kube-vip.yml against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the kube-vip prod already runs. ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine checks: the Cilium template must use the endpoint var; the endpoint must derive from kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal that environment's VIP and be none of its node IPs (the [parity] contract, now machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium. Every new assertion mutation-tested and bites. One honest negative recorded: reverting only the BASE default is not caught, because both overlays patch the key — the check reads effective per-environment values, so the base default is unguarded. APPLY ORDER, and this is the operator's step — nothing was applied: PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01 incident), it will not proceed past an unready pod, existing datapath survives an agent restart, and .9 is already live before the change lands. DEV SEPARATELY, operator present. Same mechanics but live dev agents dial 192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that. ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook. NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that test's gap list. Refs R-PLAT-01, R-NET-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-06 20:24:51 -05:00
1 parent 3ade1d440f
commit 6f61c9622d
@@ -27,19 +27,85 @@ k3s_servers_group: k3s_servers
 # a kubeconfig is a credential.)
 cluster_kubeconfig_name: dev.kubeconfig

-# The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP.
+# --- TWO API ADDRESSES, AND THEY ARE NOT INTERCHANGEABLE (split 2026-09-06) ----------
+# This used to be ONE variable, k3s_api_host, consumed by BOTH the cold-bootstrap join and
+# Cilium's k8sServiceHost. That conflation is what pointed every cilium-agent on the LIVE
+# prod cluster at 192.168.2.10 — k3s-prod-1's own address — on a cluster whose entire
+# purpose is HA: verified 2026-09-06, all three agents carried
+# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held 192.168.2.9. Lose that one node
+# and no service/policy update reaches the CNI until it returns (the existing datapath
+# survives on Cilium's cache, so it fails SILENTLY). The two consumers have genuinely
+# different requirements, so they now have two variables.
+#
+#   k3s_api_host            THE FOUNDER'S OWN ADDRESS. Bootstrap-time only.
+#   k3s_apiserver_endpoint  THE HA ENDPOINT (the kube-vip VIP). Steady-state.
+#
+# DO NOT "simplify" these back into one. See each block below for why each half must be
+# the address it is.
+
+# THE FOUNDER'S OWN NODE IP — a BOOTSTRAP-TIME address, deliberately NOT the VIP.
 # NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node
-# IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san /
-# secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux
-# HelmRelease stay in PARITY (docs/02 contract).
+# IP — so we read the real 192.168.1.x from k3s_node_ip instead.
+#
+# EXACTLY TWO CONSUMERS, and both of them need the founder specifically:
+#   1. roles/k3s_server — the secondary `--server https://<host>:<port>` JOIN URL. On a COLD
+#      build the VIP DOES NOT EXIST YET at join time: kube-vip is a k3s auto-deploy manifest
+#      that only lands once a cluster exists, so a join aimed at the VIP is a join aimed at
+#      an unanswered address and the cold build is unjoinable. The founder is up (it just ran
+#      --cluster-init), which is the whole reason this variable is the founder.
+#   2. roles/k3s_server — one entry of k3s_tls_sans, so the founder's own address is a SAN
+#      alongside the VIP and every other server's IP.
+#
+# ANYTHING THAT RUNS AGAINST A CLUSTER THAT ALREADY EXISTS MUST NOT USE THIS. It is a
+# single point of failure by construction. Use k3s_apiserver_endpoint below.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}"
+
+# Shared by both halves: k3s serves the API on 6443 in every environment, so the port is
+# not a per-environment knob and is not duplicated per address.
 k3s_api_port: 6443

+# THE HA kube-apiserver ENDPOINT — a VIRTUAL address that reaches the cluster wherever the
+# control plane currently is, not a machine. OPERATOR DECISION 2026-09-06: in-cluster
+# components dial the VIP, as kube-vip was always intended to provide.
+#
+# DERIVED from kube_vip_address (below), never restated: an environment overrides the VIP in
+# ONE place (group_vars/k3s_prod/vars.yml sets kube_vip_address: 192.168.2.9) and this
+# follows automatically, so prod needs no override of its own and a fourth environment
+# cannot set one of the two and forget the other.
+#
+# NAMED FOR THE ROLE, NOT THE MECHANISM. Consumers want "the address that reaches the
+# apiserver, HA"; that it is currently floated by kube-vip is an implementation detail. If
+# the mechanism is ever something else (an external LB, a DNS name), this one line moves and
+# no consumer changes.
+#
+# CONSUMERS:
+#   - roles/cilium -> cilium-values.yaml.j2 -> k8sServiceHost.  [parity] with the Flux side:
+#     clusters/base/cluster-vars.yaml K3S_API_HOST (and each overlay's patch) MUST equal
+#     THIS value for that environment, or Flux and the bootstrap fight over the CNI's
+#     control-plane path. See platform/cilium/helmrelease.yaml's PARITY CONTRACT header.
+#
+# TLS IS ALREADY COVERED — verified, not assumed: k3s_tls_sans (roles/k3s_server/defaults)
+# has always included kube_vip_address, and on 2026-09-06 the prod apiserver was reached
+# through https://192.168.2.9:6443 with FULL CA verification and tls-server-name=192.168.2.9
+# (infra/ansible/.cache/prod.kubeconfig, no insecure-skip-tls-verify). So the serving cert
+# needs no change and no k3s restart for this switch.
+#
+# ORDERING CONSEQUENCE, and it is why playbooks/site.yml changed too: Cilium now needs the
+# VIP to be UP before it installs. site.yml therefore imports kube-vip.yml BETWEEN k3s.yml
+# and cilium.yml, and roles/kube_vip waits for the VIP to answer before returning. kube-vip
+# can precede the CNI because its DaemonSet is hostNetwork with
+# KUBERNETES_SERVICE_HOST=127.0.0.1 — it needs neither a CNI nor kube-proxy.
+k3s_apiserver_endpoint: "{{ kube_vip_address }}"
+
 # Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover
 # endpoint for the kube-apiserver, fronted by an operator pfSense port-forward at
-# 192.168.1.49:6443. Shared here so BOTH kube_vip AND the k3s_server tls-san list
-# agree on the address (the serving cert must carry this IP or TLS via the VIP
-# fails to validate).
+# 192.168.1.49:6443. Shared here so THREE things agree on the one address:
+#   - roles/kube_vip          — what to ARP and lease-elect for;
+#   - roles/k3s_server        — the tls-san entry (the serving cert must carry this IP or
+#                               TLS via the VIP fails to validate);
+#   - k3s_apiserver_endpoint  — above; what in-cluster components actually dial.
+# THIS is the one line an environment overrides (prod: 192.168.2.9, in
+# group_vars/k3s_prod/vars.yml); all three follow.
 kube_vip_address: "192.168.1.49"

 # Controller (dev container) reaches the kube-API DIRECTLY via the pfSense NAT
@@ -51,12 +51,18 @@ k3s_servers_group: k3s_prod
 # than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential.
 cluster_kubeconfig_name: prod.kubeconfig

-# --- k3s API host — safe to set now, the primary's real IP is already known -------
+# --- k3s API host — THE FOUNDER'S ADDRESS, bootstrap-time only --------------------
 # Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
 # dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
 # has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
 # address (see the KEY DIFFERENCE note there) — so point straight at it instead of
 # needing a k3s_node_ip split.
+#
+# WHAT THIS IS *NOT*, since 2026-09-06: it is not the endpoint anything steady-state
+# dials. That is k3s_apiserver_endpoint (group_vars/all.yml), which derives from
+# kube_vip_address below and therefore needs NO override here. k3s_api_host is consumed
+# only by the secondary `--server` cold-join URL and by one tls-san entry. Read the
+# TWO API ADDRESSES block at the top of group_vars/all.yml before touching either.
 k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
 # by template reference, so it needs no separate override here — it picks up the real
@@ -75,8 +81,15 @@ k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
 # round (control-plane .49, ingress pool .53-.60).
 #
 # This is the address kube-vip floats for the API and that k3s tls-san's onto its
-# serving cert; secondaries use it as their --server join target. Dev's equivalent is
-# kube_vip_address: 192.168.1.49 in group_vars/all.yml.
+# serving cert. Dev's equivalent is kube_vip_address: 192.168.1.49 in group_vars/all.yml.
+#
+# CORRECTION 2026-09-06: an earlier version of this comment claimed "secondaries use it as
+# their --server join target". They do not, and they must not — on a COLD build the VIP is
+# not up when the secondaries join (kube-vip is a k3s auto-deploy manifest and needs a
+# cluster first), so the join uses k3s_api_host, the founder. What DOES use this address is
+# k3s_apiserver_endpoint (group_vars/all.yml, derived from this line), i.e. Cilium's
+# k8sServiceHost and any other steady-state client. Overriding kube_vip_address here moves
+# BOTH the VIP and prod's Cilium endpoint together, which is the point of deriving it.
 #
 # HISTORY: an earlier pass here wrongly set this to 192.168.2.20 — that is the INGRESS
 # VIP, not the control-plane VIP. Corrected. Keep the two distinct.