Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details

Browse Source 
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-06 20:24:51 -05:00
1 parent 3ade1d440f
commit 6f61c9622d
21 files changed
+555 -65
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml
+10 -5
View File
@@ -47,12 +47,17 @@
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
# infra/PROD-PROVISIONING.md).
# * kube-vip is NOT imported by playbooks/site.yml, so the floating endpoint is a separate
# operator step. Observed and reported, NOT asserted — R-PLAT-01 does not fix the
# bring-up order (R-TEST-06).
# * `K3S_API_HOST` (Cilium's k8sServiceHost) is the PRIMARY NODE's address in both
#
# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
# * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
# A real single-node dependency, ESCALATED to the PM rather than asserted.
# It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
# The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
# contract and per environment.
# * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
# the order is asserted — not as a tidiness preference but because the CNI's endpoint is
# the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
# API server.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
conformance/tests/l0-static/ha-control-plane-as-code/check.sh
+172 -14
View File
@@ -64,6 +64,14 @@
# is not that environment's ingress Gateway VIP, and lies OUTSIDE that
# environment's Cilium LoadBalancer pool — the two allocators can never be
# handed the same address;
# CNI PATH: the CNI dials the FLOATING endpoint, not a machine (added 2026-09-06). The
# two API addresses stay SPLIT — k3s_api_host is the founder and feeds only
# the cold `--server` join, k3s_apiserver_endpoint derives from
# kube_vip_address and feeds Cilium's k8sServiceHost — the Flux overlay's
# K3S_API_HOST equals that environment's VIP and is none of its node
# addresses ([parity], both halves), the HelmRelease keeps a per-environment
# token rather than a literal, and site.yml floats the VIP BETWEEN k3s and
# Cilium with roles/kube_vip blocking until it answers;
# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
# retargeted by variables only (docs/02 "the *same* Ansible bootstrap
# runs"), so "exercise the same multi-node topology" is a property of the
@@ -84,20 +92,25 @@
# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
# - THAT kube-vip IS PART OF THE ONE-COMMAND BRING-UP. playbooks/site.yml chains
# node-baseline -> k3s -> cilium -> verify -> flux-bootstrap and does NOT import
# kube-vip.yml, so the floating endpoint is a separate operator step. That is recorded
# here as an observation for the PM, NOT asserted: R-PLAT-01 requires an HA control
# plane, and does not name kube-vip or fix the bring-up order (R-TEST-06 — do not
# invent a requirement's meaning).
# - CILIUM'S OWN API ENDPOINT. `K3S_API_HOST` (clusters/*/kustomization.yaml), which
# becomes Cilium's `k8sServiceHost` for kube-proxy replacement, is the PRIMARY NODE's
# address in both environments — not the floating VIP — even though the prod overlay's
# own comment says it "is the prod CONTROL-PLANE VIP (kube-vip), NOT ... the single node
# address .10". That is a real single-node dependency in the CNI's control-plane path
# and it is ESCALATED to the PM rather than asserted: making this check fail would turn
# the merge gate red over a design question this workstream may not decide (R-TEST-06).
# - IPv6 / dual-stack, and any claim about the edge network.
#
# TWO FORMER GAPS ARE NOW ASSERTED, and the reasoning is worth keeping (2026-09-06):
# * CILIUM'S OWN API ENDPOINT used to be listed here as an ESCALATION — `K3S_API_HOST`
# (clusters/*/kustomization.yaml), which becomes Cilium's `k8sServiceHost`, was each
# environment's PRIMARY NODE address rather than the floating VIP, contradicting the prod
# overlay's own comment. Making the check fail over it would have turned the merge gate
# red on a design question this workstream may not decide (R-TEST-06), so it was reported
# instead. THE OPERATOR DECIDED THE VIP on 2026-09-06, which turns the question into a
# repository invariant, and invariants belong in assertions — see the "THE CNI DIALS THE
# VIP" section. Note what made it assertable: the fix SPLIT one variable in two, so there
# is finally an HA-valued name (`k3s_apiserver_endpoint`) to point the contract at.
# * kube-vip WAS NOT PART OF THE ONE-COMMAND BRING-UP (site.yml chained node-baseline ->
# k3s -> cilium -> verify -> flux-bootstrap), which was recorded here as an observation
# and not asserted, because R-PLAT-01 does not fix the bring-up order. It is asserted now
# for a different reason that IS in scope: with the CNI's endpoint being the VIP, an
# unfloated VIP means a cold build installs a CNI that cannot reach its API server. The
# ordering is no longer a preference about tidiness, it is a dependency of the control
# plane coming up at all.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -470,6 +483,149 @@ check(re.search(r"tls-san:", K3S_TEXT) is not None,
"/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
"no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")
# ============================ THE CNI DIALS THE VIP, NOT THE FOUNDER (added 2026-09-06)
# This section used to be an ESCALATION in the header's gap list, not an assertion: Cilium's
# k8sServiceHost was each environment's PRIMARY NODE address, so on a 3-server HA control
# plane the CNI's control-plane path depended on one machine. Measured on live prod
# 2026-09-06: all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10
# (k3s-prod-1's own address) while kube-vip held 192.168.2.9. That failure is SILENT — the
# datapath keeps serving from Cilium's cache, so nothing looks broken while no service or
# policy update lands at all.
#
# The operator decided the VIP (2026-09-06), so this is now a repository property and belongs
# here rather than in the gap list. THIS IS THE RIGHT LAYER because the claim is about what
# the configuration in git WOULD build, in BOTH environments, including the next one — a live
# cluster can only tell you about itself.
#
# WHAT MAKES THIS CHECKABLE AT ALL is that the fix split one variable into two. Asserting
# "Cilium uses the HA address" requires an HA address to name; before the split there was
# only k3s_api_host, which had to be the founder for the cold join. So the assertions below
# check the SPLIT as much as the value: the join keeps the founder, the CNI gets the VIP, and
# the Flux half agrees with the Ansible half per environment.
CIL_ROLE = ANS / "roles" / "cilium"
cil_values_tpl = CIL_ROLE / "templates" / "cilium-values.yaml.j2"
check(cil_values_tpl.is_file(),
"roles/cilium ships no cilium-values.yaml.j2 — the bootstrap's CNI values, including the "
"API endpoint its kube-proxy replacement dials, are not authored")
if cil_values_tpl.is_file():
# Parse the rendered-shape template, never its prose: strip comment lines first, exactly
# as the kube-vip parser above does, so a reassuring comment cannot satisfy the check.
cil_body = "\n".join(l for l in cil_values_tpl.read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#"))
m = re.search(r"^k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", cil_body, re.M)
check(m is not None,
"roles/cilium's values template sets no k8sServiceHost — with kube-proxy disabled "
"(ADR-0001) the agent has no API endpoint to dial and the CNI cannot come up")
if m is not None:
host_expr = m.group(1).strip()
check("k3s_apiserver_endpoint" in host_expr,
f"roles/cilium renders k8sServiceHost={host_expr!r}. It MUST be the HA endpoint "
"variable {{ k3s_apiserver_endpoint }} (= kube_vip_address). Any node-valued "
"expression — k3s_api_host, a primary hostvar, a literal IP — makes the CNI's "
"control-plane path depend on ONE server of an HA control plane, and the failure "
"is silent: the datapath keeps running from Cilium's cache while no service or "
"policy update ever lands (measured on prod 2026-09-06)")
check("k3s_api_host" not in host_expr,
f"roles/cilium renders k8sServiceHost={host_expr!r}, which references "
"k3s_api_host — the FOUNDER's own address. That variable exists only for the "
"cold-bootstrap `--server` join and one tls-san entry; it is a single point of "
"failure by construction and must never be a steady-state endpoint")
# The HA endpoint variable must BE the VIP, and must be DERIVED from kube_vip_address rather
# than restated: an environment overrides one line (kube_vip_address) and both the floating
# address and the CNI's endpoint move together. A literal here could disagree with the VIP the
# same environment actually floats, which is the two-places-one-fact bug this repo keeps
# hitting.
endpoint_def = str(ALL_VARS.get("k3s_apiserver_endpoint", ""))
check("kube_vip_address" in endpoint_def,
f"group_vars/all.yml defines k3s_apiserver_endpoint as {endpoint_def!r}. It MUST derive "
"from kube_vip_address, so that overriding the VIP for an environment also moves what "
"that environment's CNI dials. A restated literal can silently disagree with the VIP "
"kube-vip is actually floating")
check("kube_vip_address" not in str(ALL_VARS.get("k3s_api_host", "")),
"group_vars/all.yml's k3s_api_host resolves from kube_vip_address — the two API "
"addresses have been re-merged. k3s_api_host is the secondary join target and on a COLD "
"build no VIP exists yet, so a VIP-valued join target makes a cold bootstrap unjoinable")
# THE [parity] CONTRACT, now machine-checked instead of asserted in a comment: the Flux
# overlay's K3S_API_HOST becomes Cilium's k8sServiceHost under the HelmRelease, and the
# Ansible bootstrap renders k3s_apiserver_endpoint for the same field. If the two disagree,
# Flux and the bootstrap describe different control-plane paths and fight over the CNI.
for e in envs:
L = e.label
ov = overlay_vars(getattr(e, "overlay", ""))
k3s_api_host_knob = as_ip(ov.get("K3S_API_HOST", ""))
vip = as_ip(e.var("kube_vip_address"))
check(k3s_api_host_knob is not None,
f"{L}: overlay {getattr(e, 'overlay', '')!r} resolves no usable K3S_API_HOST — that "
"token becomes Cilium's k8sServiceHost via platform/cilium/helmrelease.yaml, and "
"kustomize-controller runs with StrictPostBuildSubstitutions so an unresolvable one "
"fails the whole platform Kustomization")
if k3s_api_host_knob is not None and vip is not None:
check(k3s_api_host_knob == vip,
f"{L}: [parity] BROKEN — the Flux overlay sets K3S_API_HOST={k3s_api_host_knob} "
f"but this environment's control-plane VIP (kube_vip_address, which is what "
f"k3s_apiserver_endpoint and therefore the Ansible-side k8sServiceHost resolve "
f"to) is {vip}. Flux and the Ansible bootstrap would render DIFFERENT API "
"endpoints into the same CNI and fight over it. If they differ because the "
"overlay still holds a NODE address, that is the 2026-09-06 defect: the CNI of "
"an HA control plane must not dial one server")
check(k3s_api_host_knob not in set(
as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
for h in hosts_in(str(e.var("k3s_servers_group")))),
f"{L}: the overlay's K3S_API_HOST {k3s_api_host_knob} is one of this "
"environment's own NODE addresses. Cilium's kube-proxy replacement would dial a "
"single machine on a control plane built specifically to survive losing one")
# The Flux half must stay a per-environment TOKEN. A literal IP in the shared HelmRelease
# would ship one environment's control-plane address to every environment (R-ENV-02), and an
# empty value is the 2026-09-01 incident verbatim (k8sServiceHost: "" made the agent's config
# init container fall back to the in-cluster ClusterIP 10.43.0.1:443, which only the agent
# being replaced could translate: deadlocked on 2 of 3 nodes).
hr = ROOT / "platform" / "cilium" / "helmrelease.yaml"
check(hr.is_file(), "platform/cilium/helmrelease.yaml is missing — nothing adopts the CNI "
"config after the bootstrap, so the parity contract has only one side")
if hr.is_file():
hr_body = "\n".join(l for l in hr.read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#"))
hm = re.search(r"^\s*k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", hr_body, re.M)
check(hm is not None and hm.group(1).strip() == "${K3S_API_HOST}",
f"platform/cilium/helmrelease.yaml sets k8sServiceHost="
f"{(hm.group(1) if hm else None)!r} instead of the cluster-vars token "
"\"${K3S_API_HOST}\" — a literal there would ship ONE environment's control-plane "
"address to every environment (R-ENV-02), and an empty value is the 2026-09-01 "
"incident verbatim")
# ORDERING: the VIP must be floating BEFORE the CNI is installed against it. site.yml is the
# one-command bring-up, so this is a property of the authored sequence, not of an operator's
# memory. (kube-vip can precede the CNI because its DaemonSet is hostNetwork with
# KUBERNETES_SERVICE_HOST=127.0.0.1 — asserted separately below/above via the manifest parse.)
site_play = PLAYBOOKS / "site.yml"
check(site_play.is_file(), "playbooks/site.yml is missing — there is no one-command bring-up "
"whose ORDER can be checked")
if site_play.is_file():
imports = re.findall(r"import_playbook:\s*(\S+)", site_play.read_text())
for needed in ("k3s.yml", "kube-vip.yml", "cilium.yml"):
check(needed in imports,
f"playbooks/site.yml does not import {needed} (imports: {imports}). Cilium's "
"k8sServiceHost is the control-plane VIP, so kube-vip is a PREREQUISITE of the "
"CNI, not an optional extra step — leaving it out of the bring-up means a cold "
"build installs a CNI pointed at an address nothing announces")
if all(n in imports for n in ("k3s.yml", "kube-vip.yml", "cilium.yml")):
check(imports.index("k3s.yml") < imports.index("kube-vip.yml") < imports.index("cilium.yml"),
f"playbooks/site.yml imports {imports} — the order MUST be k3s.yml -> "
"kube-vip.yml -> cilium.yml. kube-vip after cilium installs the CNI against an "
"unannounced VIP (the 2026-09-01 shape: agents that cannot reach the API server "
"on a cold cluster); kube-vip before k3s cannot work at all, because the VIP "
"ships as a k3s auto-deploy manifest and needs a cluster to apply it")
check(re.search(r"wait_for", VIP_TEXT) is not None
and re.search(r"kube_vip_address", VIP_TEXT) is not None,
"roles/kube_vip never WAITS for the VIP to answer — templating the manifest is not the "
"same as having a floating endpoint (k3s still has to apply it, the image still has to "
"pull, leader election still has to settle). Since the CNI is installed immediately "
"afterward against this exact address, an unverified VIP turns a missing endpoint into "
"unreachable cilium-agents diagnosed from the far side")
# ============================================ FLOATING ENDPOINT: kube-vip on every server
# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
@@ -565,7 +721,9 @@ print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
"R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
"embedded etcd on every server with no external datastore, a leader-elected "
"control-plane-only VIP on every server that is SAN'd into the serving cert and "
"collides with no other allocator, from one code path for both environments. The LIVE "
"collides with no other allocator, floated BEFORE the CNI that dials it (and the CNI "
"dials that VIP, not the founder, in both environments and on both sides of the "
"[parity] contract), from one code path for both environments. The LIVE "
"cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
"SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
PY