jakman85andClaude Opus 5
6f61c9622d
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
89 lines
5.6 KiB
YAML
89 lines
5.6 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
|
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
|
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
|
|
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
|
|
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
|
|
#
|
|
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
|
|
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
|
|
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
|
|
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
|
|
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
|
|
# cluster cannot tell you what the NEXT environment brought up from this repo would look
|
|
# like. The observed half is asserted separately at L2 by
|
|
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
|
|
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
|
|
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
|
|
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
|
|
# environment's Flux overlay are parsed and cross-checked against each other,
|
|
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
|
|
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
|
|
# ONE --cluster-init founder and every other server joining it with --server;
|
|
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
|
|
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
|
|
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
|
|
# in the apiserver serving cert's SANs together with every server's own address, and
|
|
# it is neither a node address nor the ingress Gateway VIP nor inside that
|
|
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
|
|
# roles and plays retargeted by variables only.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
|
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
|
# Chainsaw Test wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
|
|
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
|
|
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
|
|
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
|
|
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
|
|
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
|
|
# infra/PROD-PROVISIONING.md).
|
|
#
|
|
# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
|
|
# * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
|
|
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
|
|
# It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
|
|
# The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
|
|
# contract and per environment.
|
|
# * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
|
|
# the order is asserted — not as a tidiness preference but because the CNI's endpoint is
|
|
# the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
|
|
# API server.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: ha-control-plane-as-code
|
|
labels:
|
|
conformance.home.k3s/R-PLAT-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-PLAT-01"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
|
|
environment's Flux overlay are parsed and cross-checked, then every environment is
|
|
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
|
|
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
|
|
VIP that is in the serving cert's SANs and collides with no other allocator — built by
|
|
one code path for both environments.
|
|
spec:
|
|
steps:
|
|
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|