home-network-k3s-impl/conformance/tests/l0-static/ha-control-plane-as-code/check.sh

#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — ha-control-plane-as-code
# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade
#         HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the
#         same multi-node topology." — spec docs/00-vision-and-scope.md).
#
# The elaborating spec text this check quotes and asserts against:
#   docs/01-architecture-overview.md  "Kubernetes distro | **k3s** (HA, embedded etcd)"
#   docs/02-environments-and-provisioning.md, table:
#       "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd
#        (k3s server on all 3)"   (the SAME cell for dev/test and prod)
#       "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
#   docs/02 §"k3s topology decisions":
#       "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)."
#       "Nodes are both control-plane and workload (3-node converged cluster); with only 3
#        nodes we do not isolate control-plane nodes."
#       "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
#        step differs"
#
# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives)
# ---------------------------------------------------------------------
# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers:
#
#   AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control
#       plane in BOTH environments, from ONE code path": provable with no cluster, and
#       ONLY provable here. A live cluster cannot tell you whether the *next* environment
#       brought up from this repo will also be HA, whether a second `--cluster-init`
#       founder could exist, or whether an external datastore is configured anywhere. That
#       is this check.
#   OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all
#       etcd members, with the API answering through the floating VIP": that is cluster
#       state and lives in tests/l2-behavioral/ha-control-plane (same requirement label).
#
# Neither half is sufficient alone, and this file claims only the first.
#
# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod"
# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the
# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every
# group_vars/<group>/vars.yml that overrides them — and identifies WHICH environment is
# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by
# its group name. Add a fourth prod node, rename a group, or add a third environment and
# the assertions follow; nothing here restates an address, a hostname or a node count.
#
# GIVEN the repository as git tracks it,
# WHEN  the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's
#       Flux overlay are parsed and cross-checked against each other,
# THEN  DISTRO:   k3s is the distro, pinned, installed by the one role, and no node is
#                 installed as an `agent` (an agent is not a control-plane member);
#       SERVERS:  every environment's server group resolves to a quorum-capable, ODD
#                 number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to
#                 EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while
#                 every other server joins it with `--server` (so a second founder — the
#                 split-brain shape — cannot be authored);
#       ETCD:     the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no
#                 `--disable-etcd` anywhere in the provisioning code, so every server is
#                 an etcd member (docs/02's "no external datastore", "on all 3");
#       ENDPOINT: the API has a floating endpoint that survives losing the founder — the
#                 kube-vip VIP is control-plane-only (never Service LB, or it would fight
#                 Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and
#                 baked into the apiserver serving cert's SAN list ALONG WITH every
#                 server's own address (so TLS validates whichever node answers);
#       ADDRESSES: each environment's control-plane VIP is not one of its node addresses,
#                 is not that environment's ingress Gateway VIP, and lies OUTSIDE that
#                 environment's Cilium LoadBalancer pool — the two allocators can never be
#                 handed the same address;
#       CNI PATH: the CNI dials the FLOATING endpoint, not a machine (added 2026-09-06). The
#                 two API addresses stay SPLIT — k3s_api_host is the founder and feeds only
#                 the cold `--server` join, k3s_apiserver_endpoint derives from
#                 kube_vip_address and feeds Cilium's k8sServiceHost — the Flux overlay's
#                 K3S_API_HOST equals that environment's VIP and is none of its node
#                 addresses ([parity], both halves), the HelmRelease keeps a per-environment
#                 token rather than a literal, and site.yml floats the VIP BETWEEN k3s and
#                 Cilium with roles/kube_vip blocking until it answers;
#       ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
#                 retargeted by variables only (docs/02 "the *same* Ansible bootstrap
#                 runs"), so "exercise the same multi-node topology" is a property of the
#                 code rather than a coincidence of two hand-written inventories.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
#   - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with
#     a healthy quorum, that kube-vip is actually floating the VIP, and that the API
#     answers through it. All of that is asserted at L2 by
#     tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test.
#   - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds,
#     the VIP moves, the API stays up) is a fault-injection claim. The nearest existing
#     coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's
#     data-availability claim and does NOT assert control-plane survival. A dedicated
#     "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and
#     is NOT claimed here or at L2.
#   - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged
#     mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
#     decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
#     check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
#   - IPv6 / dual-stack, and any claim about the edge network.
#
# TWO FORMER GAPS ARE NOW ASSERTED, and the reasoning is worth keeping (2026-09-06):
#   * CILIUM'S OWN API ENDPOINT used to be listed here as an ESCALATION — `K3S_API_HOST`
#     (clusters/*/kustomization.yaml), which becomes Cilium's `k8sServiceHost`, was each
#     environment's PRIMARY NODE address rather than the floating VIP, contradicting the prod
#     overlay's own comment. Making the check fail over it would have turned the merge gate
#     red on a design question this workstream may not decide (R-TEST-06), so it was reported
#     instead. THE OPERATOR DECIDED THE VIP on 2026-09-06, which turns the question into a
#     repository invariant, and invariants belong in assertions — see the "THE CNI DIALS THE
#     VIP" section. Note what made it assertable: the fix SPLIT one variable in two, so there
#     is finally an HA-valued name (`k3s_apiserver_endpoint`) to point the contract at.
#   * kube-vip WAS NOT PART OF THE ONE-COMMAND BRING-UP (site.yml chained node-baseline ->
#     k3s -> cilium -> verify -> flux-bootstrap), which was recorded here as an observation
#     and not asserted, because R-PLAT-01 does not fix the bring-up order. It is asserted now
#     for a different reason that IS in scope: with the CNI's endpoint being the VIP, an
#     unfloated VIP means a cold build installs a CNI that cannot reach its API server. The
#     ordering is no longer a preference about tidiness, it is a dependency of the control
#     plane coming up at all.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
PYTHON="${PYTHON:-python3}"

cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress
import re
import sys
from pathlib import Path

try:
    import yaml
except ImportError:                                  # pragma: no cover
    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
          file=sys.stderr)
    raise SystemExit(1)

ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0


def check(cond, msg):
    global CHECKS
    CHECKS += 1
    if not cond:
        FAILURES.append(msg)


def note(msg):
    NOTES.append(msg)


def die():
    print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
          file=sys.stderr)
    for f in FAILURES:
        print(f"  - {f}", file=sys.stderr)
    raise SystemExit(1)


ANS = ROOT / "infra" / "ansible"
INV = ANS / "inventory" / "hosts.ini"
GV = ANS / "inventory" / "group_vars"
K3S_ROLE = ANS / "roles" / "k3s_server"
VIP_ROLE = ANS / "roles" / "kube_vip"
PLAYBOOKS = ANS / "playbooks"
CLUSTERS = ROOT / "clusters"

for p, why in ((INV, "the inventory that defines the node set"),
               (K3S_ROLE, "the role that installs the k3s control plane"),
               (VIP_ROLE, "the role that floats the control-plane VIP"),
               (CLUSTERS, "the Flux overlays that identify each environment")):
    check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA "
                      "control plane is authored at all (R-PLAT-01)")
if FAILURES:
    die()

K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml")))
VIP_TEXT = "\n".join(p.read_text(errors="replace")
                     for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2"))))


# ===================================================================== inventory parse
# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a
# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]`
# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'.
ini_hosts: dict[str, list[str]] = {}
ini_children: dict[str, list[str]] = {}
host_vars: dict[str, dict[str, str]] = {}
cur = None
for line in INV.read_text().splitlines():
    s = line.strip()
    if not s or s.startswith("#"):
        continue
    m = re.match(r"^\[([^\]]+)\]$", s)
    if m:
        cur = m.group(1)
        continue
    if cur is None or cur.endswith(":vars"):
        continue
    host = s.split()[0]
    if cur.endswith(":children"):
        ini_children.setdefault(cur.split(":")[0], []).append(host)
    else:
        ini_hosts.setdefault(cur, []).append(host)
        hv = host_vars.setdefault(host, {})
        for kv in s.split()[1:]:
            if "=" in kv:
                k, v = kv.split("=", 1)
                hv[k] = v


def hosts_in(group: str, seen=None) -> set[str]:
    seen = seen or set()
    if group in seen:
        return set()
    seen.add(group)
    out = set(ini_hosts.get(group, []))
    for child in ini_children.get(group, []):
        out |= hosts_in(child, seen)
    return out


def load_yaml(p: Path) -> dict:
    try:
        d = yaml.safe_load(p.read_text()) or {}
    except yaml.YAMLError:
        return {}
    return d if isinstance(d, dict) else {}


ALL_VARS = load_yaml(GV / "all.yml")


# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM
# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds
# the default pair; every group_vars/<g>/vars.yml that overrides `k3s_primary_group` is
# another environment. Nothing below knows the words "dev" or "prod" as group names.
class Env:
    def __init__(self, label, vars_, path):
        self.label = label
        self.vars = vars_
        self.path = path

    def var(self, name, default=None):
        # group-scoped value wins, else the cluster-wide default from all.yml.
        if name in self.vars:
            return self.vars[name]
        return ALL_VARS.get(name, default)


envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")]
for vf in sorted(GV.glob("*/vars.yml")):
    v = load_yaml(vf)
    if "k3s_primary_group" in v or "k3s_servers_group" in v:
        envs.append(Env(str(vf.relative_to(ROOT)), v, vf))

check(len(envs) >= 2,
      "only ONE environment is derivable from the inventory's R-ENV-02 seam "
      "(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires "
      "dev/test to exercise the same multi-node topology, so two must be authored")

# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay.
# roles/flux_bootstrap's own default supplies the value for an environment that does not
# override it, so this is read from code, never guessed from a group name.
fb_default = None
fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml"
if fb_defaults_file.is_file():
    fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path")
check(isinstance(fb_default, str) and fb_default.startswith("clusters/"),
      "roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from "
      "an inventory environment to its Flux overlay is not expressed in code, so this check "
      "cannot tell which environment is prod")

prod_envs = []
for e in envs:
    overlay = e.var("flux_bootstrap_env_path", fb_default)
    e.overlay = overlay if isinstance(overlay, str) else ""
    if e.overlay.rstrip("/").endswith("/prod"):
        prod_envs.append(e)
check(len(prod_envs) == 1,
      f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay "
      f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in "
      "R-PLAT-01 has no unambiguous referent")


# ============================================== overlay knob values (per environment)
# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map;
# each clusters/<env>/kustomization.yaml overrides values with JSON6902 patches written as
#     - op: replace
#       path: /data/<KEY>
#       value: "<v>"
# (verified shape across both overlays). A key not patched keeps the base value.
BASE_VARS: dict[str, str] = {}
base_cm = CLUSTERS / "base" / "cluster-vars.yaml"
if base_cm.is_file():
    d = load_yaml(base_cm)
    BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()}
check(len(BASE_VARS) > 5,
      "clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface "
      "cannot be read, so the VIP/pool separation cannot be checked")

_PATCH_RE = re.compile(
    r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n\s*value:\s*(?P<v>\S.*?)\s*$",
    re.M)


def overlay_vars(overlay: str) -> dict[str, str]:
    out = dict(BASE_VARS)
    f = ROOT / overlay / "kustomization.yaml"
    if not f.is_file():
        return out
    for m in _PATCH_RE.finditer(f.read_text()):
        out[m.group("k")] = m.group("v").strip().strip('"').strip("'")
    return out


# ============================================================ PER-ENVIRONMENT TOPOLOGY
def as_ip(s):
    try:
        return ipaddress.ip_address(str(s).strip())
    except ValueError:
        return None


for e in envs:
    L = e.label
    pg = e.var("k3s_primary_group")
    sg = e.var("k3s_servers_group")
    check(isinstance(pg, str) and isinstance(sg, str) and pg and sg,
          f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot "
          "resolve 'my environment's primary/servers' and would only ever build dev")
    if not (isinstance(pg, str) and isinstance(sg, str)):
        continue
    primaries = hosts_in(pg)
    servers = hosts_in(sg)

    # --- SERVERS: quorum-capable, one founder, prod exactly three ---------------------
    check(len(primaries) == 1,
          f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run "
          "`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no "
          "cluster at all")
    check(primaries and primaries <= servers,
          f"{L}: the primary {sorted(primaries)} is not a member of the servers group "
          f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or "
          "it is not part of the HA control plane it founds")
    n = len(servers)
    check(n >= 3,
          f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes "
          "'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd "
          "cluster of fewer than 3 members tolerates ZERO losses — it is not HA")
    check(n % 2 == 1,
          f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd "
          f"member count adds a member without adding fault tolerance and introduces a tie: "
          f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}")
    if n:
        tolerance = n - (n // 2 + 1)
        check(tolerance >= 1,
              f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate "
              f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one")
    if e in prod_envs:
        check(n == 3,
              f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: "
              "'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'")
    else:
        check(n >= 2,
              f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer "
              "nodes than prod but requires dev/test to 'exercise the same multi-node "
              "topology' — one node exercises no topology")

    # --- ADDRESSES: the VIP can collide with nothing ----------------------------------
    vip_raw = e.var("kube_vip_address")
    vip = as_ip(vip_raw)
    check(vip is not None,
          f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has "
          "no floating endpoint, so losing the founder takes the API endpoint with it")
    node_ips = {}
    for h in sorted(servers):
        ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
        check(ip is not None,
              f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join "
              "target and the serving-cert SAN list all derive from it")
        if ip is not None:
            node_ips[h] = ip
    if vip is not None:
        check(vip not in set(node_ips.values()),
              f"{L}: kube_vip_address {vip} is ALSO a node address "
              f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a "
              "node's own address cannot float off that node")
        ov = overlay_vars(getattr(e, "overlay", ""))
        gw = as_ip(ov.get("GATEWAY_VIP", ""))
        lo = as_ip(ov.get("CILIUM_LB_POOL_START", ""))
        hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", ""))
        check(gw is not None and lo is not None and hi is not None,
              f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + "
              "CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and "
              "the Service-LB allocator cannot be checked")
        if gw is not None:
            check(vip != gw,
                  f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME "
                  f"address ({vip}) — kube-vip and Cilium would both ARP for it")
        if lo is not None and hi is not None:
            check(not (lo <= vip <= hi),
                  f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium "
                  f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a "
                  "Service, and the two allocators would fight over ARP for it")
            check(lo <= hi,
                  f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be "
                  "used to prove the VIP is outside it")

# ==================================================================== DISTRO: k3s only
check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M)
      is not None,
      "roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane "
      "version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)")
check("get.k3s.io" in K3S_TEXT,
      "roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the "
      "distro (ADR-0001) and no other distro may stand in for it")
check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None,
      "roles/k3s_server never composes the `server` subcommand — without it k3s installs an "
      "AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')")
for bad, why in (
    (r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 "
                             "nodes (no external datastore)')"),
    (r"datastore-endpoint\s*:", "an external datastore via config.yaml"),
    (r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not "
                        "an etcd member (docs/02: etcd 'across all 3 nodes')"),
    (r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"),
):
    hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*"))
            if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini")
            and re.search(bad, p.read_text(errors="replace"))]
    check(not hits,
          f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA "
          f"mode, and {bad!r} contradicts it")

# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH
check(re.search(r"--cluster-init", K3S_TEXT) is not None,
      "roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never "
      "founded, so there is no HA datastore to join")
init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'",
                      K3S_TEXT)
check(init_when is not None,
      "the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an "
      "ungated founder flag would re-found the cluster on every server (split brain)")
join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
check(join is not None,
      "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
      "— without a join target the servers are separate single-node clusters")
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
                      K3S_TEXT, re.S)
check(join_when is not None,
      "the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
      "would also try to join itself")
k3s_play = PLAYBOOKS / "k3s.yml"
check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint")
if k3s_play.is_file():
    play_txt = k3s_play.read_text()
    modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt)
    check(sorted(modes) == ["primary", "secondary"],
          f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and "
          "one secondary play must exist, or the founder/joiner split is not authored")
    check(re.search(r"serial:\s*1", play_txt) is not None,
          "playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — "
          "converging two of three servers at once drops embedded etcd below quorum, which "
          "is exactly the outage an HA control plane exists to avoid")
    hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M)
    check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns),
          f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal "
          "group name means this play can only ever build ONE environment, so prod cannot "
          "be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')")

# =========================================== SERVING-CERT SANs: VIP + every server
sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", ""))
check("kube_vip_address" in sans,
      "roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's "
      "serving cert would carry no SAN for the floating VIP, so every client reaching the "
      "API through it fails TLS validation and the HA endpoint is unusable")
check("k3s_servers_group" in sans and "k3s_node_ip" in sans,
      "k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the "
      "cert must be valid for WHICHEVER server answers behind the VIP, and per-environment "
      "resolution of that set is what keeps one role serving both environments (R-ENV-02)")
check(re.search(r"tls-san:", K3S_TEXT) is not None,
      "the SAN list is passed only as an install flag and never persisted to "
      "/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
      "no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")

# ============================ THE CNI DIALS THE VIP, NOT THE FOUNDER (added 2026-09-06)
# This section used to be an ESCALATION in the header's gap list, not an assertion: Cilium's
# k8sServiceHost was each environment's PRIMARY NODE address, so on a 3-server HA control
# plane the CNI's control-plane path depended on one machine. Measured on live prod
# 2026-09-06: all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10
# (k3s-prod-1's own address) while kube-vip held 192.168.2.9. That failure is SILENT — the
# datapath keeps serving from Cilium's cache, so nothing looks broken while no service or
# policy update lands at all.
#
# The operator decided the VIP (2026-09-06), so this is now a repository property and belongs
# here rather than in the gap list. THIS IS THE RIGHT LAYER because the claim is about what
# the configuration in git WOULD build, in BOTH environments, including the next one — a live
# cluster can only tell you about itself.
#
# WHAT MAKES THIS CHECKABLE AT ALL is that the fix split one variable into two. Asserting
# "Cilium uses the HA address" requires an HA address to name; before the split there was
# only k3s_api_host, which had to be the founder for the cold join. So the assertions below
# check the SPLIT as much as the value: the join keeps the founder, the CNI gets the VIP, and
# the Flux half agrees with the Ansible half per environment.
CIL_ROLE = ANS / "roles" / "cilium"
cil_values_tpl = CIL_ROLE / "templates" / "cilium-values.yaml.j2"
check(cil_values_tpl.is_file(),
      "roles/cilium ships no cilium-values.yaml.j2 — the bootstrap's CNI values, including the "
      "API endpoint its kube-proxy replacement dials, are not authored")
if cil_values_tpl.is_file():
    # Parse the rendered-shape template, never its prose: strip comment lines first, exactly
    # as the kube-vip parser above does, so a reassuring comment cannot satisfy the check.
    cil_body = "\n".join(l for l in cil_values_tpl.read_text(errors="replace").splitlines()
                         if not l.lstrip().startswith("#"))
    m = re.search(r"^k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", cil_body, re.M)
    check(m is not None,
          "roles/cilium's values template sets no k8sServiceHost — with kube-proxy disabled "
          "(ADR-0001) the agent has no API endpoint to dial and the CNI cannot come up")
    if m is not None:
        host_expr = m.group(1).strip()
        check("k3s_apiserver_endpoint" in host_expr,
              f"roles/cilium renders k8sServiceHost={host_expr!r}. It MUST be the HA endpoint "
              "variable {{ k3s_apiserver_endpoint }} (= kube_vip_address). Any node-valued "
              "expression — k3s_api_host, a primary hostvar, a literal IP — makes the CNI's "
              "control-plane path depend on ONE server of an HA control plane, and the failure "
              "is silent: the datapath keeps running from Cilium's cache while no service or "
              "policy update ever lands (measured on prod 2026-09-06)")
        check("k3s_api_host" not in host_expr,
              f"roles/cilium renders k8sServiceHost={host_expr!r}, which references "
              "k3s_api_host — the FOUNDER's own address. That variable exists only for the "
              "cold-bootstrap `--server` join and one tls-san entry; it is a single point of "
              "failure by construction and must never be a steady-state endpoint")

# The HA endpoint variable must BE the VIP, and must be DERIVED from kube_vip_address rather
# than restated: an environment overrides one line (kube_vip_address) and both the floating
# address and the CNI's endpoint move together. A literal here could disagree with the VIP the
# same environment actually floats, which is the two-places-one-fact bug this repo keeps
# hitting.
endpoint_def = str(ALL_VARS.get("k3s_apiserver_endpoint", ""))
check("kube_vip_address" in endpoint_def,
      f"group_vars/all.yml defines k3s_apiserver_endpoint as {endpoint_def!r}. It MUST derive "
      "from kube_vip_address, so that overriding the VIP for an environment also moves what "
      "that environment's CNI dials. A restated literal can silently disagree with the VIP "
      "kube-vip is actually floating")
check("kube_vip_address" not in str(ALL_VARS.get("k3s_api_host", "")),
      "group_vars/all.yml's k3s_api_host resolves from kube_vip_address — the two API "
      "addresses have been re-merged. k3s_api_host is the secondary join target and on a COLD "
      "build no VIP exists yet, so a VIP-valued join target makes a cold bootstrap unjoinable")

# THE [parity] CONTRACT, now machine-checked instead of asserted in a comment: the Flux
# overlay's K3S_API_HOST becomes Cilium's k8sServiceHost under the HelmRelease, and the
# Ansible bootstrap renders k3s_apiserver_endpoint for the same field. If the two disagree,
# Flux and the bootstrap describe different control-plane paths and fight over the CNI.
for e in envs:
    L = e.label
    ov = overlay_vars(getattr(e, "overlay", ""))
    k3s_api_host_knob = as_ip(ov.get("K3S_API_HOST", ""))
    vip = as_ip(e.var("kube_vip_address"))
    check(k3s_api_host_knob is not None,
          f"{L}: overlay {getattr(e, 'overlay', '')!r} resolves no usable K3S_API_HOST — that "
          "token becomes Cilium's k8sServiceHost via platform/cilium/helmrelease.yaml, and "
          "kustomize-controller runs with StrictPostBuildSubstitutions so an unresolvable one "
          "fails the whole platform Kustomization")
    if k3s_api_host_knob is not None and vip is not None:
        check(k3s_api_host_knob == vip,
              f"{L}: [parity] BROKEN — the Flux overlay sets K3S_API_HOST={k3s_api_host_knob} "
              f"but this environment's control-plane VIP (kube_vip_address, which is what "
              f"k3s_apiserver_endpoint and therefore the Ansible-side k8sServiceHost resolve "
              f"to) is {vip}. Flux and the Ansible bootstrap would render DIFFERENT API "
              "endpoints into the same CNI and fight over it. If they differ because the "
              "overlay still holds a NODE address, that is the 2026-09-06 defect: the CNI of "
              "an HA control plane must not dial one server")
        check(k3s_api_host_knob not in set(
                  as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
                  for h in hosts_in(str(e.var("k3s_servers_group")))),
              f"{L}: the overlay's K3S_API_HOST {k3s_api_host_knob} is one of this "
              "environment's own NODE addresses. Cilium's kube-proxy replacement would dial a "
              "single machine on a control plane built specifically to survive losing one")

# The Flux half must stay a per-environment TOKEN. A literal IP in the shared HelmRelease
# would ship one environment's control-plane address to every environment (R-ENV-02), and an
# empty value is the 2026-09-01 incident verbatim (k8sServiceHost: "" made the agent's config
# init container fall back to the in-cluster ClusterIP 10.43.0.1:443, which only the agent
# being replaced could translate: deadlocked on 2 of 3 nodes).
hr = ROOT / "platform" / "cilium" / "helmrelease.yaml"
check(hr.is_file(), "platform/cilium/helmrelease.yaml is missing — nothing adopts the CNI "
                    "config after the bootstrap, so the parity contract has only one side")
if hr.is_file():
    hr_body = "\n".join(l for l in hr.read_text(errors="replace").splitlines()
                        if not l.lstrip().startswith("#"))
    hm = re.search(r"^\s*k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", hr_body, re.M)
    check(hm is not None and hm.group(1).strip() == "${K3S_API_HOST}",
          f"platform/cilium/helmrelease.yaml sets k8sServiceHost="
          f"{(hm.group(1) if hm else None)!r} instead of the cluster-vars token "
          "\"${K3S_API_HOST}\" — a literal there would ship ONE environment's control-plane "
          "address to every environment (R-ENV-02), and an empty value is the 2026-09-01 "
          "incident verbatim")

# ORDERING: the VIP must be floating BEFORE the CNI is installed against it. site.yml is the
# one-command bring-up, so this is a property of the authored sequence, not of an operator's
# memory. (kube-vip can precede the CNI because its DaemonSet is hostNetwork with
# KUBERNETES_SERVICE_HOST=127.0.0.1 — asserted separately below/above via the manifest parse.)
site_play = PLAYBOOKS / "site.yml"
check(site_play.is_file(), "playbooks/site.yml is missing — there is no one-command bring-up "
                           "whose ORDER can be checked")
if site_play.is_file():
    imports = re.findall(r"import_playbook:\s*(\S+)", site_play.read_text())
    for needed in ("k3s.yml", "kube-vip.yml", "cilium.yml"):
        check(needed in imports,
              f"playbooks/site.yml does not import {needed} (imports: {imports}). Cilium's "
              "k8sServiceHost is the control-plane VIP, so kube-vip is a PREREQUISITE of the "
              "CNI, not an optional extra step — leaving it out of the bring-up means a cold "
              "build installs a CNI pointed at an address nothing announces")
    if all(n in imports for n in ("k3s.yml", "kube-vip.yml", "cilium.yml")):
        check(imports.index("k3s.yml") < imports.index("kube-vip.yml") < imports.index("cilium.yml"),
              f"playbooks/site.yml imports {imports} — the order MUST be k3s.yml -> "
              "kube-vip.yml -> cilium.yml. kube-vip after cilium installs the CNI against an "
              "unannounced VIP (the 2026-09-01 shape: agents that cannot reach the API server "
              "on a cold cluster); kube-vip before k3s cannot work at all, because the VIP "
              "ships as a k3s auto-deploy manifest and needs a cluster to apply it")
check(re.search(r"wait_for", VIP_TEXT) is not None
      and re.search(r"kube_vip_address", VIP_TEXT) is not None,
      "roles/kube_vip never WAITS for the VIP to answer — templating the manifest is not the "
      "same as having a floating endpoint (k3s still has to apply it, the image still has to "
      "pull, leader election still has to settle). Since the CNI is installed immediately "
      "afterward against this exact address, an unverified VIP turns a missing endpoint into "
      "unreachable cilium-agents diagnosed from the far side")

# ============================================ FLOATING ENDPOINT: kube-vip on every server
# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the
# thing that ships.) Parser contract: the template writes each env var as an adjacent
# `- name: X` / `value: "Y"` pair; comment lines are dropped first.
VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2"))
check(len(VIP_TEMPLATES) >= 1,
      "roles/kube_vip ships no manifest template — nothing deploys the API VIP")
vip_manifest = "\n".join(
    "\n".join(l for l in p.read_text(errors="replace").splitlines()
              if not l.lstrip().startswith("#"))
    for p in VIP_TEMPLATES)
vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?",
                          vip_manifest))
check(len(vip_env) >= 6,
      f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the "
      "parser lost its grip on the file, so the assertions below would assert nothing")
check(vip_env.get("cp_enable") == "true",
      f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane "
      "mode must be ON or it floats no API VIP at all")
check(vip_env.get("svc_enable") == "false",
      f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be "
      "false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM "
      "for the same addresses (ADR-0002 gives Service LB to Cilium)")
check(vip_env.get("vip_leaderelection") == "true",
      f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} "
      "— without leader election the VIP has no defined owner to fail over FROM, which is "
      "the whole HA mechanism")
check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}",
      f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the "
      "environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on "
      "prod (R-ENV-02) and would not match the serving-cert SAN list")
check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None,
      "the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its "
      "node, which is the opposite of a floating endpoint")
vip_play = PLAYBOOKS / "kube-vip.yml"
check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP")
if vip_play.is_file():
    vp = vip_play.read_text()
    vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M)
    check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0],
          f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers "
          "group through the same environment-scoped variable the other plays use, so the "
          "manifest exists on EVERY server and the VIP survives losing any one of them")
check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None
      and re.search(r"k3s_data_dir", VIP_TEXT) is not None,
      "roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies "
      "<data-dir>/server/manifests, so an environment that moves --data-dir would write the "
      "manifest somewhere k3s never reads and come up with no VIP at all")

# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS
# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
# step differs". A prod-only role or playbook would be a fork of the control-plane path.
role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()}
play_names = {p.name for p in PLAYBOOKS.glob("*.yml")}
env_words = re.compile(r"prod|dev", re.I)
forked = sorted(n for n in (role_names | play_names)
                if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml"))
check(not forked,
      f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by "
      "the SAME code retargeted by variables (docs/02), not by a per-environment fork")
for e in envs:
    if e.path.name != "vars.yml":
        continue
    # Every key a non-default environment sets must be a variable something already reads,
    # or it is an invented knob nothing consumes (a silent no-op that looks like config).
    consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*"))
                          if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh"))
    for k in sorted(e.vars):
        if k.startswith("ansible_"):
            continue
        check(consumers.count(k) > 1,
              f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an "
              "environment override that nothing reads is not a scoped override, it is dead "
              "configuration (and would make prod silently inherit a dev default)")

# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 40,
      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
      "would be meaningless")

if FAILURES:
    die()
for n in NOTES:
    print(f"  note: {n}")
summary = "; ".join(
    f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]="
    f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers"
    for e in envs)
print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
      "R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
      "embedded etcd on every server with no external datastore, a leader-elected "
      "control-plane-only VIP on every server that is SAN'd into the serving cert and "
      "collides with no other allocator, floated BEFORE the CNI that dials it (and the CNI "
      "dials that VIP, not the founder, in both environments and on both sides of the "
      "[parity] contract), from one code path for both environments. The LIVE "
      "cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
      "SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
PY