jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
730 lines
42 KiB
Bash
Executable File
730 lines
42 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# L0 static check (offline, no cluster) — ha-control-plane-as-code
|
|
# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade
|
|
# HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the
|
|
# same multi-node topology." — spec docs/00-vision-and-scope.md).
|
|
#
|
|
# The elaborating spec text this check quotes and asserts against:
|
|
# docs/01-architecture-overview.md "Kubernetes distro | **k3s** (HA, embedded etcd)"
|
|
# docs/02-environments-and-provisioning.md, table:
|
|
# "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd
|
|
# (k3s server on all 3)" (the SAME cell for dev/test and prod)
|
|
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
|
|
# docs/02 §"k3s topology decisions":
|
|
# "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)."
|
|
# "Nodes are both control-plane and workload (3-node converged cluster); with only 3
|
|
# nodes we do not isolate control-plane nodes."
|
|
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
|
# step differs"
|
|
#
|
|
# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives)
|
|
# ---------------------------------------------------------------------
|
|
# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers:
|
|
#
|
|
# AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control
|
|
# plane in BOTH environments, from ONE code path": provable with no cluster, and
|
|
# ONLY provable here. A live cluster cannot tell you whether the *next* environment
|
|
# brought up from this repo will also be HA, whether a second `--cluster-init`
|
|
# founder could exist, or whether an external datastore is configured anywhere. That
|
|
# is this check.
|
|
# OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all
|
|
# etcd members, with the API answering through the floating VIP": that is cluster
|
|
# state and lives in tests/l2-behavioral/ha-control-plane (same requirement label).
|
|
#
|
|
# Neither half is sufficient alone, and this file claims only the first.
|
|
#
|
|
# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod"
|
|
# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the
|
|
# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every
|
|
# group_vars/<group>/vars.yml that overrides them — and identifies WHICH environment is
|
|
# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by
|
|
# its group name. Add a fourth prod node, rename a group, or add a third environment and
|
|
# the assertions follow; nothing here restates an address, a hostname or a node count.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's
|
|
# Flux overlay are parsed and cross-checked against each other,
|
|
# THEN DISTRO: k3s is the distro, pinned, installed by the one role, and no node is
|
|
# installed as an `agent` (an agent is not a control-plane member);
|
|
# SERVERS: every environment's server group resolves to a quorum-capable, ODD
|
|
# number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to
|
|
# EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while
|
|
# every other server joins it with `--server` (so a second founder — the
|
|
# split-brain shape — cannot be authored);
|
|
# ETCD: the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no
|
|
# `--disable-etcd` anywhere in the provisioning code, so every server is
|
|
# an etcd member (docs/02's "no external datastore", "on all 3");
|
|
# ENDPOINT: the API has a floating endpoint that survives losing the founder — the
|
|
# kube-vip VIP is control-plane-only (never Service LB, or it would fight
|
|
# Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and
|
|
# baked into the apiserver serving cert's SAN list ALONG WITH every
|
|
# server's own address (so TLS validates whichever node answers);
|
|
# ADDRESSES: each environment's control-plane VIP is not one of its node addresses,
|
|
# is not that environment's ingress Gateway VIP, and lies OUTSIDE that
|
|
# environment's Cilium LoadBalancer pool — the two allocators can never be
|
|
# handed the same address;
|
|
# CNI PATH: the CNI dials the FLOATING endpoint, not a machine (added 2026-09-06). The
|
|
# two API addresses stay SPLIT — k3s_api_host is the founder and feeds only
|
|
# the cold `--server` join, k3s_apiserver_endpoint derives from
|
|
# kube_vip_address and feeds Cilium's k8sServiceHost — the Flux overlay's
|
|
# K3S_API_HOST equals that environment's VIP and is none of its node
|
|
# addresses ([parity], both halves), the HelmRelease keeps a per-environment
|
|
# token rather than a literal, and site.yml floats the VIP BETWEEN k3s and
|
|
# Cilium with roles/kube_vip blocking until it answers;
|
|
# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
|
|
# retargeted by variables only (docs/02 "the *same* Ansible bootstrap
|
|
# runs"), so "exercise the same multi-node topology" is a property of the
|
|
# code rather than a coincidence of two hand-written inventories.
|
|
#
|
|
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
|
# - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with
|
|
# a healthy quorum, that kube-vip is actually floating the VIP, and that the API
|
|
# answers through it. All of that is asserted at L2 by
|
|
# tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test.
|
|
# - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds,
|
|
# the VIP moves, the API stays up) is a fault-injection claim. The nearest existing
|
|
# coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's
|
|
# data-availability claim and does NOT assert control-plane survival. A dedicated
|
|
# "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and
|
|
# is NOT claimed here or at L2.
|
|
# - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged
|
|
# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
|
|
# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
|
|
# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
|
|
# - IPv6 / dual-stack, and any claim about the edge network.
|
|
#
|
|
# TWO FORMER GAPS ARE NOW ASSERTED, and the reasoning is worth keeping (2026-09-06):
|
|
# * CILIUM'S OWN API ENDPOINT used to be listed here as an ESCALATION — `K3S_API_HOST`
|
|
# (clusters/*/kustomization.yaml), which becomes Cilium's `k8sServiceHost`, was each
|
|
# environment's PRIMARY NODE address rather than the floating VIP, contradicting the prod
|
|
# overlay's own comment. Making the check fail over it would have turned the merge gate
|
|
# red on a design question this workstream may not decide (R-TEST-06), so it was reported
|
|
# instead. THE OPERATOR DECIDED THE VIP on 2026-09-06, which turns the question into a
|
|
# repository invariant, and invariants belong in assertions — see the "THE CNI DIALS THE
|
|
# VIP" section. Note what made it assertable: the fix SPLIT one variable in two, so there
|
|
# is finally an HA-valued name (`k3s_apiserver_endpoint`) to point the contract at.
|
|
# * kube-vip WAS NOT PART OF THE ONE-COMMAND BRING-UP (site.yml chained node-baseline ->
|
|
# k3s -> cilium -> verify -> flux-bootstrap), which was recorded here as an observation
|
|
# and not asserted, because R-PLAT-01 does not fix the bring-up order. It is asserted now
|
|
# for a different reason that IS in scope: with the CNI's endpoint being the VIP, an
|
|
# unfloated VIP means a cold build installs a CNI that cannot reach its API server. The
|
|
# ordering is no longer a preference about tidiness, it is a dependency of the control
|
|
# plane coming up at all.
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
|
PYTHON="${PYTHON:-python3}"
|
|
|
|
cd "${ROOT}"
|
|
"${PYTHON}" - "${ROOT}" <<'PY'
|
|
import ipaddress
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
try:
|
|
import yaml
|
|
except ImportError: # pragma: no cover
|
|
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
|
file=sys.stderr)
|
|
raise SystemExit(1)
|
|
|
|
ROOT = Path(sys.argv[1])
|
|
FAILURES: list[str] = []
|
|
NOTES: list[str] = []
|
|
CHECKS = 0
|
|
|
|
|
|
def check(cond, msg):
|
|
global CHECKS
|
|
CHECKS += 1
|
|
if not cond:
|
|
FAILURES.append(msg)
|
|
|
|
|
|
def note(msg):
|
|
NOTES.append(msg)
|
|
|
|
|
|
def die():
|
|
print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
|
|
file=sys.stderr)
|
|
for f in FAILURES:
|
|
print(f" - {f}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
|
|
|
|
ANS = ROOT / "infra" / "ansible"
|
|
INV = ANS / "inventory" / "hosts.ini"
|
|
GV = ANS / "inventory" / "group_vars"
|
|
K3S_ROLE = ANS / "roles" / "k3s_server"
|
|
VIP_ROLE = ANS / "roles" / "kube_vip"
|
|
PLAYBOOKS = ANS / "playbooks"
|
|
CLUSTERS = ROOT / "clusters"
|
|
|
|
for p, why in ((INV, "the inventory that defines the node set"),
|
|
(K3S_ROLE, "the role that installs the k3s control plane"),
|
|
(VIP_ROLE, "the role that floats the control-plane VIP"),
|
|
(CLUSTERS, "the Flux overlays that identify each environment")):
|
|
check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA "
|
|
"control plane is authored at all (R-PLAT-01)")
|
|
if FAILURES:
|
|
die()
|
|
|
|
K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml")))
|
|
VIP_TEXT = "\n".join(p.read_text(errors="replace")
|
|
for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2"))))
|
|
|
|
|
|
# ===================================================================== inventory parse
|
|
# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a
|
|
# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]`
|
|
# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'.
|
|
ini_hosts: dict[str, list[str]] = {}
|
|
ini_children: dict[str, list[str]] = {}
|
|
host_vars: dict[str, dict[str, str]] = {}
|
|
cur = None
|
|
for line in INV.read_text().splitlines():
|
|
s = line.strip()
|
|
if not s or s.startswith("#"):
|
|
continue
|
|
m = re.match(r"^\[([^\]]+)\]$", s)
|
|
if m:
|
|
cur = m.group(1)
|
|
continue
|
|
if cur is None or cur.endswith(":vars"):
|
|
continue
|
|
host = s.split()[0]
|
|
if cur.endswith(":children"):
|
|
ini_children.setdefault(cur.split(":")[0], []).append(host)
|
|
else:
|
|
ini_hosts.setdefault(cur, []).append(host)
|
|
hv = host_vars.setdefault(host, {})
|
|
for kv in s.split()[1:]:
|
|
if "=" in kv:
|
|
k, v = kv.split("=", 1)
|
|
hv[k] = v
|
|
|
|
|
|
def hosts_in(group: str, seen=None) -> set[str]:
|
|
seen = seen or set()
|
|
if group in seen:
|
|
return set()
|
|
seen.add(group)
|
|
out = set(ini_hosts.get(group, []))
|
|
for child in ini_children.get(group, []):
|
|
out |= hosts_in(child, seen)
|
|
return out
|
|
|
|
|
|
def load_yaml(p: Path) -> dict:
|
|
try:
|
|
d = yaml.safe_load(p.read_text()) or {}
|
|
except yaml.YAMLError:
|
|
return {}
|
|
return d if isinstance(d, dict) else {}
|
|
|
|
|
|
ALL_VARS = load_yaml(GV / "all.yml")
|
|
|
|
|
|
# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM
|
|
# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds
|
|
# the default pair; every group_vars/<g>/vars.yml that overrides `k3s_primary_group` is
|
|
# another environment. Nothing below knows the words "dev" or "prod" as group names.
|
|
class Env:
|
|
def __init__(self, label, vars_, path):
|
|
self.label = label
|
|
self.vars = vars_
|
|
self.path = path
|
|
|
|
def var(self, name, default=None):
|
|
# group-scoped value wins, else the cluster-wide default from all.yml.
|
|
if name in self.vars:
|
|
return self.vars[name]
|
|
return ALL_VARS.get(name, default)
|
|
|
|
|
|
envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")]
|
|
for vf in sorted(GV.glob("*/vars.yml")):
|
|
v = load_yaml(vf)
|
|
if "k3s_primary_group" in v or "k3s_servers_group" in v:
|
|
envs.append(Env(str(vf.relative_to(ROOT)), v, vf))
|
|
|
|
check(len(envs) >= 2,
|
|
"only ONE environment is derivable from the inventory's R-ENV-02 seam "
|
|
"(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires "
|
|
"dev/test to exercise the same multi-node topology, so two must be authored")
|
|
|
|
# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay.
|
|
# roles/flux_bootstrap's own default supplies the value for an environment that does not
|
|
# override it, so this is read from code, never guessed from a group name.
|
|
fb_default = None
|
|
fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml"
|
|
if fb_defaults_file.is_file():
|
|
fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path")
|
|
check(isinstance(fb_default, str) and fb_default.startswith("clusters/"),
|
|
"roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from "
|
|
"an inventory environment to its Flux overlay is not expressed in code, so this check "
|
|
"cannot tell which environment is prod")
|
|
|
|
prod_envs = []
|
|
for e in envs:
|
|
overlay = e.var("flux_bootstrap_env_path", fb_default)
|
|
e.overlay = overlay if isinstance(overlay, str) else ""
|
|
if e.overlay.rstrip("/").endswith("/prod"):
|
|
prod_envs.append(e)
|
|
check(len(prod_envs) == 1,
|
|
f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay "
|
|
f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in "
|
|
"R-PLAT-01 has no unambiguous referent")
|
|
|
|
|
|
# ============================================== overlay knob values (per environment)
|
|
# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map;
|
|
# each clusters/<env>/kustomization.yaml overrides values with JSON6902 patches written as
|
|
# - op: replace
|
|
# path: /data/<KEY>
|
|
# value: "<v>"
|
|
# (verified shape across both overlays). A key not patched keeps the base value.
|
|
BASE_VARS: dict[str, str] = {}
|
|
base_cm = CLUSTERS / "base" / "cluster-vars.yaml"
|
|
if base_cm.is_file():
|
|
d = load_yaml(base_cm)
|
|
BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()}
|
|
check(len(BASE_VARS) > 5,
|
|
"clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface "
|
|
"cannot be read, so the VIP/pool separation cannot be checked")
|
|
|
|
_PATCH_RE = re.compile(
|
|
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n\s*value:\s*(?P<v>\S.*?)\s*$",
|
|
re.M)
|
|
|
|
|
|
def overlay_vars(overlay: str) -> dict[str, str]:
|
|
out = dict(BASE_VARS)
|
|
f = ROOT / overlay / "kustomization.yaml"
|
|
if not f.is_file():
|
|
return out
|
|
for m in _PATCH_RE.finditer(f.read_text()):
|
|
out[m.group("k")] = m.group("v").strip().strip('"').strip("'")
|
|
return out
|
|
|
|
|
|
# ============================================================ PER-ENVIRONMENT TOPOLOGY
|
|
def as_ip(s):
|
|
try:
|
|
return ipaddress.ip_address(str(s).strip())
|
|
except ValueError:
|
|
return None
|
|
|
|
|
|
for e in envs:
|
|
L = e.label
|
|
pg = e.var("k3s_primary_group")
|
|
sg = e.var("k3s_servers_group")
|
|
check(isinstance(pg, str) and isinstance(sg, str) and pg and sg,
|
|
f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot "
|
|
"resolve 'my environment's primary/servers' and would only ever build dev")
|
|
if not (isinstance(pg, str) and isinstance(sg, str)):
|
|
continue
|
|
primaries = hosts_in(pg)
|
|
servers = hosts_in(sg)
|
|
|
|
# --- SERVERS: quorum-capable, one founder, prod exactly three ---------------------
|
|
check(len(primaries) == 1,
|
|
f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run "
|
|
"`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no "
|
|
"cluster at all")
|
|
check(primaries and primaries <= servers,
|
|
f"{L}: the primary {sorted(primaries)} is not a member of the servers group "
|
|
f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or "
|
|
"it is not part of the HA control plane it founds")
|
|
n = len(servers)
|
|
check(n >= 3,
|
|
f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes "
|
|
"'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd "
|
|
"cluster of fewer than 3 members tolerates ZERO losses — it is not HA")
|
|
check(n % 2 == 1,
|
|
f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd "
|
|
f"member count adds a member without adding fault tolerance and introduces a tie: "
|
|
f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}")
|
|
if n:
|
|
tolerance = n - (n // 2 + 1)
|
|
check(tolerance >= 1,
|
|
f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate "
|
|
f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one")
|
|
if e in prod_envs:
|
|
check(n == 3,
|
|
f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: "
|
|
"'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'")
|
|
else:
|
|
check(n >= 2,
|
|
f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer "
|
|
"nodes than prod but requires dev/test to 'exercise the same multi-node "
|
|
"topology' — one node exercises no topology")
|
|
|
|
# --- ADDRESSES: the VIP can collide with nothing ----------------------------------
|
|
vip_raw = e.var("kube_vip_address")
|
|
vip = as_ip(vip_raw)
|
|
check(vip is not None,
|
|
f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has "
|
|
"no floating endpoint, so losing the founder takes the API endpoint with it")
|
|
node_ips = {}
|
|
for h in sorted(servers):
|
|
ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
|
|
check(ip is not None,
|
|
f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join "
|
|
"target and the serving-cert SAN list all derive from it")
|
|
if ip is not None:
|
|
node_ips[h] = ip
|
|
if vip is not None:
|
|
check(vip not in set(node_ips.values()),
|
|
f"{L}: kube_vip_address {vip} is ALSO a node address "
|
|
f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a "
|
|
"node's own address cannot float off that node")
|
|
ov = overlay_vars(getattr(e, "overlay", ""))
|
|
gw = as_ip(ov.get("GATEWAY_VIP", ""))
|
|
lo = as_ip(ov.get("CILIUM_LB_POOL_START", ""))
|
|
hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", ""))
|
|
check(gw is not None and lo is not None and hi is not None,
|
|
f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + "
|
|
"CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and "
|
|
"the Service-LB allocator cannot be checked")
|
|
if gw is not None:
|
|
check(vip != gw,
|
|
f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME "
|
|
f"address ({vip}) — kube-vip and Cilium would both ARP for it")
|
|
if lo is not None and hi is not None:
|
|
check(not (lo <= vip <= hi),
|
|
f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium "
|
|
f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a "
|
|
"Service, and the two allocators would fight over ARP for it")
|
|
check(lo <= hi,
|
|
f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be "
|
|
"used to prove the VIP is outside it")
|
|
|
|
# ==================================================================== DISTRO: k3s only
|
|
check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M)
|
|
is not None,
|
|
"roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane "
|
|
"version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)")
|
|
check("get.k3s.io" in K3S_TEXT,
|
|
"roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the "
|
|
"distro (ADR-0001) and no other distro may stand in for it")
|
|
check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None,
|
|
"roles/k3s_server never composes the `server` subcommand — without it k3s installs an "
|
|
"AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')")
|
|
for bad, why in (
|
|
(r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 "
|
|
"nodes (no external datastore)')"),
|
|
(r"datastore-endpoint\s*:", "an external datastore via config.yaml"),
|
|
(r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not "
|
|
"an etcd member (docs/02: etcd 'across all 3 nodes')"),
|
|
(r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"),
|
|
):
|
|
hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*"))
|
|
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini")
|
|
and re.search(bad, p.read_text(errors="replace"))]
|
|
check(not hits,
|
|
f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA "
|
|
f"mode, and {bad!r} contradicts it")
|
|
|
|
# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH
|
|
check(re.search(r"--cluster-init", K3S_TEXT) is not None,
|
|
"roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never "
|
|
"founded, so there is no HA datastore to join")
|
|
init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'",
|
|
K3S_TEXT)
|
|
check(init_when is not None,
|
|
"the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an "
|
|
"ungated founder flag would re-found the cluster on every server (split brain)")
|
|
join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
|
|
check(join is not None,
|
|
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
|
|
"— without a join target the servers are separate single-node clusters")
|
|
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
|
|
K3S_TEXT, re.S)
|
|
check(join_when is not None,
|
|
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
|
|
"would also try to join itself")
|
|
k3s_play = PLAYBOOKS / "k3s.yml"
|
|
check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint")
|
|
if k3s_play.is_file():
|
|
play_txt = k3s_play.read_text()
|
|
modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt)
|
|
check(sorted(modes) == ["primary", "secondary"],
|
|
f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and "
|
|
"one secondary play must exist, or the founder/joiner split is not authored")
|
|
check(re.search(r"serial:\s*1", play_txt) is not None,
|
|
"playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — "
|
|
"converging two of three servers at once drops embedded etcd below quorum, which "
|
|
"is exactly the outage an HA control plane exists to avoid")
|
|
hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M)
|
|
check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns),
|
|
f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal "
|
|
"group name means this play can only ever build ONE environment, so prod cannot "
|
|
"be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')")
|
|
|
|
# =========================================== SERVING-CERT SANs: VIP + every server
|
|
sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", ""))
|
|
check("kube_vip_address" in sans,
|
|
"roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's "
|
|
"serving cert would carry no SAN for the floating VIP, so every client reaching the "
|
|
"API through it fails TLS validation and the HA endpoint is unusable")
|
|
check("k3s_servers_group" in sans and "k3s_node_ip" in sans,
|
|
"k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the "
|
|
"cert must be valid for WHICHEVER server answers behind the VIP, and per-environment "
|
|
"resolution of that set is what keeps one role serving both environments (R-ENV-02)")
|
|
check(re.search(r"tls-san:", K3S_TEXT) is not None,
|
|
"the SAN list is passed only as an install flag and never persisted to "
|
|
"/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
|
|
"no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")
|
|
|
|
# ============================ THE CNI DIALS THE VIP, NOT THE FOUNDER (added 2026-09-06)
|
|
# This section used to be an ESCALATION in the header's gap list, not an assertion: Cilium's
|
|
# k8sServiceHost was each environment's PRIMARY NODE address, so on a 3-server HA control
|
|
# plane the CNI's control-plane path depended on one machine. Measured on live prod
|
|
# 2026-09-06: all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10
|
|
# (k3s-prod-1's own address) while kube-vip held 192.168.2.9. That failure is SILENT — the
|
|
# datapath keeps serving from Cilium's cache, so nothing looks broken while no service or
|
|
# policy update lands at all.
|
|
#
|
|
# The operator decided the VIP (2026-09-06), so this is now a repository property and belongs
|
|
# here rather than in the gap list. THIS IS THE RIGHT LAYER because the claim is about what
|
|
# the configuration in git WOULD build, in BOTH environments, including the next one — a live
|
|
# cluster can only tell you about itself.
|
|
#
|
|
# WHAT MAKES THIS CHECKABLE AT ALL is that the fix split one variable into two. Asserting
|
|
# "Cilium uses the HA address" requires an HA address to name; before the split there was
|
|
# only k3s_api_host, which had to be the founder for the cold join. So the assertions below
|
|
# check the SPLIT as much as the value: the join keeps the founder, the CNI gets the VIP, and
|
|
# the Flux half agrees with the Ansible half per environment.
|
|
CIL_ROLE = ANS / "roles" / "cilium"
|
|
cil_values_tpl = CIL_ROLE / "templates" / "cilium-values.yaml.j2"
|
|
check(cil_values_tpl.is_file(),
|
|
"roles/cilium ships no cilium-values.yaml.j2 — the bootstrap's CNI values, including the "
|
|
"API endpoint its kube-proxy replacement dials, are not authored")
|
|
if cil_values_tpl.is_file():
|
|
# Parse the rendered-shape template, never its prose: strip comment lines first, exactly
|
|
# as the kube-vip parser above does, so a reassuring comment cannot satisfy the check.
|
|
cil_body = "\n".join(l for l in cil_values_tpl.read_text(errors="replace").splitlines()
|
|
if not l.lstrip().startswith("#"))
|
|
m = re.search(r"^k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", cil_body, re.M)
|
|
check(m is not None,
|
|
"roles/cilium's values template sets no k8sServiceHost — with kube-proxy disabled "
|
|
"(ADR-0001) the agent has no API endpoint to dial and the CNI cannot come up")
|
|
if m is not None:
|
|
host_expr = m.group(1).strip()
|
|
check("k3s_apiserver_endpoint" in host_expr,
|
|
f"roles/cilium renders k8sServiceHost={host_expr!r}. It MUST be the HA endpoint "
|
|
"variable {{ k3s_apiserver_endpoint }} (= kube_vip_address). Any node-valued "
|
|
"expression — k3s_api_host, a primary hostvar, a literal IP — makes the CNI's "
|
|
"control-plane path depend on ONE server of an HA control plane, and the failure "
|
|
"is silent: the datapath keeps running from Cilium's cache while no service or "
|
|
"policy update ever lands (measured on prod 2026-09-06)")
|
|
check("k3s_api_host" not in host_expr,
|
|
f"roles/cilium renders k8sServiceHost={host_expr!r}, which references "
|
|
"k3s_api_host — the FOUNDER's own address. That variable exists only for the "
|
|
"cold-bootstrap `--server` join and one tls-san entry; it is a single point of "
|
|
"failure by construction and must never be a steady-state endpoint")
|
|
|
|
# The HA endpoint variable must BE the VIP, and must be DERIVED from kube_vip_address rather
|
|
# than restated: an environment overrides one line (kube_vip_address) and both the floating
|
|
# address and the CNI's endpoint move together. A literal here could disagree with the VIP the
|
|
# same environment actually floats, which is the two-places-one-fact bug this repo keeps
|
|
# hitting.
|
|
endpoint_def = str(ALL_VARS.get("k3s_apiserver_endpoint", ""))
|
|
check("kube_vip_address" in endpoint_def,
|
|
f"group_vars/all.yml defines k3s_apiserver_endpoint as {endpoint_def!r}. It MUST derive "
|
|
"from kube_vip_address, so that overriding the VIP for an environment also moves what "
|
|
"that environment's CNI dials. A restated literal can silently disagree with the VIP "
|
|
"kube-vip is actually floating")
|
|
check("kube_vip_address" not in str(ALL_VARS.get("k3s_api_host", "")),
|
|
"group_vars/all.yml's k3s_api_host resolves from kube_vip_address — the two API "
|
|
"addresses have been re-merged. k3s_api_host is the secondary join target and on a COLD "
|
|
"build no VIP exists yet, so a VIP-valued join target makes a cold bootstrap unjoinable")
|
|
|
|
# THE [parity] CONTRACT, now machine-checked instead of asserted in a comment: the Flux
|
|
# overlay's K3S_API_HOST becomes Cilium's k8sServiceHost under the HelmRelease, and the
|
|
# Ansible bootstrap renders k3s_apiserver_endpoint for the same field. If the two disagree,
|
|
# Flux and the bootstrap describe different control-plane paths and fight over the CNI.
|
|
for e in envs:
|
|
L = e.label
|
|
ov = overlay_vars(getattr(e, "overlay", ""))
|
|
k3s_api_host_knob = as_ip(ov.get("K3S_API_HOST", ""))
|
|
vip = as_ip(e.var("kube_vip_address"))
|
|
check(k3s_api_host_knob is not None,
|
|
f"{L}: overlay {getattr(e, 'overlay', '')!r} resolves no usable K3S_API_HOST — that "
|
|
"token becomes Cilium's k8sServiceHost via platform/cilium/helmrelease.yaml, and "
|
|
"kustomize-controller runs with StrictPostBuildSubstitutions so an unresolvable one "
|
|
"fails the whole platform Kustomization")
|
|
if k3s_api_host_knob is not None and vip is not None:
|
|
check(k3s_api_host_knob == vip,
|
|
f"{L}: [parity] BROKEN — the Flux overlay sets K3S_API_HOST={k3s_api_host_knob} "
|
|
f"but this environment's control-plane VIP (kube_vip_address, which is what "
|
|
f"k3s_apiserver_endpoint and therefore the Ansible-side k8sServiceHost resolve "
|
|
f"to) is {vip}. Flux and the Ansible bootstrap would render DIFFERENT API "
|
|
"endpoints into the same CNI and fight over it. If they differ because the "
|
|
"overlay still holds a NODE address, that is the 2026-09-06 defect: the CNI of "
|
|
"an HA control plane must not dial one server")
|
|
check(k3s_api_host_knob not in set(
|
|
as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
|
|
for h in hosts_in(str(e.var("k3s_servers_group")))),
|
|
f"{L}: the overlay's K3S_API_HOST {k3s_api_host_knob} is one of this "
|
|
"environment's own NODE addresses. Cilium's kube-proxy replacement would dial a "
|
|
"single machine on a control plane built specifically to survive losing one")
|
|
|
|
# The Flux half must stay a per-environment TOKEN. A literal IP in the shared HelmRelease
|
|
# would ship one environment's control-plane address to every environment (R-ENV-02), and an
|
|
# empty value is the 2026-09-01 incident verbatim (k8sServiceHost: "" made the agent's config
|
|
# init container fall back to the in-cluster ClusterIP 10.43.0.1:443, which only the agent
|
|
# being replaced could translate: deadlocked on 2 of 3 nodes).
|
|
hr = ROOT / "platform" / "cilium" / "helmrelease.yaml"
|
|
check(hr.is_file(), "platform/cilium/helmrelease.yaml is missing — nothing adopts the CNI "
|
|
"config after the bootstrap, so the parity contract has only one side")
|
|
if hr.is_file():
|
|
hr_body = "\n".join(l for l in hr.read_text(errors="replace").splitlines()
|
|
if not l.lstrip().startswith("#"))
|
|
hm = re.search(r"^\s*k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", hr_body, re.M)
|
|
check(hm is not None and hm.group(1).strip() == "${K3S_API_HOST}",
|
|
f"platform/cilium/helmrelease.yaml sets k8sServiceHost="
|
|
f"{(hm.group(1) if hm else None)!r} instead of the cluster-vars token "
|
|
"\"${K3S_API_HOST}\" — a literal there would ship ONE environment's control-plane "
|
|
"address to every environment (R-ENV-02), and an empty value is the 2026-09-01 "
|
|
"incident verbatim")
|
|
|
|
# ORDERING: the VIP must be floating BEFORE the CNI is installed against it. site.yml is the
|
|
# one-command bring-up, so this is a property of the authored sequence, not of an operator's
|
|
# memory. (kube-vip can precede the CNI because its DaemonSet is hostNetwork with
|
|
# KUBERNETES_SERVICE_HOST=127.0.0.1 — asserted separately below/above via the manifest parse.)
|
|
site_play = PLAYBOOKS / "site.yml"
|
|
check(site_play.is_file(), "playbooks/site.yml is missing — there is no one-command bring-up "
|
|
"whose ORDER can be checked")
|
|
if site_play.is_file():
|
|
imports = re.findall(r"import_playbook:\s*(\S+)", site_play.read_text())
|
|
for needed in ("k3s.yml", "kube-vip.yml", "cilium.yml"):
|
|
check(needed in imports,
|
|
f"playbooks/site.yml does not import {needed} (imports: {imports}). Cilium's "
|
|
"k8sServiceHost is the control-plane VIP, so kube-vip is a PREREQUISITE of the "
|
|
"CNI, not an optional extra step — leaving it out of the bring-up means a cold "
|
|
"build installs a CNI pointed at an address nothing announces")
|
|
if all(n in imports for n in ("k3s.yml", "kube-vip.yml", "cilium.yml")):
|
|
check(imports.index("k3s.yml") < imports.index("kube-vip.yml") < imports.index("cilium.yml"),
|
|
f"playbooks/site.yml imports {imports} — the order MUST be k3s.yml -> "
|
|
"kube-vip.yml -> cilium.yml. kube-vip after cilium installs the CNI against an "
|
|
"unannounced VIP (the 2026-09-01 shape: agents that cannot reach the API server "
|
|
"on a cold cluster); kube-vip before k3s cannot work at all, because the VIP "
|
|
"ships as a k3s auto-deploy manifest and needs a cluster to apply it")
|
|
check(re.search(r"wait_for", VIP_TEXT) is not None
|
|
and re.search(r"kube_vip_address", VIP_TEXT) is not None,
|
|
"roles/kube_vip never WAITS for the VIP to answer — templating the manifest is not the "
|
|
"same as having a floating endpoint (k3s still has to apply it, the image still has to "
|
|
"pull, leader election still has to settle). Since the CNI is installed immediately "
|
|
"afterward against this exact address, an unverified VIP turns a missing endpoint into "
|
|
"unreachable cilium-agents diagnosed from the far side")
|
|
|
|
# ============================================ FLOATING ENDPOINT: kube-vip on every server
|
|
# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
|
|
# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
|
|
# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the
|
|
# thing that ships.) Parser contract: the template writes each env var as an adjacent
|
|
# `- name: X` / `value: "Y"` pair; comment lines are dropped first.
|
|
VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2"))
|
|
check(len(VIP_TEMPLATES) >= 1,
|
|
"roles/kube_vip ships no manifest template — nothing deploys the API VIP")
|
|
vip_manifest = "\n".join(
|
|
"\n".join(l for l in p.read_text(errors="replace").splitlines()
|
|
if not l.lstrip().startswith("#"))
|
|
for p in VIP_TEMPLATES)
|
|
vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?",
|
|
vip_manifest))
|
|
check(len(vip_env) >= 6,
|
|
f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the "
|
|
"parser lost its grip on the file, so the assertions below would assert nothing")
|
|
check(vip_env.get("cp_enable") == "true",
|
|
f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane "
|
|
"mode must be ON or it floats no API VIP at all")
|
|
check(vip_env.get("svc_enable") == "false",
|
|
f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be "
|
|
"false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM "
|
|
"for the same addresses (ADR-0002 gives Service LB to Cilium)")
|
|
check(vip_env.get("vip_leaderelection") == "true",
|
|
f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} "
|
|
"— without leader election the VIP has no defined owner to fail over FROM, which is "
|
|
"the whole HA mechanism")
|
|
check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}",
|
|
f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the "
|
|
"environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on "
|
|
"prod (R-ENV-02) and would not match the serving-cert SAN list")
|
|
check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None,
|
|
"the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its "
|
|
"node, which is the opposite of a floating endpoint")
|
|
vip_play = PLAYBOOKS / "kube-vip.yml"
|
|
check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP")
|
|
if vip_play.is_file():
|
|
vp = vip_play.read_text()
|
|
vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M)
|
|
check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0],
|
|
f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers "
|
|
"group through the same environment-scoped variable the other plays use, so the "
|
|
"manifest exists on EVERY server and the VIP survives losing any one of them")
|
|
check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None
|
|
and re.search(r"k3s_data_dir", VIP_TEXT) is not None,
|
|
"roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies "
|
|
"<data-dir>/server/manifests, so an environment that moves --data-dir would write the "
|
|
"manifest somewhere k3s never reads and come up with no VIP at all")
|
|
|
|
# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS
|
|
# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
|
# step differs". A prod-only role or playbook would be a fork of the control-plane path.
|
|
role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()}
|
|
play_names = {p.name for p in PLAYBOOKS.glob("*.yml")}
|
|
env_words = re.compile(r"prod|dev", re.I)
|
|
forked = sorted(n for n in (role_names | play_names)
|
|
if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml"))
|
|
check(not forked,
|
|
f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by "
|
|
"the SAME code retargeted by variables (docs/02), not by a per-environment fork")
|
|
for e in envs:
|
|
if e.path.name != "vars.yml":
|
|
continue
|
|
# Every key a non-default environment sets must be a variable something already reads,
|
|
# or it is an invented knob nothing consumes (a silent no-op that looks like config).
|
|
consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*"))
|
|
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh"))
|
|
for k in sorted(e.vars):
|
|
if k.startswith("ansible_"):
|
|
continue
|
|
check(consumers.count(k) > 1,
|
|
f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an "
|
|
"environment override that nothing reads is not a scoped override, it is dead "
|
|
"configuration (and would make prod silently inherit a dev default)")
|
|
|
|
# --------------------------------------------------------------------------- verdict
|
|
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
|
check(CHECKS >= 40,
|
|
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
|
"would be meaningless")
|
|
|
|
if FAILURES:
|
|
die()
|
|
for n in NOTES:
|
|
print(f" note: {n}")
|
|
summary = "; ".join(
|
|
f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]="
|
|
f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers"
|
|
for e in envs)
|
|
print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
|
|
"R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
|
|
"embedded etcd on every server with no external datastore, a leader-elected "
|
|
"control-plane-only VIP on every server that is SAN'd into the serving cert and "
|
|
"collides with no other allocator, floated BEFORE the CNI that dials it (and the CNI "
|
|
"dials that VIP, not the founder, in both environments and on both sides of the "
|
|
"[parity] contract), from one code path for both environments. The LIVE "
|
|
"cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
|
|
"SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
|
|
PY
|