Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Browse Source 
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 00:59:22 -05:00
1 parent d1a6628a97
commit 6a8953e288
46 files changed
+7898 -71
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml
+88
View File
@@ -0,0 +1,88 @@
# =============================================================================
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
#
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
# cluster cannot tell you what the NEXT environment brought up from this repo would look
# like. The observed half is asserted separately at L2 by
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
#
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
#
# GIVEN the repository as git tracks it,
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
# environment's Flux overlay are parsed and cross-checked against each other,
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
# ONE --cluster-init founder and every other server joining it with --server;
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
# in the apiserver serving cert's SANs together with every server's own address, and
# it is neither a node address nor the ingress Gateway VIP nor inside that
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
# roles and plays retargeted by variables only.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
# Chainsaw Test wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
# infra/PROD-PROVISIONING.md).
#
# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
# * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
# It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
# The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
# contract and per environment.
# * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
# the order is asserted — not as a tidiness preference but because the CNI's endpoint is
# the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
# API server.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: ha-control-plane-as-code
labels:
conformance.home.k3s/R-PLAT-01: "true"
annotations:
conformance.home.k3s/requirements: "R-PLAT-01"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
environment's Flux overlay are parsed and cross-checked, then every environment is
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
VIP that is in the serving cert's SANs and collides with no other allocator — built by
one code path for both environments.
spec:
steps:
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
conformance/tests/l0-static/ha-control-plane-as-code/check.sh
Executable
+729
View File
@@ -0,0 +1,729 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — ha-control-plane-as-code
# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade
# HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the
# same multi-node topology." — spec docs/00-vision-and-scope.md).
#
# The elaborating spec text this check quotes and asserts against:
# docs/01-architecture-overview.md "Kubernetes distro | **k3s** (HA, embedded etcd)"
# docs/02-environments-and-provisioning.md, table:
# "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd
# (k3s server on all 3)" (the SAME cell for dev/test and prod)
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
# docs/02 §"k3s topology decisions":
# "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)."
# "Nodes are both control-plane and workload (3-node converged cluster); with only 3
# nodes we do not isolate control-plane nodes."
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
# step differs"
#
# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives)
# ---------------------------------------------------------------------
# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers:
#
# AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control
# plane in BOTH environments, from ONE code path": provable with no cluster, and
# ONLY provable here. A live cluster cannot tell you whether the *next* environment
# brought up from this repo will also be HA, whether a second `--cluster-init`
# founder could exist, or whether an external datastore is configured anywhere. That
# is this check.
# OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all
# etcd members, with the API answering through the floating VIP": that is cluster
# state and lives in tests/l2-behavioral/ha-control-plane (same requirement label).
#
# Neither half is sufficient alone, and this file claims only the first.
#
# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod"
# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the
# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every
# group_vars/<group>/vars.yml that overrides them — and identifies WHICH environment is
# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by
# its group name. Add a fourth prod node, rename a group, or add a third environment and
# the assertions follow; nothing here restates an address, a hostname or a node count.
#
# GIVEN the repository as git tracks it,
# WHEN the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's
# Flux overlay are parsed and cross-checked against each other,
# THEN DISTRO: k3s is the distro, pinned, installed by the one role, and no node is
# installed as an `agent` (an agent is not a control-plane member);
# SERVERS: every environment's server group resolves to a quorum-capable, ODD
# number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to
# EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while
# every other server joins it with `--server` (so a second founder — the
# split-brain shape — cannot be authored);
# ETCD: the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no
# `--disable-etcd` anywhere in the provisioning code, so every server is
# an etcd member (docs/02's "no external datastore", "on all 3");
# ENDPOINT: the API has a floating endpoint that survives losing the founder — the
# kube-vip VIP is control-plane-only (never Service LB, or it would fight
# Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and
# baked into the apiserver serving cert's SAN list ALONG WITH every
# server's own address (so TLS validates whichever node answers);
# ADDRESSES: each environment's control-plane VIP is not one of its node addresses,
# is not that environment's ingress Gateway VIP, and lies OUTSIDE that
# environment's Cilium LoadBalancer pool — the two allocators can never be
# handed the same address;
# CNI PATH: the CNI dials the FLOATING endpoint, not a machine (added 2026-09-06). The
# two API addresses stay SPLIT — k3s_api_host is the founder and feeds only
# the cold `--server` join, k3s_apiserver_endpoint derives from
# kube_vip_address and feeds Cilium's k8sServiceHost — the Flux overlay's
# K3S_API_HOST equals that environment's VIP and is none of its node
# addresses ([parity], both halves), the HelmRelease keeps a per-environment
# token rather than a literal, and site.yml floats the VIP BETWEEN k3s and
# Cilium with roles/kube_vip blocking until it answers;
# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
# retargeted by variables only (docs/02 "the *same* Ansible bootstrap
# runs"), so "exercise the same multi-node topology" is a property of the
# code rather than a coincidence of two hand-written inventories.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
# - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with
# a healthy quorum, that kube-vip is actually floating the VIP, and that the API
# answers through it. All of that is asserted at L2 by
# tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test.
# - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds,
# the VIP moves, the API stays up) is a fault-injection claim. The nearest existing
# coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's
# data-availability claim and does NOT assert control-plane survival. A dedicated
# "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and
# is NOT claimed here or at L2.
# - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged
# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
# - IPv6 / dual-stack, and any claim about the edge network.
#
# TWO FORMER GAPS ARE NOW ASSERTED, and the reasoning is worth keeping (2026-09-06):
# * CILIUM'S OWN API ENDPOINT used to be listed here as an ESCALATION — `K3S_API_HOST`
# (clusters/*/kustomization.yaml), which becomes Cilium's `k8sServiceHost`, was each
# environment's PRIMARY NODE address rather than the floating VIP, contradicting the prod
# overlay's own comment. Making the check fail over it would have turned the merge gate
# red on a design question this workstream may not decide (R-TEST-06), so it was reported
# instead. THE OPERATOR DECIDED THE VIP on 2026-09-06, which turns the question into a
# repository invariant, and invariants belong in assertions — see the "THE CNI DIALS THE
# VIP" section. Note what made it assertable: the fix SPLIT one variable in two, so there
# is finally an HA-valued name (`k3s_apiserver_endpoint`) to point the contract at.
# * kube-vip WAS NOT PART OF THE ONE-COMMAND BRING-UP (site.yml chained node-baseline ->
# k3s -> cilium -> verify -> flux-bootstrap), which was recorded here as an observation
# and not asserted, because R-PLAT-01 does not fix the bring-up order. It is asserted now
# for a different reason that IS in scope: with the CNI's endpoint being the VIP, an
# unfloated VIP means a cold build installs a CNI that cannot reach its API server. The
# ordering is no longer a preference about tidiness, it is a dependency of the control
# plane coming up at all.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
def die():
print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
ANS = ROOT / "infra" / "ansible"
INV = ANS / "inventory" / "hosts.ini"
GV = ANS / "inventory" / "group_vars"
K3S_ROLE = ANS / "roles" / "k3s_server"
VIP_ROLE = ANS / "roles" / "kube_vip"
PLAYBOOKS = ANS / "playbooks"
CLUSTERS = ROOT / "clusters"
for p, why in ((INV, "the inventory that defines the node set"),
(K3S_ROLE, "the role that installs the k3s control plane"),
(VIP_ROLE, "the role that floats the control-plane VIP"),
(CLUSTERS, "the Flux overlays that identify each environment")):
check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA "
"control plane is authored at all (R-PLAT-01)")
if FAILURES:
die()
K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml")))
VIP_TEXT = "\n".join(p.read_text(errors="replace")
for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2"))))
# ===================================================================== inventory parse
# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a
# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]`
# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'.
ini_hosts: dict[str, list[str]] = {}
ini_children: dict[str, list[str]] = {}
host_vars: dict[str, dict[str, str]] = {}
cur = None
for line in INV.read_text().splitlines():
s = line.strip()
if not s or s.startswith("#"):
continue
m = re.match(r"^\[([^\]]+)\]$", s)
if m:
cur = m.group(1)
continue
if cur is None or cur.endswith(":vars"):
continue
host = s.split()[0]
if cur.endswith(":children"):
ini_children.setdefault(cur.split(":")[0], []).append(host)
else:
ini_hosts.setdefault(cur, []).append(host)
hv = host_vars.setdefault(host, {})
for kv in s.split()[1:]:
if "=" in kv:
k, v = kv.split("=", 1)
hv[k] = v
def hosts_in(group: str, seen=None) -> set[str]:
seen = seen or set()
if group in seen:
return set()
seen.add(group)
out = set(ini_hosts.get(group, []))
for child in ini_children.get(group, []):
out |= hosts_in(child, seen)
return out
def load_yaml(p: Path) -> dict:
try:
d = yaml.safe_load(p.read_text()) or {}
except yaml.YAMLError:
return {}
return d if isinstance(d, dict) else {}
ALL_VARS = load_yaml(GV / "all.yml")
# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM
# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds
# the default pair; every group_vars/<g>/vars.yml that overrides `k3s_primary_group` is
# another environment. Nothing below knows the words "dev" or "prod" as group names.
class Env:
def __init__(self, label, vars_, path):
self.label = label
self.vars = vars_
self.path = path
def var(self, name, default=None):
# group-scoped value wins, else the cluster-wide default from all.yml.
if name in self.vars:
return self.vars[name]
return ALL_VARS.get(name, default)
envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")]
for vf in sorted(GV.glob("*/vars.yml")):
v = load_yaml(vf)
if "k3s_primary_group" in v or "k3s_servers_group" in v:
envs.append(Env(str(vf.relative_to(ROOT)), v, vf))
check(len(envs) >= 2,
"only ONE environment is derivable from the inventory's R-ENV-02 seam "
"(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires "
"dev/test to exercise the same multi-node topology, so two must be authored")
# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay.
# roles/flux_bootstrap's own default supplies the value for an environment that does not
# override it, so this is read from code, never guessed from a group name.
fb_default = None
fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml"
if fb_defaults_file.is_file():
fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path")
check(isinstance(fb_default, str) and fb_default.startswith("clusters/"),
"roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from "
"an inventory environment to its Flux overlay is not expressed in code, so this check "
"cannot tell which environment is prod")
prod_envs = []
for e in envs:
overlay = e.var("flux_bootstrap_env_path", fb_default)
e.overlay = overlay if isinstance(overlay, str) else ""
if e.overlay.rstrip("/").endswith("/prod"):
prod_envs.append(e)
check(len(prod_envs) == 1,
f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay "
f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in "
"R-PLAT-01 has no unambiguous referent")
# ============================================== overlay knob values (per environment)
# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map;
# each clusters/<env>/kustomization.yaml overrides values with JSON6902 patches written as
# - op: replace
# path: /data/<KEY>
# value: "<v>"
# (verified shape across both overlays). A key not patched keeps the base value.
BASE_VARS: dict[str, str] = {}
base_cm = CLUSTERS / "base" / "cluster-vars.yaml"
if base_cm.is_file():
d = load_yaml(base_cm)
BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()}
check(len(BASE_VARS) > 5,
"clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface "
"cannot be read, so the VIP/pool separation cannot be checked")
_PATCH_RE = re.compile(
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n\s*value:\s*(?P<v>\S.*?)\s*$",
re.M)
def overlay_vars(overlay: str) -> dict[str, str]:
out = dict(BASE_VARS)
f = ROOT / overlay / "kustomization.yaml"
if not f.is_file():
return out
for m in _PATCH_RE.finditer(f.read_text()):
out[m.group("k")] = m.group("v").strip().strip('"').strip("'")
return out
# ============================================================ PER-ENVIRONMENT TOPOLOGY
def as_ip(s):
try:
return ipaddress.ip_address(str(s).strip())
except ValueError:
return None
for e in envs:
L = e.label
pg = e.var("k3s_primary_group")
sg = e.var("k3s_servers_group")
check(isinstance(pg, str) and isinstance(sg, str) and pg and sg,
f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot "
"resolve 'my environment's primary/servers' and would only ever build dev")
if not (isinstance(pg, str) and isinstance(sg, str)):
continue
primaries = hosts_in(pg)
servers = hosts_in(sg)
# --- SERVERS: quorum-capable, one founder, prod exactly three ---------------------
check(len(primaries) == 1,
f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run "
"`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no "
"cluster at all")
check(primaries and primaries <= servers,
f"{L}: the primary {sorted(primaries)} is not a member of the servers group "
f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or "
"it is not part of the HA control plane it founds")
n = len(servers)
check(n >= 3,
f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes "
"'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd "
"cluster of fewer than 3 members tolerates ZERO losses — it is not HA")
check(n % 2 == 1,
f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd "
f"member count adds a member without adding fault tolerance and introduces a tie: "
f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}")
if n:
tolerance = n - (n // 2 + 1)
check(tolerance >= 1,
f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate "
f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one")
if e in prod_envs:
check(n == 3,
f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: "
"'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'")
else:
check(n >= 2,
f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer "
"nodes than prod but requires dev/test to 'exercise the same multi-node "
"topology' — one node exercises no topology")
# --- ADDRESSES: the VIP can collide with nothing ----------------------------------
vip_raw = e.var("kube_vip_address")
vip = as_ip(vip_raw)
check(vip is not None,
f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has "
"no floating endpoint, so losing the founder takes the API endpoint with it")
node_ips = {}
for h in sorted(servers):
ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
check(ip is not None,
f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join "
"target and the serving-cert SAN list all derive from it")
if ip is not None:
node_ips[h] = ip
if vip is not None:
check(vip not in set(node_ips.values()),
f"{L}: kube_vip_address {vip} is ALSO a node address "
f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a "
"node's own address cannot float off that node")
ov = overlay_vars(getattr(e, "overlay", ""))
gw = as_ip(ov.get("GATEWAY_VIP", ""))
lo = as_ip(ov.get("CILIUM_LB_POOL_START", ""))
hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", ""))
check(gw is not None and lo is not None and hi is not None,
f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + "
"CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and "
"the Service-LB allocator cannot be checked")
if gw is not None:
check(vip != gw,
f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME "
f"address ({vip}) — kube-vip and Cilium would both ARP for it")
if lo is not None and hi is not None:
check(not (lo <= vip <= hi),
f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium "
f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a "
"Service, and the two allocators would fight over ARP for it")
check(lo <= hi,
f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be "
"used to prove the VIP is outside it")
# ==================================================================== DISTRO: k3s only
check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M)
is not None,
"roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane "
"version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)")
check("get.k3s.io" in K3S_TEXT,
"roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the "
"distro (ADR-0001) and no other distro may stand in for it")
check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None,
"roles/k3s_server never composes the `server` subcommand — without it k3s installs an "
"AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')")
for bad, why in (
(r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 "
"nodes (no external datastore)')"),
(r"datastore-endpoint\s*:", "an external datastore via config.yaml"),
(r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not "
"an etcd member (docs/02: etcd 'across all 3 nodes')"),
(r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"),
):
hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*"))
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini")
and re.search(bad, p.read_text(errors="replace"))]
check(not hits,
f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA "
f"mode, and {bad!r} contradicts it")
# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH
check(re.search(r"--cluster-init", K3S_TEXT) is not None,
"roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never "
"founded, so there is no HA datastore to join")
init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'",
K3S_TEXT)
check(init_when is not None,
"the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an "
"ungated founder flag would re-found the cluster on every server (split brain)")
join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
check(join is not None,
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
"— without a join target the servers are separate single-node clusters")
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
K3S_TEXT, re.S)
check(join_when is not None,
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
"would also try to join itself")
k3s_play = PLAYBOOKS / "k3s.yml"
check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint")
if k3s_play.is_file():
play_txt = k3s_play.read_text()
modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt)
check(sorted(modes) == ["primary", "secondary"],
f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and "
"one secondary play must exist, or the founder/joiner split is not authored")
check(re.search(r"serial:\s*1", play_txt) is not None,
"playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — "
"converging two of three servers at once drops embedded etcd below quorum, which "
"is exactly the outage an HA control plane exists to avoid")
hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M)
check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns),
f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal "
"group name means this play can only ever build ONE environment, so prod cannot "
"be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')")
# =========================================== SERVING-CERT SANs: VIP + every server
sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", ""))
check("kube_vip_address" in sans,
"roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's "
"serving cert would carry no SAN for the floating VIP, so every client reaching the "
"API through it fails TLS validation and the HA endpoint is unusable")
check("k3s_servers_group" in sans and "k3s_node_ip" in sans,
"k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the "
"cert must be valid for WHICHEVER server answers behind the VIP, and per-environment "
"resolution of that set is what keeps one role serving both environments (R-ENV-02)")
check(re.search(r"tls-san:", K3S_TEXT) is not None,
"the SAN list is passed only as an install flag and never persisted to "
"/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
"no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")
# ============================ THE CNI DIALS THE VIP, NOT THE FOUNDER (added 2026-09-06)
# This section used to be an ESCALATION in the header's gap list, not an assertion: Cilium's
# k8sServiceHost was each environment's PRIMARY NODE address, so on a 3-server HA control
# plane the CNI's control-plane path depended on one machine. Measured on live prod
# 2026-09-06: all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10
# (k3s-prod-1's own address) while kube-vip held 192.168.2.9. That failure is SILENT — the
# datapath keeps serving from Cilium's cache, so nothing looks broken while no service or
# policy update lands at all.
#
# The operator decided the VIP (2026-09-06), so this is now a repository property and belongs
# here rather than in the gap list. THIS IS THE RIGHT LAYER because the claim is about what
# the configuration in git WOULD build, in BOTH environments, including the next one — a live
# cluster can only tell you about itself.
#
# WHAT MAKES THIS CHECKABLE AT ALL is that the fix split one variable into two. Asserting
# "Cilium uses the HA address" requires an HA address to name; before the split there was
# only k3s_api_host, which had to be the founder for the cold join. So the assertions below
# check the SPLIT as much as the value: the join keeps the founder, the CNI gets the VIP, and
# the Flux half agrees with the Ansible half per environment.
CIL_ROLE = ANS / "roles" / "cilium"
cil_values_tpl = CIL_ROLE / "templates" / "cilium-values.yaml.j2"
check(cil_values_tpl.is_file(),
"roles/cilium ships no cilium-values.yaml.j2 — the bootstrap's CNI values, including the "
"API endpoint its kube-proxy replacement dials, are not authored")
if cil_values_tpl.is_file():
# Parse the rendered-shape template, never its prose: strip comment lines first, exactly
# as the kube-vip parser above does, so a reassuring comment cannot satisfy the check.
cil_body = "\n".join(l for l in cil_values_tpl.read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#"))
m = re.search(r"^k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", cil_body, re.M)
check(m is not None,
"roles/cilium's values template sets no k8sServiceHost — with kube-proxy disabled "
"(ADR-0001) the agent has no API endpoint to dial and the CNI cannot come up")
if m is not None:
host_expr = m.group(1).strip()
check("k3s_apiserver_endpoint" in host_expr,
f"roles/cilium renders k8sServiceHost={host_expr!r}. It MUST be the HA endpoint "
"variable {{ k3s_apiserver_endpoint }} (= kube_vip_address). Any node-valued "
"expression — k3s_api_host, a primary hostvar, a literal IP — makes the CNI's "
"control-plane path depend on ONE server of an HA control plane, and the failure "
"is silent: the datapath keeps running from Cilium's cache while no service or "
"policy update ever lands (measured on prod 2026-09-06)")
check("k3s_api_host" not in host_expr,
f"roles/cilium renders k8sServiceHost={host_expr!r}, which references "
"k3s_api_host — the FOUNDER's own address. That variable exists only for the "
"cold-bootstrap `--server` join and one tls-san entry; it is a single point of "
"failure by construction and must never be a steady-state endpoint")
# The HA endpoint variable must BE the VIP, and must be DERIVED from kube_vip_address rather
# than restated: an environment overrides one line (kube_vip_address) and both the floating
# address and the CNI's endpoint move together. A literal here could disagree with the VIP the
# same environment actually floats, which is the two-places-one-fact bug this repo keeps
# hitting.
endpoint_def = str(ALL_VARS.get("k3s_apiserver_endpoint", ""))
check("kube_vip_address" in endpoint_def,
f"group_vars/all.yml defines k3s_apiserver_endpoint as {endpoint_def!r}. It MUST derive "
"from kube_vip_address, so that overriding the VIP for an environment also moves what "
"that environment's CNI dials. A restated literal can silently disagree with the VIP "
"kube-vip is actually floating")
check("kube_vip_address" not in str(ALL_VARS.get("k3s_api_host", "")),
"group_vars/all.yml's k3s_api_host resolves from kube_vip_address — the two API "
"addresses have been re-merged. k3s_api_host is the secondary join target and on a COLD "
"build no VIP exists yet, so a VIP-valued join target makes a cold bootstrap unjoinable")
# THE [parity] CONTRACT, now machine-checked instead of asserted in a comment: the Flux
# overlay's K3S_API_HOST becomes Cilium's k8sServiceHost under the HelmRelease, and the
# Ansible bootstrap renders k3s_apiserver_endpoint for the same field. If the two disagree,
# Flux and the bootstrap describe different control-plane paths and fight over the CNI.
for e in envs:
L = e.label
ov = overlay_vars(getattr(e, "overlay", ""))
k3s_api_host_knob = as_ip(ov.get("K3S_API_HOST", ""))
vip = as_ip(e.var("kube_vip_address"))
check(k3s_api_host_knob is not None,
f"{L}: overlay {getattr(e, 'overlay', '')!r} resolves no usable K3S_API_HOST — that "
"token becomes Cilium's k8sServiceHost via platform/cilium/helmrelease.yaml, and "
"kustomize-controller runs with StrictPostBuildSubstitutions so an unresolvable one "
"fails the whole platform Kustomization")
if k3s_api_host_knob is not None and vip is not None:
check(k3s_api_host_knob == vip,
f"{L}: [parity] BROKEN — the Flux overlay sets K3S_API_HOST={k3s_api_host_knob} "
f"but this environment's control-plane VIP (kube_vip_address, which is what "
f"k3s_apiserver_endpoint and therefore the Ansible-side k8sServiceHost resolve "
f"to) is {vip}. Flux and the Ansible bootstrap would render DIFFERENT API "
"endpoints into the same CNI and fight over it. If they differ because the "
"overlay still holds a NODE address, that is the 2026-09-06 defect: the CNI of "
"an HA control plane must not dial one server")
check(k3s_api_host_knob not in set(
as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
for h in hosts_in(str(e.var("k3s_servers_group")))),
f"{L}: the overlay's K3S_API_HOST {k3s_api_host_knob} is one of this "
"environment's own NODE addresses. Cilium's kube-proxy replacement would dial a "
"single machine on a control plane built specifically to survive losing one")
# The Flux half must stay a per-environment TOKEN. A literal IP in the shared HelmRelease
# would ship one environment's control-plane address to every environment (R-ENV-02), and an
# empty value is the 2026-09-01 incident verbatim (k8sServiceHost: "" made the agent's config
# init container fall back to the in-cluster ClusterIP 10.43.0.1:443, which only the agent
# being replaced could translate: deadlocked on 2 of 3 nodes).
hr = ROOT / "platform" / "cilium" / "helmrelease.yaml"
check(hr.is_file(), "platform/cilium/helmrelease.yaml is missing — nothing adopts the CNI "
"config after the bootstrap, so the parity contract has only one side")
if hr.is_file():
hr_body = "\n".join(l for l in hr.read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#"))
hm = re.search(r"^\s*k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", hr_body, re.M)
check(hm is not None and hm.group(1).strip() == "${K3S_API_HOST}",
f"platform/cilium/helmrelease.yaml sets k8sServiceHost="
f"{(hm.group(1) if hm else None)!r} instead of the cluster-vars token "
"\"${K3S_API_HOST}\" — a literal there would ship ONE environment's control-plane "
"address to every environment (R-ENV-02), and an empty value is the 2026-09-01 "
"incident verbatim")
# ORDERING: the VIP must be floating BEFORE the CNI is installed against it. site.yml is the
# one-command bring-up, so this is a property of the authored sequence, not of an operator's
# memory. (kube-vip can precede the CNI because its DaemonSet is hostNetwork with
# KUBERNETES_SERVICE_HOST=127.0.0.1 — asserted separately below/above via the manifest parse.)
site_play = PLAYBOOKS / "site.yml"
check(site_play.is_file(), "playbooks/site.yml is missing — there is no one-command bring-up "
"whose ORDER can be checked")
if site_play.is_file():
imports = re.findall(r"import_playbook:\s*(\S+)", site_play.read_text())
for needed in ("k3s.yml", "kube-vip.yml", "cilium.yml"):
check(needed in imports,
f"playbooks/site.yml does not import {needed} (imports: {imports}). Cilium's "
"k8sServiceHost is the control-plane VIP, so kube-vip is a PREREQUISITE of the "
"CNI, not an optional extra step — leaving it out of the bring-up means a cold "
"build installs a CNI pointed at an address nothing announces")
if all(n in imports for n in ("k3s.yml", "kube-vip.yml", "cilium.yml")):
check(imports.index("k3s.yml") < imports.index("kube-vip.yml") < imports.index("cilium.yml"),
f"playbooks/site.yml imports {imports} — the order MUST be k3s.yml -> "
"kube-vip.yml -> cilium.yml. kube-vip after cilium installs the CNI against an "
"unannounced VIP (the 2026-09-01 shape: agents that cannot reach the API server "
"on a cold cluster); kube-vip before k3s cannot work at all, because the VIP "
"ships as a k3s auto-deploy manifest and needs a cluster to apply it")
check(re.search(r"wait_for", VIP_TEXT) is not None
and re.search(r"kube_vip_address", VIP_TEXT) is not None,
"roles/kube_vip never WAITS for the VIP to answer — templating the manifest is not the "
"same as having a floating endpoint (k3s still has to apply it, the image still has to "
"pull, leader election still has to settle). Since the CNI is installed immediately "
"afterward against this exact address, an unverified VIP turns a missing endpoint into "
"unreachable cilium-agents diagnosed from the far side")
# ============================================ FLOATING ENDPOINT: kube-vip on every server
# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the
# thing that ships.) Parser contract: the template writes each env var as an adjacent
# `- name: X` / `value: "Y"` pair; comment lines are dropped first.
VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2"))
check(len(VIP_TEMPLATES) >= 1,
"roles/kube_vip ships no manifest template — nothing deploys the API VIP")
vip_manifest = "\n".join(
"\n".join(l for l in p.read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#"))
for p in VIP_TEMPLATES)
vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?",
vip_manifest))
check(len(vip_env) >= 6,
f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the "
"parser lost its grip on the file, so the assertions below would assert nothing")
check(vip_env.get("cp_enable") == "true",
f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane "
"mode must be ON or it floats no API VIP at all")
check(vip_env.get("svc_enable") == "false",
f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be "
"false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM "
"for the same addresses (ADR-0002 gives Service LB to Cilium)")
check(vip_env.get("vip_leaderelection") == "true",
f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} "
"— without leader election the VIP has no defined owner to fail over FROM, which is "
"the whole HA mechanism")
check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}",
f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the "
"environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on "
"prod (R-ENV-02) and would not match the serving-cert SAN list")
check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None,
"the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its "
"node, which is the opposite of a floating endpoint")
vip_play = PLAYBOOKS / "kube-vip.yml"
check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP")
if vip_play.is_file():
vp = vip_play.read_text()
vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M)
check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0],
f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers "
"group through the same environment-scoped variable the other plays use, so the "
"manifest exists on EVERY server and the VIP survives losing any one of them")
check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None
and re.search(r"k3s_data_dir", VIP_TEXT) is not None,
"roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies "
"<data-dir>/server/manifests, so an environment that moves --data-dir would write the "
"manifest somewhere k3s never reads and come up with no VIP at all")
# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS
# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
# step differs". A prod-only role or playbook would be a fork of the control-plane path.
role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()}
play_names = {p.name for p in PLAYBOOKS.glob("*.yml")}
env_words = re.compile(r"prod|dev", re.I)
forked = sorted(n for n in (role_names | play_names)
if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml"))
check(not forked,
f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by "
"the SAME code retargeted by variables (docs/02), not by a per-environment fork")
for e in envs:
if e.path.name != "vars.yml":
continue
# Every key a non-default environment sets must be a variable something already reads,
# or it is an invented knob nothing consumes (a silent no-op that looks like config).
consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*"))
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh"))
for k in sorted(e.vars):
if k.startswith("ansible_"):
continue
check(consumers.count(k) > 1,
f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an "
"environment override that nothing reads is not a scoped override, it is dead "
"configuration (and would make prod silently inherit a dev default)")
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 40,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
die()
for n in NOTES:
print(f" note: {n}")
summary = "; ".join(
f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]="
f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers"
for e in envs)
print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
"R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
"embedded etcd on every server with no external datastore, a leader-elected "
"control-plane-only VIP on every server that is SAN'd into the serving cert and "
"collides with no other allocator, floated BEFORE the CNI that dials it (and the CNI "
"dials that VIP, not the founder, in both environments and on both sides of the "
"[parity] contract), from one code path for both environments. The LIVE "
"cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
"SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
PY