feat(conformance): cover R-PLAT-01, R-NET-04, R-SEC-03 — 88.2% -> 94.1%, and fix an unrecordable-proof defect
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0 safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS. R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control plane" is half authored and half observable. l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init founder per environment; odd, quorum-capable server count with tolerance >= 1; tls-sans carry the VIP and every server IP; kube-vip leader-elected with cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside its environment's LB pool and not a node IP. l2-behavioral/ha-control-plane: control-plane AND etcd role on every node, every node a REGISTERED etcd member, exactly 3 when the cluster says CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every server's own address — which proves the VIP fronts the API and that every server runs an apiserver, not one founder plus two joiners. An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's LAN and fails against dev (nodes behind the pfSense NAT), making the test a property of where it was launched. Hence the in-cluster Job. R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the totality/disjointness of the selecting rules, not another per-service check. l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the same closed set in schema + Kyverno policy + renderer; EXACTLY ONE CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn; nothing selects `internet` (absence of a rule IS the path); RFC1918 containment computed with `ipaddress` plus every address each overlay assigns itself; the tunnel default route and blackhole confined to table 51820 behind an ip rule that must carry a from/fwmark/iif selector; and the nft chain `policy accept` with every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test does not assert. R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable". l0-static/auditability-proofs-producible (78 assertions) drives the real machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer the spec's wording demands (fault injection and DR drill => L3), taken from status.py's own verdict; run.sh --requirements must resolve and pass the gate's --require assertion; each proof declares a sentinel record_run.py can attribute, computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted against the real tools — a green JUnit whose log holds only a SKIP token, and a ledger row claiming `proven` at the root commit — which must come back not-proven and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted, not hand-listed. *** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel, so record_run.py could never attribute a pass to them — the isolation and TLS audit proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared, and status.py requires all of a requirement's tests to be proven). Each now ends with an additive terminal sentinel step reached only when every prior step passed. Verified: all three carry it. TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role defaults, so flipping the template to true still passed — now parses the DaemonSet template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja variable while the template carries its NAME, matching nothing and passing vacuously — templates are now rendered against role defaults first. 30 perturbations across the three checks, all caught, plus 16 synthetic L2 perturbations via a fake-kubectl shim. R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's widgets.yaml carries only datetime and search, kubernetes.yaml is empty, automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress only — no health source is even reachable. It has already drifted (crafty authors a route with no tile). A test today would go red on both; it must ship WITH the config fix. VERIFIED: full CI gate PASS. Neither cluster mutated. Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
5ec1460532
commit
cb1916f092
12 files changed
+2488
No files matched your search
@@ -0,0 +1,83 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
||||
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
||||
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
|
||||
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
|
||||
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
|
||||
#
|
||||
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
|
||||
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
|
||||
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
|
||||
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
|
||||
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
|
||||
# cluster cannot tell you what the NEXT environment brought up from this repo would look
|
||||
# like. The observed half is asserted separately at L2 by
|
||||
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
|
||||
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
|
||||
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
|
||||
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
|
||||
# environment's Flux overlay are parsed and cross-checked against each other,
|
||||
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
|
||||
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
|
||||
# ONE --cluster-init founder and every other server joining it with --server;
|
||||
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
|
||||
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
|
||||
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
|
||||
# in the apiserver serving cert's SANs together with every server's own address, and
|
||||
# it is neither a node address nor the ingress Gateway VIP nor inside that
|
||||
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
|
||||
# roles and plays retargeted by variables only.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
||||
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
|
||||
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
|
||||
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
|
||||
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
|
||||
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
|
||||
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
|
||||
# infra/PROD-PROVISIONING.md).
|
||||
# * kube-vip is NOT imported by playbooks/site.yml, so the floating endpoint is a separate
|
||||
# operator step. Observed and reported, NOT asserted — R-PLAT-01 does not fix the
|
||||
# bring-up order (R-TEST-06).
|
||||
# * `K3S_API_HOST` (Cilium's k8sServiceHost) is the PRIMARY NODE's address in both
|
||||
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
|
||||
# A real single-node dependency, ESCALATED to the PM rather than asserted.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: ha-control-plane-as-code
|
||||
labels:
|
||||
conformance.home.k3s/R-PLAT-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-PLAT-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
|
||||
environment's Flux overlay are parsed and cross-checked, then every environment is
|
||||
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
|
||||
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
|
||||
VIP that is in the serving cert's SANs and collides with no other allocator — built by
|
||||
one code path for both environments.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,571 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — ha-control-plane-as-code
|
||||
# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade
|
||||
# HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the
|
||||
# same multi-node topology." — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The elaborating spec text this check quotes and asserts against:
|
||||
# docs/01-architecture-overview.md "Kubernetes distro | **k3s** (HA, embedded etcd)"
|
||||
# docs/02-environments-and-provisioning.md, table:
|
||||
# "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd
|
||||
# (k3s server on all 3)" (the SAME cell for dev/test and prod)
|
||||
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
|
||||
# docs/02 §"k3s topology decisions":
|
||||
# "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)."
|
||||
# "Nodes are both control-plane and workload (3-node converged cluster); with only 3
|
||||
# nodes we do not isolate control-plane nodes."
|
||||
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||||
# step differs"
|
||||
#
|
||||
# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives)
|
||||
# ---------------------------------------------------------------------
|
||||
# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers:
|
||||
#
|
||||
# AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control
|
||||
# plane in BOTH environments, from ONE code path": provable with no cluster, and
|
||||
# ONLY provable here. A live cluster cannot tell you whether the *next* environment
|
||||
# brought up from this repo will also be HA, whether a second `--cluster-init`
|
||||
# founder could exist, or whether an external datastore is configured anywhere. That
|
||||
# is this check.
|
||||
# OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all
|
||||
# etcd members, with the API answering through the floating VIP": that is cluster
|
||||
# state and lives in tests/l2-behavioral/ha-control-plane (same requirement label).
|
||||
#
|
||||
# Neither half is sufficient alone, and this file claims only the first.
|
||||
#
|
||||
# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod"
|
||||
# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the
|
||||
# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every
|
||||
# group_vars/<group>/vars.yml that overrides them — and identifies WHICH environment is
|
||||
# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by
|
||||
# its group name. Add a fourth prod node, rename a group, or add a third environment and
|
||||
# the assertions follow; nothing here restates an address, a hostname or a node count.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's
|
||||
# Flux overlay are parsed and cross-checked against each other,
|
||||
# THEN DISTRO: k3s is the distro, pinned, installed by the one role, and no node is
|
||||
# installed as an `agent` (an agent is not a control-plane member);
|
||||
# SERVERS: every environment's server group resolves to a quorum-capable, ODD
|
||||
# number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to
|
||||
# EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while
|
||||
# every other server joins it with `--server` (so a second founder — the
|
||||
# split-brain shape — cannot be authored);
|
||||
# ETCD: the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no
|
||||
# `--disable-etcd` anywhere in the provisioning code, so every server is
|
||||
# an etcd member (docs/02's "no external datastore", "on all 3");
|
||||
# ENDPOINT: the API has a floating endpoint that survives losing the founder — the
|
||||
# kube-vip VIP is control-plane-only (never Service LB, or it would fight
|
||||
# Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and
|
||||
# baked into the apiserver serving cert's SAN list ALONG WITH every
|
||||
# server's own address (so TLS validates whichever node answers);
|
||||
# ADDRESSES: each environment's control-plane VIP is not one of its node addresses,
|
||||
# is not that environment's ingress Gateway VIP, and lies OUTSIDE that
|
||||
# environment's Cilium LoadBalancer pool — the two allocators can never be
|
||||
# handed the same address;
|
||||
# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
|
||||
# retargeted by variables only (docs/02 "the *same* Ansible bootstrap
|
||||
# runs"), so "exercise the same multi-node topology" is a property of the
|
||||
# code rather than a coincidence of two hand-written inventories.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with
|
||||
# a healthy quorum, that kube-vip is actually floating the VIP, and that the API
|
||||
# answers through it. All of that is asserted at L2 by
|
||||
# tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test.
|
||||
# - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds,
|
||||
# the VIP moves, the API stays up) is a fault-injection claim. The nearest existing
|
||||
# coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's
|
||||
# data-availability claim and does NOT assert control-plane survival. A dedicated
|
||||
# "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and
|
||||
# is NOT claimed here or at L2.
|
||||
# - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged
|
||||
# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
|
||||
# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
|
||||
# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
|
||||
# - THAT kube-vip IS PART OF THE ONE-COMMAND BRING-UP. playbooks/site.yml chains
|
||||
# node-baseline -> k3s -> cilium -> verify -> flux-bootstrap and does NOT import
|
||||
# kube-vip.yml, so the floating endpoint is a separate operator step. That is recorded
|
||||
# here as an observation for the PM, NOT asserted: R-PLAT-01 requires an HA control
|
||||
# plane, and does not name kube-vip or fix the bring-up order (R-TEST-06 — do not
|
||||
# invent a requirement's meaning).
|
||||
# - CILIUM'S OWN API ENDPOINT. `K3S_API_HOST` (clusters/*/kustomization.yaml), which
|
||||
# becomes Cilium's `k8sServiceHost` for kube-proxy replacement, is the PRIMARY NODE's
|
||||
# address in both environments — not the floating VIP — even though the prod overlay's
|
||||
# own comment says it "is the prod CONTROL-PLANE VIP (kube-vip), NOT ... the single node
|
||||
# address .10". That is a real single-node dependency in the CNI's control-plane path
|
||||
# and it is ESCALATED to the PM rather than asserted: making this check fail would turn
|
||||
# the merge gate red over a design question this workstream may not decide (R-TEST-06).
|
||||
# - IPv6 / dual-stack, and any claim about the edge network.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
ANS = ROOT / "infra" / "ansible"
|
||||
INV = ANS / "inventory" / "hosts.ini"
|
||||
GV = ANS / "inventory" / "group_vars"
|
||||
K3S_ROLE = ANS / "roles" / "k3s_server"
|
||||
VIP_ROLE = ANS / "roles" / "kube_vip"
|
||||
PLAYBOOKS = ANS / "playbooks"
|
||||
CLUSTERS = ROOT / "clusters"
|
||||
|
||||
for p, why in ((INV, "the inventory that defines the node set"),
|
||||
(K3S_ROLE, "the role that installs the k3s control plane"),
|
||||
(VIP_ROLE, "the role that floats the control-plane VIP"),
|
||||
(CLUSTERS, "the Flux overlays that identify each environment")):
|
||||
check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA "
|
||||
"control plane is authored at all (R-PLAT-01)")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml")))
|
||||
VIP_TEXT = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2"))))
|
||||
|
||||
|
||||
# ===================================================================== inventory parse
|
||||
# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a
|
||||
# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]`
|
||||
# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'.
|
||||
ini_hosts: dict[str, list[str]] = {}
|
||||
ini_children: dict[str, list[str]] = {}
|
||||
host_vars: dict[str, dict[str, str]] = {}
|
||||
cur = None
|
||||
for line in INV.read_text().splitlines():
|
||||
s = line.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^\[([^\]]+)\]$", s)
|
||||
if m:
|
||||
cur = m.group(1)
|
||||
continue
|
||||
if cur is None or cur.endswith(":vars"):
|
||||
continue
|
||||
host = s.split()[0]
|
||||
if cur.endswith(":children"):
|
||||
ini_children.setdefault(cur.split(":")[0], []).append(host)
|
||||
else:
|
||||
ini_hosts.setdefault(cur, []).append(host)
|
||||
hv = host_vars.setdefault(host, {})
|
||||
for kv in s.split()[1:]:
|
||||
if "=" in kv:
|
||||
k, v = kv.split("=", 1)
|
||||
hv[k] = v
|
||||
|
||||
|
||||
def hosts_in(group: str, seen=None) -> set[str]:
|
||||
seen = seen or set()
|
||||
if group in seen:
|
||||
return set()
|
||||
seen.add(group)
|
||||
out = set(ini_hosts.get(group, []))
|
||||
for child in ini_children.get(group, []):
|
||||
out |= hosts_in(child, seen)
|
||||
return out
|
||||
|
||||
|
||||
def load_yaml(p: Path) -> dict:
|
||||
try:
|
||||
d = yaml.safe_load(p.read_text()) or {}
|
||||
except yaml.YAMLError:
|
||||
return {}
|
||||
return d if isinstance(d, dict) else {}
|
||||
|
||||
|
||||
ALL_VARS = load_yaml(GV / "all.yml")
|
||||
|
||||
|
||||
# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM
|
||||
# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds
|
||||
# the default pair; every group_vars/<g>/vars.yml that overrides `k3s_primary_group` is
|
||||
# another environment. Nothing below knows the words "dev" or "prod" as group names.
|
||||
class Env:
|
||||
def __init__(self, label, vars_, path):
|
||||
self.label = label
|
||||
self.vars = vars_
|
||||
self.path = path
|
||||
|
||||
def var(self, name, default=None):
|
||||
# group-scoped value wins, else the cluster-wide default from all.yml.
|
||||
if name in self.vars:
|
||||
return self.vars[name]
|
||||
return ALL_VARS.get(name, default)
|
||||
|
||||
|
||||
envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")]
|
||||
for vf in sorted(GV.glob("*/vars.yml")):
|
||||
v = load_yaml(vf)
|
||||
if "k3s_primary_group" in v or "k3s_servers_group" in v:
|
||||
envs.append(Env(str(vf.relative_to(ROOT)), v, vf))
|
||||
|
||||
check(len(envs) >= 2,
|
||||
"only ONE environment is derivable from the inventory's R-ENV-02 seam "
|
||||
"(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires "
|
||||
"dev/test to exercise the same multi-node topology, so two must be authored")
|
||||
|
||||
# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay.
|
||||
# roles/flux_bootstrap's own default supplies the value for an environment that does not
|
||||
# override it, so this is read from code, never guessed from a group name.
|
||||
fb_default = None
|
||||
fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml"
|
||||
if fb_defaults_file.is_file():
|
||||
fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path")
|
||||
check(isinstance(fb_default, str) and fb_default.startswith("clusters/"),
|
||||
"roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from "
|
||||
"an inventory environment to its Flux overlay is not expressed in code, so this check "
|
||||
"cannot tell which environment is prod")
|
||||
|
||||
prod_envs = []
|
||||
for e in envs:
|
||||
overlay = e.var("flux_bootstrap_env_path", fb_default)
|
||||
e.overlay = overlay if isinstance(overlay, str) else ""
|
||||
if e.overlay.rstrip("/").endswith("/prod"):
|
||||
prod_envs.append(e)
|
||||
check(len(prod_envs) == 1,
|
||||
f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay "
|
||||
f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in "
|
||||
"R-PLAT-01 has no unambiguous referent")
|
||||
|
||||
|
||||
# ============================================== overlay knob values (per environment)
|
||||
# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map;
|
||||
# each clusters/<env>/kustomization.yaml overrides values with JSON6902 patches written as
|
||||
# - op: replace
|
||||
# path: /data/<KEY>
|
||||
# value: "<v>"
|
||||
# (verified shape across both overlays). A key not patched keeps the base value.
|
||||
BASE_VARS: dict[str, str] = {}
|
||||
base_cm = CLUSTERS / "base" / "cluster-vars.yaml"
|
||||
if base_cm.is_file():
|
||||
d = load_yaml(base_cm)
|
||||
BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()}
|
||||
check(len(BASE_VARS) > 5,
|
||||
"clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface "
|
||||
"cannot be read, so the VIP/pool separation cannot be checked")
|
||||
|
||||
_PATCH_RE = re.compile(
|
||||
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n\s*value:\s*(?P<v>\S.*?)\s*$",
|
||||
re.M)
|
||||
|
||||
|
||||
def overlay_vars(overlay: str) -> dict[str, str]:
|
||||
out = dict(BASE_VARS)
|
||||
f = ROOT / overlay / "kustomization.yaml"
|
||||
if not f.is_file():
|
||||
return out
|
||||
for m in _PATCH_RE.finditer(f.read_text()):
|
||||
out[m.group("k")] = m.group("v").strip().strip('"').strip("'")
|
||||
return out
|
||||
|
||||
|
||||
# ============================================================ PER-ENVIRONMENT TOPOLOGY
|
||||
def as_ip(s):
|
||||
try:
|
||||
return ipaddress.ip_address(str(s).strip())
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
for e in envs:
|
||||
L = e.label
|
||||
pg = e.var("k3s_primary_group")
|
||||
sg = e.var("k3s_servers_group")
|
||||
check(isinstance(pg, str) and isinstance(sg, str) and pg and sg,
|
||||
f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot "
|
||||
"resolve 'my environment's primary/servers' and would only ever build dev")
|
||||
if not (isinstance(pg, str) and isinstance(sg, str)):
|
||||
continue
|
||||
primaries = hosts_in(pg)
|
||||
servers = hosts_in(sg)
|
||||
|
||||
# --- SERVERS: quorum-capable, one founder, prod exactly three ---------------------
|
||||
check(len(primaries) == 1,
|
||||
f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run "
|
||||
"`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no "
|
||||
"cluster at all")
|
||||
check(primaries and primaries <= servers,
|
||||
f"{L}: the primary {sorted(primaries)} is not a member of the servers group "
|
||||
f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or "
|
||||
"it is not part of the HA control plane it founds")
|
||||
n = len(servers)
|
||||
check(n >= 3,
|
||||
f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes "
|
||||
"'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd "
|
||||
"cluster of fewer than 3 members tolerates ZERO losses — it is not HA")
|
||||
check(n % 2 == 1,
|
||||
f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd "
|
||||
f"member count adds a member without adding fault tolerance and introduces a tie: "
|
||||
f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}")
|
||||
if n:
|
||||
tolerance = n - (n // 2 + 1)
|
||||
check(tolerance >= 1,
|
||||
f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate "
|
||||
f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one")
|
||||
if e in prod_envs:
|
||||
check(n == 3,
|
||||
f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: "
|
||||
"'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'")
|
||||
else:
|
||||
check(n >= 2,
|
||||
f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer "
|
||||
"nodes than prod but requires dev/test to 'exercise the same multi-node "
|
||||
"topology' — one node exercises no topology")
|
||||
|
||||
# --- ADDRESSES: the VIP can collide with nothing ----------------------------------
|
||||
vip_raw = e.var("kube_vip_address")
|
||||
vip = as_ip(vip_raw)
|
||||
check(vip is not None,
|
||||
f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has "
|
||||
"no floating endpoint, so losing the founder takes the API endpoint with it")
|
||||
node_ips = {}
|
||||
for h in sorted(servers):
|
||||
ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
|
||||
check(ip is not None,
|
||||
f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join "
|
||||
"target and the serving-cert SAN list all derive from it")
|
||||
if ip is not None:
|
||||
node_ips[h] = ip
|
||||
if vip is not None:
|
||||
check(vip not in set(node_ips.values()),
|
||||
f"{L}: kube_vip_address {vip} is ALSO a node address "
|
||||
f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a "
|
||||
"node's own address cannot float off that node")
|
||||
ov = overlay_vars(getattr(e, "overlay", ""))
|
||||
gw = as_ip(ov.get("GATEWAY_VIP", ""))
|
||||
lo = as_ip(ov.get("CILIUM_LB_POOL_START", ""))
|
||||
hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", ""))
|
||||
check(gw is not None and lo is not None and hi is not None,
|
||||
f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + "
|
||||
"CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and "
|
||||
"the Service-LB allocator cannot be checked")
|
||||
if gw is not None:
|
||||
check(vip != gw,
|
||||
f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME "
|
||||
f"address ({vip}) — kube-vip and Cilium would both ARP for it")
|
||||
if lo is not None and hi is not None:
|
||||
check(not (lo <= vip <= hi),
|
||||
f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium "
|
||||
f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a "
|
||||
"Service, and the two allocators would fight over ARP for it")
|
||||
check(lo <= hi,
|
||||
f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be "
|
||||
"used to prove the VIP is outside it")
|
||||
|
||||
# ==================================================================== DISTRO: k3s only
|
||||
check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M)
|
||||
is not None,
|
||||
"roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane "
|
||||
"version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)")
|
||||
check("get.k3s.io" in K3S_TEXT,
|
||||
"roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the "
|
||||
"distro (ADR-0001) and no other distro may stand in for it")
|
||||
check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None,
|
||||
"roles/k3s_server never composes the `server` subcommand — without it k3s installs an "
|
||||
"AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')")
|
||||
for bad, why in (
|
||||
(r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 "
|
||||
"nodes (no external datastore)')"),
|
||||
(r"datastore-endpoint\s*:", "an external datastore via config.yaml"),
|
||||
(r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not "
|
||||
"an etcd member (docs/02: etcd 'across all 3 nodes')"),
|
||||
(r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"),
|
||||
):
|
||||
hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*"))
|
||||
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini")
|
||||
and re.search(bad, p.read_text(errors="replace"))]
|
||||
check(not hits,
|
||||
f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA "
|
||||
f"mode, and {bad!r} contradicts it")
|
||||
|
||||
# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH
|
||||
check(re.search(r"--cluster-init", K3S_TEXT) is not None,
|
||||
"roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never "
|
||||
"founded, so there is no HA datastore to join")
|
||||
init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'",
|
||||
K3S_TEXT)
|
||||
check(init_when is not None,
|
||||
"the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an "
|
||||
"ungated founder flag would re-found the cluster on every server (split brain)")
|
||||
join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
|
||||
check(join is not None,
|
||||
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
|
||||
"— without a join target the servers are separate single-node clusters")
|
||||
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
|
||||
K3S_TEXT, re.S)
|
||||
check(join_when is not None,
|
||||
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
|
||||
"would also try to join itself")
|
||||
k3s_play = PLAYBOOKS / "k3s.yml"
|
||||
check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint")
|
||||
if k3s_play.is_file():
|
||||
play_txt = k3s_play.read_text()
|
||||
modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt)
|
||||
check(sorted(modes) == ["primary", "secondary"],
|
||||
f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and "
|
||||
"one secondary play must exist, or the founder/joiner split is not authored")
|
||||
check(re.search(r"serial:\s*1", play_txt) is not None,
|
||||
"playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — "
|
||||
"converging two of three servers at once drops embedded etcd below quorum, which "
|
||||
"is exactly the outage an HA control plane exists to avoid")
|
||||
hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M)
|
||||
check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns),
|
||||
f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal "
|
||||
"group name means this play can only ever build ONE environment, so prod cannot "
|
||||
"be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')")
|
||||
|
||||
# =========================================== SERVING-CERT SANs: VIP + every server
|
||||
sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", ""))
|
||||
check("kube_vip_address" in sans,
|
||||
"roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's "
|
||||
"serving cert would carry no SAN for the floating VIP, so every client reaching the "
|
||||
"API through it fails TLS validation and the HA endpoint is unusable")
|
||||
check("k3s_servers_group" in sans and "k3s_node_ip" in sans,
|
||||
"k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the "
|
||||
"cert must be valid for WHICHEVER server answers behind the VIP, and per-environment "
|
||||
"resolution of that set is what keeps one role serving both environments (R-ENV-02)")
|
||||
check(re.search(r"tls-san:", K3S_TEXT) is not None,
|
||||
"the SAN list is passed only as an install flag and never persisted to "
|
||||
"/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
|
||||
"no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")
|
||||
|
||||
# ============================================ FLOATING ENDPOINT: kube-vip on every server
|
||||
# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
|
||||
# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
|
||||
# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the
|
||||
# thing that ships.) Parser contract: the template writes each env var as an adjacent
|
||||
# `- name: X` / `value: "Y"` pair; comment lines are dropped first.
|
||||
VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2"))
|
||||
check(len(VIP_TEMPLATES) >= 1,
|
||||
"roles/kube_vip ships no manifest template — nothing deploys the API VIP")
|
||||
vip_manifest = "\n".join(
|
||||
"\n".join(l for l in p.read_text(errors="replace").splitlines()
|
||||
if not l.lstrip().startswith("#"))
|
||||
for p in VIP_TEMPLATES)
|
||||
vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?",
|
||||
vip_manifest))
|
||||
check(len(vip_env) >= 6,
|
||||
f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the "
|
||||
"parser lost its grip on the file, so the assertions below would assert nothing")
|
||||
check(vip_env.get("cp_enable") == "true",
|
||||
f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane "
|
||||
"mode must be ON or it floats no API VIP at all")
|
||||
check(vip_env.get("svc_enable") == "false",
|
||||
f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be "
|
||||
"false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM "
|
||||
"for the same addresses (ADR-0002 gives Service LB to Cilium)")
|
||||
check(vip_env.get("vip_leaderelection") == "true",
|
||||
f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} "
|
||||
"— without leader election the VIP has no defined owner to fail over FROM, which is "
|
||||
"the whole HA mechanism")
|
||||
check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}",
|
||||
f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the "
|
||||
"environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on "
|
||||
"prod (R-ENV-02) and would not match the serving-cert SAN list")
|
||||
check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None,
|
||||
"the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its "
|
||||
"node, which is the opposite of a floating endpoint")
|
||||
vip_play = PLAYBOOKS / "kube-vip.yml"
|
||||
check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP")
|
||||
if vip_play.is_file():
|
||||
vp = vip_play.read_text()
|
||||
vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M)
|
||||
check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0],
|
||||
f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers "
|
||||
"group through the same environment-scoped variable the other plays use, so the "
|
||||
"manifest exists on EVERY server and the VIP survives losing any one of them")
|
||||
check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None
|
||||
and re.search(r"k3s_data_dir", VIP_TEXT) is not None,
|
||||
"roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies "
|
||||
"<data-dir>/server/manifests, so an environment that moves --data-dir would write the "
|
||||
"manifest somewhere k3s never reads and come up with no VIP at all")
|
||||
|
||||
# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS
|
||||
# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||||
# step differs". A prod-only role or playbook would be a fork of the control-plane path.
|
||||
role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()}
|
||||
play_names = {p.name for p in PLAYBOOKS.glob("*.yml")}
|
||||
env_words = re.compile(r"prod|dev", re.I)
|
||||
forked = sorted(n for n in (role_names | play_names)
|
||||
if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml"))
|
||||
check(not forked,
|
||||
f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by "
|
||||
"the SAME code retargeted by variables (docs/02), not by a per-environment fork")
|
||||
for e in envs:
|
||||
if e.path.name != "vars.yml":
|
||||
continue
|
||||
# Every key a non-default environment sets must be a variable something already reads,
|
||||
# or it is an invented knob nothing consumes (a silent no-op that looks like config).
|
||||
consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*"))
|
||||
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh"))
|
||||
for k in sorted(e.vars):
|
||||
if k.startswith("ansible_"):
|
||||
continue
|
||||
check(consumers.count(k) > 1,
|
||||
f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an "
|
||||
"environment override that nothing reads is not a scoped override, it is dead "
|
||||
"configuration (and would make prod silently inherit a dev default)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 40,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
summary = "; ".join(
|
||||
f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]="
|
||||
f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers"
|
||||
for e in envs)
|
||||
print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
|
||||
"R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
|
||||
"embedded etcd on every server with no external datastore, a leader-elected "
|
||||
"control-plane-only VIP on every server that is SAN'd into the serving cert and "
|
||||
"collides with no other allocator, from one code path for both environments. The LIVE "
|
||||
"cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
|
||||
"SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
|
||||
PY
|
||||
Reference in new issue
Block a user