jakman85andClaude Opus 5
cb1916f092
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0 safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS. R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control plane" is half authored and half observable. l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init founder per environment; odd, quorum-capable server count with tolerance >= 1; tls-sans carry the VIP and every server IP; kube-vip leader-elected with cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside its environment's LB pool and not a node IP. l2-behavioral/ha-control-plane: control-plane AND etcd role on every node, every node a REGISTERED etcd member, exactly 3 when the cluster says CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every server's own address — which proves the VIP fronts the API and that every server runs an apiserver, not one founder plus two joiners. An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's LAN and fails against dev (nodes behind the pfSense NAT), making the test a property of where it was launched. Hence the in-cluster Job. R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the totality/disjointness of the selecting rules, not another per-service check. l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the same closed set in schema + Kyverno policy + renderer; EXACTLY ONE CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn; nothing selects `internet` (absence of a rule IS the path); RFC1918 containment computed with `ipaddress` plus every address each overlay assigns itself; the tunnel default route and blackhole confined to table 51820 behind an ip rule that must carry a from/fwmark/iif selector; and the nft chain `policy accept` with every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test does not assert. R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable". l0-static/auditability-proofs-producible (78 assertions) drives the real machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer the spec's wording demands (fault injection and DR drill => L3), taken from status.py's own verdict; run.sh --requirements must resolve and pass the gate's --require assertion; each proof declares a sentinel record_run.py can attribute, computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted against the real tools — a green JUnit whose log holds only a SKIP token, and a ledger row claiming `proven` at the root commit — which must come back not-proven and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted, not hand-listed. *** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel, so record_run.py could never attribute a pass to them — the isolation and TLS audit proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared, and status.py requires all of a requirement's tests to be proven). Each now ends with an additive terminal sentinel step reached only when every prior step passed. Verified: all three carry it. TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role defaults, so flipping the template to true still passed — now parses the DaemonSet template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja variable while the template carries its NAME, matching nothing and passing vacuously — templates are now rendered against role defaults first. 30 perturbations across the three checks, all caught, plus 16 synthetic L2 perturbations via a fake-kubectl shim. R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's widgets.yaml carries only datetime and search, kubernetes.yaml is empty, automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress only — no health source is even reachable. It has already drifted (crafty authors a route with no tile). A test today would go red on both; it must ship WITH the config fix. VERIFIED: full CI gate PASS. Neither cluster mutated. Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]>
84 lines
5.3 KiB
YAML
84 lines
5.3 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
|
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
|
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
|
|
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
|
|
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
|
|
#
|
|
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
|
|
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
|
|
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
|
|
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
|
|
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
|
|
# cluster cannot tell you what the NEXT environment brought up from this repo would look
|
|
# like. The observed half is asserted separately at L2 by
|
|
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
|
|
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
|
|
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
|
|
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
|
|
# environment's Flux overlay are parsed and cross-checked against each other,
|
|
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
|
|
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
|
|
# ONE --cluster-init founder and every other server joining it with --server;
|
|
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
|
|
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
|
|
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
|
|
# in the apiserver serving cert's SANs together with every server's own address, and
|
|
# it is neither a node address nor the ingress Gateway VIP nor inside that
|
|
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
|
|
# roles and plays retargeted by variables only.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
|
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
|
# Chainsaw Test wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
|
|
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
|
|
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
|
|
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
|
|
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
|
|
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
|
|
# infra/PROD-PROVISIONING.md).
|
|
# * kube-vip is NOT imported by playbooks/site.yml, so the floating endpoint is a separate
|
|
# operator step. Observed and reported, NOT asserted — R-PLAT-01 does not fix the
|
|
# bring-up order (R-TEST-06).
|
|
# * `K3S_API_HOST` (Cilium's k8sServiceHost) is the PRIMARY NODE's address in both
|
|
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
|
|
# A real single-node dependency, ESCALATED to the PM rather than asserted.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: ha-control-plane-as-code
|
|
labels:
|
|
conformance.home.k3s/R-PLAT-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-PLAT-01"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
|
|
environment's Flux overlay are parsed and cross-checked, then every environment is
|
|
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
|
|
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
|
|
VIP that is in the serving cert's SANs and collides with no other allocator — built by
|
|
one code path for both environments.
|
|
spec:
|
|
steps:
|
|
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|