jakman85andClaude Opus 5
5fa278f61c
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
99 lines
6.4 KiB
YAML
99 lines
6.4 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
|
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
|
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
|
|
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
|
|
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
|
|
#
|
|
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
|
|
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
|
|
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
|
|
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
|
|
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
|
|
# cluster cannot tell you what the NEXT environment brought up from this repo would look
|
|
# like. The observed half is asserted separately at L2 by
|
|
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
|
|
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
|
|
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
|
|
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
|
|
# environment's Flux overlay are parsed and cross-checked against each other,
|
|
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
|
|
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
|
|
# ONE --cluster-init founder and every other server joining it with --server;
|
|
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
|
|
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
|
|
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
|
|
# in the apiserver serving cert's SANs together with every server's own address, and
|
|
# it is neither a node address nor the ingress Gateway VIP nor inside that
|
|
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
|
|
# roles and plays retargeted by variables only.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
|
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
|
# Chainsaw Test wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
|
|
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
|
|
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
|
|
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
|
|
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
|
|
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
|
|
# infra/PROD-PROVISIONING.md).
|
|
#
|
|
# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
|
|
# * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
|
|
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
|
|
# It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
|
|
# The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
|
|
# contract and per environment.
|
|
# * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
|
|
# the order is asserted — not as a tidiness preference but because the CNI's endpoint is
|
|
# the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
|
|
# API server.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: ha-control-plane-as-code
|
|
labels:
|
|
conformance.home.k3s/R-PLAT-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-PLAT-01"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE. The invariant spans THREE independent subtrees — infra/ (the Ansible
|
|
# roles + OpenTofu that create the control-plane nodes and the VIP), clusters/ (the
|
|
# CONTROL_PLANE_VIP knob and its per-overlay values) and platform/cilium/ (the HelmRelease
|
|
# whose k8sServiceHost IS that VIP). ci/touched-requirements.sh keys infra/* to NOTHING,
|
|
# clusters/* to R-GITOPS-01/R-ENV-02 and platform/cilium/* to the R-NET-* family: not one
|
|
# of them attributes R-PLAT-01, so no edit that can break HA control-plane-as-code selects
|
|
# this test. Note this is the same VIP whose empty substitution took the CNI down on 2/3
|
|
# dev nodes (2026-09-01) — the two halves of that incident are guarded here and in
|
|
# flux-postbuild-substitution.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
|
|
environment's Flux overlay are parsed and cross-checked, then every environment is
|
|
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
|
|
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
|
|
VIP that is in the serving cert's SANs and collides with no other allocator — built by
|
|
one code path for both environments.
|
|
spec:
|
|
steps:
|
|
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|