jakman85andClaude Opus 5
cb1916f092
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0 safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS. R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control plane" is half authored and half observable. l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init founder per environment; odd, quorum-capable server count with tolerance >= 1; tls-sans carry the VIP and every server IP; kube-vip leader-elected with cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside its environment's LB pool and not a node IP. l2-behavioral/ha-control-plane: control-plane AND etcd role on every node, every node a REGISTERED etcd member, exactly 3 when the cluster says CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every server's own address — which proves the VIP fronts the API and that every server runs an apiserver, not one founder plus two joiners. An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's LAN and fails against dev (nodes behind the pfSense NAT), making the test a property of where it was launched. Hence the in-cluster Job. R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the totality/disjointness of the selecting rules, not another per-service check. l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the same closed set in schema + Kyverno policy + renderer; EXACTLY ONE CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn; nothing selects `internet` (absence of a rule IS the path); RFC1918 containment computed with `ipaddress` plus every address each overlay assigns itself; the tunnel default route and blackhole confined to table 51820 behind an ip rule that must carry a from/fwmark/iif selector; and the nft chain `policy accept` with every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test does not assert. R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable". l0-static/auditability-proofs-producible (78 assertions) drives the real machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer the spec's wording demands (fault injection and DR drill => L3), taken from status.py's own verdict; run.sh --requirements must resolve and pass the gate's --require assertion; each proof declares a sentinel record_run.py can attribute, computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted against the real tools — a green JUnit whose log holds only a SKIP token, and a ledger row claiming `proven` at the root commit — which must come back not-proven and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted, not hand-listed. *** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel, so record_run.py could never attribute a pass to them — the isolation and TLS audit proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared, and status.py requires all of a requirement's tests to be proven). Each now ends with an additive terminal sentinel step reached only when every prior step passed. Verified: all three carry it. TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role defaults, so flipping the template to true still passed — now parses the DaemonSet template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja variable while the template carries its NAME, matching nothing and passing vacuously — templates are now rendered against role defaults first. 30 perturbations across the three checks, all caught, plus 16 synthetic L2 perturbations via a fake-kubectl shim. R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's widgets.yaml carries only datetime and search, kubernetes.yaml is empty, automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress only — no health source is even reachable. It has already drifted (crafty authors a route with no tile). A test today would go red on both; it must ship WITH the config fix. VERIFIED: full CI gate PASS. Neither cluster mutated. Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]>
107 lines
7.2 KiB
YAML
107 lines
7.2 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
|
|
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
|
|
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
|
|
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
|
|
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
|
|
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
|
|
# one-time checks", with the acceptance gate "The five auditability proofs above can
|
|
# be produced on demand."
|
|
#
|
|
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
|
|
# -----------------------------------------------------------------------------------
|
|
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
|
|
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
|
|
# failure modes are asserted instead, each by DRIVING the real machinery:
|
|
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
|
|
# the spec's own words require (fault injection / a DR drill => L3), taken from
|
|
# lib/status.py's own verdict so this check cannot disagree with the status report;
|
|
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
|
|
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
|
|
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
|
|
# attribute — a proof that passes but records nothing leaves the operator with a green
|
|
# terminal and no evidence;
|
|
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
|
|
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
|
|
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
|
|
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
|
|
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
|
|
# ci/ or run.sh ran before this test existed.
|
|
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
|
|
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
|
|
# so a sixth proof added to the spec fails here instead of being silently unverified.
|
|
#
|
|
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
|
|
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
|
|
# anyone needs it.
|
|
#
|
|
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
|
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
|
|
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
|
|
# and two synthetic forgeries are presented to them,
|
|
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
|
|
# documented one-command invocation selects and asserts them, each is capturable as a
|
|
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
|
|
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
|
|
# it directly, and this Chainsaw Test wraps the SAME script.
|
|
#
|
|
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
|
|
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
|
|
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
|
|
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
|
|
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
|
|
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
|
|
# command, capturable, and that the chain refuses forgeries — not that isolation holds
|
|
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
|
|
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
|
|
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
|
|
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
|
|
# would need signed history.
|
|
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
|
|
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
|
|
# not re-asserted here.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: auditability-proofs-producible
|
|
labels:
|
|
conformance.home.k3s/R-SEC-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-SEC-03"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree and the harness's own tooling, when status.py, the requirement
|
|
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
|
|
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
|
|
of docs/07's five auditability proofs has an executable test at the layer the spec
|
|
requires, is selectable and asserted by the documented one-command invocation, is
|
|
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
|
|
evidence.
|
|
spec:
|
|
steps:
|
|
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|
|
# ---------------------------------------------------------------------------------------------
|
|
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
|
|
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
|
|
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
|
|
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
|
|
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
|
|
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
|
|
# the wrong trade (R-TEST-06).
|