Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat(conformance): cover R-PLAT-01, R-NET-04, R-SEC-03 — 88.2% -> 94.1%, and fix an unrecordable-proof defect

Browse Source 
Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0
safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS.

R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control
plane" is half authored and half observable.
  l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from
  the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with
  no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init
  founder per environment; odd, quorum-capable server count with tolerance >= 1;
  tls-sans carry the VIP and every server IP; kube-vip leader-elected with
  cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside
  its environment's LB pool and not a node IP.
  l2-behavioral/ha-control-plane: control-plane AND etcd role on every node,
  every node a REGISTERED etcd member, exactly 3 when the cluster says
  CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every
  server's own address — which proves the VIP fronts the API and that every server
  runs an apiserver, not one founder plus two joiners.

  An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's
  LAN and fails against dev (nodes behind the pfSense NAT), making the test a
  property of where it was launched. Hence the in-cluster Job.

R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the
totality/disjointness of the selecting rules, not another per-service check.
  l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the
  same closed set in schema + Kyverno policy + renderer; EXACTLY ONE
  CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn;
  nothing selects `internet` (absence of a rule IS the path); RFC1918 containment
  computed with `ipaddress` plus every address each overlay assigns itself; the
  tunnel default route and blackhole confined to table 51820 behind an ip rule that
  must carry a from/fwmark/iif selector; and the nft chain `policy accept` with
  every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test
  does not assert.

R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable".
  l0-static/auditability-proofs-producible (78 assertions) drives the real
  machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer
  the spec's wording demands (fault injection and DR drill => L3), taken from
  status.py's own verdict; run.sh --requirements must resolve and pass the gate's
  --require assertion; each proof declares a sentinel record_run.py can attribute,
  computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted
  against the real tools — a green JUnit whose log holds only a SKIP token, and a
  ledger row claiming `proven` at the root commit — which must come back not-proven
  and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted,
  not hand-listed.

*** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's
default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel,
so record_run.py could never attribute a pass to them — the isolation and TLS audit
proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared,
and status.py requires all of a requirement's tests to be proven). Each now ends
with an additive terminal sentinel step reached only when every prior step passed.
Verified: all three carry it.

TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this
repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role
defaults, so flipping the template to true still passed — now parses the DaemonSet
template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja
variable while the template carries its NAME, matching nothing and passing
vacuously — templates are now rendered against role defaults first. 30 perturbations
across the three checks, all caught, plus 16 synthetic L2 perturbations via a
fake-kubectl shim.

R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's
widgets.yaml carries only datetime and search, kubernetes.yaml is empty,
automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress
only — no health source is even reachable. It has already drifted (crafty authors a
route with no tile). A test today would go red on both; it must ship WITH the
config fix.

VERIFIED: full CI gate PASS. Neither cluster mutated.

Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-06 19:35:18 -05:00
1 parent 5ec1460532
commit cb1916f092
12 files changed
+2488
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml
+106
View File
@@ -0,0 +1,106 @@
# =============================================================================
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
# one-time checks", with the acceptance gate "The five auditability proofs above can
# be produced on demand."
#
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
# -----------------------------------------------------------------------------------
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
# failure modes are asserted instead, each by DRIVING the real machinery:
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
# the spec's own words require (fault injection / a DR drill => L3), taken from
# lib/status.py's own verdict so this check cannot disagree with the status report;
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
# attribute — a proof that passes but records nothing leaves the operator with a green
# terminal and no evidence;
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
# ci/ or run.sh ran before this test existed.
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
# so a sixth proof added to the spec fails here instead of being silently unverified.
#
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
# anyone needs it.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
# and two synthetic forgeries are presented to them,
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
# documented one-command invocation selects and asserts them, each is capturable as a
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
# it directly, and this Chainsaw Test wraps the SAME script.
#
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
# command, capturable, and that the chain refuses forgeries — not that isolation holds
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
# would need signed history.
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
# not re-asserted here.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: auditability-proofs-producible
labels:
conformance.home.k3s/R-SEC-03: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree and the harness's own tooling, when status.py, the requirement
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
of docs/07's five auditability proofs has an executable test at the layer the spec
requires, is selectable and asserted by the documented one-command invocation, is
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
evidence.
spec:
steps:
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh
# ---------------------------------------------------------------------------------------------
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
# the wrong trade (R-TEST-06).
conformance/tests/l0-static/auditability-proofs-producible/check.sh
Executable
+431
View File
@@ -0,0 +1,431 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — auditability-proofs-producible
# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can
# prove isolation, egress path, cert validity, and backup integrity)."
# — spec docs/00-vision-and-scope.md).
#
# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md
# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove":
# - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a
# denied-probe test.
# - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
# - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
# - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert
# status + external check.
# - **Backups**: "irreplaceable data is restorable" — via a DR drill.
# "These proofs are part of the platform acceptance suite, not one-time checks."
# and its acceptance gate: "The five auditability proofs above can be produced on demand."
#
# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY
# ------------------------------------------------------------
# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01,
# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and
# would keep passing while every proof rotted into a scaffold. Four separate failure modes
# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted
# by DRIVING the real machinery rather than by reading files:
#
# (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a
# requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and,
# where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an
# executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this
# check RUNS lib/status.py and reads its verdict, so it cannot disagree with the
# repository's own status report.
# (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh
# --requirements <the five IDs>` must resolve to a non-empty test set and must ASSERT
# their coverage (that flag is the merge gate's own path: it passes --require to
# coverage.py). This check drives run.sh itself, so if the resolver, the labels or the
# gate wiring break, R-SEC-03 fails.
# (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator
# with a green terminal and no evidence. For each Chainsaw-layer proof, the test must
# declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with
# record_run.py's OWN parser, so deleting a test's sentinel makes this check fail
# instead of silently making the requirement unprovable.
# (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause
# that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the
# real tools, in a throwaway directory:
# * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log
# carries only that test's SKIP token — the exact shape of a self-skipping L2/L3
# test. record_run.py must refuse to call it `proven`.
# * a ledger row claiming `proven` at a commit that PREDATES the requirement's
# implementing code — status.py must degrade it to `stale-proof`, never green.
# Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing
# in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof
# were themselves unexercised in the merge gate until this check.
#
# The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec
# repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's
# mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather
# than quietly ignoring it.
#
# WHY L0: every clause above is a property of the repository and its own tooling, provable
# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact
# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true
# with one.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
# - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one
# documented command, capturable as evidence, and that the evidence chain refuses
# forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR
# restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the
# proof, which is what is asserted.
# - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this
# suite queries Hubble, and this check does not pretend otherwise: it asserts the
# denied-probe half, which is the half that is automated. The Hubble half remains an
# operator-interactive capability — a real, named gap in R-SEC-03's coverage.
# - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status
# report's prose, the ledger's provenance fields) is a usability property, not a testable
# one.
# - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review;
# nothing here can prove a row was never removed from it (that would need signed history).
# - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than
# the five bullets docs/07 enumerates. The five are what the spec made concrete and are
# what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim,
# tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
export SPEC_REPO
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
ROOT = Path(sys.argv[1])
CONF = ROOT / "conformance"
LIB = CONF / "lib"
FAILURES: list[str] = []
NOTES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def note(msg):
NOTES.append(msg)
def die():
print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} "
"assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
def run(cmd, **kw) -> subprocess.CompletedProcess:
return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw)
# ================================================= THE FIVE PROOFS docs/07 ENUMERATES
# key = the bullet label docs/07 uses (verified against the spec below when it is mounted)
# reqs = the requirement(s) whose verifying test IS that proof
# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection
# test" / "a DR drill", an L2 happy-path test does not discharge the proof.
PROOFS = {
"Isolation": {"reqs": ["R-NET-01"], "layers": {"L2", "L3"},
"why": '"service X cannot reach service Y" — via Hubble flow logs + a '
"denied-probe test"},
"Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"},
"why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'},
"Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"},
"why": '"with VPN down, X leaks nothing" — via a fault-injection test'},
"TLS": {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"},
"why": '"every endpoint serves a valid, non-self-signed, unexpired cert" '
"— via cert status + external check"},
"Backups": {"reqs": ["R-STATE-03"], "layers": {"L3"},
"why": '"irreplaceable data is restorable" — via a DR drill'},
}
PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]})
# ---- SELF-POLICING: the proof list must still be the SPEC's proof list ---------------
spec = Path(os.environ.get("SPEC_REPO", ""))
sec_doc = spec / "docs" / "07-security-and-secrets.md"
if sec_doc.is_file():
text = sec_doc.read_text()
m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text,
re.M | re.S)
check(m is not None,
"docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — "
"the proof list this check is coupled to has moved or been removed, so the mapping "
"below can no longer be trusted (R-TEST-06: follow the spec, never the other way)")
if m:
labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M)
check(set(labels) == set(PROOFS),
f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check "
f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying "
"test, and a proof this check does not know about would be silently unverified")
check(len(labels) == 5,
f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance "
"gate says 'The five auditability proofs above can be produced on demand' — the "
"prose and the list have drifted apart")
else:
note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be "
"re-derived from docs/07, so the mapping below was used as written. Every other "
"assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)")
# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH
# Driven through lib/status.py so this check and the repository's own status report cannot
# disagree about what is executable and what is a scaffold.
tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-"))
try:
out = tmp / "status"
st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
"--registry", str(CONF / "requirements" / "registry.yaml"),
"--tests", str(CONF / "tests"), "--out", str(out)])
check(st.returncode == 0,
f"lib/status.py exited {st.returncode} — the report an operator would use to find "
f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}")
status = {}
sj = out / "status.json"
if sj.is_file():
status = json.loads(sj.read_text())
check(bool(status.get("by_requirement")),
"lib/status.py produced no per-requirement report — the audit posture of each "
"requirement is not derivable, so R-SEC-03 cannot be assessed at all")
by_req = status.get("by_requirement", {})
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
rec = by_req.get(rid)
check(rec is not None,
f"proof '{label}' maps to {rid}, which the status report does not know — "
"either the registry lost the requirement or this mapping is stale")
if not rec:
continue
tests = (rec.get("verification") or {}).get("tests") or []
executable = [t for t in tests if t.get("state", "").startswith("executable")]
check(bool(executable),
f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are "
f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: "
f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the "
"operator cannot prove this property")
at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]]
check(bool(at_layer),
f"proof '{label}' ({rid}) has no executable test at layer(s) "
f"{sorted(spec_proof['layers'])} — it has "
f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. "
f"docs/07 requires this proof to be produced {spec_proof['why']}, and a "
"weaker layer does not discharge it (a rendering check is not a drill)")
# ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND
# The resolver: --requirements must translate the five R-IDs into real test dirs.
resolver = run(["bash", "-c",
f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; '
f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"'])
dirs = [l for l in resolver.stdout.splitlines() if l.strip()]
check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS),
f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) "
f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's "
"acceptance gate depends on ('produced on demand') cannot select the proofs")
# The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a
# zero exit here is the repository's own statement that all five proofs are covered.
gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)])
check(gate.returncode == 0,
f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited "
f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit "
f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}")
# ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE
# Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass
# token can pass forever and never be recorded as proof of anything.
sys.path.insert(0, str(LIB))
os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1")
sys.dont_write_bytecode = True
import record_run as rr # noqa: E402
smap = rr.sentinel_map(CONF / "tests")
check(len(smap) >= 10,
f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its "
"grip on the test tree, so the assertions below would assert nothing")
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
rec = by_req.get(rid) or {}
tests = (rec.get("verification") or {}).get("tests") or []
names = [t.get("name") for t in tests
if t.get("state", "").startswith("executable")
and t.get("layer") in spec_proof["layers"]]
for name in names:
ent = smap.get(name)
check(ent is not None,
f"proof '{label}': record_run.py does not know the test {name!r} — its "
"result could never be attributed to a ledger row")
if not ent:
continue
if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS:
continue # offline layers are truthful by JUnit alone
check(bool(ent.get("pass")),
f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, "
"so lib/record_run.py can never record it as proven. It would go green "
"in the terminal and leave the operator with no evidence — exactly the "
"gap the ledger exists to close (docs/07: the proofs are 'part of the "
"platform acceptance suite, not one-time checks')")
check(bool(ent.get("terminal_pass")),
f"proof '{label}': the test {name!r} has no TERMINAL pass token, so "
"record_run.py cannot decide its verdict")
# =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE
# Both tools' own guards, which NOTHING in ci/ or run.sh drives today.
for tool in ("record_run.py", "status.py"):
r = run([sys.executable, str(LIB / tool), "--selftest"])
check(r.returncode == 0,
f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a "
"self-skipped or stale run from being recorded as proof are broken, so nothing "
f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}")
# ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token.
# This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof.
forge_target, forge_skip = None, None
for label, spec_proof in sorted(PROOFS.items()):
for rid in spec_proof["reqs"]:
for name, ent in smap.items():
if rid in (ent.get("requirements") or []) and ent.get("skip") \
and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
forge_target, forge_skip = name, ent["skip"][0]
break
if forge_target:
break
if forge_target:
break
check(forge_target is not None,
"no Chainsaw-layer proof test declares a skip token, so the forgery attempt below "
"could not be mounted. That is not a pass: it means this check could not verify "
"that a self-skip is refused as proof, which is the whole point of the ledger")
if forge_target:
run_dir = tmp / "20260101T000000Z-forged"
run_dir.mkdir(parents=True)
(run_dir / "report.junit.xml").write_text(
'<?xml version="1.0" encoding="UTF-8"?>\n'
'<testsuites><testsuite name="chainsaw" tests="1" failures="0" errors="0">'
f'<testcase name="{forge_target}" classname="forged"/>'
"</testsuite></testsuites>\n")
(run_dir / "run.log").write_text(
f"=== RUN chainsaw/{forge_target}\n"
f" === STDOUT\n {forge_skip}\n")
forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir),
"--dry-run", "--ledger", str(tmp / "never-written.jsonl"),
"--tests", str(CONF / "tests"), "--repo", str(ROOT)])
verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b",
forged.stdout, re.M)
check(verdicts and all(v != "proven" for v in verdicts),
f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: "
"a JUnit saying `passed` with nothing in the log but that test's own SKIP token "
f"({forge_skip}). A self-skip must never become proof — with this hole every "
"audit proof in docs/07 could be produced by a cluster that skipped it")
check(not (tmp / "never-written.jsonl").exists(),
"record_run.py --dry-run WROTE to the ledger — a tool that records while "
"claiming not to cannot be trusted to record only real runs")
# ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code.
# status.py must degrade it to stale-proof (not green), or an old green run would keep
# vouching for code it never executed.
root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"])
old_sha = (root_sha.stdout.strip().splitlines() or [""])[0]
check(len(old_sha) == 40,
"the repository's root commit could not be resolved, so the stale-proof guard could "
"not be exercised (that is a gap in this check, not a pass)")
if len(old_sha) == 40 and forge_target:
stale_ledger = tmp / "stale.jsonl"
rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0]
stale_ledger.write_text(json.dumps({
"ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev",
"chainsaw_version": "v0.0.0", "kube_version": "unknown",
"layer": smap[forge_target].get("layer", "L3"), "test": forge_target,
"requirements": [rid], "junit_result": "passed",
"sentinel": (smap[forge_target].get("terminal_pass") or "X"),
"proof": "proven", "run_id": "20260101T000000Z-forged",
"note": "synthetic row written by the R-SEC-03 conformance check",
}) + "\n")
out2 = tmp / "status-stale"
st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
"--registry", str(CONF / "requirements" / "registry.yaml"),
"--tests", str(CONF / "tests"), "--out", str(out2),
"--ledger", str(stale_ledger)])
check(st2.returncode == 0,
f"lib/status.py failed against a synthetic ledger: "
f"{(st2.stderr or st2.stdout).strip()[:300]}")
s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {}
rec2 = (s2.get("by_requirement") or {}).get(rid) or {}
per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])}
check(per.get(forge_target) == "stale-proof",
f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT "
f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A "
"proof from a run that predates the code it vouches for is not evidence, and "
"honouring it would let one old green run certify every later change")
check((rec2.get("proof") or {}).get("state") != "proven",
f"{rid} is reported PROVEN on the strength of a row from the root commit — the "
"freshness guard is not holding")
# ---- the ledger itself is machine-readable, and no row proves a test that is gone ----
ledger = CONF / "runs" / "ledger.jsonl"
if ledger.is_file():
rows, bad = [], []
for i, line in enumerate(ledger.read_text().splitlines(), 1):
if not line.strip():
continue
try:
rows.append(json.loads(line))
except json.JSONDecodeError:
bad.append(i)
check(not bad,
f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record "
"is not machine-readable, so no proof can be derived from it")
for r in rows:
for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof",
"run_id"):
check(key in r,
f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without "
"provenance cannot be audited")
if r.get("proof") == "proven":
check(bool(r.get("sentinel")),
f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — "
"the sentinel is the only thing separating a real pass from a self-skip")
check(r.get("test") in smap,
f"a ledger row claims `proven` for {r.get('test')!r}, which is not a "
"test in the suite any more — a proof of something that no longer exists")
check("-dirty" not in str(r.get("git_sha", "")),
f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree "
f"({r.get('git_sha')}) — what ran is not what is committed")
else:
note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions "
"were skipped; every other assertion ran, and the forgery probes above use "
"synthetic ledgers, not this file")
finally:
shutil.rmtree(tmp, ignore_errors=True)
# --------------------------------------------------------------------------- verdict
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
check(CHECKS >= 30,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
"would be meaningless")
if FAILURES:
die()
for n in NOTES:
print(f" note: {n}")
print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 "
f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the "
f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves "
"and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can "
"attribute, both evidence tools' self-tests pass, and two independent forgeries — a "
"self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. "
"R-SEC-03's producibility half only: whether each proof PASSES is its own "
"requirement's test, and the Hubble half of the isolation proof is automated nowhere "
"(see the header's gap list).")
PY