jakman85andClaude Opus 5
6e77150ae1
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
93 lines
3.6 KiB
Python
Executable File
93 lines
3.6 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""Emit a JUnit report marking every discovered Chainsaw test as *skipped*.
|
|
|
|
Used in Phase 0 (and any run where no cluster is reachable): the behavior tests are
|
|
valid scaffolds but cannot execute without a cluster, so instead of failing the suite we
|
|
emit a JUnit file whose testcases are `<skipped>` with the reason. This keeps the
|
|
merge/phase gate green while the coverage report still shows which requirements have a
|
|
verifying test. As later phases deliver a dev/test cluster, run.sh hands execution to
|
|
Chainsaw, which emits the real pass/fail JUnit and this shim is no longer used.
|
|
|
|
Each testcase carries its verified R-* IDs in the `classname` + a property so the JUnit
|
|
remains requirement-tagged (R-TEST-04) even in skipped form.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import re
|
|
import sys
|
|
import xml.etree.ElementTree as ET
|
|
from pathlib import Path
|
|
from xml.dom import minidom
|
|
|
|
LABEL_PREFIX = "conformance.home.k3s"
|
|
LABEL_LINE_RE = re.compile(
|
|
r"^\s*" + re.escape(LABEL_PREFIX) + r"/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true", re.I
|
|
)
|
|
|
|
|
|
def collect(path: Path) -> tuple[str, list[str], str]:
|
|
text = path.read_text()
|
|
name_m = re.search(r"^\s*name:\s*(\S+)", text, re.M)
|
|
name = name_m.group(1) if name_m else path.parent.name
|
|
reqs = sorted({m.group(1).upper() for m in
|
|
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
|
|
rel = str(path)
|
|
layer = ("L0" if "l0-static" in rel
|
|
else "L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
|
else "L3" if "l3-fault-injection" in rel else "META")
|
|
return name, reqs, layer
|
|
|
|
|
|
def main() -> int:
|
|
ap = argparse.ArgumentParser(description=__doc__)
|
|
here = Path(__file__).resolve().parent.parent
|
|
ap.add_argument("--tests", type=Path, default=here / "tests")
|
|
ap.add_argument("--out", type=Path,
|
|
default=here / "_artifacts" / "report.junit.xml")
|
|
ap.add_argument("--reason", default="no cluster reachable (Phase 0 scaffold): "
|
|
"behavior test not executed")
|
|
args = ap.parse_args()
|
|
|
|
files = sorted(args.tests.rglob("chainsaw-test.yaml"))
|
|
suite = ET.Element("testsuite", {
|
|
"name": "home-k3s-conformance (scaffold/skipped)",
|
|
"tests": str(len(files)),
|
|
"failures": "0",
|
|
"errors": "0",
|
|
"skipped": str(len(files)),
|
|
})
|
|
for f in files:
|
|
name, reqs, layer = collect(f)
|
|
tc = ET.SubElement(suite, "testcase", {
|
|
# classname carries layer + requirement tags so the JUnit stays traceable.
|
|
"classname": f"{layer}:{','.join(reqs) if reqs else 'UNTAGGED'}",
|
|
"name": name,
|
|
})
|
|
props = ET.SubElement(tc, "properties")
|
|
ET.SubElement(props, "property",
|
|
{"name": "requirements", "value": ",".join(reqs)})
|
|
ET.SubElement(props, "property", {"name": "layer", "value": layer})
|
|
ET.SubElement(props, "property", {"name": "path", "value": str(f)})
|
|
ET.SubElement(tc, "skipped", {"message": args.reason})
|
|
|
|
root = ET.Element("testsuites", {
|
|
"name": "home-k3s-conformance",
|
|
"tests": str(len(files)),
|
|
"skipped": str(len(files)),
|
|
"failures": "0",
|
|
"errors": "0",
|
|
})
|
|
root.append(suite)
|
|
|
|
args.out.parent.mkdir(parents=True, exist_ok=True)
|
|
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
|
args.out.write_text(xml)
|
|
print(f"[junit] wrote skipped JUnit for {len(files)} scaffolded test(s) -> {args.out}",
|
|
file=sys.stderr)
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|