Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)

Browse Source 
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.

=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.

Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.

Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.

=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.

R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.

R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.

R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.

Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.

Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.

Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-08-31 23:26:49 -05:00
1 parent 9f59c36000
commit 6e77150ae1
21 files changed
+1450 -97
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

conformance/run.sh
+45 -30
View File
@@ -8,7 +8,7 @@
#
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
# ./run.sh --layer l2 # one layer
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
@@ -280,26 +280,40 @@ run_chainsaw() {
return 0
}
# --- L1 rendering: cluster-INDEPENDENT render checks (docs/14) --------------------------
# L1 verifies ServiceDefinition -> rendered-resources against golden expectations; it needs
# no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer + golden
# diff + structural asserts). We run those directly and emit a REAL pass/fail JUnit, instead
# of skipping L1 with the cluster-gated layers (T-0.3). Honors the same --layer/--requirements
# selection as the rest of the suite (select_dirs), filtered to the l1-rendering subtree.
l1_selected_dirs() {
select_dirs | grep '/l1-rendering/' || true
# --- L0/L1: the cluster-INDEPENDENT layers (docs/14) ------------------------------------
# docs/14 puts TWO layers in CI without a cluster:
# L0 static/policy tests/l0-static/** each dir carries a `check.sh` (repo invariants)
# L1 rendering tests/l1-rendering/** each dir carries a `render-check.sh` (render == golden)
# Both are executed directly here and emit a REAL pass/fail JUnit, instead of being skipped
# with the cluster-gated layers. Honors the same --layer/--requirements selection as the rest
# of the suite (select_dirs), filtered to those two subtrees.
OFFLINE_SUBTREES='/l0-static/|/l1-rendering/'
offline_selected_dirs() {
select_dirs | grep -E "${OFFLINE_SUBTREES}" || true
}
run_l1_offline() {
local -a l1dirs
mapfile -t l1dirs < <(l1_selected_dirs)
if [ "${#l1dirs[@]}" -eq 0 ]; then
log "No L1 render tests match the selection (layer=${LAYER} reqs='${REQS}')."
# True when EVERY requested layer is cluster-independent, so the offline result is the
# whole result and run.sh must exit on it (the ci/l0.sh + ci/l1.sh merge-gate paths).
layers_are_offline_only() {
local l
[ "${LAYER}" = all ] && return 1
for l in ${LAYER//,/ }; do
case "$l" in l0|l1) ;; *) return 1 ;; esac
done
return 0
}
run_offline_checks() {
local -a dirs
mapfile -t dirs < <(offline_selected_dirs)
if [ "${#dirs[@]}" -eq 0 ]; then
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
return 0
fi
log "Running ${#l1dirs[@]} L1 render check(s) OFFLINE (no cluster; docs/14)."
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${l1dirs[@]}"
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
}
###############################################################################
@@ -331,29 +345,30 @@ case "${MODE}" in
;;
esac
# L1 rendering is cluster-independent (docs/14): run the selected L1 render checks offline
# and treat their result as authoritative REGARDLESS of cluster availability. A render-check
# failure is a real failure (nothing cluster-gated about it), so it always blocks.
L1_RC=0
run_l1_offline || L1_RC=$?
if [ "${LAYER}" = l1 ]; then
# L1-only run (the ci/l1.sh merge-gate path): the offline render result IS the result.
if [ "${L1_RC}" -eq 0 ]; then
log "Done (L1 render). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
OFFLINE_RC=0
run_offline_checks || OFFLINE_RC=$?
if layers_are_offline_only; then
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
if [ "${OFFLINE_RC}" -eq 0 ]; then
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
else
log "L1 render checks FAILED (rc=${L1_RC}). JUnit: ${JUNIT_OUT}"
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
fi
exit "${L1_RC}"
exit "${OFFLINE_RC}"
fi
if [ "${L1_RC}" -ne 0 ]; then
log "L1 render checks FAILED (rc=${L1_RC}) — blocking (render failure is not cluster-gated)."
exit "${L1_RC}"
if [ "${OFFLINE_RC}" -ne 0 ]; then
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
exit "${OFFLINE_RC}"
fi
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
# cluster-gated layers. (L1 already ran offline above and passed.)
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
log "NOTE: the offline L0/L1 results above already ran for real; this shim covers L2/L3 only."
log "Also running structural lint so scaffolds stay valid."
lint_tests || { log "lint failed"; exit 1; }
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \