jakman85andClaude Opus 5
6e77150ae1
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
83 lines
3.5 KiB
Bash
83 lines
3.5 KiB
Bash
#!/usr/bin/env bash
|
|
# Shared helpers for the conformance harness. Sourced by run.sh; not executed directly.
|
|
# shellcheck shell=bash
|
|
|
|
# --- Requirement-tagging convention (single source of truth) --------------------------
|
|
# A Chainsaw Test declares the requirement(s) it verifies with a per-requirement label:
|
|
#
|
|
# metadata:
|
|
# labels:
|
|
# conformance.home.k3s/R-NET-01: "true" # one label per verified R-* ID
|
|
# annotations:
|
|
# conformance.home.k3s/requirements: "R-NET-01, R-NET-02" # human summary
|
|
# conformance.home.k3s/layer: "L2"
|
|
# conformance.home.k3s/safety-critical: "false"
|
|
#
|
|
# The LABEL is authoritative and does double duty:
|
|
# * `chainsaw test --selector conformance.home.k3s/R-NET-01=true` runs the subset.
|
|
# * lib/coverage.py reads the same labels to map requirement -> verifying test(s).
|
|
# The annotation is a convenience for humans/reports and MUST agree with the labels
|
|
# (coverage.py flags a mismatch). This is the convention referenced in README.md.
|
|
export CONF_LABEL_PREFIX="conformance.home.k3s"
|
|
|
|
# Absolute path to the conformance root (dir containing run.sh). Set by run.sh.
|
|
: "${CONF_ROOT:?CONF_ROOT must be set before sourcing common.sh}"
|
|
|
|
# --- Test discovery -------------------------------------------------------------------
|
|
# Every leaf test is a directory containing a `chainsaw-test.yaml`.
|
|
conf_all_test_files() {
|
|
find "${CONF_ROOT}/tests" -type f -name 'chainsaw-test.yaml' 2>/dev/null | sort
|
|
}
|
|
|
|
# Directories of tests that carry ANY of the given requirement labels (space-separated
|
|
# R-IDs). Used to translate `--requirements R-NET-01,R-NET-06` into a set of test dirs
|
|
# so we run each matching test exactly once (union, deduplicated).
|
|
conf_dirs_for_requirements() {
|
|
local reqs="$1" f dir rid matched
|
|
while IFS= read -r f; do
|
|
[ -n "$f" ] || continue
|
|
matched=0
|
|
for rid in $reqs; do
|
|
# Match a label line: conformance.home.k3s/R-NET-01: "true"
|
|
if grep -Eq "^[[:space:]]*${CONF_LABEL_PREFIX}/${rid}:[[:space:]]*[\"']?true" "$f"; then
|
|
matched=1; break
|
|
fi
|
|
done
|
|
[ "$matched" = 1 ] && dirname "$f"
|
|
done < <(conf_all_test_files) | sort -u
|
|
}
|
|
|
|
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
|
|
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
|
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
|
# this is what lets `run.sh --layer l2,l3` (the ci/conformance.sh L2/L3 stage) select both
|
|
# layers in one pass instead of erroring on an unknown "l2,l3" layer.
|
|
conf_dirs_for_layer() {
|
|
local layers="$1" layer
|
|
local -a roots=()
|
|
for layer in ${layers//,/ }; do
|
|
case "$layer" in
|
|
all) roots=("${CONF_ROOT}/tests"); break ;;
|
|
l0) roots+=("${CONF_ROOT}/tests/l0-static") ;;
|
|
l1) roots+=("${CONF_ROOT}/tests/l1-rendering") ;;
|
|
l2) roots+=("${CONF_ROOT}/tests/l2-behavioral") ;;
|
|
l3) roots+=("${CONF_ROOT}/tests/l3-fault-injection") ;;
|
|
meta) roots+=("${CONF_ROOT}/tests/meta") ;;
|
|
*) echo "unknown layer: $layer" >&2; return 2 ;;
|
|
esac
|
|
done
|
|
[ "${#roots[@]}" -gt 0 ] || return 0
|
|
find "${roots[@]}" -type f -name 'chainsaw-test.yaml' -exec dirname {} \; 2>/dev/null | sort -u
|
|
}
|
|
|
|
# Normalize a comma/space separated requirement list to space-separated, uppercased.
|
|
conf_normalize_reqs() {
|
|
echo "$1" | tr ',' ' ' | tr '[:lower:]' '[:upper:]' | xargs
|
|
}
|
|
|
|
# Is a live cluster reachable? (Preflight for L1/L2/L3 execution.)
|
|
conf_cluster_reachable() {
|
|
command -v kubectl >/dev/null 2>&1 || return 1
|
|
kubectl cluster-info >/dev/null 2>&1
|
|
}
|