jakman85andClaude Opus 5
8eb5baffa4
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).
THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.
Adds:
* SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
failed run), no exec, no secrets. Results come back as a tarball on stdout read via
pods/log, so no execution channel into the runner pod is ever granted.
* ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
the rest of the cluster.
* Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
like apps-vms, so a test identity can never stall the platform or app tiers.
* A kyverno test with seven cases, six of them real escalation paths.
VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack
evil-image DENIED at .../image/
tag-not-digest DENIED at .../image/ (mutable tag is republishable)
host-namespaces DENIED at /spec/template/spec/hostPID/
extra-volume DENIED by the volumes foreach
injected-args DENIED by the args foreach
good-run ADMITTED
ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.
IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.
NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
170 lines
9.7 KiB
Bash
Executable File
170 lines
9.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/l0.sh — Stage 2+3: ServiceDefinition schema validation + Kyverno L0 policy gate.
|
|
# Both run in CI WITHOUT a cluster (docs/14 §"Where each layer runs") — live now.
|
|
#
|
|
# Stage 2 schema : catalog/*.yaml validated against the ServiceDefinition
|
|
# schema (owner: T-0.3, schemas/).
|
|
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
|
# present, no plaintext secrets, required labels/egress-class,
|
|
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
|
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
|
# a per-resource policy cannot express (secret hygiene across every
|
|
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
|
# Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
|
|
# test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
|
|
# replaces what used to be a hand-written "UNCONDITIONAL" stage per
|
|
# whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
|
|
# bulk-tier): three near-identical 12-line rationales, three hardcoded
|
|
# check.sh paths and three copies of the same rc 0/3/* case block, which
|
|
# did not scale and — worse — failed SILENTLY, because whoever added the
|
|
# next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
|
|
# test declares its own scope and the runner enforces it. The reasoning and
|
|
# the incidents those stages recorded were moved INTO the tests they
|
|
# describe (see each chainsaw-test.yaml's `scope` annotation) and the
|
|
# general mechanism is documented in conformance/lib/common.sh.
|
|
#
|
|
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
|
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
|
# rather than a false pass — the gate then knows the seam is unwired, not green.
|
|
set -euo pipefail
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/lib.sh"
|
|
|
|
rc=0
|
|
|
|
# ---- Stage 2: ServiceDefinition schema validation ---------------------------
|
|
log "Stage: ServiceDefinition schema validation (catalog/ against schemas/)"
|
|
if ! entrypoint_ready "$SCHEMA_VALIDATE_CMD"; then
|
|
skip "schema validator not present at '${SCHEMA_VALIDATE_CMD}' (T-0.3 / schemas/ not delivered yet) — UNWIRED, not green"
|
|
summary "- schema: **SKIPPED** (validator entrypoint \`${SCHEMA_VALIDATE_CMD}\` missing — T-0.3)"
|
|
else
|
|
if [ -d "$CATALOG_DIR" ] && compgen -G "${CATALOG_DIR}/*.y*ml" >/dev/null 2>&1; then
|
|
if "$SCHEMA_VALIDATE_CMD" "${CATALOG_DIR}"/*.y*ml; then
|
|
ok "schema validation passed"
|
|
summary "- schema: **passed**"
|
|
else
|
|
err "schema validation failed"; summary "- schema: **FAILED**"; rc=1
|
|
fi
|
|
else
|
|
log "no ServiceDefinitions in ${CATALOG_DIR} yet — validating validator is callable only"
|
|
"$SCHEMA_VALIDATE_CMD" --version >/dev/null 2>&1 || true
|
|
ok "schema validator present; no catalog entries to validate"
|
|
summary "- schema: **passed** (no catalog entries yet)"
|
|
fi
|
|
fi
|
|
|
|
# ---- Stage 3: Kyverno L0 policy gate ----------------------------------------
|
|
log "Stage: Kyverno L0 policy gate (kyverno test ${KYVERNO_POLICY_DIR})"
|
|
# `kyverno test <dir>` discovers kyverno-test.yaml files recursively.
|
|
if ! have kyverno; then
|
|
skip "kyverno CLI not installed (pin v${KYVERNO_VERSION}) — cannot run L0 gate"
|
|
summary "- kyverno-l0: **SKIPPED** (kyverno CLI not installed)"
|
|
# Missing tool is an environment gap, not a code pass -> exit 3 unless already failing.
|
|
[ "$rc" -eq 0 ] && rc=3
|
|
elif [ ! -d "$KYVERNO_POLICY_DIR" ] || ! compgen -G "${KYVERNO_POLICY_DIR}/**/kyverno-test.yaml" >/dev/null 2>&1 \
|
|
&& ! find "$KYVERNO_POLICY_DIR" -name 'kyverno-test.yaml' 2>/dev/null | grep -q .; then
|
|
skip "no kyverno-test.yaml under '${KYVERNO_POLICY_DIR}' (T-0.5 / platform/kyverno not delivered yet) — UNWIRED, not green"
|
|
summary "- kyverno-l0: **SKIPPED** (no policy tests yet — T-0.5)"
|
|
else
|
|
# The ClusterPolicies carry a Flux postBuild token in validationFailureAction
|
|
# (${KYVERNO_VALIDATION_ACTION}, substituted in-cluster from
|
|
# clusters/base/cluster-vars.yaml). The offline `kyverno test` CLI does NOT run
|
|
# Flux substitution, so the literal token is an invalid enum value and those
|
|
# policies fail to load (no result rows -> "Not found"). Render a throwaway copy
|
|
# with the base/prod value (Enforce — the strict posture the gate must prove)
|
|
# before testing. This does not change what the policies enforce in-cluster.
|
|
: "${KYVERNO_VALIDATION_ACTION:=Enforce}"
|
|
l0_render="$(mktemp -d)"
|
|
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
|
|
find "$l0_render" -type f -name '*.yaml' -print0 \
|
|
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
|
|
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
|
|
# error, bad match block, malformed JMESPath) does not fail the run: every result row
|
|
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
|
|
#
|
|
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
|
|
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
|
|
# rows came back `Skip / Invalid Policy`, and the summary read:
|
|
# Test Summary: 9 tests passed and 0 tests failed
|
|
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
|
|
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
|
|
#
|
|
# This is not specific to that policy: a typo in restrict-host-namespaces or
|
|
# secret-provenance would silently disable it and the gate would stay green. Scan the
|
|
# output for the load failure the exit code hides.
|
|
l0_kout="$(mktemp)"
|
|
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
|
|
if grep -q 'Invalid Policy' "$l0_kout"; then
|
|
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
|
|
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
|
|
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
|
|
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
|
|
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
|
|
rc=1
|
|
else
|
|
ok "Kyverno L0 policy tests passed"
|
|
summary "- kyverno-l0: **passed**"
|
|
fi
|
|
else
|
|
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
|
|
fi
|
|
rm -f "$l0_kout"
|
|
rm -rf "$l0_render"
|
|
fi
|
|
|
|
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
|
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
|
# than what Kyverno alone can express over single resources: whole-repo invariants
|
|
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
|
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
|
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
|
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
|
#
|
|
# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
|
|
# --------------------------------------------------------------
|
|
# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
|
|
# whose inputs live under the subtree its requirements map to. It is WRONG for one that
|
|
# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
|
|
# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
|
|
# most likely to break the invariant attributes no requirement at all, so requirement-scoped
|
|
# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
|
|
# in every L0 test that DECLARES that scope, so those run on every gate invocation.
|
|
#
|
|
# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
|
|
# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
|
|
# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
|
|
# the gate's own rc 3 below, never to rc 0.
|
|
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
|
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
|
|
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
|
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
|
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi
|
|
else
|
|
load_touched_requirements
|
|
l0_static_rc=0
|
|
"$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
|
|
${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
|
|
case "$l0_static_rc" in
|
|
0) ok "L0 static conformance checks passed"
|
|
summary "- l0-static: **passed**" ;;
|
|
3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
|
|
summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
|
|
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
|
*) err "L0 static conformance checks failed"
|
|
summary "- l0-static: **FAILED**"
|
|
rc=1 ;;
|
|
esac
|
|
# Per-check step-summary lines, so the gate output still NAMES what ran — the generated
|
|
# replacement for the one hardcoded `summary` line each deleted stage carried.
|
|
emit_offline_summary
|
|
fi
|
|
|
|
case "$rc" in
|
|
0) ok "L0 stage complete" ;;
|
|
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
|
*) err "L0 stage failed" ;;
|
|
esac
|
|
exit "$rc"
|