fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate
gate / gate (pull_request) Successful in 1m40s
gate / conformance (pull_request) Has been skipped

`catalog/render.py --selftest` executes 106 assertions on the renderer's output
contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed —
so all 106 were dev-time only, and the merge gate could not tell a sound renderer
from a gutted one. Recorded as the top follow-up of
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.

`ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines
of invocation; the rest is the rationale comment.

WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives
conformance/lib/{record_run,status}.py --selftest "because nothing else drives
them". That is the wrong hook: the check serves R-SEC-03 and its subject is the
evidence chain — can a forged or stale run become an audit proof? The renderer is
not in that chain, and hanging it there would make the check's name, its intent
annotation and its requirement mapping untrue. This stage's subject already IS
catalog/render.py, so the renderer's own contract test belongs in it.

The two assertions are independent, and that is the point: render-golden catches
golden-vs-renderer drift but is satisfied by ANY renderer whose output was
re-rendered — including one that just lost a contract invariant and had all ten
goldens regenerated in the same commit. That is precisely the hole the follow-up
named. The selftest catches it; render-golden cannot.

It runs before the empty-catalog exit 3 because the selftest renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must
not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is
source in this repo, so once python3 and render.py are both present (asserted
above it) there is no legitimate environmental cause for "could not run".

PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via
the stage's existing RENDER_PY override, so catalog/render.py — another branch's
file this session — was never touched:
  * deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie
    fix) -> stage rc 1;
  * OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion
    "parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the
    cookie entry recorded as UNGATED, and one render-golden's diff would miss.

AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage
render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this
repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be
swallowed by CI.

render.py itself is unchanged. No conformance/ path is touched, no gate stage is
added, nothing is deployed by this.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-01 21:21:49 -05:00
1 parent 08deb208a1
commit d44490736c
2 files changed
+259 -1

No files matched your search

+54 -1
View File
@@ -1,5 +1,10 @@
#!/usr/bin/env bash
# ci/render-golden.sh — Stage: render==golden reproducibility gate.
# ci/render-golden.sh — Stage: the renderer is trustworthy.
# (1) catalog/render.py passes its OWN output-contract self-test, and
# (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
# Two independent assertions about one subject, catalog/render.py. (1) was added
# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
# block above the invocation for why it lives here and not in a conformance check.
#
# WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
# invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
summary "- render-golden: **SKIPPED** (renderer missing)"
exit 3
fi
# --- The renderer's OWN self-test, before anything is compared ---------------
# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
# assertions was dev-time only and a merge gate could not tell a sound renderer from a
# gutted one.
#
# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
# that chain, and hanging it there would make the check's name and its requirement mapping
# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
#
# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
# memory), so an empty catalog must not be able to skip it.
#
# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
# output was re-rendered, including one that just lost a contract invariant and had all ten
# goldens regenerated in the same commit. The self-test catches exactly that.
#
# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
# run its self-test" has no legitimate environmental cause once python3 and render.py are
# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
selftest_out="$(mktemp)"
if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
ok "render.py --selftest passed"
summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
else
err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
err " Every golden under apps/ is produced by this renderer, so the goldens below cannot"
err " be trusted even if they reproduce byte-for-byte. Reproduce with:"
err " ${PYTHON} catalog/render.py --selftest"
sed 's/^/ /' "$selftest_out" >&2 || true
summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
rm -f "$selftest_out"
exit 1
fi
rm -f "$selftest_out"
if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
summary "- render-golden: **SKIPPED** (empty catalog)"