jakman85andClaude Opus 5
d44490736c
`catalog/render.py --selftest` executes 106 assertions on the renderer's output
contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed —
so all 106 were dev-time only, and the merge gate could not tell a sound renderer
from a gutted one. Recorded as the top follow-up of
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
`ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines
of invocation; the rest is the rationale comment.
WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives
conformance/lib/{record_run,status}.py --selftest "because nothing else drives
them". That is the wrong hook: the check serves R-SEC-03 and its subject is the
evidence chain — can a forged or stale run become an audit proof? The renderer is
not in that chain, and hanging it there would make the check's name, its intent
annotation and its requirement mapping untrue. This stage's subject already IS
catalog/render.py, so the renderer's own contract test belongs in it.
The two assertions are independent, and that is the point: render-golden catches
golden-vs-renderer drift but is satisfied by ANY renderer whose output was
re-rendered — including one that just lost a contract invariant and had all ten
goldens regenerated in the same commit. That is precisely the hole the follow-up
named. The selftest catches it; render-golden cannot.
It runs before the empty-catalog exit 3 because the selftest renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must
not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is
source in this repo, so once python3 and render.py are both present (asserted
above it) there is no legitimate environmental cause for "could not run".
PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via
the stage's existing RENDER_PY override, so catalog/render.py — another branch's
file this session — was never touched:
* deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie
fix) -> stage rc 1;
* OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion
"parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the
cookie entry recorded as UNGATED, and one render-golden's diff would miss.
AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage
render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this
repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be
swallowed by CI.
render.py itself is unchanged. No conformance/ path is touched, no gate stage is
added, nothing is deployed by this.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
174 lines
8.7 KiB
Bash
Executable File
174 lines
8.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/render-golden.sh — Stage: the renderer is trustworthy.
|
|
# (1) catalog/render.py passes its OWN output-contract self-test, and
|
|
# (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
|
|
# Two independent assertions about one subject, catalog/render.py. (1) was added
|
|
# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
|
|
# block above the invocation for why it lives here and not in a conformance check.
|
|
#
|
|
# WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
|
|
# invocation of catalog/render.py. If a golden is hand-edited, or a catalog
|
|
# ServiceDefinition is changed without re-rendering, the goldens drift from
|
|
# their source of truth and Flux deploys something no one reviewed. The three
|
|
# L1 render spot-checks (conformance/tests/l1-rendering/*) only cover a couple
|
|
# of assertions on a couple of services — they let a real source/golden
|
|
# mismatch through (image-automation objects present in gitea/nextcloud but
|
|
# absent everywhere else). This stage closes that hole: it renders EVERY
|
|
# onboarded catalog service with the canonical command and FAILS on ANY diff
|
|
# vs the committed golden.
|
|
#
|
|
# CANONICAL RENDER (source of truth for apps/):
|
|
# python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
|
|
#
|
|
# The `--no-image-automation` flag is canonical because image automation is
|
|
# authored-but-DISABLED (DECISIONS-LOG: "Phase 4 image-automation merged but
|
|
# DISABLED pending a Gitea write-cred"; pihole/all apps onboarded via
|
|
# `--no-image-automation`). The Flux image-reflector/-automation controllers
|
|
# are NOT installed in-cluster, so emitting ImageRepository/ImagePolicy CRs
|
|
# would leave apps referencing controllers that don't exist. Every onboarded
|
|
# app must render identically under this ONE command.
|
|
#
|
|
# Runs in CI WITHOUT a cluster (pure render + diff). Exit codes per ci/lib.sh:
|
|
# 0 = pass, 1 = fail (blocks merge), 3 = skip (renderer/tooling missing).
|
|
set -euo pipefail
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/lib.sh"
|
|
|
|
APPS_DIR="${APPS_DIR:-${REPO_ROOT}/apps}"
|
|
RENDER_PY="${RENDER_PY:-${REPO_ROOT}/catalog/render.py}"
|
|
PYTHON="${PYTHON:-python3}"
|
|
|
|
# Catalog services that are intentionally NOT onboarded to the apps tree yet
|
|
# (computed into a wave by the renderer but with NO committed manifest — see
|
|
# DESIGN-waves.md §"Catalog services not yet onboarded"). They render as a
|
|
# by-product of `--all catalog` but must NOT have a golden; the gate ignores
|
|
# their rendered output. Onboarding one = add its golden and drop it here.
|
|
#
|
|
: "${RENDER_GOLDEN_NOT_ONBOARDED:=keycloak vm-demo}"
|
|
|
|
log "Stage: render==golden reproducibility (canonical render vs committed apps/)"
|
|
|
|
if ! entrypoint_ready "$PYTHON"; then
|
|
skip "python3 not available — cannot run the renderer. UNWIRED, not green."
|
|
summary "- render-golden: **SKIPPED** (python3 missing)"
|
|
exit 3
|
|
fi
|
|
if [ ! -f "$RENDER_PY" ]; then
|
|
skip "renderer not present at '${RENDER_PY}' (catalog/render.py) — UNWIRED, not green."
|
|
summary "- render-golden: **SKIPPED** (renderer missing)"
|
|
exit 3
|
|
fi
|
|
# --- The renderer's OWN self-test, before anything is compared ---------------
|
|
# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
|
|
# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
|
|
# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
|
|
# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
|
|
# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
|
|
# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
|
|
# assertions was dev-time only and a merge gate could not tell a sound renderer from a
|
|
# gutted one.
|
|
#
|
|
# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
|
|
# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
|
|
# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
|
|
# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
|
|
# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
|
|
# that chain, and hanging it there would make the check's name and its requirement mapping
|
|
# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
|
|
# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
|
|
#
|
|
# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
|
|
# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
|
|
# memory), so an empty catalog must not be able to skip it.
|
|
#
|
|
# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
|
|
# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
|
|
# output was re-rendered, including one that just lost a contract invariant and had all ten
|
|
# goldens regenerated in the same commit. The self-test catches exactly that.
|
|
#
|
|
# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
|
|
# run its self-test" has no legitimate environmental cause once python3 and render.py are
|
|
# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
|
|
log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
|
|
selftest_out="$(mktemp)"
|
|
if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
|
|
ok "render.py --selftest passed"
|
|
summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
|
|
else
|
|
err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
|
|
err " Every golden under apps/ is produced by this renderer, so the goldens below cannot"
|
|
err " be trusted even if they reproduce byte-for-byte. Reproduce with:"
|
|
err " ${PYTHON} catalog/render.py --selftest"
|
|
sed 's/^/ /' "$selftest_out" >&2 || true
|
|
summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
|
|
rm -f "$selftest_out"
|
|
exit 1
|
|
fi
|
|
rm -f "$selftest_out"
|
|
|
|
if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
|
|
skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
|
|
summary "- render-golden: **SKIPPED** (empty catalog)"
|
|
exit 3
|
|
fi
|
|
|
|
# --- Render every service with the ONE canonical command into a temp dir -----
|
|
TMP_RENDER="$(mktemp -d)"
|
|
trap 'rm -rf "$TMP_RENDER"' EXIT
|
|
|
|
if ! "$PYTHON" "$RENDER_PY" --all "$CATALOG_DIR" -o "$TMP_RENDER" --wave-layout --no-image-automation; then
|
|
err "canonical render failed to run"
|
|
summary "- render-golden: **FAILED** (renderer error)"
|
|
exit 1
|
|
fi
|
|
|
|
# Drop the not-onboarded services' rendered output — they have no golden by design.
|
|
for svc in $RENDER_GOLDEN_NOT_ONBOARDED; do
|
|
find "$TMP_RENDER" -type f -name "${svc}.yaml" -delete
|
|
done
|
|
|
|
# --- Compare EVERY rendered service against its committed golden --------------
|
|
# For each rendered file (relative path wave-N/<name>.yaml):
|
|
# - the committed golden apps/wave-N/<name>.yaml MUST exist, and
|
|
# - it MUST be byte-identical to the freshly rendered output.
|
|
# Any missing golden or any diff is a hard FAIL (blocks merge).
|
|
fail=0
|
|
checked=0
|
|
while IFS= read -r -d '' rendered; do
|
|
rel="${rendered#"$TMP_RENDER"/}"
|
|
golden="${APPS_DIR}/${rel}"
|
|
checked=$((checked + 1))
|
|
if [ ! -f "$golden" ]; then
|
|
err "MISSING GOLDEN: catalog renders '${rel}' but apps/${rel} does not exist"
|
|
err " -> re-render (see canonical command in this script's header) and commit, or add it to RENDER_GOLDEN_NOT_ONBOARDED"
|
|
fail=1
|
|
continue
|
|
fi
|
|
if ! diff -u "$golden" "$rendered" >/dev/null 2>&1; then
|
|
err "GOLDEN DRIFT: apps/${rel} does not match the canonical render:"
|
|
diff -u "$golden" "$rendered" >&2 || true
|
|
fail=1
|
|
fi
|
|
done < <(find "$TMP_RENDER" -type f -print0)
|
|
|
|
if [ "$checked" -eq 0 ]; then
|
|
err "renderer produced no output — cannot verify goldens"
|
|
summary "- render-golden: **FAILED** (renderer produced nothing)"
|
|
exit 1
|
|
fi
|
|
|
|
if [ "$fail" -ne 0 ]; then
|
|
err "render==golden check FAILED — the committed apps/ goldens are NOT reproducible"
|
|
err " from the canonical render. A golden was hand-edited, or a catalog source"
|
|
err " changed without re-rendering. Fix:"
|
|
err " ${PYTHON} catalog/render.py --all catalog -o apps --wave-layout --no-image-automation"
|
|
err " then remove the not-onboarded outputs (${RENDER_GOLDEN_NOT_ONBOARDED// /, }) and commit."
|
|
summary "- render-golden: **FAILED** (goldens drifted from canonical render)"
|
|
exit 1
|
|
fi
|
|
|
|
ok "render==golden: all ${checked} onboarded service golden(s) reproduce from the canonical render"
|
|
summary "- render-golden: **passed** (${checked} services; canonical \`--no-image-automation\` render)"
|
|
exit 0
|