--- date: 2026-10-01 branch: fix/render-selftest-runs-in-gate title: catalog/render.py's 106 assertions were dev-time only; the gate now runs them and blocks on them --- ## What changed Single subject, one file: **`ci/render-golden.sh` now invokes `catalog/render.py --selftest`** and exits 1 when it fails. Nothing else. No renderer change, no new conformance test, no new gate stage. This closes the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`: > Nothing in `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the > twenty-odd assertions in that selftest, including the six I just added, are dev-time only: a > future edit that drops either flag *and* re-renders the goldens would pass the entire merge > gate. Re-measured here and confirmed: before this commit, `grep -rl -- --selftest` over `ci/` and `.gitea/` returned **nothing** for the renderer. The selftest executes **106** assertions (`check()` invocations counted at runtime via `sys.settrace`, without editing `render.py`; 109 static call sites, three of them on failure-only branches). All 106 were unexercised by the merge gate. ### The lead was WRONG, and here is why The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the one-line fix. That line is real and it does drive selftests: for tool in ("record_run.py", "status.py"): r = run([sys.executable, str(LIB / tool), "--selftest"]) But it is **the wrong hook**. That loop is section (d) of a check that serves **R-SEC-03**, and its subject is the *evidence chain* — can a self-skipped run, or a stale ledger row, become an audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not. Hanging the renderer there would have made the check's name, its `intent` annotation and its requirement mapping (`R-SEC-03`) all untrue, and a future decision about the renderer could then only be expressed by weakening a *security-auditability* check. That is the precise coupling the cookie-collision entry refused for its own check, and it applies here. The second option that entry offered — a new scope-declaring L0 test — is the supported mechanism for a **whole-repo invariant** (`ci/l0.sh` Stage 3b's `--include-scope whole-repo` union, ratcheted by `tests/l0-static/whole-repo-invariants-wired`). It is the right shape for an invariant quantified over the tracked tree. It is the wrong shape for *"a script in this repo must pass its own unit test"*: it would need to declare requirements it does not serve, and it would be two new files of several hundred lines by this tree's conventions for what is a six-line invocation. **The honest home is `ci/render-golden.sh`** — the gate stage whose subject *already is* `catalog/render.py`. It already resolves `$PYTHON` and `$RENDER_PY`, already reports rc 3 when either is missing, already exits 1 on renderer error, and `ci/gate.sh` already maps it through `for s in lint l0 render-golden l1` where a FAIL is BLOCKED and a SKIP is UNDETERMINED. The stage's two assertions are independent and both are needed: - `render-golden` catches golden-vs-renderer drift, but is satisfied by **any** renderer whose output was re-rendered — including one that just lost a contract invariant and had all ten goldens regenerated in the same commit. That is exactly the hole named in the quote above. - `--selftest` catches that, and is blind to golden drift. The file header and the stage name were updated so neither is a lie: the stage is now "the renderer is trustworthy", with two numbered assertions, and the step summary emits its own `- render-selftest:` line so the gate output still NAMES what ran. The selftest runs **before** the empty-catalog `exit 3` on purpose: it renders synthetic ServiceDefinitions in memory and is catalog-independent, so an empty catalog must not be able to skip it. `catalog/render.py` itself was **not touched** (owned by a concurrent branch this session), and neither was any file under `conformance/`. ## What I verified - **The selftest runs and passes: rc 0, `render.py selftest: PASS`, 106 assertions executed.** Counted at runtime, not from the source: a `sys.settrace` hook counting `check()` calls, in a throwaway script outside the repo, with no edit to `render.py`. - **The stage passes with it wired in**, host run of the single stage: [gate] Stage: render==golden reproducibility (canonical render vs committed apps/) [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions) [ ok ] render.py --selftest passed [ ok ] render==golden: all 10 onboarded service golden(s) reproduce from the canonical render rc **0**. The golden comparison still reports all 10 goldens, so the new invocation did not displace what the stage already asserted. - **IT CAN FAIL — provoked twice, against a mutated COPY of the renderer, never against the tree.** `ci/render-golden.sh` already honours `RENDER_PY`, so each mutant lived in `/tmp/claude-1000/desmond-selftest/` and the repo's `catalog/render.py` was never modified — which matters, because that file belongs to a concurrent branch this session. - **Mutation 1 — the regression this selftest was written for.** Deleted the `f"--cookie-name=_oauth2_proxy_{s['name']}"` emission from `_oauth2_sidecar_container()`, i.e. reverted the fleet-wide cookie-collision fix of PR #28. Stage rc **1**. Note the shape of the failure, because it is a real (minor) weakness and is listed as a follow-up: the selftest died with an uncaught `StopIteration` out of `next(a for a in container["args"] if a.startswith("--cookie-name="))` at `render.py:2641`, rather than printing its collected `SELFTEST FAIL:` lines. It is correctly non-zero and the stage correctly blocks, but the operator gets a traceback instead of the assertion message. - **Mutation 2 — a clean, named assertion failure.** `OAUTH2_COOKIE_REFRESH = "2m30s"` -> `"10m"`, widening the refresh past the measured 300s access-token lifetime. Stage rc **1**: [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract. [fail] Every golden under apps/ is produced by this renderer, so the goldens below cannot [fail] be trusted even if they reproduce byte-for-byte. Reproduce with: [fail] python3 catalog/render.py --selftest SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A refresh at or past the access-token lifetime never renews in time, which is the same logout it was added to stop This is the mutation that proves the wiring's value most directly: mutation 2 would **not** be caught by `render-golden`'s diff if the goldens were regenerated in the same commit, and it is one of the two flags the cookie-collision entry explicitly recorded as **ungated**. - **THE NON-ZERO PROPAGATES ALL THE WAY TO `gate.sh`'s rc — provoked, in the pinned toolbox, not reasoned about.** `RENDER_PY=/desmond-selftest/render-mutant2.py ./ci/gate.sh`: [ ok ] stage spec-immutability: pass [ ok ] stage decisions-log: pass [ ok ] stage lint: pass [ ok ] stage l0: pass [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions) [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract. SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). ... [fail] stage render-golden: FAIL [ ok ] stage l1: pass ... [fail] render-golden: FAILED -> blocks merge [fail] MERGE GATE: BLOCKED `gate.sh` rc **1**. Three links in the chain, each measured: selftest rc 1 -> stage rc 1 -> gate rc 1 / `MERGE GATE: BLOCKED`. This mattered enough to provoke rather than assume: this repo has a documented habit of exit codes getting flattened, and `.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, which tolerates rc 3 — so a stage that degraded a failure to a skip would be swallowed by CI. rc 1 is not tolerated there, and rc 1 is what this produces. - **`spec-immutability` genuinely RAN and passed, it did not skip.** The two-mount worktree shim plus `-e SPEC_REPO=/home-network-k3s` (the fix recorded in the 2026-09-30 cookie entry) works: `stage spec-immutability: pass`, not SKIPPED. R-TEST-06 is therefore actually covered by these runs. - **No collision with PR #29.** The brief warned that `feat/oauth2-proxy-auth-failure-visible` had added checks under `conformance/tests/l0-static/`. Measured — `git diff origin/main...origin/feat/oauth2-proxy-auth-failure-visible --name-only` — it touches **no `conformance/` path at all**: `decisions/`, `platform/oauth2-proxy/` (a promtool test, an alert rule, its kustomization), `runbooks/`. Disjoint from this branch either way, since this branch touches only `ci/render-golden.sh` and this file. - **Priya's files were not touched.** `catalog/render.py`, `schemas/servicedefinition.schema.json`, `apps/wave-*` and `conformance/tests/l1-rendering/` are all untouched by this branch — confirmed by `git diff --name-only`, which lists exactly two paths. - **`catalog/render.py --selftest` was the LAST unwired selftest in the tree.** Measured: `grep -rl -- --selftest` across the repo finds selftest-capable artifacts in exactly four places, and after this commit every one is driven by the gate — `conformance/lib/record_run.py` and `conformance/lib/status.py` by `auditability-proofs-producible/check.sh:291`, the Keycloak client-secret reconciler (embedded in `platform/keycloak/client-secret-reconciler.yaml`) by `keycloak-client-secret-reconciler/check.sh:633`, and `catalog/render.py` by this change. ## What I did NOT verify - **Local gate: rc 3 (UNDETERMINED), NOT 0. This is not green and must not be reported as green.** Measured in the pinned toolbox on `eeb6211`: [ ok ] spec-immutability: green [ ok ] approval-policy: green [ ok ] decisions-log: green [ ok ] lint: green [ ok ] l0: green [ ok ] render-golden: green [ ok ] l1: green [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED. [fail] Stages that did not run: gitea-settings Nothing FAILED. The single cause is `gitea-settings` **SKIPPED** — `SETTINGS_API_TOKEN` is not set in my environment, so branch protection and runner registration were not read; CI has that token and the stage is expected to run there. **L2/L3 `conformance`** also skipped (`CLUSTER_AVAILABLE` unset) but is a *declared* gap that `gate.sh` treats as not-applicable rather than undetermined, so it is not what produced the 3. Touched requirements **R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are therefore **UNVERIFIED at L2/L3**, all four. The stage this branch changes, `render-golden`, is green **and** was independently proven able to turn the whole gate red (above). - **`approval-policy` passed LOCALLY only because there is no pull_request event** (`not a pull_request event (event='none') — approval policy N/A`). That is not evidence that it passes. This diff touches `ci/**`, a sensitive path, so in CI the stage is expected to **FAIL** until a non-author approving review exists. That is §1 working, fail-closed, not a defect. - **The selftest's own quality is unchanged and ungated.** I wired its *invocation*; I did not review its 106 assertions, and nothing ratchets them. Deleting the body of `_selftest()` would still pass this stage, because an empty selftest returns 0. The repo already has the right answer for that — `conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py` injects regressions one at a time and requires `--selftest` to catch every one — and applying that pattern to `catalog/render.py` is the follow-up, deliberately not done here: it is a second subject, it touches `conformance/`, and it would need to live beside the renderer whose file another branch owns this session. **Until it exists, "the selftest runs" is proven and "the selftest is falsifiable" rests on my two hand-run mutations, which are not in the gate.** - **Mutation 1's traceback is not fixed.** `render.py:2641`'s `next(...)` has no default, so the absence of `--cookie-name` raises `StopIteration` and the selftest's collected `SELFTEST FAIL:` lines are never printed. Correct verdict, poor diagnostics, and it suppresses the other assertions that mutation also broke. The one-line fix (`next(..., None)` plus a `check()`) is in `catalog/render.py`, which is another branch's file this session, so it is a follow-up and not a thing I touched. - **Nothing about the live cluster.** No `kubectl` of any kind was run on this branch — not even read-only; this change is offline CI wiring and needed none. No `apply`/`edit`/`patch`, no `flux reconcile`. Merging this deploys nothing: it changes only what the gate refuses. - **CI's own run of this branch.** Quoted in the PR body from the commit's `/statuses`, with descriptions read rather than colours. A *skipped* Gitea job reports `success` with "Has been skipped" and certifies nothing. - **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so even a fully green PR proves the L0/L1 offline subset and nothing about runtime behaviour. - **I did not re-measure branch protection, required approvals, or the runner.** Those are CLAUDE.md §2 claims and §2b says re-measure before relying on them; this branch relies on none of them beyond the PR flow itself, and I read the gate's own output rather than the API.