fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate #33

2026-10-02T02:29:54Z

gitea-admin commented

2026-10-02 02:29:54 +00:00

The defect

catalog/render.py --selftest executes 106 assertions on the renderer's output contract —
default-deny covering both directions, the Kyverno namespace hook label, a valid auth-mode, the
oauth2-proxy sidecar's --cookie-name / --cookie-domain / --cookie-refresh flags, the
placeholder-substitution guard, the observability refusals.

Nothing in ci/ or .gitea/ had ever invoked it. Re-measured here, not inherited: grep -rl -- --selftest over both trees returns nothing for the renderer. So all 106 assertions were
dev-time only, and the merge gate could not tell a sound renderer from a gutted one. This is the
top follow-up of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md (PR #28):

a future edit that drops either flag and re-renders the goldens would pass the entire merge
gate.

The change

One file, ci/render-golden.sh: run --selftest and exit 1 when it fails. catalog/render.py
is not touched (another branch owns it this session). No conformance/ path is touched. No
gate stage is added. Nothing is deployed by this — it changes only what the gate refuses.

The lead was wrong; where it actually went

The brief named conformance/tests/l0-static/auditability-proofs-producible/check.sh:291 as the
one-line fix. That line is real and does drive conformance/lib/{record_run,status}.py --selftest
"because nothing else drives them" — but it is the wrong hook. That loop is section (d) of a
check serving R-SEC-03, whose subject is the evidence chain: can a forged or stale run
become an audit proof? record_run.py and status.py are in that chain; catalog/render.py
is not. Hanging the renderer there would make the check's name, its intent annotation and its
requirement mapping untrue, and a future decision about the renderer could then only be expressed
by weakening a security-auditability check.

A new scope-declaring L0 test — the other option that entry offered — is the right shape for a
whole-repo invariant, not for "a script in this repo must pass its own unit test": it would
declare requirements it does not serve, for several hundred lines by this tree's conventions.

ci/render-golden.sh is the honest home — the gate stage whose subject already is
catalog/render.py. The two assertions are independent, and that is the point: render-golden
catches golden-vs-renderer drift but is satisfied by any renderer whose output was
re-rendered, including one that just lost a contract invariant and had all ten goldens
regenerated in the same commit. That is exactly the hole named above. The selftest catches it;
the golden diff cannot.

The stage header and summary were updated so neither is a lie, and a separate
- render-selftest: step-summary line is emitted so the gate output still NAMES what ran. The
selftest runs before the empty-catalog exit 3 because it renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must not be able to
skip it.

Proven to FAIL, not merely to pass

Against mutated copies of the renderer, via the stage's existing RENDER_PY override, so
catalog/render.py was never modified:

Deleting the --cookie-name emission from _oauth2_sidecar_container() — i.e. reverting
PR #28's fleet-wide cookie-collision fix. Stage rc 1. (It dies with an uncaught
StopIteration at render.py:2641 rather than printing its SELFTEST FAIL: lines — correct
verdict, poor diagnostics; logged as a follow-up, the fix is in another branch's file.)
OAUTH2_COOKIE_REFRESH 2m30s -> 10m, widening past the measured 300s access-token
lifetime. Stage rc 1, clean named assertion:
```
[fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
    SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A
    refresh at or past the access-token lifetime never renews in time, which is the same
    logout it was added to stop
```
This is the mutation that proves the wiring's value: render-golden's diff would not catch
it if the goldens were regenerated in the same commit, and --cookie-refresh is one of the two
flags PR #28 recorded as ungated.

And proven to PROPAGATE

Exit codes get flattened in this repo, and .gitea/workflows/gate.yml:138 runs
./ci/gate.sh || [ "$?" -eq 3 ] — a failure degraded to a skip would be swallowed by CI. So the
chain was provoked, in the pinned toolbox, not reasoned about:

[fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
[fail] stage render-golden: FAIL
[fail] render-golden: FAILED -> blocks merge
[fail] MERGE GATE: BLOCKED

gate.sh rc 1. Three links, each measured: selftest rc 1 -> stage rc 1 -> gate rc 1.

Gate on this branch: rc 3 (UNDETERMINED), NOT green

Pinned toolbox, commit d444907:

[ ok ] spec-immutability: green      (the spec repo was MOUNTED — it ran, it did not skip)
[ ok ] approval-policy: green        (locally only: "not a pull_request event" — see below)
[ ok ] decisions-log: green
[ ok ] lint: green
[ ok ] l0: green
[ ok ] render-golden: green
[ ok ] l1: green
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
[fail]   Stages that did not run: gitea-settings

Nothing FAILED. The single cause of the 3 is gitea-settings SKIPPED — no
SETTINGS_API_TOKEN in my environment; CI has it. L2/L3 conformance also skipped
(CLUSTER_AVAILABLE unset) but is a declared gap. Touched requirements
R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06 are UNVERIFIED at L2/L3.

Expect `approval-policy` to FAIL in CI, and that is correct

This diff touches ci/**, a sensitive path in infra/gitea/approval-policy.json ("the
pipeline itself — the thing doing the approving"). The stage is fail-closed and will block
until a non-author approving review exists. A change to the gate cannot be gated by the gate.
required_approvals=1 applies to every PR here regardless of path. The local approval-policy: green above is not evidence — it passed only because there was no pull_request event.

When reading the checks: a skipped Gitea job reports success with the description "Has
been skipped". Read descriptions, not colours.

Not covered

The selftest's quality is unchanged and unratcheted — an empty _selftest() returns 0 and
would still pass. The repo's answer already exists
(conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py injects
regressions and requires --selftest to catch each one); applying it to the renderer is a
second subject and a follow-up. Until then, "the selftest runs" is gated and "the selftest is
falsifiable" rests on two hand-run mutations that are not in the gate.
render.py:2641's next(...) needs a default so mutation 1 reports an assertion instead of a
traceback. One line, in another branch's file.
No cluster access of any kind was used, not even read-only. No apply/edit/patch, no
flux reconcile.

Full honest accounting: decisions/2026-10-01-render-selftest-runs-in-gate.md.

🤖 Generated with Claude Code

## The defect `catalog/render.py --selftest` executes **106 assertions** on the renderer's output contract — default-deny covering both directions, the Kyverno namespace hook label, a valid `auth-mode`, the oauth2-proxy sidecar's `--cookie-name` / `--cookie-domain` / `--cookie-refresh` flags, the placeholder-substitution guard, the observability refusals. **Nothing in `ci/` or `.gitea/` had ever invoked it.** Re-measured here, not inherited: `grep -rl -- --selftest` over both trees returns nothing for the renderer. So all 106 assertions were dev-time only, and the merge gate could not tell a sound renderer from a gutted one. This is the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md` (PR #28): > a future edit that drops either flag *and* re-renders the goldens would pass the entire merge > gate. ## The change One file, `ci/render-golden.sh`: run `--selftest` and `exit 1` when it fails. `catalog/render.py` is **not** touched (another branch owns it this session). No `conformance/` path is touched. No gate stage is added. Nothing is deployed by this — it changes only what the gate refuses. ### The lead was wrong; where it actually went The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the one-line fix. That line is real and does drive `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them" — but it is **the wrong hook**. That loop is section (d) of a check serving **R-SEC-03**, whose subject is the *evidence chain*: can a forged or stale run become an audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not. Hanging the renderer there would make the check's name, its `intent` annotation and its requirement mapping untrue, and a future decision about the renderer could then only be expressed by weakening a security-auditability check. A new scope-declaring L0 test — the other option that entry offered — is the right shape for a **whole-repo invariant**, not for *"a script in this repo must pass its own unit test"*: it would declare requirements it does not serve, for several hundred lines by this tree's conventions. **`ci/render-golden.sh` is the honest home** — the gate stage whose subject already *is* `catalog/render.py`. The two assertions are independent, and that is the point: `render-golden` catches golden-vs-renderer drift but is satisfied by **any** renderer whose output was re-rendered, including one that just lost a contract invariant and had all ten goldens regenerated in the same commit. That is exactly the hole named above. The selftest catches it; the golden diff cannot. The stage header and summary were updated so neither is a lie, and a separate `- render-selftest:` step-summary line is emitted so the gate output still NAMES what ran. The selftest runs *before* the empty-catalog `exit 3` because it renders synthetic ServiceDefinitions in memory and is catalog-independent — an empty catalog must not be able to skip it. ## Proven to FAIL, not merely to pass Against mutated **copies** of the renderer, via the stage's existing `RENDER_PY` override, so `catalog/render.py` was never modified: 1. **Deleting the `--cookie-name` emission** from `_oauth2_sidecar_container()` — i.e. reverting PR #28's fleet-wide cookie-collision fix. Stage rc **1**. (It dies with an uncaught `StopIteration` at `render.py:2641` rather than printing its `SELFTEST FAIL:` lines — correct verdict, poor diagnostics; logged as a follow-up, the fix is in another branch's file.) 2. **`OAUTH2_COOKIE_REFRESH` `2m30s` -> `10m`**, widening past the measured 300s access-token lifetime. Stage rc **1**, clean named assertion: [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract. SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A refresh at or past the access-token lifetime never renews in time, which is the same logout it was added to stop This is the mutation that proves the wiring's value: `render-golden`'s diff would **not** catch it if the goldens were regenerated in the same commit, and `--cookie-refresh` is one of the two flags PR #28 recorded as **ungated**. ## And proven to PROPAGATE Exit codes get flattened in this repo, and `.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]` — a failure degraded to a skip would be swallowed by CI. So the chain was provoked, in the pinned toolbox, not reasoned about: [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract. [fail] stage render-golden: FAIL [fail] render-golden: FAILED -> blocks merge [fail] MERGE GATE: BLOCKED `gate.sh` rc **1**. Three links, each measured: selftest rc 1 -> stage rc 1 -> gate rc 1. ## Gate on this branch: rc 3 (UNDETERMINED), NOT green Pinned toolbox, commit `d444907`: [ ok ] spec-immutability: green (the spec repo was MOUNTED — it ran, it did not skip) [ ok ] approval-policy: green (locally only: "not a pull_request event" — see below) [ ok ] decisions-log: green [ ok ] lint: green [ ok ] l0: green [ ok ] render-golden: green [ ok ] l1: green [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED. [fail] Stages that did not run: gitea-settings Nothing FAILED. The single cause of the 3 is `gitea-settings` **SKIPPED** — no `SETTINGS_API_TOKEN` in my environment; CI has it. L2/L3 `conformance` also skipped (`CLUSTER_AVAILABLE` unset) but is a *declared* gap. Touched requirements **R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are **UNVERIFIED at L2/L3**. ## Expect `approval-policy` to FAIL in CI, and that is correct This diff touches **`ci/**`**, a sensitive path in `infra/gitea/approval-policy.json` ("the pipeline itself — the thing doing the approving"). The stage is **fail-closed** and will block until a **non-author approving review** exists. A change to the gate cannot be gated by the gate. `required_approvals=1` applies to every PR here regardless of path. The local `approval-policy: green` above is **not** evidence — it passed only because there was no `pull_request` event. When reading the checks: a **skipped** Gitea job reports `success` with the description **"Has been skipped"**. Read descriptions, not colours. ## Not covered - The selftest's *quality* is unchanged and unratcheted — an empty `_selftest()` returns 0 and would still pass. The repo's answer already exists (`conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py` injects regressions and requires `--selftest` to catch each one); applying it to the renderer is a second subject and a follow-up. **Until then, "the selftest runs" is gated and "the selftest is falsifiable" rests on two hand-run mutations that are not in the gate.** - `render.py:2641`'s `next(...)` needs a default so mutation 1 reports an assertion instead of a traceback. One line, in another branch's file. - No cluster access of any kind was used, not even read-only. No `apply`/`edit`/`patch`, no `flux reconcile`. Full honest accounting: `decisions/2026-10-01-render-selftest-runs-in-gate.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-02 02:29:54 +00:00

fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate

gate / gate (pull_request) Successful in 1m40s

Details

gate / conformance (pull_request) Has been skipped

Details

d44490736c

`catalog/render.py --selftest` executes 106 assertions on the renderer's output
contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed —
so all 106 were dev-time only, and the merge gate could not tell a sound renderer
from a gutted one. Recorded as the top follow-up of
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.

`ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines
of invocation; the rest is the rationale comment.

WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives
conformance/lib/{record_run,status}.py --selftest "because nothing else drives
them". That is the wrong hook: the check serves R-SEC-03 and its subject is the
evidence chain — can a forged or stale run become an audit proof? The renderer is
not in that chain, and hanging it there would make the check's name, its intent
annotation and its requirement mapping untrue. This stage's subject already IS
catalog/render.py, so the renderer's own contract test belongs in it.

The two assertions are independent, and that is the point: render-golden catches
golden-vs-renderer drift but is satisfied by ANY renderer whose output was
re-rendered — including one that just lost a contract invariant and had all ten
goldens regenerated in the same commit. That is precisely the hole the follow-up
named. The selftest catches it; render-golden cannot.

It runs before the empty-catalog exit 3 because the selftest renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must
not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is
source in this repo, so once python3 and render.py are both present (asserted
above it) there is no legitimate environmental cause for "could not run".

PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via
the stage's existing RENDER_PY override, so catalog/render.py — another branch's
file this session — was never touched:
  * deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie
    fix) -> stage rc 1;
  * OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion
    "parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the
    cookie entry recorded as UNGATED, and one render-golden's diff would miss.

AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage
render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this
repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be
swallowed by CI.

render.py itself is unchanged. No conformance/ path is touched, no gate stage is
added, nothing is deployed by this.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin commented

2026-10-02 02:34:00 +00:00

CI has run on `d444907`. Read the descriptions, not the colours.

gate / gate (pull_request)         failure   "Failing after 1m40s"
gate / conformance (pull_request)  success   "Has been skipped"

The second one is green and certifies nothing — it is the trap CLAUDE.md §2b names.

The first one is red for exactly one reason, read from the job log (run 185, job 0) rather
than from the tick:

[fail] This PR changes the pipeline or the security posture it rests on, and has NO approving
[fail]   review from anyone other than its author ('gitea-admin').
         sensitive paths touched:
           ci/render-golden.sh
[fail] stage approval-policy: FAIL
[fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
[fail] MERGE GATE: BLOCKED

That is fail-closed working, not a defect. A change to the gate cannot be gated by the gate.
It clears when a non-author reviews it.

Every other stage is green in CI, and the thing this PR adds ran there:

[ ok ] stage decisions-log: pass
[ ok ] stage lint: pass
[ ok ] stage l0: pass
[gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
[ ok ] render.py --selftest passed
[ ok ] stage render-golden: pass
[ ok ] stage l1: pass
[ ok ] stage gitea-settings: pass
[skip] stage spec-immutability: SKIPPED (not run)
[skip] stage conformance: SKIPPED (not run)

So the selftest is now executed by CI, on a real PR, with a timestamp — which is the whole point
of the change.

Two notes on the skips, so nobody reads them as coverage:

spec-immutability skipped in CI because the workflow does not check the spec repo out — the
documented tolerated skip. It did run and pass in my local toolbox run, where the spec repo
was mounted, so R-TEST-06 is covered there.
conformance (L2/L3) skipped: vars.CLUSTER_AVAILABLE is unset. Touched requirements
R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06 are UNVERIFIED at runtime.

CI is a stricter run than my local one on every axis except the spec repo (gitea-settings
passes there; it skipped locally for want of a token, which is why my local rc was 3 and not 0),
and it agrees with it.

🤖 Generated with Claude Code

## CI has run on `d444907`. Read the descriptions, not the colours. gate / gate (pull_request) failure "Failing after 1m40s" gate / conformance (pull_request) success "Has been skipped" The second one is green and certifies **nothing** — it is the trap CLAUDE.md §2b names. The first one is red for **exactly one** reason, read from the job log (run 185, job 0) rather than from the tick: [fail] This PR changes the pipeline or the security posture it rests on, and has NO approving [fail] review from anyone other than its author ('gitea-admin'). sensitive paths touched: ci/render-golden.sh [fail] stage approval-policy: FAIL [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge [fail] MERGE GATE: BLOCKED **That is fail-closed working, not a defect.** A change to the gate cannot be gated by the gate. It clears when a non-author reviews it. Every other stage is green in CI, and the thing this PR adds **ran there**: [ ok ] stage decisions-log: pass [ ok ] stage lint: pass [ ok ] stage l0: pass [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions) [ ok ] render.py --selftest passed [ ok ] stage render-golden: pass [ ok ] stage l1: pass [ ok ] stage gitea-settings: pass [skip] stage spec-immutability: SKIPPED (not run) [skip] stage conformance: SKIPPED (not run) So the selftest is now executed by CI, on a real PR, with a timestamp — which is the whole point of the change. Two notes on the skips, so nobody reads them as coverage: - `spec-immutability` skipped in CI because the workflow does not check the spec repo out — the documented tolerated skip. It **did** run and pass in my local toolbox run, where the spec repo was mounted, so R-TEST-06 is covered there. - `conformance` (L2/L3) skipped: `vars.CLUSTER_AVAILABLE` is unset. Touched requirements **R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are **UNVERIFIED at runtime**. CI is a *stricter* run than my local one on every axis except the spec repo (`gitea-settings` passes there; it skipped locally for want of a token, which is why my local rc was 3 and not 0), and it agrees with it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin referenced this issue from a commit

2026-10-02 03:09:05 +00:00

feat(auth): declare API paths per service, so --api-route can ship at all

gitea-admin referenced this pull request

2026-10-02 03:11:07 +00:00

feat(auth): declare API paths per service, so --api-route can ship at all #34

jakman85 approved these changes

2026-10-02 04:29:17 +00:00

jakman85 merged commit 0c7e8dd3ba into main

2026-10-02 04:33:02 +00:00

jakman85 referenced this issue from a commit

2026-10-02 04:33:04 +00:00