fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate

`catalog/render.py --selftest` executes 106 assertions on the renderer's output
contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed —
so all 106 were dev-time only, and the merge gate could not tell a sound renderer
from a gutted one. Recorded as the top follow-up of
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.

`ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines
of invocation; the rest is the rationale comment.

WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives
conformance/lib/{record_run,status}.py --selftest "because nothing else drives
them". That is the wrong hook: the check serves R-SEC-03 and its subject is the
evidence chain — can a forged or stale run become an audit proof? The renderer is
not in that chain, and hanging it there would make the check's name, its intent
annotation and its requirement mapping untrue. This stage's subject already IS
catalog/render.py, so the renderer's own contract test belongs in it.

The two assertions are independent, and that is the point: render-golden catches
golden-vs-renderer drift but is satisfied by ANY renderer whose output was
re-rendered — including one that just lost a contract invariant and had all ten
goldens regenerated in the same commit. That is precisely the hole the follow-up
named. The selftest catches it; render-golden cannot.

It runs before the empty-catalog exit 3 because the selftest renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must
not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is
source in this repo, so once python3 and render.py are both present (asserted
above it) there is no legitimate environmental cause for "could not run".

PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via
the stage's existing RENDER_PY override, so catalog/render.py — another branch's
file this session — was never touched:
  * deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie
    fix) -> stage rc 1;
  * OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion
    "parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the
    cookie entry recorded as UNGATED, and one render-golden's diff would miss.

AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage
render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this
repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be
swallowed by CI.

render.py itself is unchanged. No conformance/ path is touched, no gate stage is
added, nothing is deployed by this.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/jackett.yaml

@@ -100,11 +100,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_jackett
+        - --cookie-domain=jackett.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -125,6 +128,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-0/pihole.yaml

@@ -108,11 +108,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_pihole
+        - --cookie-domain=pihole.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -133,6 +136,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-0/qbittorrent.yaml

@@ -171,11 +171,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_qbittorrent
+        - --cookie-domain=qbittorrent.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -196,6 +199,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-1/radarr.yaml

@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_radarr
+        - --cookie-domain=radarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-1/sonarr.yaml

@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

catalog/render.py

@@ -133,6 +133,36 @@ OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
 OAUTH2_PROXY_PORT = 4180
 KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
 
+# --- session lifetime -----------------------------------------------------------------
+# Keycloak's `homelab` realm issues access tokens with a 300-SECOND lifetime. MEASURED on a
+# live session, not read off a config page: `expires - created = 300.000s` exactly, with
+# `refresh_token:true` on the same line — so the proxy HOLDS a refresh token it was never
+# told to spend. `--cookie-refresh` defaults to 0, which the v7.8.1 binary's own help spells
+# "0 to disable", so every one of these sidecars dropped its session five minutes after
+# sign-in and sent the user back to Keycloak. Refresh at HALF the lifetime: two chances to
+# renew before expiry, so one failed refresh is not an immediate logout.
+#
+# NOTE for anyone tempted to add `--session-cookie-minimal` to shrink the cookie: it is
+# MUTUALLY EXCLUSIVE with this (provoked: "cookie_refresh > 0 requires oauth tokens in
+# sessions"). Dropping --pass-access-token is a least-privilege argument, not a size fix.
+OAUTH2_ACCESS_TOKEN_LIFETIME_S = 300
+OAUTH2_COOKIE_REFRESH = "2m30s"
+
+# --- metrics --------------------------------------------------------------------------
+# oauth2-proxy serves /metrics only when --metrics-address is given; with it absent the
+# listener never starts, so NO per-app sidecar was scrapable at all (measured: `ports` was
+# `['http']` on all six). 44180 is the platform's existing convention, not a guess — the
+# SHARED proxy already runs `--metrics-address=0.0.0.0:44180` with a `metrics` port and 2
+# ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
+# which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
+#
+# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
+# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
+# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
+# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+# are added. Do not read this flag as "metrics are working".
+OAUTH2_PROXY_METRICS_PORT = 44180
+
 
 # =============================================================================
 # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -971,7 +1001,11 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     auth.group (R-IDP-06: a valid session is not enough — the user must be in an authorized group,
     else /oauth2 returns 403). client-id/secret + cookie-secret come by name from the SOPS-encrypted
     `<app>-oauth2-proxy` Secret (R-IDP-04). Single provider => --skip-provider-button 302s straight
-    to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy."""
+    to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy.
+
+    --cookie-name and --cookie-domain are DERIVED per app (see the block comment on those two
+    args): every instance holds a different cookie-secret, so sharing one cookie name across the
+    zone makes the instances evict each other's sessions."""
     route = s["ingressRoutes"][0]
     host = route["host"]
     upstream_port = route["port"]
@@ -991,11 +1025,64 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     args += [
         "--skip-provider-button=true",
         f"--redirect-url=https://{host}/oauth2/callback",
-        f"--cookie-domain=.{ZONE}",
+        # ── COOKIE ISOLATION (fleet fix 2026-09-30; crafty's 2026-09-29 fix, generalized) ──────
+        # EVERY oauth2-proxy on this platform — this sidecar, its four siblings, and the SHARED
+        # forward-auth proxy (platform/oauth2-proxy) — is an INDEPENDENT instance holding its OWN
+        # `cookie-secret`. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
+        # DISTINCT cookie-secret values, so no instance can decrypt any other's cookie.
+        #
+        # Left at oauth2-proxy's DEFAULTS (`_oauth2_proxy` / `_oauth2_proxy_csrf`) and scoped to
+        # `.${BASE_DOMAIN}`, the browser therefore keeps ONE cookie pair for ALL of them:
+        # whichever proxy authenticates last OVERWRITES every other's session. The loser logs
+        # "No valid authentication in request" and bounces to Keycloak, which — the SSO session
+        # still being live — redirects straight back and overwrites the cookie again. The
+        # user-visible symptom is "I signed into Keycloak but I'm still not signed in", and it
+        # sends you hunting in Keycloak, where nothing is wrong. Silent, intermittent, and
+        # misattributed: the exact shape this repo keeps getting burned by.
+        #
+        # WHY IT WAS *INTERMITTENT*, AND WHY RELOADING NEVER HELPED. The loser does not merely
+        # fail to read the cookie — it calls ClearSession, which EXPIRES the cookie on its own
+        # --cookie-domain, i.e. the whole zone. And the *arr UIs poll: measured on prod, radarr's
+        # rejections are `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate`,
+        # 155 of them in 12h against 1 successful auth. Every poll carrying a cookie radarr
+        # cannot decrypt calls ClearSession again — so A LEFT-OPEN RADARR TAB DELETES JACKETT'S
+        # COOKIE EVERY FEW SECONDS, from another namespace, with no user action at all. That is
+        # the intermittency, and it is why signing in again fixed nothing for longer than the
+        # gap between two polls. Host-scoping the cookie (below) is what contains the blast
+        # radius of a ClearSession to the app that issued it.
+        #
+        # DERIVED, NOT LISTED. Both flags come from the ServiceDefinition (`s['name']` and the
+        # route host), so a NEW forward-auth app is isolated the moment it is onboarded. A
+        # per-service table would leave the next app defaulting into the collision, which is how
+        # twelve ServiceMonitors acquired one identical wrong port name in a single edit.
+        # conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct asserts both properties
+        # over the WHOLE tree and asserts THIS function derives rather than lists.
+        #
+        #   --cookie-name   a unique name per instance, so two sessions cannot alias at all.
+        #                   This is the fix; the one below is defence in depth.
+        #   --cookie-domain HOST-scoped. A per-app reverse proxy has no reason to broadcast its
+        #                   session cookie to every other host in the zone — a zone-wide cookie
+        #                   is transmitted to every sibling app on `.${BASE_DOMAIN}`, so any
+        #                   FUTURE name collision would again be platform-wide instead of
+        #                   contained. The SHARED forward-auth proxy keeps the wide scope,
+        #                   because cross-host forward-auth is precisely its job.
+        #
+        # --whitelist-domain deliberately stays ZONE-WIDE: it governs permitted post-login
+        # REDIRECT targets, not cookie scope. Narrowing it here would break the sign-in bounce.
+        f"--cookie-name=_oauth2_proxy_{s['name']}",
+        f"--cookie-domain={host}",
         f"--whitelist-domain=.{ZONE}",
         "--cookie-secure=true",
+        # Spend the refresh token the proxy already holds, at half the 300s access-token
+        # lifetime. Without this the session dies 5 minutes after sign-in. See
+        # OAUTH2_ACCESS_TOKEN_LIFETIME_S above for the measurement and the
+        # --session-cookie-minimal incompatibility.
+        f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
         "--pass-access-token=true",
         "--set-xauthrequest=true",
+        # Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
+        # see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
+        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
     ]
     secret = f"{s['name']}-oauth2-proxy"
     return {
@@ -1017,7 +1104,10 @@ def _oauth2_sidecar_container(s: dict) -> dict:
                 "valueFrom": {"secretKeyRef": {"name": secret, "key": "cookie-secret"}},
             },
         ],
-        "ports": [{"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"}],
+        "ports": [
+            {"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"},
+            {"name": "metrics", "containerPort": OAUTH2_PROXY_METRICS_PORT, "protocol": "TCP"},
+        ],
         "resources": {
             "requests": {"cpu": "25m", "memory": "64Mi"},
             "limits": {"memory": "128Mi"},
@@ -2475,6 +2565,86 @@ def _selftest() -> int:
         not any(a.startswith("--ssl-upstream-insecure-skip-verify") for a in side["args"]),
         "http upstream must NOT skip TLS verify (that flag is only for crafty's self-signed https)",
     )
+    # COOKIE ISOLATION. Every oauth2-proxy instance holds its own cookie-secret, so two instances
+    # sharing a cookie NAME on a shared DOMAIN evict each other's sessions silently. Both flags
+    # must be DERIVED from this service, not defaulted and not listed.
+    check(
+        "--cookie-name=_oauth2_proxy_fa" in side["args"],
+        "sidecar must declare a cookie name DERIVED from the service name — the oauth2-proxy "
+        "default (_oauth2_proxy) is shared with every other instance on the zone",
+    )
+    check(
+        "--cookie-domain=fa.home.keithjackson.dev" in side["args"],
+        "sidecar cookie-domain must be HOST-scoped (the app's own host), not zone-wide",
+    )
+    check(
+        f"--whitelist-domain=.{ZONE}" in side["args"],
+        "whitelist-domain must stay ZONE-wide — it governs post-login redirect targets, not "
+        "cookie scope; narrowing it breaks the sign-in bounce",
+    )
+    # SESSION LIFETIME. `--cookie-refresh` must be present AND strictly shorter than the
+    # measured 300s access-token lifetime, or the session dies mid-use. Asserted as a parsed
+    # duration rather than a string match, so "2m30s" -> "150s" -> "2.5m" stays green and a
+    # silent widening past the token lifetime does not.
+    refresh = next((a.split("=", 1)[1] for a in side["args"]
+                    if a.startswith("--cookie-refresh=")), None)
+    check(refresh is not None,
+          "sidecar must set --cookie-refresh; the v7.8.1 default is 0 = DISABLED, so the "
+          "session dies when the access token expires and the user is bounced to Keycloak")
+    if refresh is not None:
+        _m = re.fullmatch(r"(?:(\d+)m)?(?:(\d+(?:\.\d+)?)s)?", refresh)
+        _secs = (int(_m.group(1) or 0) * 60 + float(_m.group(2) or 0)) if _m else -1
+        check(0 < _secs < OAUTH2_ACCESS_TOKEN_LIFETIME_S,
+              f"--cookie-refresh={refresh} parses to {_secs}s, which is not inside "
+              f"(0, {OAUTH2_ACCESS_TOKEN_LIFETIME_S}s). A refresh at or past the access-token "
+              f"lifetime never renews in time, which is the same logout it was added to stop")
+    check(
+        not any(a.startswith("--session-cookie-minimal") for a in side["args"]),
+        "--session-cookie-minimal is MUTUALLY EXCLUSIVE with --cookie-refresh "
+        "('cookie_refresh > 0 requires oauth tokens in sessions') — the proxy refuses to start",
+    )
+    # METRICS. The listener does not exist unless the flag is given, and the port must be
+    # DECLARED or nothing can ever select it (neither a Service port nor a PodMonitor).
+    check(
+        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}" in side["args"],
+        "sidecar must start the /metrics listener; without --metrics-address it is not merely "
+        "unscraped, it is not serving at all",
+    )
+    check(
+        [p["name"] for p in side["ports"]] == ["http", "metrics"]
+        and {p["containerPort"] for p in side["ports"]}
+        == {OAUTH2_PROXY_PORT, OAUTH2_PROXY_METRICS_PORT},
+        "sidecar must declare BOTH the http and metrics container ports — an undeclared port "
+        "cannot be named by a Service port or a PodMonitor, so the flag alone is inert",
+    )
+    # Two forward-auth apps must not be able to collide. Renders a SECOND one and compares, so a
+    # future refactor to a constant (or to a per-service table with a copy-paste slip) fails here
+    # rather than silently in a browser six weeks later.
+    fa2_side = next(
+        c
+        for c in next(
+            d[1]
+            for d in render(
+                {
+                    "name": "fa2",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "fa2.home.keithjackson.dev", "port": 9118}],
+                    "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+                }
+            )
+            if d[1]["kind"] == "Deployment"
+        )["spec"]["template"]["spec"]["containers"]
+        if c["name"] == "oauth2-proxy"
+    )
+
+    def _cookie_name(container):
+        return next(a for a in container["args"] if a.startswith("--cookie-name="))
+
+    check(
+        _cookie_name(side) != _cookie_name(fa2_side),
+        "two forward-auth apps rendered IDENTICAL cookie names — they would overwrite each "
+        "other's session on every sign-in (the 2026-09-29 crafty defect, fleet-wide)",
+    )
     check(
         side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
         "sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",

ci/render-golden.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
-# ci/render-golden.sh — Stage: render==golden reproducibility gate.
+# ci/render-golden.sh — Stage: the renderer is trustworthy.
+#   (1) catalog/render.py passes its OWN output-contract self-test, and
+#   (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
+# Two independent assertions about one subject, catalog/render.py. (1) was added
+# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
+# block above the invocation for why it lives here and not in a conformance check.
 #
 # WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
 # invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
   summary "- render-golden: **SKIPPED** (renderer missing)"
   exit 3
 fi
+# --- The renderer's OWN self-test, before anything is compared ---------------
+# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
+# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
+# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
+# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
+# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
+# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
+# assertions was dev-time only and a merge gate could not tell a sound renderer from a
+# gutted one.
+#
+# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
+# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
+# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
+# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
+# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
+# that chain, and hanging it there would make the check's name and its requirement mapping
+# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
+# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
+#
+# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
+# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
+# memory), so an empty catalog must not be able to skip it.
+#
+# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
+# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
+# output was re-rendered, including one that just lost a contract invariant and had all ten
+# goldens regenerated in the same commit. The self-test catches exactly that.
+#
+# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
+# run its self-test" has no legitimate environmental cause once python3 and render.py are
+# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
+log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
+selftest_out="$(mktemp)"
+if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
+  ok "render.py --selftest passed"
+  summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
+else
+  err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
+  err "  Every golden under apps/ is produced by this renderer, so the goldens below cannot"
+  err "  be trusted even if they reproduce byte-for-byte. Reproduce with:"
+  err "    ${PYTHON} catalog/render.py --selftest"
+  sed 's/^/    /' "$selftest_out" >&2 || true
+  summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
+  rm -f "$selftest_out"
+  exit 1
+fi
+rm -f "$selftest_out"
+
 if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
   skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
   summary "- render-golden: **SKIPPED** (empty catalog)"

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/chainsaw-test.yaml

@@ -0,0 +1,152 @@
+# =============================================================================
+# L0 STATIC — no two oauth2-proxy instances can share a browser cookie
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. Every oauth2-proxy on this platform is an INDEPENDENT instance with
+# its OWN `cookie-secret`: five per-app reverse-proxy sidecars (jackett, pihole, qbittorrent,
+# radarr, sonarr), crafty's, and the SHARED forward-auth proxy. Measured on prod 2026-09-30:
+# seven `*-oauth2-proxy` Secrets, SEVEN DISTINCT cookie-secret values — so no instance can
+# decrypt any other's cookie.
+#
+# They were all emitting oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and
+# `_oauth2_proxy_csrf`), scoped to `.${BASE_DOMAIN}`. A cookie is keyed by (name, domain, path),
+# so the browser kept ONE pair for ALL of them. Whichever proxy authenticated last OVERWROTE
+# every other's session; the loser logged `No valid authentication in request`, bounced to
+# Keycloak, and — the SSO session still being live — was redirected straight back and overwrote
+# the cookie again. Six live instances, one cookie, mutual eviction.
+#
+# WHY IT IS THE WORST KIND OF BUG. The symptom is "I signed into Keycloak but I'm still not
+# signed in", which points the investigator at Keycloak, where nothing is wrong. It is
+# INTERMITTENT — it only appears once a browser has visited a SECOND gated app, so the obvious
+# single-app reproduction succeeds — and it is SILENT: every Deployment stays Ready, every
+# Kustomization stays Ready, the proxy's own log line names an authentication failure rather
+# than a collision. It went unnoticed across five services for the whole life of the pattern,
+# and was found only as a side effect of investigating something else (2026-09-29, crafty).
+#
+# WHY A COMMIT IS NOT ENOUGH, AND THIS IS A GATE STAGE. The collision is a property of the
+# PATTERN, not of any one app: the next forward-auth service onboarded inherits the defect by
+# doing nothing at all, because a missing flag IS the bug. Nothing about its absence is loud.
+# This is the same shape as the twelve inert ServiceMonitors in this tree — one omission,
+# copied, invisible for weeks — and the same remedy: make the omission fail the merge.
+#
+# GIVEN every oauth2-proxy instance this repo authors — a container in any manifest (the
+#       rendered sidecars in apps/, crafty's hand-authored one in platform/) and the shared
+#       proxy's HelmRelease values,
+# WHEN  each instance's EFFECTIVE cookie name is resolved (its declared `--cookie-name` /
+#       `OAUTH2_PROXY_COOKIE_NAME`, or oauth2-proxy's built-in default `_oauth2_proxy` when it
+#       declares none — an omission is a NAME, not an absence, and that is the whole bug),
+# THEN  (1) no two instances share an effective cookie name;
+#       (2) no instance's effective name is another's name + `_csrf` — oauth2-proxy derives the
+#           CSRF cookie that way, so `_oauth2_proxy` and `_oauth2_proxy_csrf` as two SESSION
+#           names would collide on the second cookie while looking distinct on the first;
+#       (3) the DEFAULT name has exactly ONE owner and it is the instance declared in
+#           SHARED_DEFAULT_COOKIE_OWNER. This is the deliberate ASYMMETRY. The shared
+#           forward-auth proxy MUST keep `_oauth2_proxy`: it is the single session every
+#           OTHER forward-auth-gated surface on the platform rides on, so renaming it signs
+#           every one of them out at once. The rule is therefore NOT "every instance must
+#           declare a name" — it is "the default is a NAME LIKE ANY OTHER, held by exactly one
+#           REGISTERED owner". That formulation needs no exemption from (1), and it closes the
+#           hole an exemption would leave: if the shared proxy is ever removed, an app silently
+#           inheriting the default would still FAIL here instead of quietly becoming its heir;
+#       (4) every instance that is NOT that owner is HOST-SCOPED: its `--cookie-domain` equals
+#           the host of its own `--redirect-url`. See the argument below;
+#       (5) catalog/render.py DERIVES both flags from the ServiceDefinition (the service name
+#           and the route host) and does not carry a per-service table. A list is how twelve
+#           ServiceMonitors acquired one identical wrong port name in a single edit, and a list
+#           leaves the NEXT app defaulting into the collision;
+#       (6) the SHIPPED renderer is proven END-TO-END on synthetic ServiceDefinitions, run as a
+#           subprocess: two forward-auth fixtures must yield two different, non-default cookie
+#           names, each scoped to its own host. Without this, once the tree is compliant (1)-(4)
+#           are satisfied by a tree with nothing left to catch, and a guard whose only evidence
+#           is "nothing failed" cannot be told apart from one that stopped working;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope:
+#           whole-repo`, so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into
+#           EVERY gate run, and ci/gate.sh blocks the merge when the l0 stage fails. It
+#           deliberately adds NO stage to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired
+#           fails the gate if anyone adds one.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps `apps/*` to R-IDP-02/R-IDP-06
+# today, but the instances live in THREE unrelated subtrees (apps/, platform/crafty/,
+# platform/oauth2-proxy/) and the file most likely to reintroduce the defect is
+# catalog/render.py, which maps to R-GITOPS-02 and to no identity requirement at all. A
+# requirement-scoped selection would filter this check out of exactly the change that breaks it.
+#
+# ON ASSERTION (4), HOST-SCOPED COOKIE-DOMAIN — the argument, and the decision.
+#   AGAINST asserting it (overreach): distinct NAMES already close the eviction bug completely.
+#   Two cookies with different names do not overwrite each other however wide their domains are,
+#   so (4) fixes nothing that (1) has not already fixed. `--cookie-domain` is also a legitimate
+#   knob — a future proxy fronting two hostnames of one app would need a wider value — and a
+#   check that fails correct configurations is a check someone eventually disables.
+#   FOR asserting it (and this wins): a zone-wide cookie is TRANSMITTED BY THE BROWSER TO EVERY
+#   HOST IN THE ZONE. qBittorrent's session token is therefore sent to pihole, to radarr, to
+#   sonarr, and to every other name under `.${BASE_DOMAIN}` — including hosts behind no proxy at
+#   all. That is a session-credential disclosure across trust boundaries that no one needs and
+#   nobody asked for; the sidecar's session is only ever meaningful to the one app it fronts.
+#   And it is what makes a FUTURE name collision catastrophic rather than contained: with
+#   host-scoped cookies, two instances that accidentally share a name cannot even see each
+#   other's cookie, so the failure this whole test exists to prevent degrades from platform-wide
+#   to nothing. Defence in depth on the exact axis that already failed once.
+#   DECISION: assert it, for every instance except the registered shared proxy — whose whole
+#   job IS cross-host forward-auth, and which is exempted by name and with a reason, not by
+#   loosening the rule. `--whitelist-domain` is deliberately NOT touched: it governs permitted
+#   post-login REDIRECT targets, not cookie scope, and narrowing it breaks the sign-in bounce.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py
+# and the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it
+# directly, and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is STATIC, with no cluster and no browser):
+#   - That anyone can actually sign in. No browser round-trip is performed, and none can be
+#     from L0. The 2026-09-29 crafty entry recorded the same gap and it is still open: the
+#     symptom has never been REPRODUCED in a browser, so this is a latent-bug fix reasoned from
+#     measured configuration, not a fix watched to work.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b). The
+#     live flags were measured separately, once, by hand; nothing here re-measures them, and a
+#     hand-applied drift would not be visible to this check.
+#   - That the cookie-secrets really differ. That is a Secret comparison and belongs in-cluster;
+#     it was measured out of band (seven distinct values, compared without printing any of
+#     them). If they were ever made IDENTICAL the collision would become harmless — but sharing
+#     one session-encryption key across trust zones is a worse idea than the bug it would mask,
+#     so this check does not offer it as an alternative and does not look for it.
+#   - Anything about the CONTENT of a session, group authorization, or the CSRF cookie's own
+#     behaviour. Names and domains only.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-cookie-names-distinct
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+    conformance.home.k3s/R-IDP-06: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02, R-IDP-06"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the instances span apps/wave-0, apps/wave-1, platform/crafty and
+    # platform/oauth2-proxy, and the generator that emits five of them is catalog/render.py.
+    # No single requirement path-maps to that set, and the renderer maps to no identity
+    # requirement at all, so a requirement-scoped selection would skip this check on precisely
+    # the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No two oauth2-proxy instances can share a browser cookie: every instance's effective
+      cookie name is unique, the built-in default belongs to exactly one registered owner (the
+      shared forward-auth proxy, which cannot be renamed without signing every gated app out),
+      every other instance is host-scoped, and catalog/render.py derives both flags from the
+      ServiceDefinition rather than from a per-service list — proven end-to-end on synthetic
+      fixtures so the guard stays falsifiable once the tree is clean.
+spec:
+  steps:
+    - name: assert-no-two-oauth2-proxy-instances-share-a-cookie
+      description: >-
+        Inventory every oauth2-proxy container and HelmRelease in the tree, resolve each one's
+        effective cookie name (declared, or oauth2-proxy's default when omitted), assert
+        pairwise distinctness including the session/CSRF derivation, ratchet the single
+        registered owner of the default name, assert host-scoped cookie-domain on every other
+        instance, assert catalog/render.py derives rather than lists, and prove the shipped
+        renderer on two synthetic forward-auth ServiceDefinitions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/check.sh

@@ -0,0 +1,498 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no two oauth2-proxy instances can share a browser cookie.
+# See chainsaw-test.yaml's header for WHY this exists, what it measured, and the argument
+# for assertion (4). In one paragraph:
+#
+# Every oauth2-proxy here holds its OWN cookie-secret (seven distinct values on prod,
+# 2026-09-30), so two instances sharing a cookie NAME on a shared DOMAIN cannot read each
+# other's session — they OVERWRITE it. The loser bounces to Keycloak, is redirected straight
+# back by the live SSO session, and overwrites the winner in turn. The symptom is "I signed
+# into Keycloak but I'm still not signed in", it only shows up after a second gated app is
+# visited, and nothing in the cluster reports an error. A missing --cookie-name IS the bug,
+# so the next forward-auth app inherits it by doing nothing.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner
+# turns into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope)
+# only if the tree contains no oauth2-proxy instance at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+# Matches the IMAGE every instance runs and the CHART the shared one installs; if NEITHER
+# appears anywhere, the forward-auth pattern has left this repo and the check has no subject.
+#
+# ANCHORED ON `image:` DELIBERATELY. A bare `oauth2-proxy/oauth2-proxy:` also matches
+# platform/oauth2-proxy/forward-auth-cec.yaml's Envoy cluster reference
+# (`serviceName: oauth2-proxy/oauth2-proxy:80` — namespace/service:port, not a registry path).
+# That file configures the ext_authz listener that CALLS the proxy; it runs no proxy and holds
+# no cookie flags, so counting it made the grep/parse cross-check below report a phantom
+# unaudited instance.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy|chart:[[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" "${ROOT}/catalog" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy image or chart found under apps/, platform/ or catalog/ — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: no two oauth2-proxy instances can share a browser cookie.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather
+than line-based parsing: a container's args live in a nested structure and a HelmRelease's
+flags live in a values map, and a line-based reader of either is precisely the approximate
+parser that silently stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# oauth2-proxy's OWN built-in defaults (pkg/apis/options/cookie.go): the session cookie is
+# `_oauth2_proxy` and the CSRF cookie is that name with `_csrf` appended. An instance that
+# declares nothing is NOT unconfigured — it has claimed this exact name, which is why the
+# inventory below treats the default as a name like any other rather than as an absence.
+DEFAULT_COOKIE_NAME = "_oauth2_proxy"
+CSRF_SUFFIX = "_csrf"
+
+# =============================================================================================
+# REGISTER — SHARED_DEFAULT_COOKIE_OWNER. Exactly one entry, and it must match a real instance.
+#
+# The one instance permitted to hold DEFAULT_COOKIE_NAME, and the one permitted a zone-wide
+# cookie-domain. Both exemptions belong to the SAME object for the SAME reason, so they are one
+# register rather than two: the shared forward-auth proxy is not a per-app sidecar, it is the
+# single session that every forward-auth-gated surface on the platform rides on. Renaming its
+# cookie signs all of them out simultaneously, and host-scoping it would break the cross-host
+# forward-auth that is its entire job.
+#
+# Shrink-only, and exact in BOTH directions: an entry that matches no instance FAILS as stale
+# (the proxy was renamed or removed and the exemption is now a hole), and an instance holding
+# the default WITHOUT an entry FAILS as an unregistered heir. Adding an entry is a review event
+# on a sensitive path.
+#
+# Keyed by "<relpath>:<instance id>".
+# =============================================================================================
+SHARED_DEFAULT_COOKIE_OWNER: dict[str, str] = {
+    "platform/oauth2-proxy/helmrelease.yaml:oauth2-proxy":
+        "The SHARED forward-auth proxy (ADR-0014, platform/oauth2-proxy/README.md). It is the "
+        "single authenticator behind every forward-auth-gated surface that is NOT fronted by "
+        "its own sidecar, so its cookie is the platform session: renaming it signs every one "
+        "of them out at once, and that is why the 2026-09-29 crafty fix changed only the "
+        "sidecar side. Its cookie-domain is zone-wide for the same structural reason — "
+        "cross-host forward-auth is what it is for.",
+}
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
+
+    Auditing a golden would be auditing a copy of something already audited through apps/.
+    Same exclusion the sibling l0-static checks make, and the same "reference, not
+    declaration" distinction ci/touched-requirements.sh draws for fixture paths.
+    """
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: a sidecar sits under spec.template.spec.containers in a
+    Deployment, but the same container shape appears under a CronJob's jobTemplate, a
+    DaemonSet, or a Pod, and a reader that hardcoded one path would silently stop seeing an
+    instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def flag_from_args(args: list, flag: str):
+    """Value of `--flag=value` or `--flag value` in an argv list, else None."""
+    pref = "--" + flag
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith(pref + "="):
+            return a.split("=", 1)[1]
+        if a == pref and i + 1 < len(args) and isinstance(args[i + 1], str):
+            return args[i + 1]
+    return None
+
+
+def host_of(url):
+    """Host part of an https://host/path URL, else None."""
+    if not isinstance(url, str):
+        return None
+    m = re.match(r"^https?://([^/]+)", url)
+    return m.group(1) if m else None
+
+
+# ============================================================ (0) INVENTORY
+# instances: list of dicts {rel, ident, kind, declared_name, cookie_domain, redirect_host}
+instances: list[dict] = []
+unparseable: list[str] = []
+# Files the grep gate says contain an instance, so the parse can be cross-checked against it.
+grep_files: set[str] = set()
+
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES
+    # such as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent
+    # runs, so a bare rglob audited up to 18 duplicates of every file — which both invents
+    # FAILs and, worse, can satisfy a "something declares X" check from a copy rather than from
+    # real source. Same fix as egress-paths-selected-by-rules. .claude/ is gitignored.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if not txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    # Same anchoring as check.sh's skip gate: `image:` (or a chart name), never a bare path —
+    # see the comment there for the Envoy `serviceName: oauth2-proxy/oauth2-proxy:80` near-miss.
+    if re.search(r"image:\s*\S*oauth2-proxy|chart:\s*oauth2-proxy", txt):
+        grep_files.add(rel)
+    elif "oauth2-proxy" not in txt:
+        continue
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        unparseable.append(f"{rel} ({exc})")
+        continue
+    for d in docs:
+        # --- shape 1: a container running the oauth2-proxy image -------------------------
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            args = [a for a in (list(c.get("args") or []) + list(c.get("command") or []))]
+            env = {e.get("name"): e.get("value")
+                   for e in (c.get("env") or []) if isinstance(e, dict) and "value" in e}
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            instances.append({
+                "rel": rel,
+                "ident": f"{owner}/{name or 'oauth2-proxy'}",
+                "kind": "container",
+                "declared_name": flag_from_args(args, "cookie-name")
+                                 or env.get("OAUTH2_PROXY_COOKIE_NAME"),
+                "cookie_domain": flag_from_args(args, "cookie-domain")
+                                 or env.get("OAUTH2_PROXY_COOKIE_DOMAIN"),
+                "redirect_host": host_of(flag_from_args(args, "redirect-url")
+                                         or env.get("OAUTH2_PROXY_REDIRECT_URL")),
+            })
+        # --- shape 2: the shared proxy's HelmRelease values ------------------------------
+        if d.get("kind") != "HelmRelease":
+            continue
+        spec = d.get("spec") or {}
+        chart = (((spec.get("chart") or {}).get("spec") or {}).get("chart") or "")
+        if "oauth2-proxy" not in str(chart):
+            continue
+        values = spec.get("values") or {}
+        extra = values.get("extraArgs") or {}
+        if isinstance(extra, list):          # the chart also accepts a bare argv list
+            extra = {a.lstrip("-").split("=", 1)[0]: a.split("=", 1)[1]
+                     for a in extra if isinstance(a, str) and "=" in a}
+        cfg = values.get("config") or {}
+        get = lambda k: extra.get(k, cfg.get(k))     # noqa: E731 - two-source lookup, used 3x
+        instances.append({
+            "rel": rel,
+            "ident": str((d.get("metadata") or {}).get("name") or "oauth2-proxy"),
+            "kind": "helmrelease",
+            "declared_name": get("cookie-name"),
+            "cookie_domain": get("cookie-domain"),
+            "redirect_host": host_of(get("redirect-url")),
+        })
+
+for u in unparseable:
+    check(False, f"{u} is not parseable YAML — every oauth2-proxy instance in it went unaudited")
+
+for ins in instances:
+    ins["key"] = f"{ins['rel']}:{ins['ident']}"
+    # THE CORE MODELLING DECISION: an omitted --cookie-name is not "unset", it is a CLAIM on
+    # oauth2-proxy's built-in default. Resolving it here is what lets assertion (1) below be a
+    # plain distinctness test with no special case, and it is exactly the step whose absence
+    # made the defect invisible to every reviewer who read these manifests.
+    ins["effective_name"] = ins["declared_name"] or DEFAULT_COOKIE_NAME
+
+# Anti-vacuous. The floor is deliberately low (a real removal should not turn this check red),
+# so the load-bearing guard is the grep/parse CROSS-CHECK beneath it: every file the cheap
+# text scan says holds an instance must yield at least one PARSED instance. A parser that
+# quietly stopped matching would keep the floor and fail this.
+check(len(instances) >= 5,
+      f"only {len(instances)} oauth2-proxy instance(s) parsed out of the tree. This repo "
+      f"authors more than that (five rendered sidecars, crafty's, and the shared proxy), so "
+      f"the inventory has stopped seeing instances and every assertion below would pass "
+      f"vacuously while a collision went unaudited")
+
+parsed_files = {ins["rel"] for ins in instances}
+for rel in sorted(grep_files - parsed_files):
+    check(False,
+          f"{rel} contains an oauth2-proxy image or chart reference but NO instance was parsed "
+          f"out of it. Either it uses a shape iter_containers()/the HelmRelease reader does not "
+          f"recognise, or the file stopped parsing. Its cookie configuration is UNAUDITED, which "
+          f"is the same state the whole fleet was in before this check existed")
+
+# ============================================================ (1) DISTINCT EFFECTIVE NAMES
+by_name: dict[str, list[dict]] = {}
+for ins in instances:
+    by_name.setdefault(ins["effective_name"], []).append(ins)
+
+for nm, group in sorted(by_name.items()):
+    if len(group) < 2:
+        continue
+    who = ", ".join(sorted(g["key"] for g in group))
+    default_note = (
+        " Every one of them OMITS --cookie-name, so they are all claiming oauth2-proxy's "
+        "built-in default. Omitting the flag is not neutral: it IS this collision."
+        if nm == DEFAULT_COOKIE_NAME else "")
+    check(False,
+          f"{len(group)} oauth2-proxy instances share the cookie name '{nm}': {who}.{default_note}\n"
+          f"       WHY THIS IS A BUG AND NOT A STYLE RULE: these are INDEPENDENT instances with "
+          f"DIFFERENT cookie-secrets (measured on prod 2026-09-30: seven distinct values), so "
+          f"neither can DECRYPT the other's cookie — it can only OVERWRITE it. A browser keys a "
+          f"cookie by (name, domain, path), so signing into one app destroys the session of every "
+          f"other app sharing that key. The loser logs 'No valid authentication in request', "
+          f"bounces to Keycloak, is redirected straight back because the SSO session is still "
+          f"live, and overwrites the winner in turn. The user sees 'I signed into Keycloak but "
+          f"I'm still not signed in' and goes looking in Keycloak, where nothing is wrong.\n"
+          f"       FIX: give each per-app instance its own --cookie-name. For a service rendered "
+          f"from catalog/, do NOT hand-edit the golden — catalog/render.py derives the flag from "
+          f"the service name; re-render instead (ci/render-golden.sh's header has the canonical "
+          f"command). Exactly one instance may hold '{DEFAULT_COOKIE_NAME}', and it must be the "
+          f"one registered in SHARED_DEFAULT_COOKIE_OWNER.")
+
+# ============================================================ (2) SESSION vs CSRF DERIVATION
+# oauth2-proxy names the CSRF cookie <cookie-name>_csrf. So a SESSION name that happens to equal
+# another instance's name + "_csrf" collides on the CSRF cookie while assertion (1) sees two
+# perfectly distinct strings. This is the near-miss the obvious fix invites: naming the shared
+# proxy's neighbour `_oauth2_proxy_csrf` reads as "clearly different" and is not.
+effective = {ins["effective_name"] for ins in instances}
+for ins in sorted(instances, key=lambda i: i["key"]):
+    base = ins["effective_name"]
+    if not base.endswith(CSRF_SUFFIX):
+        continue
+    victim = base[: -len(CSRF_SUFFIX)]
+    check(victim not in effective,
+          f"{ins['key']} uses the SESSION cookie name '{base}', which is exactly the CSRF cookie "
+          f"oauth2-proxy derives for the instance named '{victim}' (it appends '{CSRF_SUFFIX}'). "
+          f"The two session names look distinct and the CSRF cookies collide, so sign-in breaks "
+          f"with a state-mismatch instead of a session loss — the same bug wearing a different "
+          f"error message. Pick a name that is not another instance's name + '{CSRF_SUFFIX}'")
+
+# ============================================================ (3) THE DEFAULT HAS ONE OWNER
+default_holders = sorted(ins["key"] for ins in instances
+                         if ins["effective_name"] == DEFAULT_COOKIE_NAME)
+registered = set(SHARED_DEFAULT_COOKIE_OWNER)
+
+for key in sorted(registered - set(ins["key"] for ins in instances)):
+    check(False,
+          f"SHARED_DEFAULT_COOKIE_OWNER registers '{key}', but no such oauth2-proxy instance "
+          f"exists in the tree. The register is SHRINK-ONLY and exact: a stale entry is a "
+          f"standing exemption for an object that is gone, so the next instance to inherit the "
+          f"default name would be silently excused. Remove the entry, or restore the instance")
+
+for key in default_holders:
+    check(key in registered,
+          f"{key} holds oauth2-proxy's DEFAULT cookie name '{DEFAULT_COOKIE_NAME}' but is NOT "
+          f"registered in SHARED_DEFAULT_COOKIE_OWNER. Exactly ONE instance may hold the default, "
+          f"and it is the SHARED forward-auth proxy — whose cookie is the platform-wide session "
+          f"every forward-auth-gated surface rides on, so renaming IT signs them all out at once. "
+          f"Every other instance must declare its own --cookie-name. If this instance really is "
+          f"the new shared proxy, register it here with the reason; if it is a per-app sidecar, "
+          f"give it a unique name")
+
+check(len(default_holders) <= 1,
+      f"{len(default_holders)} instances hold the default cookie name: {', '.join(default_holders)}. "
+      f"The default is a NAME, not an absence — it can have exactly one owner")
+
+# ============================================================ (4) HOST-SCOPED COOKIE DOMAIN
+# See the chainsaw-test header for the argument (distinct names alone already fix the eviction;
+# host-scoping is asserted anyway because a zone-wide cookie is TRANSMITTED to every host in the
+# zone, and because it makes a future name collision contained instead of platform-wide).
+for ins in sorted(instances, key=lambda i: i["key"]):
+    if ins["key"] in registered:
+        continue                          # the shared proxy: zone-wide by design, registered
+    dom, rhost = ins["cookie_domain"], ins["redirect_host"]
+    check(dom is not None,
+          f"{ins['key']} declares no --cookie-domain. oauth2-proxy then scopes the cookie to the "
+          f"request host, which happens to be correct — but it is correct BY ACCIDENT and says "
+          f"nothing to the next reader. Declare it explicitly as this app's own host")
+    if dom is None:
+        continue
+    check(not dom.startswith("."),
+          f"{ins['key']} sets --cookie-domain={dom}. A leading dot makes the cookie ZONE-WIDE: "
+          f"the browser sends this app's session token to every other host under that domain, "
+          f"including hosts behind no proxy at all, and it is what turns any future cookie-name "
+          f"collision into a platform-wide outage instead of a contained one. Scope it to this "
+          f"app's own host. Only the registered shared forward-auth proxy may be zone-wide, "
+          f"because cross-host forward-auth is its job")
+    if rhost:
+        check(dom == rhost,
+              f"{ins['key']} sets --cookie-domain={dom} but its --redirect-url host is {rhost}. "
+              f"The cookie is then set for a domain the sign-in flow never returns to (or for a "
+              f"wider one than this app owns), so the session either does not stick or leaks "
+              f"beyond the app. Both flags describe the SAME app: make them agree")
+
+# ============================================================ (5) THE RENDERER DERIVES
+# Five of these instances are GENERATED. Auditing only the goldens would let someone "fix" the
+# tree by hand and leave the generator emitting the defect for the next service onboarded — and
+# the next re-render would then silently revert the hand fix.
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these instances cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed; assertions (5) below cannot run, and the generator is unaudited")
+    if fn:
+        cookie_name_lines = [ln for ln in fn.splitlines() if "--cookie-name" in ln]
+        check(bool(cookie_name_lines),
+              "catalog/render.py's _oauth2_sidecar_container() emits NO --cookie-name at all, so "
+              "every service it renders claims oauth2-proxy's default and collides with every "
+              "other. This is the defect itself, back in the one place that reintroduces it "
+              "fleet-wide in a single edit")
+        check(any("s['name']" in ln or 's["name"]' in ln for ln in cookie_name_lines),
+              "catalog/render.py emits --cookie-name without DERIVING it from the "
+              "ServiceDefinition's own name. A constant collides with every sibling by "
+              "construction; a per-service table leaves the NEXT app onboarded defaulting into "
+              "the collision, which is how twelve ServiceMonitors acquired one identical wrong "
+              "port name in a single edit. Derive it from s['name']")
+        cookie_dom_lines = [ln for ln in fn.splitlines() if "--cookie-domain" in ln]
+        check(any("{host}" in ln for ln in cookie_dom_lines),
+              "catalog/render.py's sidecar --cookie-domain is not derived from the route host. A "
+              "zone-wide literal (`.{ZONE}`) broadcasts every app's session cookie across the "
+              "whole zone — see assertion (4)")
+        check(not any(re.search(r"--cookie-name=_oauth2_proxy_[a-z0-9]", ln) for ln in cookie_name_lines),
+              "catalog/render.py hardcodes a literal service name into --cookie-name. Whatever "
+              "service that is, every OTHER service now renders the same constant and they all "
+              "collide. Derive the name instead")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(4) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on two synthetic ServiceDefinitions
+# and asserts the property directly, so the guard stays falsifiable.
+def _sd(name: str, port: int) -> str:
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
+    }, sort_keys=False)
+
+
+def _sidecar_args(rendered: str) -> list:
+    for doc in yaml.safe_load_all(rendered):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    fixtures = {"cookiefixa": 8111, "cookiefixb": 8112}
+    rendered_args: dict[str, list] = {}
+    with tempfile.TemporaryDirectory() as td:
+        for svc, port in fixtures.items():
+            f = Path(td) / f"{svc}.yaml"
+            f.write_text(_sd(svc, port))
+            proc = subprocess.run([sys.executable, str(render_py), str(f)],
+                                  capture_output=True, text=True)
+            check(proc.returncode == 0,
+                  f"the shipped catalog/render.py failed on a synthetic forward-auth fixture "
+                  f"({svc}): rc={proc.returncode} {proc.stderr.strip()[:200]}")
+            if proc.returncode == 0:
+                rendered_args[svc] = _sidecar_args(proc.stdout)
+
+    names, domains = {}, {}
+    for svc, args in rendered_args.items():
+        names[svc] = flag_from_args(args, "cookie-name")
+        domains[svc] = flag_from_args(args, "cookie-domain")
+        check(bool(args),
+              f"the shipped renderer produced no oauth2-proxy sidecar for synthetic forward-auth "
+              f"fixture {svc} — the sidecar is how a forward-auth app is gated at all (R-IDP-02)")
+        check(names[svc] not in (None, DEFAULT_COOKIE_NAME),
+              f"the shipped renderer gave synthetic fixture {svc} the cookie name "
+              f"{names[svc]!r}. A newly-onboarded forward-auth app would therefore default "
+              f"straight into the fleet collision — which is exactly how five services acquired "
+              f"this defect without anyone making a decision")
+        check(domains[svc] == f"{svc}.home.keithjackson.dev",
+              f"the shipped renderer scoped synthetic fixture {svc}'s cookie to "
+              f"{domains[svc]!r} rather than to its own host — a newly-onboarded app would "
+              f"broadcast its session cookie across the whole zone")
+        check(any(isinstance(a, str) and a.startswith("--whitelist-domain=.") for a in args),
+              f"synthetic fixture {svc} lost its ZONE-WIDE --whitelist-domain. That flag governs "
+              f"permitted post-login REDIRECT targets, not cookie scope; narrowing it with the "
+              f"cookie domain breaks the sign-in bounce")
+
+    if len(rendered_args) == 2:
+        a, b = sorted(rendered_args)
+        check(names[a] != names[b],
+              f"the shipped renderer gave two DIFFERENT synthetic services the SAME cookie name "
+              f"({names[a]!r}). Whatever the committed goldens currently look like, the generator "
+              f"will collide every forward-auth app it renders from now on")
+        check(domains[a] != domains[b],
+              f"the shipped renderer gave two different synthetic services the same cookie domain "
+              f"({domains[a]!r}) — the cookies would share a scope even with distinct names")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{i['key']} -> {i['effective_name']} @ {i['cookie_domain'] or '<host>'}"
+                 for i in sorted(instances, key=lambda x: x["key"]))
+print(f"oauth2-proxy instances audited ({len(instances)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-cookie-names-distinct): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-cookie-names-distinct — {CHECKS} assertions (R-IDP-02, R-IDP-06)")
+PY

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -76,11 +76,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_paas-demo
+        - --cookie-domain=paas-demo.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -101,6 +104,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

decisions/2026-09-30-jackett-oauth2-proxy-session.md

@@ -0,0 +1,273 @@
+---
+date: 2026-09-30
+branch: fix/jackett-oauth2-proxy-session
+title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
+---
+
+## What changed
+
+Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
+never fixes it, and I am never re-prompted for login if it is a login issue."*
+
+**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
+deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.
+
+Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2
+and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as
+written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1
+is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top;
+I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all
+three; do not rest the case on the first.
+
+**1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy
+instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
+`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
+pods' own logs, interleaved to the second:
+
+    15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
+    15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
+    15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
+    15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
+    15:27:59  radarr   "cookie signature not valid, removing session"
+    15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
+    15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
+    15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie
+
+`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
+`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
+by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
+out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
+so plain expiry does not explain it"): it was never expiry, and it was never Jackett.
+
+It also explains the half of the report that looked like user error. **Reloading genuinely does not
+help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and
+Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against
+**1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+`/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot
+decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every
+few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals.
+
+**A counter-hypothesis was put to me and it does not survive bounding.** The observation was that
+Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from
+Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0
+restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the
+ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the
+fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** —
+from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and
+15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The
+ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty)
+manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly
+bounded, Crafty is evidence **for** the fix.
+
+**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
+`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
+`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
+(disabled), so the proxy holds a refresh token it has no mandate to spend.
+
+**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
+AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
+consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
+split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
+`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
+browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
+JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
+re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
+nothing logged as a failure anywhere.
+
+### The fix has one home, and it is shared — this is the blocker
+
+The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
+apps to a sibling branch. **That split is not implementable, and I found out by trying it.**
+
+`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
+shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
+declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
+— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
+gate rejected it exactly as designed:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
+    [fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)
+
+So I made the change at its real source instead and **measured the blast radius** before committing
+to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
+apps, and `render-golden` requires all five be regenerated in the same commit:
+
+    CHANGED: apps/wave-0/jackett.yaml
+    CHANGED: apps/wave-0/pihole.yaml
+    CHANGED: apps/wave-0/qbittorrent.yaml
+    CHANGED: apps/wave-1/radarr.yaml
+    CHANGED: apps/wave-1/sonarr.yaml
+
+Four of those five are the files I was explicitly told not to touch, and all four are being edited
+right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
+direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
+branch owns, and two branches rewriting the same generated block across five files is precisely the
+cross-branch conflict this repo's §7 exists to prevent.
+
+I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
+as a single renderer commit on one branch. The intended content, validated below:
+
+    --cookie-name=_oauth2_proxy_{name}        # per-app, derived
+    --cookie-domain={host}                    # host-scoped, derived
+    --whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
+    --cookie-refresh=2m30s                    # half the measured 300 s token lifetime
+    (drop --pass-access-token, --set-xauthrequest)
+
+`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
+measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
+either a per-app hook or four more measurements.
+
+**Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true
+and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries
+`--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent,
+radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never
+starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is
+already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's
+`args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding
+`--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie
+change. It belongs in the same single renderer commit — which is an argument for cutting that commit
+soon, not for me to hand-edit a golden a second time after the gate already refused the first.
+(Crafty is separately hand-written and could take the metrics flag on its own.)
+
+**Crafty is unaffected by all of this** — its sidecar is hand-written in
+`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
+2026-09-29.
+
+### What this branch does land
+
+`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
+the app's own login apart (including the upstream `-` field that means the proxy answered a request
+the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
+302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
+runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
+than the file merely existing.
+
+## What I verified
+
+- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
+  cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
+  radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
+  qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
+  operator's whole report.
+- **Six instances / seven pods are in the collision set** — the brief said five; the shared
+  forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
+  pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
+  `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
+- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
+  never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
+  can read each other's cookie — the precondition for the eviction.
+- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
+  `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
+- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
+- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
+  `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
+  two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
+- **The proposed argument list starts, against the pinned image and the real issuer.** Ran
+  `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
+  against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
+  `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
+  domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
+  `API route - Path: ^/api/`.
+- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
+  the proxy would have refused to start. Read off `--help`.
+- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
+  assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
+  session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
+  `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
+  session, and it is unavailable while refresh is on. Dropping those two flags is justified on
+  **least privilege and inertness**, not size.
+- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
+  `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
+  `/oauth2/auth` is never called.
+- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
+  inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
+  renderer-only change listed all five apps as changed (transcript above).
+- **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`,
+  `startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and
+  `--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3**
+  `No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a
+  24 h window over a 13.9 h pod and is dominated by the pre-fix process.
+- **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156
+  distinct user-visible failures.
+- **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is
+  `['http']` on all six per-app sidecars; only the shared proxy has
+  `--metrics-address=0.0.0.0:44180` and a `metrics` port.
+- **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green:
+  `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`.
+  Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE`
+  unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**,
+  not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path
+  `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly
+  circulated shim mounts it at its host path, where `SPEC_REPO` never finds it.
+- **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts
+  reported, and **only one of them ran**:
+
+      gate / gate (pull_request)         state=success   desc='Successful in 1m41s'
+      gate / conformance (pull_request)  state=success   desc='Has been skipped'
+
+  That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The
+  conformance job did not execute, so the green tick beside it certifies nothing. Read together with
+  the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran.
+  Anyone reading the PR's check mark alone would conclude otherwise.
+- **`main` is protected**, re-measured this session rather than assumed:
+  `GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`.
+
+## What I did NOT verify
+
+- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
+  Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
+  change ships. **This is the single most important line in this entry.**
+- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
+  is proven to start, but nobody has added an indexer through the proposed configuration. The human
+  test is below, for whoever lands the renderer change.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
+  read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
+  the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
+  Sonarr's direct `:9117` access.
+- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
+  `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
+  4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
+  this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
+  this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
+  `--session-cookie-minimal` is ruled out (above).
+- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
+  the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
+  credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
+  and the realm is where to look next.
+- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
+  PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
+- **The four sibling apps still collide with each other** regardless of what happens to Jackett.
+  The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
+- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
+  is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are
+  **unverified by me this session** — I did not re-measure them against the Gitea API.
+- **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the
+  better instrument (an exact process boundary rather than a wall-clock window against a 72 h
+  retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a
+  per-app ratio over a window longer than the current pod's life is not something I measured at all.
+- **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three
+  seconds and the mechanism is documented and present, but I did not enumerate every jackett failure
+  and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven
+  sufficient** — the coordinator's framing, and I agree with it on this evidence.
+- **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the
+  oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
+- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
+  per-namespace `for` loops are edited from the scripted forms I actually used.
+
+### The human test — for whoever lands the renderer fix
+
+After it merges and the pod rolls (watch for a **new** pod name at 2/2):
+
+1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
+2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
+   today** — it is the 15:31:29 line. Return to the Jackett tab.
+3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
+4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
+   test; the session currently dies at five.
+5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
+   call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
+   did not take; check the running pod's args before assuming anything else.
+6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
+   A 401 page on reload means the regex is too broad — that is a regression.

decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md

@@ -0,0 +1,378 @@
+---
+date: 2026-09-30
+branch: fix/oauth2-proxy-cookie-collision-fleet
+title: The cookie collision was never Crafty's — it was the renderer's, and it shipped to five apps
+---
+
+## What changed
+
+Follow-up #3 of `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`: *"Any further per-app
+reverse-proxy sidecar needs a unique `--cookie-name` too. The collision is a property of the
+pattern, not of Crafty."*
+
+It is a property of the pattern, and this branch found where the pattern actually lives.
+
+**Read the scope claim carefully before the detail.** This branch fixes THREE defects in the
+forward-auth sidecar contract, all in one function, all landing on all five apps at once:
+
+| Defect | Fix | Measured evidence |
+|---|---|---|
+| Six independent instances, seven different cookie-secrets, **one** cookie name on **one** zone-wide domain — so they overwrite and `ClearSession`-delete each other's sessions | `--cookie-name=_oauth2_proxy_<app>` + host-scoped `--cookie-domain` | 7/7 distinct secrets; 6 instances on the default name; radarr logs the eviction fingerprint 8× in 12h |
+| The session dies **5 minutes** after sign-in — `--cookie-refresh` defaults to 0 = *disabled*, so the refresh token the proxy already holds is never spent | `--cookie-refresh=2m30s` (half the lifetime) | access-token lifetime = **300.000s exactly**, `refresh_token:true`, flag absent |
+| **No** per-app sidecar serves `/metrics` at all — without `--metrics-address` the listener never starts | `--metrics-address=0.0.0.0:44180` + a `metrics` container port | `ports` was `['http']` on all six |
+
+Rejection *volume* is dominated by the second defect, not the first: ~95% of radarr's rejections
+carry no cookie at all. **Do not merge this expecting one number to drop** — three mechanisms were
+tangled together, and separating them is most of the value here.
+
+**Boundary note.** My brief originally reserved `apps/wave-0/jackett.yaml` for a concurrent branch.
+That split is not achievable and has been revoked: the file is generated, the fix is one renderer
+edit, and `render-golden` requires all five goldens regenerated in the *same commit*. I own all
+five. The proof I ran before touching jackett is under "What I verified".
+
+**The defect was not in the four manifests it was reported in.** `apps/wave-0/pihole.yaml`,
+`apps/wave-0/qbittorrent.yaml`, `apps/wave-1/radarr.yaml`, `apps/wave-1/sonarr.yaml` — and
+`apps/wave-0/jackett.yaml` — are **generated goldens**. One function emits all five sidecars:
+`catalog/render.py::_oauth2_sidecar_container()`, which hardcoded `--cookie-domain=.{ZONE}` and
+emitted no `--cookie-name` at all. Hand-editing a golden is not a fix twice over: `ci/render-golden.sh`
+FAILS on the drift (provoked, not assumed — see below), and the next re-render silently reverts it.
+
+So the fix is in the generator, and both flags are now **derived** from the ServiceDefinition:
+
+    --cookie-name=_oauth2_proxy_{s['name']}     unique per service, by construction
+    --cookie-domain={host}                      the app's own route host, not the zone
+
+`--whitelist-domain` deliberately stays zone-wide — it governs permitted post-login *redirect*
+targets, not cookie scope. `catalog/render.py --selftest` gained four assertions, including one
+that renders a **second** synthetic forward-auth service and requires the two cookie names to
+differ, so a future refactor to a constant fails in the renderer's own test.
+
+Re-rendering with the canonical command regenerated **five** goldens, not four. Jackett is the
+fifth, and this is the one thing in this branch the PM brief did not anticipate — see
+"the jackett collision" below. The three `conformance/tests/l1-rendering/*/golden/` fixtures are
+expected-output copies of the same renderer, so they are regenerated too; that is not collateral
+damage, it is the L1 stage doing its job (it FAILED on the first gate run and named the drift).
+
+**The new gate check** is `conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/`,
+declaring `conformance.home.k3s/scope: whole-repo` so `ci/l0.sh`'s `--include-scope` union runs it
+on **every** gate invocation. No bespoke `ci/l0.sh` stage was added — `whole-repo-invariants-wired`
+fails the gate if anyone adds one, and the scope annotation is the supported mechanism.
+
+It inventories every oauth2-proxy instance the repo authors (containers at any depth, plus the
+shared proxy's HelmRelease values) and resolves each one's **effective** cookie name — its declared
+name, **or oauth2-proxy's built-in `_oauth2_proxy` when it declares none**. That single modelling
+decision is the whole check: an omitted flag is not an absence, it is a *claim on a name*, and
+treating it as an absence is precisely why five reviewers read these manifests and saw nothing.
+Then: (1) no two effective names collide; (2) no name is another's name + `_csrf`, because that is
+how oauth2-proxy derives the CSRF cookie and two such session names look distinct while their CSRF
+cookies collide; (3) the default name has exactly one owner, and it is the one registered in
+`SHARED_DEFAULT_COOKIE_OWNER`; (4) every non-registered instance is host-scoped; (5)
+`catalog/render.py` derives rather than lists; (6) the **shipped** renderer is proven end-to-end on
+two synthetic ServiceDefinitions run as a subprocess.
+
+**On the shared proxy — "distinct" or "explicitly declared"?** The brief asked me to decide, and
+the answer is *neither, quite*. The shared forward-auth proxy at `platform/oauth2-proxy/` **must
+keep** `_oauth2_proxy`: it is the single session every forward-auth-gated surface without its own
+sidecar rides on, so renaming it signs all of them out at once. Requiring *every* instance to
+declare a name would therefore need an exemption, and an exemption is a hole: delete the shared
+proxy and whichever app next omitted the flag would silently inherit the default with the rule
+still satisfied. So the assertion is plain **distinctness over effective names**, with the default
+treated as a name like any other — held by **exactly one owner, named in a shrink-only register
+with its reason**. The shared proxy needs no exemption from the rule; it is simply the registered
+holder of that one name. A stale register entry FAILS (the exemption would outlive its object) and
+an unregistered holder FAILS (no silent heirs). Both were provoked and are quoted below.
+
+**On host-scoped `--cookie-domain` — overreach, or not?** Asserted, and here is the argument both
+ways. *Against:* distinct names alone close the eviction bug completely — two differently-named
+cookies do not overwrite each other however wide their domains are — so this assertion fixes
+nothing (1) has not already fixed, and a check that fails a legitimate configuration is a check
+someone eventually disables. *For, and this wins:* a zone-wide cookie is **transmitted by the
+browser to every host in the zone**. qBittorrent's session token was being sent to pihole, radarr,
+sonarr and every other name under the zone, including hosts behind no proxy at all — a
+session-credential disclosure across trust boundaries that nothing needs. And it is what makes a
+*future* name collision catastrophic rather than contained: with host-scoped cookies, two
+instances that accidentally share a name cannot see each other's cookie at all. Defence in depth
+on the exact axis that already failed once. The escape hatch is the same register, by name and
+with a reason, rather than a loosened rule.
+
+### The jackett boundary, and why it could not hold
+
+`apps/wave-0/jackett.yaml` carries the identical defect and was initially assigned to a concurrent
+branch, with an instruction to me not to touch it. **That separation is not achievable** — the
+ownership split was drawn across a generated file — and I did not discover this by reasoning about
+it, I provoked it. With the renderer fixed and jackett's golden left at its `main` content,
+`./ci/render-golden.sh` exits **1**:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render:
+    -        - --cookie-domain=.${BASE_DOMAIN}
+    +        - --cookie-name=_oauth2_proxy_jackett
+    +        - --cookie-domain=jackett.${BASE_DOMAIN}
+    [fail] render==golden check FAILED
+
+Jackett's golden is an *output* of the file that contains the defect. There are exactly three
+options and two of them are worse: hand-edit four goldens and leave the generator broken (fails
+`render-golden`, and reverts on the next render); add a per-service carve-out to the renderer so
+jackett alone keeps defaulting (ships the exact "list, don't derive" anti-pattern this check
+exists to forbid, and leaves jackett broken *by design*); or fix the generator and regenerate all
+five. I took the third. Jackett's golden here is **machine output of a one-line generator change**,
+not an independent edit of Mira's file, and it is byte-identical to what her branch must produce.
+
+**Consequence:** the jackett branch has no cookie work left in it — this commit fixes jackett
+identically and by the same mechanism, and both branches independently arrived at the *same two
+values* (`_oauth2_proxy_jackett`, `jackett.${BASE_DOMAIN}`), which is a good sign for the
+diagnosis. Its hand-edit of the rendered file will fail `render-golden` on its own terms, so it
+should be closed rather than rebased. **What must not be lost with it** is its evidence: the
+interleaved radarr/jackett log showing `removing session` deleting a clean session 26s after login,
+and the `expires − created = 300.000s` measurement that produced `--cookie-refresh` here. Both are
+carried into this entry and into the renderer's comments; the branch itself is now redundant, the
+work behind it is not.
+
+**The general lesson, which is worth more than this bug:** an ownership split across
+`apps/wave-*/*.yaml` cannot hold, because those files are *outputs*. Two agents assigned two
+rendered apps are assigned the same function and will collide in `render-golden` no matter how
+carefully they stay in their own files. Ownership of a rendered app is ownership of
+`catalog/render.py`, and briefs should say so.
+
+### What this branch deliberately does NOT do
+
+- **`--api-route=^/api/` is NOT added**, and not because of scope discipline alone: it would be
+  **wrong** renderer-wide. It was measured on jackett, and **PiHole does not serve `/api/`** — so
+  a renderer constant applies one app's path shape to five. Doing it properly means declaring API
+  path prefixes per service, and the `auth` block is `additionalProperties: false`, so that is a
+  *schema change*. Separate PR. (Note the flag is **`--api-route`, singular** — verified in the
+  pinned binary's own `--help`. The plural `--api-routes` does not exist in v7.8.1 and the proxy
+  refuses to start with it. That error was propagating through briefs; it stops here.)
+- **`--session-cookie-minimal` is NOT added** and must not be, as a cookie-size fix: it is
+  **mutually exclusive** with `--cookie-refresh` (*"cookie_refresh > 0 requires oauth tokens in
+  sessions"*). Dropping `--pass-access-token` is a least-privilege argument, never a size one.
+- **Metrics are NOT scraped, only served.** The flag and container port are the prerequisite;
+  a container port is not a Service port. These Services expose the app port and `:4180`, and each
+  namespace's default-deny baseline is `ingress: []`, so Prometheus can neither resolve nor reach
+  `:44180` until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+  exist. That wiring is someone else's follow-up and I have deliberately not faked its start.
+  **Nobody should read this commit as "the sidecars have metrics now."**
+- **The new check was NOT widened to cover cookie-refresh or metrics.** Its subject is cookie
+  *identity*; those two share neither its inputs nor its failure mode. Bundling them would make
+  the check's name a lie and couple unrelated ratchets, so a future metrics decision could only
+  be expressed by weakening a *cookie* check. The cost of that discipline is named honestly in
+  the follow-ups: those two flags are currently **ungated**.
+
+## What I verified
+
+- **The premise, in-cluster, rather than inherited from the handoff** (CLAUDE.md §2b). Read-only,
+  `kubectl --context k3s-prod`, through the pinned toolbox.
+  - **Seven** `*-oauth2-proxy` Secrets carry a `cookie-secret`: crafty, jackett, pihole,
+    qbittorrent, radarr, sonarr, and the shared `oauth2-proxy-credentials`. **All seven values are
+    DISTINCT.** Compared without printing anything: each value was hashed **in-cluster-side script,
+    never echoed**, and only the first 8 hex characters of each `sha256` were emitted — enough to
+    say same-or-different, recovering nothing. Secret *key names* were listed; no `.data` or
+    `.stringData` was printed (CLAUDE.md §6). **So no instance can decrypt another's cookie — it can
+    only overwrite it. The premise holds.**
+  - **The default cookie names really were in use.** Of 8 live oauth2-proxy containers, **6 declare
+    no `--cookie-name` at all** — jackett, pihole, qbittorrent, radarr, sonarr, and both replicas of
+    the shared proxy — so all six emit `_oauth2_proxy` / `_oauth2_proxy_csrf`. All six also carry
+    `--cookie-domain=.home.keithjackson.dev`. **One cookie pair, six independent instances.**
+  - Crafty's live pod (`crafty-647c7d8cc5-5spv9`) already carries
+    `--cookie-name=_oauth2_proxy_crafty` and `--cookie-domain=crafty.home.keithjackson.dev`, so
+    PR #22 is merged and reconciled. It is the only compliant per-app instance on the cluster.
+- **THE COLLISION IS REAL. IT IS NOT ESTABLISHED AS THE CAUSE OF THE REPORTED FAILURES.** These
+  are two different claims and only the first is supported. I was handed counter-evidence —
+  crafty, the app already fixed, showed the *worst* rejection ratio of the set over a 24h Loki
+  window (`crafty 81:1`, `radarr 67:1`, `jackett 39:4`) — and it is right to refuse the causal
+  claim. What I could do was bound the window. Read from **each sidecar's own log** rather than
+  Loki, because `kubectl logs` on a pod that has not restarted cannot reach a pod older than
+  itself, so a window inside the pod's age is inside the fixed spec's lifetime *by construction*:
+
+      namespace      age(h) rst  reject success    ratio  cleared  badsig
+      crafty           13.9   0       3       0      n/a        0       0   <- FIXED spec, whole life
+      radarr          422.5   0     155       1  155.0:1        8       8
+      jackett         422.6   0      39       4    9.8:1        1       1
+      pihole          395.7   0       0       0      0:0        0       0
+      qbittorrent     404.1   0       0       0      0:0        0       0
+      sonarr          422.5   0       0       0      0:0        0       0
+      oauth2-proxy    422.6   0       0       0      0:0        0       0   (both replicas)
+
+  `reject` = `No valid authentication in request`; `success` = `AuthSuccess`;
+  `cleared` = `removing session` (oauth2-proxy's `ClearSession`); `badsig` = `cookie signature
+  not valid`. Window `--since=24h`, which exceeds every pod's age only for crafty — whose pod is
+  13.9h old with **0 restarts**, so these are its complete post-fix figures.
+
+  Then I read crafty's four auth-relevant lines rather than counting them, because at n=3 the
+  content is the evidence:
+
+      03:57:20  Cookie settings: name:_oauth2_proxy_crafty secure(https):true httponly:true
+                expiry:168h0m0s domains:crafty.home.keithjackson.dev path:/
+      04:02:55  No valid authentication in request. Initiating login.
+      15:22:39  No valid authentication in request. Initiating login.
+      15:22:44  No valid authentication in request. Initiating login.
+
+  6743 log lines; those are all four. **Zero** `removing session`, **zero** `cookie signature
+  not valid`, zero anything else.
+
+  Four conclusions, kept separate:
+  1. **`crafty 81:1` is a window artefact. It is not evidence against the fix and must not be
+     cited as such.** The fixed pod emitted **3** rejections in 13h 52m; the 81 came from a 24h
+     window straddling it, i.e. from the pre-fix process.
+  2. **The eviction FINGERPRINT is gone where the fix is deployed.** `cookie signature not
+     valid` + `removing session` is what a clobbered instance logs — a cookie *is* present,
+     written with a key this instance does not hold, so it cannot decrypt it and then
+     `ClearSession` expires it on its own `--cookie-domain`. Fixed crafty: **0 and 0** across
+     its whole life. Unfixed radarr: **8 and 8** in 12h. Unfixed jackett: 1 and 1. That is a
+     real, directional signal and it is the strongest evidence available that the fix works.
+  3. **It is still not a demonstrated successful sign-in.** Crafty logged **0 `AuthSuccess`** —
+     so what is shown is *absence of the failure mode*, not a session observed working
+     end-to-end. Those three rejections are consistent with benign first-hits (one client, two
+     of them 5s apart, i.e. a page plus a `/ws` reconnect). I am comfortable saying the fix
+     works; I am not willing to say it has been watched working, and the distinction is the
+     whole reason this section exists.
+  4. **The collision is a MINORITY of the rejection volume.** radarr: 8 of 155 (5%) carry the
+     fingerprint; ~95% carry **no cookie at all**. Those are the 5-minute session expiry and
+     `302`-on-XHR — the first of which this branch now also fixes, the second of which it
+     deliberately does not.
+
+  Two cautions about the instrument. `reject` ranks **chattiness, not breakage**: radarr's are
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate` polls, so a busy *arr
+  inflates the count with nobody inconvenienced. And that same polling is the *amplifier* that
+  made this intermittent — every poll carrying an undecryptable cookie calls `ClearSession`
+  again on the zone-wide domain, so **a left-open Radarr tab deleted Jackett's cookie every few
+  seconds, from another namespace, with no user action at all.** That is why signing in again
+  never helped for longer than the gap between two polls. It is now a comment in the renderer,
+  because it is the part of this bug that no one would reconstruct from the flags alone.
+- **The four target files are generated, not authored.** Rendering `catalog/` with the canonical
+  command into a temp dir and diffing against `apps/` reported all 10 onboarded goldens
+  byte-identical before any change. Hand-editing one is drift by definition.
+- **`render-golden` FAILS if jackett is excluded** — provoked (output quoted above), rc **1**.
+  Restoring the regenerated golden returns it to `all 10 onboarded service golden(s) reproduce`,
+  rc **0**.
+- **Every flag verified against the PINNED v7.8.1 binary**, not against docs, memory, or the
+  shared proxy (which runs **v7.15.3** and therefore could not vouch for v7.8.1):
+  `podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`.
+  - `--cookie-name string ... (default "_oauth2_proxy")` — **this confirms the constant my whole
+    check is built on, from the binary itself.** The check's central claim, that an omitted flag
+    is a *claim on a name* rather than an absence, now rests on the implementation's own help
+    text rather than on my reading of its source.
+  - `--cookie-refresh duration  refresh the cookie after this duration; 0 to disable` — confirms
+    the default really is disabled.
+  - `--metrics-address string` — present in v7.8.1, so the new flag cannot crashloop the pods.
+  - `--api-route strings` — **singular**, confirming the brief's plural was wrong.
+  - `--session-cookie-minimal` — exists, and is the flag that conflicts with `--cookie-refresh`.
+- **44180 is the platform's existing convention, measured, not chosen.** The shared proxy already
+  runs `--metrics-address=0.0.0.0:44180` with a `metrics` container port, a `metrics:44180`
+  Service port, a ServiceMonitor whose endpoint is `metrics`, and **2 ready endpoint addresses** —
+  i.e. it demonstrably works on this platform.
+- **`catalog/render.py --selftest`: PASS**, now including the two-service distinctness assertion,
+  a `--cookie-refresh` assertion that PARSES the duration and requires `0 < refresh < 300s` (so
+  `2m30s` -> `150s` -> `2.5m` all stay green but a silent widening past the token lifetime does
+  not), a `--session-cookie-minimal` mutual-exclusion guard, and both metrics assertions.
+- **The check is selected by the whole-repo union.** `conformance/run.sh --layer l0 --requirements
+  r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
+  `tests/l0-static/oauth2-proxy-cookie-names-distinct` among 20 tests — i.e. it runs even when the
+  touched-requirement filter matches nothing, which is the blind spot it was scoped for.
+- **The check was tested in BOTH directions**, against throwaway trees outside the repo, never by
+  mutating the branch:
+  - **Compliant tree (this branch): rc 0, 39 assertions, 7 instances audited.**
+  - **Pre-fix tree** (`git archive HEAD` + this test dropped in): **rc 1, 26 failures**, naming all
+    five violators, the 6-way `_oauth2_proxy` collision, the zone-wide domains, the renderer
+    emitting no `--cookie-name`, and both synthetic fixtures rendering `None`.
+  - **Targeted mutations on a compliant copy**, one assertion class each, all fired and nothing
+    else did: duplicate name (sonarr←radarr) → 1 failure; CSRF derivation
+    (`_oauth2_proxy_sonarr_csrf` as a session name) → fired; zone-wide `--cookie-domain` on pihole
+    → 2 failures; crafty's `--cookie-name` deleted → unregistered-default-heir + 2-way collision;
+    shared proxy renamed → **stale register entry** failure.
+  - A **false positive was found and fixed** by this testing: `platform/oauth2-proxy/forward-auth-cec.yaml`
+    contains `serviceName: oauth2-proxy/oauth2-proxy:80` — an Envoy `namespace/service:port`
+    reference, not an image — which the first inventory scan counted as an unaudited instance. The
+    scan is now anchored on `image:`, and the near-miss is documented in the check.
+- **Gate:** toolbox `./ci/gate.sh` — exit code and stage list in the next section, because it is
+  not a pass and that belongs there.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
+  `l0`, `render-golden`, `l1`, `approval-policy`. **Skipped:** `gitea-settings` (no
+  `SETTINGS_API_TOKEN` locally) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing
+  FAILED; nothing runtime was certified either. **rc 3 is UNDETERMINED, never green.** Touched
+  subset `R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-PLAT-02 R-TEST-01
+  R-TEST-02 R-TEST-03 R-TEST-04` — **UNVERIFIED** at L2/L3, all twelve.
+- **`spec-immutability` was skipping because of a BUG IN MY TOOLING, not the environment.** The
+  worktree ENGINE shim in circulation mounts only the main checkout's `.git`, and `dev.sh`
+  computes the spec path from its own `REPO_ROOT` — which in a worktree is
+  `.claude/worktrees/agent-*/`, so `../home-network-k3s` does not resolve and it adds no spec
+  mount at all. The gate then reported the skip, and the skip read as "the one tolerated
+  environmental skip" while actually being a path bug being waved through. Adding
+  `-v /var/home/keith/source/home-network-k3s:/home-network-k3s:ro -e SPEC_REPO=/home-network-k3s`
+  makes **R-TEST-06 actually run and pass**. Worth putting in every worktree agent brief; the
+  two-mount shim in CLAUDE.md §3 should say so.
+- **CI: `gate / gate (pull_request)` reports FAILURE on `75a4029`, for exactly ONE reason, and it
+  is not this change.** Read from the job log rather than the tick:
+
+      [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  Every substantive stage is **green in CI** — `gitea-settings: pass` (CI has the token my local
+  run does not), `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. Skipped: `spec-immutability`
+  (the workflow does not check the spec repo out — the documented tolerated skip) and
+  `conformance` (declared). So CI is a **stricter, more complete** run than my local one on every
+  axis except the spec repo, and it agrees.
+  `infra/gitea/approval-policy.json` lists `conformance/**` as sensitive — *"the tests and the
+  harness; changing them changes what green MEANS"* — so this PR is **fail-closed pending a
+  non-author approving review**, which is CLAUDE.md §1 working, not a defect to route around.
+  **The red tick is the policy, and it clears when a human reviews it.** Note also that
+  `gate / conformance (pull_request)` reports `success` with description **"Has been skipped"** —
+  the precise trap CLAUDE.md §2b names; it certifies nothing.
+- **A SUCCESSFUL SIGN-IN, ANYWHERE.** The strongest post-fix evidence is the *absence* of the
+  eviction fingerprint on crafty (0 `removing session`, 0 `cookie signature not valid`, across
+  13h 52m) against radarr's 8-and-8. That is directional and I believe it. But crafty logged
+  **0 `AuthSuccess`** in the same window, so no session has been observed working end to end on a
+  fixed instance. Whoever closes this out should re-measure crafty **after real sign-in traffic**.
+- **No browser round-trip by me at all**, for any of the three defects. Reproducing the collision
+  needs two gated apps visited in one session. The concurrent jackett branch does carry an observed
+  log-level reproduction (interleaved radarr/jackett `removing session` deleting a clean session
+  26s after login) — real evidence for the MECHANISM, and it is theirs, not mine.
+- **`--cookie-refresh=2m30s` has never been exercised.** The value is derived from a measured
+  300.000s access-token lifetime and is asserted to sit inside `(0, 300s)`, but no session has been
+  held past the five-minute cliff to watch a refresh succeed. If Keycloak's refresh-token or SSO
+  idle timeouts are shorter than expected, the symptom moves rather than disappears. **This is the
+  fix most likely to need a second pass**, and the first thing to watch after merge.
+- **The `metrics` port has never been fetched.** The listener is asserted to start by flag
+  presence only; nobody has curled `:44180` on a rendered pod. And it is unreachable by design
+  until the Service/policy/ServiceMonitor wiring lands — see above.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. Cluster access was
+  read-only throughout; the one container I ran was the oauth2-proxy image locally under podman to
+  read its `--help`. The running pods are still the pre-fix spec. **Merging is what deploys**, and
+  it will roll five Deployments; every user's existing session on those five apps is invalidated by
+  the cookie rename and they will re-authenticate once. Radarr/sonarr/qbittorrent use
+  `strategy: Recreate` on RWO volumes, so there is a brief gap per app. Pick the moment — and
+  note this now changes **three** flags plus a port on all five, so a bad `--cookie-refresh` value
+  would surface as five apps misbehaving, not one.
+- **The live flags are not re-measured by the check.** git is not the cluster. I measured the live
+  pods by hand, once; nothing in the gate re-reads them, so hand-applied drift would be invisible
+  to it. A live-state assertion is L2 and is not written.
+- **`--cookie-refresh` and `--metrics-address` are UNGATED, and I found out why.** Nothing in
+  `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the twenty-odd
+  assertions in that selftest, including the six I just added, are dev-time only: a future edit
+  that drops either flag *and* re-renders the goldens would pass the entire merge gate.
+  `render-golden` catches accidental drift (golden vs renderer) but not a deliberate removal of
+  both together, and my L0 check deliberately does not cover these two flags. **The repo already
+  has the precedent for the fix and the one-line shape of it:**
+  `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` drives
+  `conformance/lib/<tool> --selftest` as a subprocess for exactly this reason — its own header
+  says it does so "because nothing [else] drives them". Adding `catalog/render.py` to that list,
+  or giving the renderer its own scope-declaring L0 test, closes it. I did not do it here because
+  it widens an already review-blocked PR into a third subject; it is the highest-value follow-up
+  in this entry.
+- **Log-derived counts are a weak instrument and I did not strengthen them.** `kubectl logs
+  --since=` returns what the kubelet still retains, so an old, busy container's window may be
+  truncated; and `No valid authentication in request` counts unauthenticated first hits and XHR
+  polls indiscriminately. The `cookie signature not valid` / `removing session` pair is the only
+  collision-specific signal in there, and it is the one I leaned on. None of this is gated —
+  there is no alert on it and no L2 check reads it.
+- **The cookie-secrets' distinctness is not gated either.** It was measured out of band and is
+  asserted nowhere. If they were ever made identical the collision would become harmless — but
+  sharing one session-encryption key across trust zones is a worse idea than the bug it would mask,
+  so the check does not offer it as an alternative and does not look for it.
+- **The shared proxy's own configuration is unchanged and untested by me.** I did not touch
+  `platform/oauth2-proxy/**` (owned by another agent this cycle); the check only reads it.
+- **Nothing about group authorization, CSRF behaviour, or session contents.** Names and domains only.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so the PR's green tick — if it goes green —
+  proves the L0/L1 offline subset and nothing about runtime.

decisions/2026-10-01-render-selftest-runs-in-gate.md

@@ -0,0 +1,205 @@
+---
+date: 2026-10-01
+branch: fix/render-selftest-runs-in-gate
+title: catalog/render.py's 106 assertions were dev-time only; the gate now runs them and blocks on them
+---
+
+## What changed
+
+Single subject, one file: **`ci/render-golden.sh` now invokes `catalog/render.py --selftest`**
+and exits 1 when it fails. Nothing else. No renderer change, no new conformance test, no new
+gate stage.
+
+This closes the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`:
+
+> Nothing in `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the
+> twenty-odd assertions in that selftest, including the six I just added, are dev-time only: a
+> future edit that drops either flag *and* re-renders the goldens would pass the entire merge
+> gate.
+
+Re-measured here and confirmed: before this commit, `grep -rl -- --selftest` over `ci/` and
+`.gitea/` returned **nothing** for the renderer. The selftest executes **106** assertions
+(`check()` invocations counted at runtime via `sys.settrace`, without editing `render.py`; 109
+static call sites, three of them on failure-only branches). All 106 were unexercised by the
+merge gate.
+
+### The lead was WRONG, and here is why
+
+The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the
+one-line fix. That line is real and it does drive selftests:
+
+    for tool in ("record_run.py", "status.py"):
+        r = run([sys.executable, str(LIB / tool), "--selftest"])
+
+But it is **the wrong hook**. That loop is section (d) of a check that serves **R-SEC-03**, and
+its subject is the *evidence chain* — can a self-skipped run, or a stale ledger row, become an
+audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not.
+Hanging the renderer there would have made the check's name, its `intent` annotation and its
+requirement mapping (`R-SEC-03`) all untrue, and a future decision about the renderer could then
+only be expressed by weakening a *security-auditability* check. That is the precise coupling the
+cookie-collision entry refused for its own check, and it applies here.
+
+The second option that entry offered — a new scope-declaring L0 test — is the supported mechanism
+for a **whole-repo invariant** (`ci/l0.sh` Stage 3b's `--include-scope whole-repo` union, ratcheted
+by `tests/l0-static/whole-repo-invariants-wired`). It is the right shape for an invariant
+quantified over the tracked tree. It is the wrong shape for *"a script in this repo must pass its
+own unit test"*: it would need to declare requirements it does not serve, and it would be two new
+files of several hundred lines by this tree's conventions for what is a six-line invocation.
+
+**The honest home is `ci/render-golden.sh`** — the gate stage whose subject *already is*
+`catalog/render.py`. It already resolves `$PYTHON` and `$RENDER_PY`, already reports rc 3 when
+either is missing, already exits 1 on renderer error, and `ci/gate.sh` already maps it through
+`for s in lint l0 render-golden l1` where a FAIL is BLOCKED and a SKIP is UNDETERMINED. The
+stage's two assertions are independent and both are needed:
+
+- `render-golden` catches golden-vs-renderer drift, but is satisfied by **any** renderer whose
+  output was re-rendered — including one that just lost a contract invariant and had all ten
+  goldens regenerated in the same commit. That is exactly the hole named in the quote above.
+- `--selftest` catches that, and is blind to golden drift.
+
+The file header and the stage name were updated so neither is a lie: the stage is now "the
+renderer is trustworthy", with two numbered assertions, and the step summary emits its own
+`- render-selftest:` line so the gate output still NAMES what ran.
+
+The selftest runs **before** the empty-catalog `exit 3` on purpose: it renders synthetic
+ServiceDefinitions in memory and is catalog-independent, so an empty catalog must not be able to
+skip it.
+
+`catalog/render.py` itself was **not touched** (owned by a concurrent branch this session), and
+neither was any file under `conformance/`.
+
+## What I verified
+
+- **The selftest runs and passes: rc 0, `render.py selftest: PASS`, 106 assertions executed.**
+  Counted at runtime, not from the source: a `sys.settrace` hook counting `check()` calls, in a
+  throwaway script outside the repo, with no edit to `render.py`.
+- **The stage passes with it wired in**, host run of the single stage:
+
+      [gate] Stage: render==golden reproducibility (canonical render vs committed apps/)
+      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
+      [ ok ] render.py --selftest passed
+      [ ok ] render==golden: all 10 onboarded service golden(s) reproduce from the canonical render
+
+  rc **0**. The golden comparison still reports all 10 goldens, so the new invocation did not
+  displace what the stage already asserted.
+- **IT CAN FAIL — provoked twice, against a mutated COPY of the renderer, never against the
+  tree.** `ci/render-golden.sh` already honours `RENDER_PY`, so each mutant lived in
+  `/tmp/claude-1000/desmond-selftest/` and the repo's `catalog/render.py` was never modified —
+  which matters, because that file belongs to a concurrent branch this session.
+  - **Mutation 1 — the regression this selftest was written for.** Deleted the
+    `f"--cookie-name=_oauth2_proxy_{s['name']}"` emission from `_oauth2_sidecar_container()`,
+    i.e. reverted the fleet-wide cookie-collision fix of PR #28. Stage rc **1**. Note the shape
+    of the failure, because it is a real (minor) weakness and is listed as a follow-up: the
+    selftest died with an uncaught `StopIteration` out of `next(a for a in container["args"] if
+    a.startswith("--cookie-name="))` at `render.py:2641`, rather than printing its collected
+    `SELFTEST FAIL:` lines. It is correctly non-zero and the stage correctly blocks, but the
+    operator gets a traceback instead of the assertion message.
+  - **Mutation 2 — a clean, named assertion failure.** `OAUTH2_COOKIE_REFRESH = "2m30s"` ->
+    `"10m"`, widening the refresh past the measured 300s access-token lifetime. Stage rc **1**:
+
+        [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
+        [fail]   Every golden under apps/ is produced by this renderer, so the goldens below cannot
+        [fail]   be trusted even if they reproduce byte-for-byte. Reproduce with:
+        [fail]     python3 catalog/render.py --selftest
+            SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A
+            refresh at or past the access-token lifetime never renews in time, which is the same
+            logout it was added to stop
+
+    This is the mutation that proves the wiring's value most directly: mutation 2 would **not**
+    be caught by `render-golden`'s diff if the goldens were regenerated in the same commit, and
+    it is one of the two flags the cookie-collision entry explicitly recorded as **ungated**.
+- **THE NON-ZERO PROPAGATES ALL THE WAY TO `gate.sh`'s rc — provoked, in the pinned toolbox, not
+  reasoned about.** `RENDER_PY=/desmond-selftest/render-mutant2.py ./ci/gate.sh`:
+
+      [ ok ] stage spec-immutability: pass
+      [ ok ] stage decisions-log: pass
+      [ ok ] stage lint: pass
+      [ ok ] stage l0: pass
+      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
+      [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
+          SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). ...
+      [fail] stage render-golden: FAIL
+      [ ok ] stage l1: pass
+      ...
+      [fail] render-golden: FAILED -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  `gate.sh` rc **1**. Three links in the chain, each measured: selftest rc 1 -> stage rc 1 ->
+  gate rc 1 / `MERGE GATE: BLOCKED`. This mattered enough to provoke rather than assume: this
+  repo has a documented habit of exit codes getting flattened, and
+  `.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, which tolerates rc 3 —
+  so a stage that degraded a failure to a skip would be swallowed by CI. rc 1 is not tolerated
+  there, and rc 1 is what this produces.
+- **`spec-immutability` genuinely RAN and passed, it did not skip.** The two-mount worktree shim
+  plus `-e SPEC_REPO=/home-network-k3s` (the fix recorded in the 2026-09-30 cookie entry) works:
+  `stage spec-immutability: pass`, not SKIPPED. R-TEST-06 is therefore actually covered by these
+  runs.
+- **No collision with PR #29.** The brief warned that `feat/oauth2-proxy-auth-failure-visible`
+  had added checks under `conformance/tests/l0-static/`. Measured —
+  `git diff origin/main...origin/feat/oauth2-proxy-auth-failure-visible --name-only` — it touches
+  **no `conformance/` path at all**: `decisions/`, `platform/oauth2-proxy/` (a promtool test, an
+  alert rule, its kustomization), `runbooks/`. Disjoint from this branch either way, since this
+  branch touches only `ci/render-golden.sh` and this file.
+- **Priya's files were not touched.** `catalog/render.py`, `schemas/servicedefinition.schema.json`,
+  `apps/wave-*` and `conformance/tests/l1-rendering/` are all untouched by this branch — confirmed
+  by `git diff --name-only`, which lists exactly two paths.
+- **`catalog/render.py --selftest` was the LAST unwired selftest in the tree.** Measured:
+  `grep -rl -- --selftest` across the repo finds selftest-capable artifacts in exactly four
+  places, and after this commit every one is driven by the gate — `conformance/lib/record_run.py`
+  and `conformance/lib/status.py` by `auditability-proofs-producible/check.sh:291`, the Keycloak
+  client-secret reconciler (embedded in `platform/keycloak/client-secret-reconciler.yaml`) by
+  `keycloak-client-secret-reconciler/check.sh:633`, and `catalog/render.py` by this change.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), NOT 0. This is not green and must not be reported as
+  green.** Measured in the pinned toolbox on `eeb6211`:
+
+      [ ok ] spec-immutability: green
+      [ ok ] approval-policy: green
+      [ ok ] decisions-log: green
+      [ ok ] lint: green
+      [ ok ] l0: green
+      [ ok ] render-golden: green
+      [ ok ] l1: green
+      [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+      [fail]   Stages that did not run: gitea-settings
+
+  Nothing FAILED. The single cause is `gitea-settings` **SKIPPED** — `SETTINGS_API_TOKEN` is not
+  set in my environment, so branch protection and runner registration were not read; CI has that
+  token and the stage is expected to run there. **L2/L3 `conformance`** also skipped
+  (`CLUSTER_AVAILABLE` unset) but is a *declared* gap that `gate.sh` treats as not-applicable
+  rather than undetermined, so it is not what produced the 3. Touched requirements
+  **R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are therefore **UNVERIFIED at L2/L3**, all four.
+  The stage this branch changes, `render-golden`, is green **and** was independently proven able
+  to turn the whole gate red (above).
+- **`approval-policy` passed LOCALLY only because there is no pull_request event**
+  (`not a pull_request event (event='none') — approval policy N/A`). That is not evidence that it
+  passes. This diff touches `ci/**`, a sensitive path, so in CI the stage is expected to **FAIL**
+  until a non-author approving review exists. That is §1 working, fail-closed, not a defect.
+- **The selftest's own quality is unchanged and ungated.** I wired its *invocation*; I did not
+  review its 106 assertions, and nothing ratchets them. Deleting the body of `_selftest()` would
+  still pass this stage, because an empty selftest returns 0. The repo already has the right
+  answer for that — `conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`
+  injects regressions one at a time and requires `--selftest` to catch every one — and applying
+  that pattern to `catalog/render.py` is the follow-up, deliberately not done here: it is a
+  second subject, it touches `conformance/`, and it would need to live beside the renderer whose
+  file another branch owns this session. **Until it exists, "the selftest runs" is proven and
+  "the selftest is falsifiable" rests on my two hand-run mutations, which are not in the gate.**
+- **Mutation 1's traceback is not fixed.** `render.py:2641`'s `next(...)` has no default, so the
+  absence of `--cookie-name` raises `StopIteration` and the selftest's collected `SELFTEST FAIL:`
+  lines are never printed. Correct verdict, poor diagnostics, and it suppresses the other
+  assertions that mutation also broke. The one-line fix (`next(..., None)` plus a `check()`) is in
+  `catalog/render.py`, which is another branch's file this session, so it is a follow-up and not
+  a thing I touched.
+- **Nothing about the live cluster.** No `kubectl` of any kind was run on this branch — not even
+  read-only; this change is offline CI wiring and needed none. No `apply`/`edit`/`patch`, no
+  `flux reconcile`. Merging this deploys nothing: it changes only what the gate refuses.
+- **CI's own run of this branch.** Quoted in the PR body from the commit's `/statuses`, with
+  descriptions read rather than colours. A *skipped* Gitea job reports `success` with
+  "Has been skipped" and certifies nothing.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so even a fully green PR proves the L0/L1
+  offline subset and nothing about runtime behaviour.
+- **I did not re-measure branch protection, required approvals, or the runner.** Those are
+  CLAUDE.md §2 claims and §2b says re-measure before relying on them; this branch relies on none
+  of them beyond the PR flow itself, and I read the gate's own output rather than the API.

runbooks/README.md

@@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
+  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
+  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
+  app never saw). Covers the cookie collision between per-app sidecars sharing the default
+  `_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in
+  *other* namespaces and is only legible with two logs side by side — the 5-minute access-token
+  session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a
+  POST body silently.
 - **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
   Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
   factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap

runbooks/kustomization.yaml

@@ -85,6 +85,11 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
+      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
+      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
+      # presents as random session loss in ONE app's log and is only legible with both side by side,
+      # so the page has to be reachable at 3am without going looking for it in git.
+      - oauth2-proxy-session=oauth2-proxy-session.md
       # Operator procedure, not an alert page (no alert links to it — the
       # alert-runbooks-resolvable check treats that as advisory). It is served
       # because the plex.tv claim window is short and is not a thing to go looking

runbooks/oauth2-proxy-session.md

@@ -0,0 +1,195 @@
+# oauth2-proxy-session — telling the auth layers apart from the sidecar log
+
+**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
+(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
+proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
+signed out at the same time: the page loads, but its buttons and saves quietly do nothing.
+
+> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
+> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)
+
+## What this means here
+
+Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
+stacked on it, and the symptom rarely names the one that refused:
+
+1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
+   form, or an error page from Keycloak's own hostname.
+2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
+   (`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
+   this page is about reading it.
+3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
+   *behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.
+
+The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
+      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'
+
+Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
+`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
+already changed once. Do not run two `dev.sh` invocations at the same time.
+
+### The four lines, and what each one means
+
+| Log line | What actually happened | Which layer |
+|---|---|---|
+| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
+| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
+| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
+| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |
+
+That last column is the one people miss. The upstream field sits between the method and the quoted
+path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
+it was proxied to the app. Count them:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
+      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
+      | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'
+
+If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
+log had `-`.
+
+## First commands
+
+**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
+go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
+      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'
+
+**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
+If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
+is not refreshing it:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
+      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
+      | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'
+
+The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
+spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.
+
+**3. Is this a cookie collision?** Two checks, and you want both.
+
+*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
+`.home.keithjackson.dev` domain is in the collision set with every other one:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
+      | grep -E "cookie-name|cookie-domain"'
+
+An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
+flag being absent, not wrong.
+
+*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
+line, and put the timestamps next to each other:
+
+    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
+      echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
+      -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'
+
+**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
+instances that genuinely cannot read each other. Compare **hashes**, never values:
+
+    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
+      printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
+      -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'
+
+Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
+before.
+
+### Do not rank apps by their failure ratio
+
+Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
+not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
+manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
+scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
+2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
+paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
+(43) — polling, not 156 user-visible failures.
+
+**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
+that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
+deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
+start — so read it there:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
+      -l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'
+
+then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
+to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.
+
+## Likely causes, most likely first
+
+1. **Cookie collision between two per-app proxies.** Several instances emit the default
+   `_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
+   authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
+   session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
+   is the whole zone — **delete a session belonging to an app in another namespace**. The victim
+   then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
+   its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
+   makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
+   `--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
+   scope — cross-host auth is its job.
+2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
+   the proxy never spends the refresh token it holds, so the session expires with the access token.
+   Set it to about half the measured token lifetime.
+3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
+   live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
+   it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
+   is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
+   browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
+   discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
+   `Accept: application/json`, which most app JavaScript does not send; make it deterministic with
+   `--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
+   start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
+4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
+   session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
+5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
+   by the app's own `/login` page is layer 3, not layer 2.
+
+## When to escalate or stop
+
+- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
+  hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
+  go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
+  agent action.
+- **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
+  (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
+  `_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
+  `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
+  is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
+  `render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
+  erased by the next render anyway. Change the renderer, then regenerate:
+
+      python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+
+  **Budget for the blast radius**: because the function is shared, any change to it rewrites the
+  sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
+  the same commit. There is no way to change one app's sidecar arguments alone. Measured
+  2026-09-30. Coordinate before starting — this is a single change, not five.
+- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
+  `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
+- **You cannot currently alert on any of this**, and that is worth knowing before you promise
+  monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
+  `metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
+  series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
+  Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
+  log-reading only.
+- **Validate an argument change before you open the PR**, not after. The pinned image will tell
+  you what it understood, without touching the cluster:
+
+      podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
+
+  Running it with your full proposed argument list prints a `Cookie settings:` line and an
+  `API route - Path:` line back at you, and rejects mutually exclusive flags outright
+  (`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
+  on a service you have just told someone is fixed.
+- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
+  that app and does nothing about the collision, which is caused by the *names* matching, not the
+  secrets differing.
+- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
+  there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
+  cookie flag will help.
+- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
+  has a unique `--cookie-name`. That combination should not occur; it means something is writing
+  that name which is not in this inventory. Record what you measured.